Comcast Gains Performance Visibility with SevOne NPM

 

Challenge:

Serving millions of cable, internet, streaming and voice customers, Comcast’s service infrastructure is comprised of thousands of network devices located throughout the country. Technical service centers work behind regional and local call centers in order to keep the network up and running and service delivery reliable.

According to Jeff Gill, senior director of network surveillance for Comcast, linking the performance data collected from those devices with the personnel most responsible for their support has been a significant challenge.

“In order to improve our system-wide standard of service excellence, we needed a tool that could very easily and quickly navigate through thousands of devices, evaluate how they were operating, and determine how that result contributed to the overall quality of service,” explains Gill.

Gill and his IT staff had been using the largest legacy performance management application in the industry to ensure quality of service standards. However, with the reporting structure associated with its lead application there was always the risk of degrading server performance to the point it would stop polling the devices it was trying to extract data from – an untenable position for a service provider with the market profile and customer base of Comcast.

Gill and his team began looking for a solution that provided more insight into how the devices within the service infrastructure were performing on a continual, rather than an intermittently polled basis. They needed the ability to see performance trends over time and link that information – on an individual, component level – with overall service performance goals. What Gill and his team really wanted was the ability to look at performance data at a high level across the entire service and then isolate problem areas and quickly resolve them before they escalated and became a serious problem for Comcast’s vast customer base.

Solution:

After evaluating a number of potential solutions, Comcast chose the IBM SevOne Network Performance Management (NPM). “A full install of SevOne NPM – including IP addressing, device discovery, racking and stacking, took only two weeks, and that was for the complete install, not a .dot release,” says Gill.

Almost immediately Gill commissioned SevOne NPM to certify components within the company’s infrastructure as part of the SevOne NPM proof-of-value. While Comcast has many common network components, what is unique is the company’s Cable Modem Termination System (CMTS) where each residential-based cable modem synchronizes with the CMTS and acts as the primary multiplexing device.

Remarkably, only two and a half weeks into the proof of concept, SevOne NPM was not only able to certify the CMTS, but also went as far as offering Comcast the bundling of selected upstream and downstream data that the company had been waiting on for the past 18 months from its incumbent performance management application vendor.

In addition to certifying individual devices in the Comcast service infrastructure, SevOne NPM was also able to respond to Gill’s requirement for enhanced visibility into those devices as a measure of overall service performance.

“With SevOne NPM, we can literally bring up a device in a particular region, state or area of the network and get an all around status of the device – from performance trends, history, current alerts and anything else happening on that router or switch that has some significance to or association with the problem,” says Gill. These features represented a sea change for Comcast in how performance problems were diagnosed and remedied.

SevOne NPM also delivered on Gill’s tertiary requirement: the ability to open up device polling to more users and more devices without significant service degradation.

“As far as the features and functionality SevOne NPM provided, I went from 8-10 users a day under my incumbent performance management vendor, which is a very small number of technical people looking into an application to determine how the service infrastructure is operating, to about 80 users on average with SevOne NPM,” says Gill.

This is the result of SevOne NPM's distributed architecture and intuitive interface, which makes it easier to access the necessary data Gill, and his team needs. “With SevOne NPM, real-time reporting might take two to five seconds, while it would, take my incumbent vendor three to four hours to run that same report, place it in a queue, so I could then go back in and access it.”

“SevOne was, conservatively, 25 percent of the investment we had already made in our legacy provider. Based on the success of the proof of value and its ability to respond and successfully adapt to our demanding service provider environment, the decision for executive management to move forward with SevOne NPM was kind of a no-brainer,” says Gill.

Results:

As Gill has learned first-hand, the rapid retrieval of the individual device data he coveted from his new performance management application provider is light years away from the way his incumbent vendor approached reporting on those very same elements. “SevOne NPM not only polls performance data in real-time for support personnel throughout our organization, but also produces a common platform for communication,” he says.

“One of the advantages SevOne NPM gives us, particularly with our large number of users, is that users can log on virtually to different boxes, distributing the load that each one of them places on the overall architecture,” explains Gill. “Instead of logging into one interface, for example, where you may have as many as 80 people running reports against one interface, they’re each logged on to different servers within the cluster. This process expedites rapid data retrieval without impacting other users on the system, as was the case with our legacy management performance application provider.”

Another important outcome in selecting SevOne NPM is more organic in nature: the tool’s ability to federate data to where it’s needed – nationally, regionally or locally. That includes the ability to proactively root out problems down to individual neighborhoods. Data collected at the national level is now accessible by technical support personnel both regionally and locally, shifting the model of proactive investigation of the service infrastructure down into the hands of technical personnel that work in the same region or area where their customers are located. The bottom line: it’s now much easier and quicker to intercept and resolve events at the point of disruption.

For Gill, the next step taken with SevOne NPM may be procedural in nature. “The simplicity of the navigation, and the way users can intuitively access the interface and get the data they want, changes the nature of our business processes,” concludes Gill. ”In a nutshell, that level of responsiveness is why we use SevOne NPM today as Comcast’s national standard for performance application management.”