Comcast Gains Performance Visibility with SevOne
- Can now poll all performance data in real-time
- Enhanced device visibility
- Expedited data retrieval without impacting other users through distributed architecture
Comcast is principally involved in the development, management and operation of broadband cable networks and in the delivery of programming content.
Headquartered in Philadelphia, Pennsylvania, Comcast employs more than 100,000 people and provides a wide variety of consumer products and services including Video, Online, Voice, Comcast Business Class and Comcast Interactive Media (CIM).
Serving 24.7 million cable customers in 39 states and the District of Columbia, including 14.1 million high-speed Internet and 5.2 million voice customers, Comcast’s service infrastructure is comprised of thousands of network devices located throughout the country. Fifteen technical service centers work behind regional and local call centers in order to keep the network up and running and service delivery reliable.
According to Jeff Gill, senior director of network surveillance for Comcast, linking the performance data collected from those devices with the personnel most responsible for their support has been a significant challenge.
“In order to improve our system-wide standard of service excellence, we needed a tool that could very easily and quickly navigate through thousands of devices, evaluate how they were operating, and determine how that result contributed to the overall quality of service,” explains Gill.
Gill and his IT staff had been using the largest legacy performance management application in the industry to ensure quality of service standards. However, with the reporting structure associated with its lead application there was always the risk of degrading server performance to the point it would stop polling the devices it was trying to extract data from – an untenable position for a service provider with the market profile and customer base of Comcast.
Gill and his team began looking for a solution that provided more insight into how the devices within the service infrastructure were performing on a continual, rather than an intermittently polled basis. They needed the ability to see performance trends over time and link that information – on an individual, component level – with overall service performance goals. What Gill and his team really wanted was the ability to look at performance data at a high level across the entire service and then isolate problem areas and quickly resolve them before they escalated and became a serious problem for Comcast’s vast customer base.
After evaluating a number of potential solutions, Comcast chose SevOne’s Performance Appliance Solutions (SevOne PASTM). “A full install of SevOne – including IP addressing, device discovery, racking and stacking, took only two weeks, and that was for the complete install, not a .dot release,” says Gill.
Almost immediately Gill commissioned SevOne to certify components within the company’s infrastructure as part of the SevOne proof-of-concept. While Comcast has many common network components, what is unique is the company’s Cable Modem Termination System (CMTS) where each residential-based cable modem synchronizes with the CMTS and acts as the primary multiplexing device.
Remarkably, only two and a half weeks into the proof of concept, SevOne was not only able to certify the CMTS, but also went as far as offering Comcast the bundling of selected upstreams and downstreams data that the company had been waiting on for the past 18 months from its incumbent performance management application vendor.
In addition to certifying individual devices in the Comcast service infrastructure, SevOne was also able to respond to Gill’s requirement for enhanced visibility into those devices as a measure of overall service performance.
“By using the Quick View features of the SevOne application, we can literally bring up a device in a particular region, state or area of the network and get an all around status of the device – from performance trends, history, current alerts and anything else happening on that router or switch that has some significance to or association with the problem,” says Gill. These features represented a sea change for Comcast in how performance problems were diagnosed and remedied.
SevOne also delivered on Gill’s tertiary requirement: the ability to open up device polling to more users and more devices without significant service degradation.
“As far as the features and functionality SevOne provided, I went from 8-10 users a day under my incumbent performance management vendor, which is a very small number of technical people looking into an application to determine how the service infrastructure is operating, to about 80 users on average with SevOne,” says Gill.
This is the result of SevOne’s distributed architecture and intuitive interface, which makes it easier to access the necessary data Gill, and his team needs. “With SevOne, real-time reporting might take two to five seconds, while it would, take my incumbent vendor three to four hours to run that same report, place it in a queue, so I could then go back in and access it.”
According to Gill, SevOne’s price tag did not hurt its chance of being selected as a wholesale replacement for Comcast’s legacy performance management application provider.
“SevOne was, conservatively, 25 percent of the investment we had already made in our legacy provider. Based on the success of the proof of concept and its ability to respond and successfully adapt to our demanding service provider environment, the decision for executive management to move forward with SevOne was kind of a no-brainer,” says Gill.
As Gill has learned first-hand, the rapid retrieval of the individual device data he coveted from his new performance management application provider is light years away from the way his incumbent vendor approached reporting on those very same elements. “SevOne not only polls performance data in real-time for support personnel throughout our organization, but also produces a common platform for communication,” he says.
“One of the advantages SevOne gives us, particularly with our large number of users, is that users can log on virtually to different boxes, distributing the load that each one of them places on the overall architecture,” explains Gill. “Instead of logging into one interface, for example, where you may have as many as 80 people running reports against one interface, they’re each logged on to different servers within the cluster. This process expedites rapid data retrieval without impacting other users on the system, as was the case with our legacy management performance application provider.”
Another important outcome in selecting SevOne is more organic in nature: the tool’s ability to federate data to where it’s needed – nationally, regionally or locally. That includes the ability to proactively root out problems down to individual neighborhoods. Data collected at the national level is now accessible by technical support personnel both regionally and locally, shifting the model of proactive investigation of the service infrastructure down into the hands of technical personnel that work in the same region or area where their customers are located. The bottom line: it’s now much easier and quicker to intercept and resolve events at the point of disruption.
For Gill, the next step taken with SevOne may be procedural in nature. “The simplicity of the navigation, and the way users can intuitively access the interface and get the data they want, changes the nature of our business processes,” concludes Gill. ”In a nutshell, that level of responsiveness is why we use SevOne today as Comcast’s national standard for performance application management.”