The Case of the Phantom 125GB Data Spike
It's always fun to hear ways in which SevOne customers use our platform to uncover the root cause of performance issues. Here's one such story that we learned earlier today…
A large cable company uses SevOne to monitor the performance of their new video platform, which serves approximately five million customers. Specifically, they monitor the two data centers that serve up the search functions for this video platform. When a customer picks up their remote to search for a show, actor, movie listing, show time, etc., SevOne monitors the infrastructure that handles those queries.
When the home-grown video platform first launched, the developers built some periodic messaging and "keep alive" signals into the system. One such message sends 24KB of data to each of the five million set top boxes at the top of each hour. This equates to 125GB of data hitting their switching/routing complex in a single microburst, every hour!
Of course, the details of the messaging were never shared with the Ops team. Every hour, the microburst would tip over their switches and load balancers and the Ops team had no idea what was happening.
How did this cable operator uncover the root cause of the problem?
They ratcheted SevOne's polling cycles down to 5 seconds. With granular views of actual infrastructure performance, they were able to determine the issue: an application-specific integrated circuit (ASIC) used by the load balancers and switches was unable to handle the flood of traffic each hour.
As a result, the developers of the video platform have trimmed their 24KB message to 10KB. Problem solved.
SevOne is capable of polling at one second granularity for high definition views of
actual network performance.You are able to detect bursts in traffic that would go
unnoticed at legacy five minute polling cycles.
Increasing polling frequency is just one recommendation in SevOne's recent whitepaper on 6 Steps to an Effective Performance Monitoring Strategy. Download the whitepaper now to learn more.