A leading global financial services firm with assets of $2.3 trillion is located in more than 60 countries with over 240,000 employees. It operates in investment banking, financial services for consumers, small business and commercial banking, financial transaction processing, asset management and private equity.
The financial services firm had made a multi-million dollar investment in packet capture technology and struggled with internal buy-in. There was a lack of management confidence in the chosen solution as well as poor operational feedback. The firm knew it needed to monitor this estate - and fast.
The financial services firm turned to a leading network monitoring solution but it was unable to provide accurate monitoring. As a result, the firm chose to evaluate and partner with SevOne. Within six hours, SevOne was deployed globally, providing full visibility into the packet capture infrastructure.
The organization was immediately able to see 16 boxes with duplex mismatches between the switches and the probes - one of which was causing major issues - in SevOne. Because SevOne highlighted the issue, they were able to fix it, significantly improving user experience. “We were blind to this issue before SevOne,” said the Technology Director, Global Network Services. It was a light bulb moment.
Given the instant visibility SevOne provided for the packet capture estate, the firm engaged in a larger PoC to determine suitability for a global solution.
“Our biggest issue was that we were never able to see all our metrics on a single screen,” said the Technology Director, Global Network Services. “SevOne changed that.” SevOne was brought in to initially monitor 32,000 network devices and 3,000 flow interfaces.
Business Benefits and Results
From reactive to proactive: In the first nine months of using SevOne, the institution’s tier 3 operations team fixed 163 formally unknown issues, which, if left unchecked, would have caused P1 service-impacting outages.
Total visualization of data utilization: The institution was using a new Cisco hashing algorithm on 40G aggregate links to route traffic between data centers but had no visibility and therefore no confidence that it was working correctly. The data centers had also been experiencing erroring Nexus uplinks, impacting VDI services and taking hundreds or thousands of users offline. SevOne allows the team to see the links begin to error before they become service-impacting and are able to mitigate the issues accordingly.
Saves 90 minutes per incident: Previously the team conducted WAN breach analysis by cobbling together disparate reports from different vendors in a PowerPoint slide. With SevOne, they have visualization of WAN utilization and flow with two clicks, saving 90 minutes of work per incident.
Provides visibility of the smoking gun: In one instance, there was a huge meltdown of an MPLS core network that took it offline for 10 hours. The team began troubleshooting in SevOne and saw that a code update for the CPU on a GSR caused SNMP to stop responding three hours later. The BGP stopped responding three days after that. The team had no idea what had actually happened - until they saw the code update was the smoking gun in SevOne. Furthermore, the team instituted a process moving forward that when making an update to the core, they also had to check the SevOne reports to compare the status before and after the update.
Granular monitoring: The team hated that their incumbent tool from the leading networking monitoring provider could not provide granular polling at scale and they weren’t able to see what was happening between each five-minute poll. With SevOne, they set up 9,100 interfaces and 3,700 CPUs at 60/180 second granularity to monitor interface utilization/errors and discards, CPU stats, latency, and QoS.
Detecting DDoS before service is impacted: The institution set up dashboards for at-a-glance visualization of 70 pieces of DMZ infrastructure across multiple data centers to monitor for DDoS attacks. This allowed operations to identify the beginnings of a DDoS attack versus back-end slow downs. In the first 6 months of deploying, the firm averted 12 P1 DDoS and security issues through early identification. As an unintended result, the team was also able to empirically prove that business day load testing was impacting production performance and they changed processes accordingly.
SevOne provided the end-to-end visibility the team needed to assure its infrastructure and provide a quality experience for its end users.