3 ms·
This article resonates a lot with me as someone that was supposed to be in devops and wound up becoming a human network monitor in the end that covered the blin
by devonkim 11y ago
This article resonates a lot with me as someone that was supposed to be in devops and wound up becoming a human network monitor in the end that covered the blind spots of the network infrastructure team's monitoring while teaching some engineers how to use tools like mtr and how to disprove that the network is not responsible and how to use Chrome Developer Tools to show network latency v. infrastructure latency.
Fundamentally, the network infrastructure team had plenty of monitoring - it was just insufficiently scoped for anything anyone but the network team cared about. The granularity of the monitoring was insufficient to catch a wave of handshake failures with third parties, yet it wasn't broad enough to detect when everyone across the enterprise was experiencing so much packet loss due to asymmetric routing that throughput to third party services dropped by an order of magnitude, too. Yet somehow every rack in the DCs had perfect layout and each cable was pinpointed to the extent that if you unplugged a cable randomly someone would come directly to that rack and switch in 3 minutes... after going through the biometric scanners and everything.
The fact that I became a part-time network engineer spending half my time doing network monitoring and troubleshooting when I was supposed to lead cloud operations and develop a devops in such an immature environment with no urgency to make it better is a large part of why I quit.