One of our servers stopped responding for 30 seconds every once in a while. Spent some time trying to figure out why. No luck. Spent some more time. Eventual

One of our servers stopped responding for 30 seconds every once in a while. Spent some time trying to figure out why. No luck. Spent some more time. Eventually I realized that the log timestamps were weird - it looked like the query had been sent a response, but the log message appeared 30 seconds later. I instrumented the servers to measure disk latency. I noticed massive spikes in latency every few hours. Couldn't figure out why. Then someone told me the servers were running on virtual machines with a shared NetApp for storage... and it all came together. Every few hours a multi-gigabyte file was delivered to each machine. This was a design that had originally been done for physical machines. With virtual machines, 30 copies of a multi-gigabyte file were being dumped to a single NetApp, filling up the file server's memory buffer and making disk latency spike since it was waiting for physical writes. Meanwhile, the server I was debugging was doing log writes in the main I/O thread, so it blocked on handling requests when this happened. I went and talked to team lead for the server. "Oh yeah, we fixed that recently, logging will be in its own thread as of next release." :facepalm: Moral of the story: 1. Talk to the people maintaining the software before you spend too much time debugging. 2. Disks do block, don't assume they won't. 3. Changes to operational setups can have significant, hard to predict impacts. If you want to hear more stories, I'm writing a weekly email with one of my programming or career mistakes: https://softwareclown.com https://softwareclown.com