That's pretty much it. There's a certain category of people with a certain attitude towards debugging. You have to be able to break it down into steps - sometim

That's pretty much it. There's a certain category of people with a certain attitude towards debugging. You have to be able to break it down into steps - sometimes from "first principles". Then work up and identify where the issue is and how to resolve it. Sometimes it requires looking at logs and simulating production conditions and other times you may have to understand low level network stack. I still remember the time trying to reproduce and fix a certain issue. We had a 3-node cluster mysql database setup with one main and two replicas. The replicas are read-only. If one replica goes down, reads shouldn't be affected at all. We had an application layer on top of this to coordinate and enforce these requirements. Every which way you look at the code, there's no reason why this shouldn't work. And yet, when one node went down in production under a very particular circumstance with the network, it took down the entire cluster - not that the cluster was not reachable - it wouldn't allow reads and just hang. Tracing it took me down the hole of understanding the precise network issue (fading memory - I think the machine wasn't responding back and tcp kept retrying) + linux tcp setting (I think tcp_delack_min) + a java networking setting + how thread safety was setup in the application (do not use synchronized at the method level). It took a week to diagnose. Recently, there was a minor issue with adapting a sample Spark UDF to work with our setup and it took the developers 3 weeks and they kept giving up after every try.