2 ms·
I've been described as being quite good at debugging, here is my approach: Short term - while debugging a problem: 1. Gather whatever information you can. Loo
by ltbarcly3 4y ago
I've been described as being quite good at debugging, here is my approach:
Short term - while debugging a problem:
1. Gather whatever information you can. Look in the logs!
2. Debugging means tracking the flow of execution from beginning to end (and then back to the front in many cases). Where along that chain is the breakdown?
3. Often you can stop after 2, since when the point of failure is determined the issue becomes obvious. However, in some cases it is not obvious. In those cases, you have to look for 'something that does not make sense' or can't be true.
Case study: timeouts
We started having errors on a service we owned. Calls would be made to our service (call it service A) and then calls to an upstream service (service B) would time out, and we didn't catch this exception so the request to A would end with a 500 response. However, when we looked at the logs for B we could see our requests from A and that they were not timing out at all, they were taking the usual amount of time. This is a huge contradiction! After a day of poking around we could reproduce the timeout with a script which called our service. Thinking it might be the load balancer, we started sending requests to the IP's of workers directly, which never timed out. On a hunch, we sent requests to the IP of the load balancer (this can't be done directly since the LB uses the hostname as part of how it routes requests to servers, but we could do this by adding the LB with the proper hostname to the /etc/hosts file.) Low and behold, the timeouts were still gone even though we were doing 'the exact same thing' as before! Removing the entry from the hosts file, the timeouts immediately came back for the same small percentage of requests.
Long term:
Being good at debugging really consists of two things, knowing how things work and spending the time required to debug things. Interestingly, one great way to learn how things work is to debug things. Whenever something does not make sense, or you can't explain it, dig in and get to the bottom of it. Every time you do this you'll learn things that you never would have guessed, and your ability to debug some random future issue will go way up when that future issue just happens to intersect something that you had to dig into previously.