5 ms·

A systematic approach to debugging

132 points by chmaynard 3y ago

boberoni 3y ago
This blog post should be turned into a GitHub issue template.
- skipkey 3y ago
  The one thing I would add is, if at all possible, write a unit test that fails, reproducing the bug. I have found when working on a code base that is only lightly unit tested, it’s one of the best ways to increase coverage.
  - Brystephor 3y ago
    100%. In general, if you're submitting a PR for a bug fix then an automated test (ideally a unit test) should be included in the PR that tests the bug fix.
aebtebeten 3y ago
see also https://lobste.rs/s/4iwbak/systematic_approach_debugging#c_sxghmn https://lobste.rs/s/4iwbak/systematic_approach_debugging#c_s...
ianmcgowan 3y ago
Another see also, a book I used to hand out to developers, support teams and DBA's: http://debuggingrules.com/ http://debuggingrules.com/. It's not CS specific, or even really computers, but the mindset is invaluable. It's also got some engaging war stories to keep you reading :_) (or for a summary of just the rules) https://courses.cs.washington.edu/courses/cse474/18wi/pdfs/lectures/Debugging-Rules.pdf https://courses.cs.washington.edu/courses/cse474/18wi/pdfs/l...
vegetablepotpie 3y ago
I really like this document. This is the process I have settled on to debug issues, and I have found this approach to be universal. Having such a finely articulated process has also been helpful to demonstrating to me why micromanagement in the business world completely destroys productivity. Management hates, hates, hates this process to the depths of their souls because they would prefer a prescribed process that outlines well defined time-boxed steps to be articulated, from the get-go for many diverse disciplines to contribute to equally. To them, Steps 1-3 should already by known and are unnecessary. Needing to go through them either means the issue is unimportant, the developer is incompetent, or there isn’t alignment between teams. Step 4 and 5 means you’re flailing. You need to provide action, not ideas. “narrowing the search space” means assigning blame; healthy teams work together, they don’t assign blame. Developers attempting to break down the problem are disrupting the harmony of the work environment. This necessary step should be avoided. Step 6 is the only step that matters. It’s more important to get to this step as quickly as possible ill informed than to arrive at it with data to support your conclusions. Following these steps are strictly necessary in many circumstances. They’re also impossible to follow under scrutiny. I say this to make a recommendation: Any good leader, to a development team, should insulate and shield their team from scrutiny so that they can carry these steps out rigorously rather than succumb to pressure and micromanage productivity into oblivion.
AlbertCory 3y ago
> Reproduce the bug. Usually if you can do that, the bug's as good as solved. It's the ones you can't reproduce that take all the time. He does explain how difficult that can be. Also, one of mine: "The hardest bug is where nothing happens." If something goes wrong, at least you can start from that. When nothing happens, you don't even know where it was supposed to.
- hinkley 3y ago
  "Nothing happens" is one of the legs in the argument of libraries over frameworks. Frameworks are notoriously bad at doing nothing and saying nothing if you don't have your inputs just so. And then you're just staring at config files for an hour hoping you will achieve satori instead of rage.
  - AlbertCory 3y ago
    while we're on the subject: Learning a UI is especially bad that way. You double click, select a menu, etc., and nothing at all happens.
    - bfLives 3y ago
      Greyed-out buttons are similarly terrible UX. Far too often, it’s impossible to figure out why a button is un-clickable.
      - eitland 3y ago
        Possibly even worse: When the UX designer decided that greyed out buttons are terrible UX and bans them, so instead of disabling them (greying them out) you have to remove them if they aren't clickable. Now, instead of being puzzled by why he can't click the button the user doesn't even know there is a button and has to start on by solving another puzzle first before solving the puzzle about how to enable the button. PS: back in the day this could be solved by a tool tip or a paragraph in the help file. Today it seems such helpers are forbidden too.
rectang 3y ago
> My approach is systematic and focused on understanding first and foremost. This is for a variety of reasons, but principally that you need to understand what is going on both to fix it and to be sure it's fixed. Feature-focused management often fails to take comprehensibility into account. Initial development hours are rationed carefully, but debugging gets an infinite budget, as if it exists outside the reality of delivering value. Eventually is a tradeoff between the amount of time you invest in making the system comprehensible and the amount of time saved in debugging. But that's only after you've thrown down table stakes (unit tests, basic documentation, high-level modularization with loose coupling, preservation of critical invariants, etc.)
- jonahx 3y ago
  > Eventually is a tradeoff between the amount of time you invest in making the system comprehensible and the amount of time saved in debugging. Well said. > But that's only after you've thrown down table stakes (unit tests, basic documentation, high-level modularization with loose coupling, preservation of critical invariants, etc.) I would actually turn this on its head, because I would rather have a comprehensible system without those table stakes features than an incomprehensible system with them. Of course, there is a feedback cycle between unit tests/docs/invariants and comprehensibility, but I have worked in systems with pretty good tests that were still very slow to work in because of their complexity. Imo everything comes second to comprehensibility. Without it you are hosed.
hinkley 3y ago
4&5 is too slow by half, probably more. As the number of potential culprits goes up the odds that you will identify the correct one first are very small. Most of what makes these processes slow is getting emotionally invested in solving the problem in as few 'moves' as possible. Occasionally that will work for you but it's not a safe long-term strategy. What you want to do instead is to formulate several hypotheses, and then figure out the relative likelihood and difficulty in testing each one. Determine if partial tests of one hypothesis also satisfy or invalidates parts of others, and then begin the process of winnowing down possibilities in a calm and collected process of elimination, balancing cost of validating an assumption versus how much of the problem space it eliminates. Once you find the bug you still have to fix it, and getting worked up makes for cures that are as bad as the disease.
- lelanthran 3y ago
  > Determine if partial tests of one hypothesis also satisfy or invalidates parts of others, and then begin the process of winnowing down possibilities in a calm and collected process of elimination, Think of it like Wordle - with each guess you try to: 1. Use as many new letters that you can (so you can eliminate them), 2. Use all the almost-correct (yellow) letters in new positions, 3. Keep all the correct (green) letters in the same position. So each guess is a test, where you try to invalidate as many letters as possible and satisfy the existing proven letters.
  - hinkley 3y ago
    Yep, and sometimes you just test things because you know Steve won't let it drop until someone does. Eliminate the zebra just so people stop thinking about zebras.
syndicatedjelly 3y ago
Really well-written, and helps head off the anxiety some people feel when a new bug report rolls in.
kmoser 3y ago
> What were the recent changes? If you narrow your search to most recent changes, you have a really good chance of finding the source of the bug. The culprit may even be you!
navane 3y ago
The one step I miss: what reasoning or assumption lead to this bug, and where could that have been repeated. This way you have a chance to sniff out bugs before they surface.
gniv 3y ago
This is great as a technical approach, but I would argue that sometimes it's not the right way. Debugging a difficult bug alone takes a lot of time and can be frustrating, especially if you're not familiar with the code base. I found that broadcasting about the bug to the extended team (if the team culture permits, of course) is more efficient. Somebody might understand that part of the project so well that they can tell you exactly where to look to solve it. Also, multiple people thinking a few moments about it and giving educated guesses can make a big difference.
onetimeuse92304 3y ago
Some important things missing. One has to define what a bug is. For example, I have many times received a bug report for something that was perfectly normal system behaviour. The reasons for this are varied from users not understanding the system, the system behaviour being complex and causing emergent phenomena that is strange at first sight but completely fine when you start to analysing it to consequences of underspecified requirements, broken communication at the time of requirement gathering, etc. It might still result in a bug, but this time a bug report for the original design. Another very important thing missing is it is absolutely crucial to gather from the user what was the expected outcome. I can't begin to tell you how many times I had a developer "fix" the bug only to find out the fix is not what the user wanted to happen. Or that there wasn't even a bug in the first place and the fix actually broke the system for somebody else. As to the forming and testing the hypothesis, the missing is that you can do it all by reading, understanding the code and simulating the problem all in your mind. I have probably fixed something like 70-80% of all bugs at this stage. What you do is you form a hypothesis and then you start tracing the pieces of data through the code of the application (aided by your understading of it). If I can find something in the code that disproves my hypothesis, I have now saved time on setting up the whole test. But I also gained more understanding of the problem which I would probably not if I just let the application fail. Even if I actually found a solution, I can have better understanding how the piece of data interacts with parts of the system and I may now be aware how to write the fix or at the very least how to do it better.
philbo 3y ago
I like to have a step zero, that comes even before you try to reproduce the issue: get your head right, essentially. If something's broken in production and the bug is hard to figure out, that often means stress. And stress is not good for helping you to solve problems. So take a few breaths, slow down, relax. Slowing down to get into the right frame of mind helps get to the _right_ fix faster, because your panicked brain is not jumping to all kinds of speculative conclusions about what the solution might be. Be calm, observe, reflect. Coincidentally, I wrote a blog post about this and my other steps for debugging not long ago: https://philbooth.me/blog/how-to-debug https://philbooth.me/blog/how-to-debug

132 points by chmaynard 3y ago

boberoni 3y ago

This blog post should be turned into a GitHub issue template.

skipkey 3y ago
The one thing I would add is, if at all possible, write a unit test that fails, reproducing the bug. I have found when working on a code base that is only lightly unit tested, it’s one of the best ways to increase coverage.
- Brystephor 3y ago
  100%. In general, if you're submitting a PR for a bug fix then an automated test (ideally a unit test) should be included in the PR that tests the bug fix.

aebtebeten 3y ago

see also https://lobste.rs/s/4iwbak/systematic_approach_debugging#c_sxghmn https://lobste.rs/s/4iwbak/systematic_approach_debugging#c_s...

ianmcgowan 3y ago

Another see also, a book I used to hand out to developers, support teams and DBA's: http://debuggingrules.com/ http://debuggingrules.com/. It's not CS specific, or even really computers, but the mindset is invaluable. It's also got some engaging war stories to keep you reading :_) (or for a summary of just the rules) https://courses.cs.washington.edu/courses/cse474/18wi/pdfs/lectures/Debugging-Rules.pdf https://courses.cs.washington.edu/courses/cse474/18wi/pdfs/l...

vegetablepotpie 3y ago

I really like this document. This is the process I have settled on to debug issues, and I have found this approach to be universal. Having such a finely articulated process has also been helpful to demonstrating to me why micromanagement in the business world completely destroys productivity. Management hates, hates, hates this process to the depths of their souls because they would prefer a prescribed process that outlines well defined time-boxed steps to be articulated, from the get-go for many diverse disciplines to contribute to equally. To them, Steps 1-3 should already by known and are unnecessary. Needing to go through them either means the issue is unimportant, the developer is incompetent, or there isn’t alignment between teams. Step 4 and 5 means you’re flailing. You need to provide action, not ideas. “narrowing the search space” means assigning blame; healthy teams work together, they don’t assign blame. Developers attempting to break down the problem are disrupting the harmony of the work environment. This necessary step should be avoided. Step 6 is the only step that matters. It’s more important to get to this step as quickly as possible ill informed than to arrive at it with data to support your conclusions. Following these steps are strictly necessary in many circumstances. They’re also impossible to follow under scrutiny. I say this to make a recommendation: Any good leader, to a development team, should insulate and shield their team from scrutiny so that they can carry these steps out rigorously rather than succumb to pressure and micromanage productivity into oblivion.

AlbertCory 3y ago

> Reproduce the bug. Usually if you can do that, the bug's as good as solved. It's the ones you can't reproduce that take all the time. He does explain how difficult that can be. Also, one of mine: "The hardest bug is where nothing happens." If something goes wrong, at least you can start from that. When nothing happens, you don't even know where it was supposed to.

hinkley 3y ago
"Nothing happens" is one of the legs in the argument of libraries over frameworks. Frameworks are notoriously bad at doing nothing and saying nothing if you don't have your inputs just so. And then you're just staring at config files for an hour hoping you will achieve satori instead of rage.
- AlbertCory 3y ago
  while we're on the subject: Learning a UI is especially bad that way. You double click, select a menu, etc., and nothing at all happens.
  - bfLives 3y ago
    Greyed-out buttons are similarly terrible UX. Far too often, it’s impossible to figure out why a button is un-clickable.
    - eitland 3y ago
      Possibly even worse: When the UX designer decided that greyed out buttons are terrible UX and bans them, so instead of disabling them (greying them out) you have to remove them if they aren't clickable. Now, instead of being puzzled by why he can't click the button the user doesn't even know there is a button and has to start on by solving another puzzle first before solving the puzzle about how to enable the button. PS: back in the day this could be solved by a tool tip or a paragraph in the help file. Today it seems such helpers are forbidden too.

rectang 3y ago

> My approach is systematic and focused on understanding first and foremost. This is for a variety of reasons, but principally that you need to understand what is going on both to fix it and to be sure it's fixed. Feature-focused management often fails to take comprehensibility into account. Initial development hours are rationed carefully, but debugging gets an infinite budget, as if it exists outside the reality of delivering value. Eventually is a tradeoff between the amount of time you invest in making the system comprehensible and the amount of time saved in debugging. But that's only after you've thrown down table stakes (unit tests, basic documentation, high-level modularization with loose coupling, preservation of critical invariants, etc.)

jonahx 3y ago
> Eventually is a tradeoff between the amount of time you invest in making the system comprehensible and the amount of time saved in debugging. Well said. > But that's only after you've thrown down table stakes (unit tests, basic documentation, high-level modularization with loose coupling, preservation of critical invariants, etc.) I would actually turn this on its head, because I would rather have a comprehensible system without those table stakes features than an incomprehensible system with them. Of course, there is a feedback cycle between unit tests/docs/invariants and comprehensibility, but I have worked in systems with pretty good tests that were still very slow to work in because of their complexity. Imo everything comes second to comprehensibility. Without it you are hosed.

hinkley 3y ago

4&5 is too slow by half, probably more. As the number of potential culprits goes up the odds that you will identify the correct one first are very small. Most of what makes these processes slow is getting emotionally invested in solving the problem in as few 'moves' as possible. Occasionally that will work for you but it's not a safe long-term strategy. What you want to do instead is to formulate several hypotheses, and then figure out the relative likelihood and difficulty in testing each one. Determine if partial tests of one hypothesis also satisfy or invalidates parts of others, and then begin the process of winnowing down possibilities in a calm and collected process of elimination, balancing cost of validating an assumption versus how much of the problem space it eliminates. Once you find the bug you still have to fix it, and getting worked up makes for cures that are as bad as the disease.

lelanthran 3y ago
> Determine if partial tests of one hypothesis also satisfy or invalidates parts of others, and then begin the process of winnowing down possibilities in a calm and collected process of elimination, Think of it like Wordle - with each guess you try to: 1. Use as many new letters that you can (so you can eliminate them), 2. Use all the almost-correct (yellow) letters in new positions, 3. Keep all the correct (green) letters in the same position. So each guess is a test, where you try to invalidate as many letters as possible and satisfy the existing proven letters.
- hinkley 3y ago
  Yep, and sometimes you just test things because you know Steve won't let it drop until someone does. Eliminate the zebra just so people stop thinking about zebras.

syndicatedjelly 3y ago

Really well-written, and helps head off the anxiety some people feel when a new bug report rolls in.

kmoser 3y ago

> What were the recent changes? If you narrow your search to most recent changes, you have a really good chance of finding the source of the bug. The culprit may even be you!

navane 3y ago

The one step I miss: what reasoning or assumption lead to this bug, and where could that have been repeated. This way you have a chance to sniff out bugs before they surface.

gniv 3y ago

This is great as a technical approach, but I would argue that sometimes it's not the right way. Debugging a difficult bug alone takes a lot of time and can be frustrating, especially if you're not familiar with the code base. I found that broadcasting about the bug to the extended team (if the team culture permits, of course) is more efficient. Somebody might understand that part of the project so well that they can tell you exactly where to look to solve it. Also, multiple people thinking a few moments about it and giving educated guesses can make a big difference.

onetimeuse92304 3y ago

Some important things missing. One has to define what a bug is. For example, I have many times received a bug report for something that was perfectly normal system behaviour. The reasons for this are varied from users not understanding the system, the system behaviour being complex and causing emergent phenomena that is strange at first sight but completely fine when you start to analysing it to consequences of underspecified requirements, broken communication at the time of requirement gathering, etc. It might still result in a bug, but this time a bug report for the original design. Another very important thing missing is it is absolutely crucial to gather from the user what was the expected outcome. I can't begin to tell you how many times I had a developer "fix" the bug only to find out the fix is not what the user wanted to happen. Or that there wasn't even a bug in the first place and the fix actually broke the system for somebody else. As to the forming and testing the hypothesis, the missing is that you can do it all by reading, understanding the code and simulating the problem all in your mind. I have probably fixed something like 70-80% of all bugs at this stage. What you do is you form a hypothesis and then you start tracing the pieces of data through the code of the application (aided by your understading of it). If I can find something in the code that disproves my hypothesis, I have now saved time on setting up the whole test. But I also gained more understanding of the problem which I would probably not if I just let the application fail. Even if I actually found a solution, I can have better understanding how the piece of data interacts with parts of the system and I may now be aware how to write the fix or at the very least how to do it better.

philbo 3y ago

I like to have a step zero, that comes even before you try to reproduce the issue: get your head right, essentially. If something's broken in production and the bug is hard to figure out, that often means stress. And stress is not good for helping you to solve problems. So take a few breaths, slow down, relax. Slowing down to get into the right frame of mind helps get to the _right_ fix faster, because your panicked brain is not jumping to all kinds of speculative conclusions about what the solution might be. Be calm, observe, reflect. Coincidentally, I wrote a blog post about this and my other steps for debugging not long ago: https://philbooth.me/blog/how-to-debug https://philbooth.me/blog/how-to-debug