8 ms·

Learning from Your Bugs

60 points by henrik_w 11y ago

jesserayadkins2 11y ago
I have a similar process myself. What I find to be helpful is to ask myself what, in particular, went wrong. If I have a handful of cases where "some api was used wrong", then I take that as a sign of a need for refactoring to make the callee easier to use. Another that I like to ask myself, is how I can use the compiler and its type checking to my advantage. As in, how can I make it more difficult for myself to make this bug again in the future? But I agree with the author's point: Nearly every bug is an opportunity to learn. You should not just fix bugs, but also to understand why they happened.
dfan 11y ago
I do this with chess. I take positions from tournament games in which I made a correctable mistake (generally not positions in which there was just a lot of calculation to be done and I wasn't up to the task) and add them as flashcards to my spaced repetition system. It is true that software engineers often treat bugs as unfortunate but unavoidable noise in the system; when one pops up, you fix it and move on. But it's really worth thinking them as an inherent part of the process of software development, one which you can improve and learn from, just like anything else.
gumby 11y ago
This is a clever and straightforward technique.
another-dave 11y ago
Sounds quite interesting to see how it will build up and to spot any patterns emerging in the results. Will give it a go over the next while. Cheers for sharing!
gitmagic 11y ago
I'm kind of doing the same but instead I write my analysis and notes in the pull request that fixes the bug. I think it's a great way for others to learn too :)
hankerfaster 11y ago
We do something similar. Any bug that is submitted by a customer (rather than QA, or a trial team, or other developers) is analyzed. 50% of our bugs are due to poor requirements. 25% are due to poor design. The other 25% are just coding problems. Most of the requirements problems derive from lack of understanding about the tradeoffs in the current system or lack of understanding what the customer actually wants. Most of the design problems are a result of excessive complexity. People often forget to simplify and refactor prior to adding a new feature. They end up having to keep a larger system in their brain when designing than they otherwise need to. Therefore they miss interactions that exist or forget to handle them in the new design. Most of the coding problems would have been caught had unit tests been clearer and closer to 100% branch/code coverage. It is amazing how many bugs lurk in the last 5% of testing. It is actually quite fun to tell a product manager that they are the cause for a bug :) Usually it is the developer that
- chriswarbo 11y ago
  > Most of the coding problems would have been caught had unit tests been clearer and closer to 100% branch/code coverage. It is amazing how many bugs lurk in the last 5% of testing. Presumably, coverage starts with the low-hanging fruit (simple, linear code; heavily-exercised code like startup routines and main loops; etc.) and grows to encompass ever more unweildy, smelly, edge-case parts of the codebase. Hence, by increasing code coverage, we're boiling off the solvent of run-of-the-mill code, such that what remains uncovered gets an ever stronger concentration of hackery, unintelligibility and black magic. By 95% coverage, the bits remaining are those parts people don't want to touch, so I'd expect the concentration of bugs to be highest there. :)
  - hankerfaster 11y ago
    precisely! As programmers that is what we care about. But really, at that point the business often doesn't care. Not when a requirement bug could cause months of lost engineer time. Our experience is requirement bugs are much more costly than a missed unit test resulting in a bug.
jondubois 11y ago
I never bothered to write down details of bugs... I rarely learn a lesson from any one specific bug - Lessons are learned by being exposed to a wide range of different bugs over many years. Eventually it becomes gut instincts. For example today I was debugging an issue with Kubernetes (it was a really cryptic error about client-server not being able to negotiate a protocol or similar). I had never used Kubernetes before and I couldn't find anything on Stack Overflow. The first thing I did was check the client and server versions (to see if they matched) - They did so that wasn't the issue. So then I had a strong 'gut feeling' that it was related to the client configs. In this case, my second hypothesis turned out to be correct. 5 years ago I would probably tested my way through 20 different hypotheses before finding the problem. To summarize my debugging strategy, it's simple; just pick all the low-hanging fruits first. The hard part is knowing what a low-hanging fruit looks like.
- MattGrommes 11y ago
  This is basically intuition, a base of experiences that leads to these "gut instinct" decisions and feelings. Doing a journal like the author describes could actually help with your intuition because you're adding more specific information from each bug into your mental storage. Now you're getting a cursory level of data from each individual bug which leads you to think each one is less important. It's like skimming 3 books on a subject instead of close reading 1. I'm just thinking out loud because I do what you describe here but I think the journaling idea could be valuable.
  - vitd 11y ago
    FWIW, journaling has been useful to me. I've run into bugs and thought, "I know I hit something just like this before. Oh wait, let me look through my debug journal and see if there's something in there!" And often there is. It saves me having to do web searches or guess and use hunches. I can always fall back on those things if it's something I haven't seen before or is manifesting in a different way. I'd go even further and say that I've often wasted time chasing down a hunch that was completely wrong.
    - 11y ago

60 points by henrik_w 11y ago

I have a similar process myself. What I find to be helpful is to ask myself what, in particular, went wrong. If I have a handful of cases where "some api was used wrong", then I take that as a sign of a need for refactoring to make the callee easier to use. Another that I like to ask myself, is how I can use the compiler and its type checking to my advantage. As in, how can I make it more difficult for myself to make this bug again in the future? But I agree with the author's point: Nearly every bug is an opportunity to learn. You should not just fix bugs, but also to understand why they happened.

dfan 11y ago

I do this with chess. I take positions from tournament games in which I made a correctable mistake (generally not positions in which there was just a lot of calculation to be done and I wasn't up to the task) and add them as flashcards to my spaced repetition system. It is true that software engineers often treat bugs as unfortunate but unavoidable noise in the system; when one pops up, you fix it and move on. But it's really worth thinking them as an inherent part of the process of software development, one which you can improve and learn from, just like anything else.

gumby 11y ago

This is a clever and straightforward technique.

another-dave 11y ago

Sounds quite interesting to see how it will build up and to spot any patterns emerging in the results. Will give it a go over the next while. Cheers for sharing!

gitmagic 11y ago

I'm kind of doing the same but instead I write my analysis and notes in the pull request that fixes the bug. I think it's a great way for others to learn too :)

hankerfaster 11y ago

We do something similar. Any bug that is submitted by a customer (rather than QA, or a trial team, or other developers) is analyzed. 50% of our bugs are due to poor requirements. 25% are due to poor design. The other 25% are just coding problems. Most of the requirements problems derive from lack of understanding about the tradeoffs in the current system or lack of understanding what the customer actually wants. Most of the design problems are a result of excessive complexity. People often forget to simplify and refactor prior to adding a new feature. They end up having to keep a larger system in their brain when designing than they otherwise need to. Therefore they miss interactions that exist or forget to handle them in the new design. Most of the coding problems would have been caught had unit tests been clearer and closer to 100% branch/code coverage. It is amazing how many bugs lurk in the last 5% of testing. It is actually quite fun to tell a product manager that they are the cause for a bug :) Usually it is the developer that

chriswarbo 11y ago
> Most of the coding problems would have been caught had unit tests been clearer and closer to 100% branch/code coverage. It is amazing how many bugs lurk in the last 5% of testing. Presumably, coverage starts with the low-hanging fruit (simple, linear code; heavily-exercised code like startup routines and main loops; etc.) and grows to encompass ever more unweildy, smelly, edge-case parts of the codebase. Hence, by increasing code coverage, we're boiling off the solvent of run-of-the-mill code, such that what remains uncovered gets an ever stronger concentration of hackery, unintelligibility and black magic. By 95% coverage, the bits remaining are those parts people don't want to touch, so I'd expect the concentration of bugs to be highest there. :)
- hankerfaster 11y ago
  precisely! As programmers that is what we care about. But really, at that point the business often doesn't care. Not when a requirement bug could cause months of lost engineer time. Our experience is requirement bugs are much more costly than a missed unit test resulting in a bug.

jondubois 11y ago

I never bothered to write down details of bugs... I rarely learn a lesson from any one specific bug - Lessons are learned by being exposed to a wide range of different bugs over many years. Eventually it becomes gut instincts. For example today I was debugging an issue with Kubernetes (it was a really cryptic error about client-server not being able to negotiate a protocol or similar). I had never used Kubernetes before and I couldn't find anything on Stack Overflow. The first thing I did was check the client and server versions (to see if they matched) - They did so that wasn't the issue. So then I had a strong 'gut feeling' that it was related to the client configs. In this case, my second hypothesis turned out to be correct. 5 years ago I would probably tested my way through 20 different hypotheses before finding the problem. To summarize my debugging strategy, it's simple; just pick all the low-hanging fruits first. The hard part is knowing what a low-hanging fruit looks like.

MattGrommes 11y ago
This is basically intuition, a base of experiences that leads to these "gut instinct" decisions and feelings. Doing a journal like the author describes could actually help with your intuition because you're adding more specific information from each bug into your mental storage. Now you're getting a cursory level of data from each individual bug which leads you to think each one is less important. It's like skimming 3 books on a subject instead of close reading 1. I'm just thinking out loud because I do what you describe here but I think the journaling idea could be valuable.
- vitd 11y ago
  FWIW, journaling has been useful to me. I've run into bugs and thought, "I know I hit something just like this before. Oh wait, let me look through my debug journal and see if there's something in there!" And often there is. It saves me having to do web searches or guess and use hunches. I can always fall back on those things if it's something I haven't seen before or is manifesting in a different way. I'd go even further and say that I've often wasted time chasing down a hunch that was completely wrong.
  - 11y ago