6 ms·

>Rob Pike of golang fame noted that while he reached for the debugger first thing, Ken Thompson would ignore what Rob was doing and just stand and think. Then

>Rob Pike of golang fame noted that while he reached for the debugger first thing, Ken Thompson would ignore what Rob was doing and just stand and think. Then after a while he'd say: "I know what's wrong" and very often be right. I feel like this is not a helpful take on debugging for most of us. I'm certainly not as good at this as Ken Thompson, and very seldom is the problem space simple enough that I can hold a complete mental model of it in my head. Instead I reach for the debugger to collect more data on what's going on. And I can usually find the issue by looking a lot faster than I can correctly theorize about it. In Debugging by Davig J. Agans, his third rule is "Quit thinking and look". And I tend to agree. In the majority of cases when I'm debugging a problem the solution only reveals itself when I see that missing piece of information. EDIT: As others have rightly pointed out, if I wrote the code I've already thought about it and have a mental model of what I wrote. And yes, I do think about what could be causing the problem. I guess it's those cases where I think it should be working in a certain way, but it isn't, where I find it most valuable to stop thinking and look. And usually in those cases I either have an incorrect or incomplete model, or I wrote something that didn't match what I was thinking.

m463 5y ago
If you wrote the code, there's likely a mental model in your head. This is why it is relatively easy to work on a project you were in on from the beginning. and why it is always a struggle to come into a project with an existing codebase. I've spent many years working on large existing codebases and it gets in the way of liking your job.
- drewcoo 5y ago
  > there's likely a mental model in your head Yes, and unless there was a simple typo, the problem is the model in your head. Most bugs come from unchallenged assumptions.
  - Jensson 5y ago
    Knowing that a particular bug can happen is usually enough to debug your mental model though. You just trace back to different assumptions you made and see which of them could cause this bug if wrong, and then fix it.
  - wruza 5y ago
    In my personal experience, most bugs come from changing a working code, especially at a refactoring. It mixes assumptions in an unpredictable way, because of chaotic code rearrangements and lots attention required to do that correctly. First-time code is much easier to challenge (if you ignore repl development phase which is not “code” yet).
xyzzy123 5y ago
If you have the luxury of 2 people on a bug, having one on strategy & idea generation and the other down in the weeds confirming hypotheses and gathering data is a common pattern. For some reason, not being on the keyboard or operating tools seems to free up a lot of mental bandwidth. Nothing gets done without someone being hands-on though so arguably yes, that role is more important. If there's just 1 person they usually have to stop and think really hard periodically and keep very good notes to avoid getting lost in the weeds.
bhk 5y ago
In my experience, a debugger is most helpful when you are unsure of the behavior of the language or libraries or machine on which you are working. It shows you what happens at various levels and validate or invalidate your assumptions or what you were led to believe. Sitting and thinking on the other hand, is best for dealing with errors in your own thinking. Ideally, we should have a well-thought-out argument for why something will work, so when it doesn't, our argument needs to be refined.
- zwieback 5y ago
  Exactly, perfect summary. Expanding on the first point, a debugger is invaluable for me if I can run my program up to a point and then use the "immediate" window to run a few API calls to see what would happen in that context.
- svachalek 5y ago
  I agree. Unfortunately over my career (going on 30 years) the ratio of my thinking to other people's thinking in the systems I have to develop is shrinking and shrinking. Modern software runs on systems layered onto systems layered on systems of systems. "Debuggers" are more sorely needed than ever but the traditional concept of a debugger is decades out of date and sorely needs to be reinvented for heterogenous, distributed, and parallel interactions.
  - wilburTheDog 5y ago
    I think this is very true. The most vexing problems I've run into lately have all been because something is going wrong in a step I don't have much control over. Like in one case the proxy was stripping out the Authorization header for PUT requests (seriously). Or in another the corporate-signed certs used by all the on-prem systems weren't validating on AWS because they don't have a valid root cert. I really wish I had a "debugger" that could trace the interconnections between all the systems my app depends on.
    - spockz 5y ago
      In both cases you would have been helped by proper error reporting and by tracing. (See OpenTelemetry) For the first the remote server could say there is no Authorization header where tracing confirms it was there when sent by the client but not by the proxy. For the second it would be really nice to see more detailed logging on the server side about why a particular peer certificate was rejected. My biggest gripe with current OpenSSL and Java TLS implementations is this lack of detailed reporting.
mannykannot 5y ago
I don't think this was intended to be an either/or choice, and "quit thinking and look" can be read as "if thinking isn't working, get more data" (and it could be read as such even if that wasn't the author's intended meaning!) Even if thinking has not solved the problem, it may help you find where to look next. One tip I have is that even if you wrote the code and think you know how it works, stepping through it in your mind might lead you to see where you are making a questionable assumption. You may think that you can do that just as well in the debugger, and you may be right, but there are two ways of using the debugger: one is to hit 'step' and see what happens; the other is to figure out what you expect will happen, and then check if your expectation was correct. There have been many times when I have stepped up to the point where the bug is about to happen, and then realized what is wrong before taking that final step.
masswerk 5y ago
Personally, in the rare cases I really do not understand the behavior, I tend to just stare at the code. (It's not a recipe or so, it's just what I tend to do.) Having subtile or no syntax coloring helps.
mcguire 5y ago
It's a learnable skill although it is quite difficult, particularly if you didn't write the code (recently). And it's very useful because at some point the debugger (or logging[1]) will let you down. [1] I myself don't use a debugger much; I'm frequently working on something distributed where a debugger is laughably useless.
coldtea 5y ago
>and very seldom is the problem space simple enough that I can hold a complete mental model of it in my head You already have a more narrow sense of the problem space, else you wouldn't know where to look with the debugger either...
jrochkind1 5y ago
So, I kind of agree with you. I had the same thought about the OP's emphasis on not going to the debugger quickly, resistance to that -- while mostly really identifying with the OP's framework. BUT. Where I think the OP is right is what you do with the debugger. I agree with OP that you shouldn't use the debugger just for generalized "seeing what is going on" or jump too quickly into "finding the problem". You should be using it to prove or disprove a hypothesis. Initially, hypotheses about your mental model, not about the problem itself. But you can and often need to use the debugger in order to build the mental model. The next part of quote from OP is: > What Ken was doing was constructing a mental model of the program. When something broke it was an error in that model. He'd think of how that error might happen or where the code might not satisfy the model. You absolutely can and I'd go so far as to say even should use the debugger as an aid to building the mental model. If you're not Ken Thompson, and depending on how familiar you are with the code, and the nature of the platform and codebase, it isn't necessarily useful to just be totally inside your head trying to "build the mental model" and think of what might be breaking it. So it's not really about how quickly you pull out the debugger. But it is absolutely good advice not to immediately start debugging ("find the problem") with it, and not to just use the debugger aimlessly, but to start with a separate "mental model phase". I see why the OP wants to tell you not to use the debugger in this phase, to help you understand it is in fact a separate phase, and because the debugger can lead you to miss the forest for the leaf. But you can and probably often should use the debugger there too, I agree. But you use it to build your mental model -- if you aren't sure you have a mental model, make some guesses (about how the code works, NOT about "what's wrong"), then use the debugger to confirm or deny them. To figure out "how does this code work, big picture" before trying to use it to figure out "why is it broken". IF you are pretty familiar with the particular platform and codebase, AND pretty experienced with doing this kind of work, you might not find the debugger helpful for that first phase, like Ken Thomson. But i'd argue it's also fine to use the debugger there, it's a question of what you are using it to do.
bsder 5y ago
I generally use the debugger to test my hypothesis. This is something that too many people overlook when debugging. My debugging goes like this: 1) I need to have a model in my head of what I think is going on even if simple. 2) I make a change X with the idea that result Y should happen. VERIFYING THIS HYPOTHESIS IS VITALLY IMPORTANT. Your change should create cause and effect. 3) I test that result Y actually happened--generally this is where I'm using the debugger. 4a) If result Y happened and result Y means my program is fixed, congratulations, you are done. 4b) If result Y happened but my program is not yet fixed, go back to make another hypothesis and go back to step 2. 4c) If result Y did not happen, undo change X, update your mental model and go back to step 1. This is where the "think hard" part mostly comes in. The toughest bugs are the ones that require you to update your mental model the most.

m463 5y ago

If you wrote the code, there's likely a mental model in your head. This is why it is relatively easy to work on a project you were in on from the beginning. and why it is always a struggle to come into a project with an existing codebase. I've spent many years working on large existing codebases and it gets in the way of liking your job.

drewcoo 5y ago
> there's likely a mental model in your head Yes, and unless there was a simple typo, the problem is the model in your head. Most bugs come from unchallenged assumptions.
- Jensson 5y ago
  Knowing that a particular bug can happen is usually enough to debug your mental model though. You just trace back to different assumptions you made and see which of them could cause this bug if wrong, and then fix it.
- wruza 5y ago
  In my personal experience, most bugs come from changing a working code, especially at a refactoring. It mixes assumptions in an unpredictable way, because of chaotic code rearrangements and lots attention required to do that correctly. First-time code is much easier to challenge (if you ignore repl development phase which is not “code” yet).

xyzzy123 5y ago

If you have the luxury of 2 people on a bug, having one on strategy & idea generation and the other down in the weeds confirming hypotheses and gathering data is a common pattern. For some reason, not being on the keyboard or operating tools seems to free up a lot of mental bandwidth. Nothing gets done without someone being hands-on though so arguably yes, that role is more important. If there's just 1 person they usually have to stop and think really hard periodically and keep very good notes to avoid getting lost in the weeds.

bhk 5y ago

In my experience, a debugger is most helpful when you are unsure of the behavior of the language or libraries or machine on which you are working. It shows you what happens at various levels and validate or invalidate your assumptions or what you were led to believe. Sitting and thinking on the other hand, is best for dealing with errors in your own thinking. Ideally, we should have a well-thought-out argument for why something will work, so when it doesn't, our argument needs to be refined.

zwieback 5y ago
Exactly, perfect summary. Expanding on the first point, a debugger is invaluable for me if I can run my program up to a point and then use the "immediate" window to run a few API calls to see what would happen in that context.
svachalek 5y ago
I agree. Unfortunately over my career (going on 30 years) the ratio of my thinking to other people's thinking in the systems I have to develop is shrinking and shrinking. Modern software runs on systems layered onto systems layered on systems of systems. "Debuggers" are more sorely needed than ever but the traditional concept of a debugger is decades out of date and sorely needs to be reinvented for heterogenous, distributed, and parallel interactions.
- wilburTheDog 5y ago
  I think this is very true. The most vexing problems I've run into lately have all been because something is going wrong in a step I don't have much control over. Like in one case the proxy was stripping out the Authorization header for PUT requests (seriously). Or in another the corporate-signed certs used by all the on-prem systems weren't validating on AWS because they don't have a valid root cert. I really wish I had a "debugger" that could trace the interconnections between all the systems my app depends on.
  - spockz 5y ago
    In both cases you would have been helped by proper error reporting and by tracing. (See OpenTelemetry) For the first the remote server could say there is no Authorization header where tracing confirms it was there when sent by the client but not by the proxy. For the second it would be really nice to see more detailed logging on the server side about why a particular peer certificate was rejected. My biggest gripe with current OpenSSL and Java TLS implementations is this lack of detailed reporting.

mannykannot 5y ago

I don't think this was intended to be an either/or choice, and "quit thinking and look" can be read as "if thinking isn't working, get more data" (and it could be read as such even if that wasn't the author's intended meaning!) Even if thinking has not solved the problem, it may help you find where to look next. One tip I have is that even if you wrote the code and think you know how it works, stepping through it in your mind might lead you to see where you are making a questionable assumption. You may think that you can do that just as well in the debugger, and you may be right, but there are two ways of using the debugger: one is to hit 'step' and see what happens; the other is to figure out what you expect will happen, and then check if your expectation was correct. There have been many times when I have stepped up to the point where the bug is about to happen, and then realized what is wrong before taking that final step.

masswerk 5y ago

Personally, in the rare cases I really do not understand the behavior, I tend to just stare at the code. (It's not a recipe or so, it's just what I tend to do.) Having subtile or no syntax coloring helps.

mcguire 5y ago

It's a learnable skill although it is quite difficult, particularly if you didn't write the code (recently). And it's very useful because at some point the debugger (or logging[1]) will let you down. [1] I myself don't use a debugger much; I'm frequently working on something distributed where a debugger is laughably useless.

coldtea 5y ago

>and very seldom is the problem space simple enough that I can hold a complete mental model of it in my head You already have a more narrow sense of the problem space, else you wouldn't know where to look with the debugger either...

jrochkind1 5y ago

So, I kind of agree with you. I had the same thought about the OP's emphasis on not going to the debugger quickly, resistance to that -- while mostly really identifying with the OP's framework. BUT. Where I think the OP is right is what you do with the debugger. I agree with OP that you shouldn't use the debugger just for generalized "seeing what is going on" or jump too quickly into "finding the problem". You should be using it to prove or disprove a hypothesis. Initially, hypotheses about your mental model, not about the problem itself. But you can and often need to use the debugger in order to build the mental model. The next part of quote from OP is: > What Ken was doing was constructing a mental model of the program. When something broke it was an error in that model. He'd think of how that error might happen or where the code might not satisfy the model. You absolutely can and I'd go so far as to say even should use the debugger as an aid to building the mental model. If you're not Ken Thompson, and depending on how familiar you are with the code, and the nature of the platform and codebase, it isn't necessarily useful to just be totally inside your head trying to "build the mental model" and think of what might be breaking it. So it's not really about how quickly you pull out the debugger. But it is absolutely good advice not to immediately start debugging ("find the problem") with it, and not to just use the debugger aimlessly, but to start with a separate "mental model phase". I see why the OP wants to tell you not to use the debugger in this phase, to help you understand it is in fact a separate phase, and because the debugger can lead you to miss the forest for the leaf. But you can and probably often should use the debugger there too, I agree. But you use it to build your mental model -- if you aren't sure you have a mental model, make some guesses (about how the code works, NOT about "what's wrong"), then use the debugger to confirm or deny them. To figure out "how does this code work, big picture" before trying to use it to figure out "why is it broken". IF you are pretty familiar with the particular platform and codebase, AND pretty experienced with doing this kind of work, you might not find the debugger helpful for that first phase, like Ken Thomson. But i'd argue it's also fine to use the debugger there, it's a question of what you are using it to do.

bsder 5y ago

I generally use the debugger to test my hypothesis. This is something that too many people overlook when debugging. My debugging goes like this: 1) I need to have a model in my head of what I think is going on even if simple. 2) I make a change X with the idea that result Y should happen. VERIFYING THIS HYPOTHESIS IS VITALLY IMPORTANT. Your change should create cause and effect. 3) I test that result Y actually happened--generally this is where I'm using the debugger. 4a) If result Y happened and result Y means my program is fixed, congratulations, you are done. 4b) If result Y happened but my program is not yet fixed, go back to make another hypothesis and go back to step 2. 4c) If result Y did not happen, undo change X, update your mental model and go back to step 1. This is where the "think hard" part mostly comes in. The toughest bugs are the ones that require you to update your mental model the most.