4 ms·

If you have a reproducible test case that runs reasonably quickly, then I think printf debugging is usually just as good as a "real" debugger, and a lot easier.

If you have a reproducible test case that runs reasonably quickly, then I think printf debugging is usually just as good as a "real" debugger, and a lot easier. I typically have my test output log open in one editor frame. I make a change to the code in another frame, save it, my build system immediately re-runs the test, and the test log immediately refreshes. So when the test isn't working, I just keep adding printfs to get more info out of the log until I figure it out. This sounds so dumb but it works out to be equivalent to some very powerful debugger features. You don't need a magical debugger that lets you modify code on-the-fly while continuing the same debug session... just change the code and re-run the test. You don't need a magical record-replay debugger that lets you go "back in time"... just add a printf earlier in the control flow and re-run the test. You don't need a magical debugger that can break when a property is modified... just temporarily modify the property to have a setter function and printf in the setter. Most importantly, though, this sort of debugging is performed using the same language and user interface I use all day to write code in the first place, so I don't have to spend time trying to remember how to do stuff in a debugger... it's just code. BUT... this is all contingent on having fast-running automated tests that can reproduce your bugs. But you should have that anyway.

eru 2y ago
> BUT... this is all contingent on having fast-running automated tests that can reproduce your bugs. But you should have that anyway. Ideally, yes. But for many bugs, getting to a reproduction is already more than half the battle. And a debugger can help you with that.
- skissane 2y ago
  > But for many bugs, getting to a reproduction is already more than half the battle. Today I am working on a bug where a token expires after 2 hours and then we fail to request a new one instead we just keep on using the now expired one, which (of course) doesn’t work. I have a script to reproduce it but it takes 2 hours to run. It would be great if there was some configuration knob to turn down the expiry just for this test so we can reproduce it faster - but there isn’t because nobody thought of that.
  - atq2119 2y ago
    Maybe it takes less than 2 hours to add such a configuration knob?
    - skissane 2y ago
      That’s a good idea except it is in another team’s service and I’m not very familiar with their code. But I can try Well, the hard part isn’t actually the code change (reasonably obvious) it is deploying my own copy of their service…
      - pantalaimon 2y ago
        Can you just set the local time two hours into the future? Or modify the expiry time locally / not use the one from the token at all.
        skissane 2y ago
        The token isn’t being verified in my service it is being verified in a remote service. I can’t easily change the clock on the remote service. It runs in a K8S pod and (AFAIK) K8S doesn’t support time namespaces. And even if it does, the pod is deployed by a custom K8S operator so unsure if I could make the operator turn that on even if it is available. And the remote service is a complex monolith which uses lots of RAM so trying to run it on my own laptop will be painful
        majewsky 2y ago
        Reminds me of the classic adage that "everything can be solved by another layer of abstraction, except for the problem of too much abstraction layers".
        kmoser 2y ago
        Or modify the token itself so it's no longer valid? (Although that might come back with "invalid token" rather than "expired token.")
whatevertrevor 2y ago
> Most importantly, though, this sort of debugging is performed using the same language and user interface I use all day to write code in the first place, so I don't have to spend time trying to remember how to do stuff in a debugger... it's just code. Absolutely! Running a commandline debugger adds additional context I have to keep in my head (syntax for all the commands, output format etc), that actively competes with context required to debug my code. Printfs work just fine without incurring that penalty, granted this argument applies less to IDE debuggers because their UX is usually intuitive.
seanmcdirmid 2y ago
The crashes/bugs I deal with are rarely straight down failures, they are often the 1 out of 100 runs kind, so printf debugging is the only way to go really. And I used to be big on using debuggers, but now I’m horribly out of practice.
- skissane 2y ago
  > The crashes/bugs I deal with are rarely straight down failures, they are often the 1 out of 100 runs kind, so printf debugging is the only way to go really. Another thing I’ve found helpful, is to write out a “system state dump” (say in JSON) to a file whenever certain errors happen. Like we had a production system that was randomly hanging and running out of DB connections. So now whenever the DB connection pool is exhausted, it dumps a JSON file to S3 listing the status of every DB connection, including the stack dump of the thread that owns it, the HTTP request that thread is serving, the user account, etc. Once we did that, it went from “we don’t understand why this application is randomly falling over” to “oh this is the thing that is consistently triggering it” When it writes a dump, it then starts a “lockout period” in which it won’t write any further dumps even if the error reoccurs. Don’t want to make a meltdown worse by getting bogged down endlessly writing out diagnostics.
  - alextingle 2y ago
    Why not just dump a core, rather than going to all that effort?
    - skissane 2y ago
      Because it is a lot easier to analyse a few megabytes of JSON than a heap dump which is gigabytes in size. Not my original idea, I got it from some Oracle products which have a similar feature (most notably Oracle Database)
- roca 2y ago
  Record-and-replay debuggers are often better than anything else for debugging intermittent failures: run the program with recording many times until you eventually get the bug, then debug that recording at your leisure; you'll never have to waste time reproducing it again.
rtpg 2y ago
I would say that like... saying "OK so `a.b` looks like this, what does `a.b.c` look like?" is a very nice flow with deep and messy objects (especially when working in languages with absolutely garbage default serialization logic like Javascript) But even with a debugger, there's still loads of value of sitting up, moving from writing a bunch of single line statements all over, and to writing a real test harness ASAP to not have to rely on the debugger. For any non-trivial problem, you'll often very quickly appreciate a properly formatted output stack, and that output will be shaped to the problem you are looking at. Very hard for an in-process debugger to have the answer for you there immediately. Being serious about debugging can even get into things like writing actual new entrypoints (scripts with arguments and whatnot) and things like adding branches togglable through environment variables. I think a lot of people's mindset in debugging is "if I walk the tightrope and put in _one more hack_ I'll find my answer" and it gets more and more precarious. Debugging is always a bit of an exercise of mental gymnastics, but if you practice being really good at printf debugging _and_ configuring program flow easily, you can be a lot less stressed out. Like if you think your problem is going to take more than 20 minutes to debug, you probably should start writing a couple helper functions to get you on the right foot.
NikkiA 2y ago
Things like variable watchpoints mean that debuggers are still the better option.
- guelo 2y ago
  You can add the condition to the print statement and grep for it if it gets too verbose.
  - NikkiA 2y ago
    Watchpoints can tell you which function is causing the variable to change, making printf's conditional cannot.
roca 2y ago
Adding print statements, rebuilding and rerunning the program and figuring out where in the logs the bug showed up this time can a lot more tedious than setting a (possibly conditional) breakpoint in a reverse-execution debugger and doing "reverse-continue".
deleted 2y ago
[deleted]

eru 2y ago

> BUT... this is all contingent on having fast-running automated tests that can reproduce your bugs. But you should have that anyway. Ideally, yes. But for many bugs, getting to a reproduction is already more than half the battle. And a debugger can help you with that.

skissane 2y ago
> But for many bugs, getting to a reproduction is already more than half the battle. Today I am working on a bug where a token expires after 2 hours and then we fail to request a new one instead we just keep on using the now expired one, which (of course) doesn’t work. I have a script to reproduce it but it takes 2 hours to run. It would be great if there was some configuration knob to turn down the expiry just for this test so we can reproduce it faster - but there isn’t because nobody thought of that.
- atq2119 2y ago
  Maybe it takes less than 2 hours to add such a configuration knob?
  - skissane 2y ago
    That’s a good idea except it is in another team’s service and I’m not very familiar with their code. But I can try Well, the hard part isn’t actually the code change (reasonably obvious) it is deploying my own copy of their service…
    - pantalaimon 2y ago
      Can you just set the local time two hours into the future? Or modify the expiry time locally / not use the one from the token at all.
      - skissane 2y ago
        The token isn’t being verified in my service it is being verified in a remote service. I can’t easily change the clock on the remote service. It runs in a K8S pod and (AFAIK) K8S doesn’t support time namespaces. And even if it does, the pod is deployed by a custom K8S operator so unsure if I could make the operator turn that on even if it is available. And the remote service is a complex monolith which uses lots of RAM so trying to run it on my own laptop will be painful
        majewsky 2y ago
        Reminds me of the classic adage that "everything can be solved by another layer of abstraction, except for the problem of too much abstraction layers".
      - kmoser 2y ago
        Or modify the token itself so it's no longer valid? (Although that might come back with "invalid token" rather than "expired token.")

whatevertrevor 2y ago

> Most importantly, though, this sort of debugging is performed using the same language and user interface I use all day to write code in the first place, so I don't have to spend time trying to remember how to do stuff in a debugger... it's just code. Absolutely! Running a commandline debugger adds additional context I have to keep in my head (syntax for all the commands, output format etc), that actively competes with context required to debug my code. Printfs work just fine without incurring that penalty, granted this argument applies less to IDE debuggers because their UX is usually intuitive.

seanmcdirmid 2y ago

The crashes/bugs I deal with are rarely straight down failures, they are often the 1 out of 100 runs kind, so printf debugging is the only way to go really. And I used to be big on using debuggers, but now I’m horribly out of practice.

skissane 2y ago
> The crashes/bugs I deal with are rarely straight down failures, they are often the 1 out of 100 runs kind, so printf debugging is the only way to go really. Another thing I’ve found helpful, is to write out a “system state dump” (say in JSON) to a file whenever certain errors happen. Like we had a production system that was randomly hanging and running out of DB connections. So now whenever the DB connection pool is exhausted, it dumps a JSON file to S3 listing the status of every DB connection, including the stack dump of the thread that owns it, the HTTP request that thread is serving, the user account, etc. Once we did that, it went from “we don’t understand why this application is randomly falling over” to “oh this is the thing that is consistently triggering it” When it writes a dump, it then starts a “lockout period” in which it won’t write any further dumps even if the error reoccurs. Don’t want to make a meltdown worse by getting bogged down endlessly writing out diagnostics.
- alextingle 2y ago
  Why not just dump a core, rather than going to all that effort?
  - skissane 2y ago
    Because it is a lot easier to analyse a few megabytes of JSON than a heap dump which is gigabytes in size. Not my original idea, I got it from some Oracle products which have a similar feature (most notably Oracle Database)
roca 2y ago
Record-and-replay debuggers are often better than anything else for debugging intermittent failures: run the program with recording many times until you eventually get the bug, then debug that recording at your leisure; you'll never have to waste time reproducing it again.

rtpg 2y ago

I would say that like... saying "OK so `a.b` looks like this, what does `a.b.c` look like?" is a very nice flow with deep and messy objects (especially when working in languages with absolutely garbage default serialization logic like Javascript) But even with a debugger, there's still loads of value of sitting up, moving from writing a bunch of single line statements all over, and to writing a real test harness ASAP to not have to rely on the debugger. For any non-trivial problem, you'll often very quickly appreciate a properly formatted output stack, and that output will be shaped to the problem you are looking at. Very hard for an in-process debugger to have the answer for you there immediately. Being serious about debugging can even get into things like writing actual new entrypoints (scripts with arguments and whatnot) and things like adding branches togglable through environment variables. I think a lot of people's mindset in debugging is "if I walk the tightrope and put in _one more hack_ I'll find my answer" and it gets more and more precarious. Debugging is always a bit of an exercise of mental gymnastics, but if you practice being really good at printf debugging _and_ configuring program flow easily, you can be a lot less stressed out. Like if you think your problem is going to take more than 20 minutes to debug, you probably should start writing a couple helper functions to get you on the right foot.

NikkiA 2y ago

Things like variable watchpoints mean that debuggers are still the better option.

guelo 2y ago
You can add the condition to the print statement and grep for it if it gets too verbose.
- NikkiA 2y ago
  Watchpoints can tell you which function is causing the variable to change, making printf's conditional cannot.

roca 2y ago

Adding print statements, rebuilding and rerunning the program and figuring out where in the logs the bug showed up this time can a lot more tedious than setting a (possibly conditional) breakpoint in a reverse-execution debugger and doing "reverse-continue".

deleted 2y ago

[deleted]