7 ms·

I have become less and less enamored of gdb (or IDE equivalent) debugging over my many years in software. Debugging is often very confusing with multithreaded c

I have become less and less enamored of gdb (or IDE equivalent) debugging over my many years in software. Debugging is often very confusing with multithreaded code. It only works in one language at a time. If you're doing Java with JNI for example, the code called through JNI is a black box. If you are doing anything involving multiple processes, or distribution, debuggers are useless. But the real problem, and top-level issue, is that you are constantly rooting around the very lowest level of your code, and it is difficult and labor-intensive to get a higher level view of what's going on. On the other hand, well-designed logging code is there when you need it, can provide a view at any level of abstraction that you'd like (if you've written the logging code), and allows you to go forward and backward in time easily and repeatedly. The project that really pushed me in this direction was working on a cluster. My piece of the system would start on one of the nodes, and delegate work to a fixed number of other nodes, or to all nodes. Out of necessity, I put a lot of work into debugging code, taking care to include the right information in each line (node id, time to nsec, source id, line number), carefully formatted to enable sorting and filtering, log rotation to make sure that I could focus on recent events or a longer timescale, and a lot of attention to the information included in each line and how it was formatted. Because this was a long-running distributed system, I would often need to gather logs from across the cluster, going back hours or days, and merge and sort the files, and then debugging could start. Going along with logging was heavy use of assertions, which were enabled even in production. I definitely did not want my code going forward doing random unanticipated crazy things past the point of an assertion failure.

mirceal 4y ago
If you're at the point where you need to attach a debugger, in production, you have already lost. I don't care how much you like debuggers and debugging. Anything, but the most trivial, single-threaded code will be problematic to debug. So? What do you do? Logging at the right level + turn up logging. Metrics - proper metrics. Core dumps. Debugging turns into an exercise of grepping/inspecting the logs/metrics and/or working with the dump. Anyone that tells you differently is massaging the truth.
- guhidalg 4y ago
  I agree. In today's world of cloud services, good logging is the single most important way to debug issues.
10000truths 4y ago
Working with gdb is quite the learning curve. Luckily, it’s mostly unnecessary, except in the rare case where you actually need register-level debugging or hardware watchpoints. But it can still be done if you’re willing to suffer through the docs. For debugging multithreaded code, you can run a gdb script in batch/non-interactive mode, and direct the gdb output to some out-of-band channel, like a file or other terminal session. For debugging JNI, you can try running Java under gdb and set a breakpoint on the extension library entry function before it’s loaded.
- geophile 4y ago
  ... or you could just log. I don't understand the extreme attachment to interactive debugging that I see in some of the comments here. It's like people are so attached to this idea that they will do anything to make it work as each complication is added. Furthermore, the amount of attention and manual labor needed to use a debugger effectively is just staggering. One mistake, e.g. you step over a function you should have stepped into, and you've got to start over. Or you could just log. Yes, writing logging code is no fun. But you don't even have to do it all at once. In fact, you probably shouldn't. I often found that I didn't know what to log until I had to investigate some problems turned up in integration testing. So I added logging to solve the problem, and then benefitted from it forever.
  - GeorgeTirebiter 4y ago
    The lure of gdb is that you do not need to write the logging code. But I am reminded of the diddy: "If a system has a Powerful Debugger, what does that say about the apparent need for a Powerful Debugger?" --- I read that as: if you have a kick-a* debugger, you have a really complicated system. Maybe the answer is to make the system less complicated.
    - geophile 4y ago
      That doesn't make sense to me. Complication is mainly a property of the software you're writing, not the language in which the software is written.
jayd16 4y ago
> If you are doing anything involving multiple processes, or distribution, debuggers are useless How so? Can't you just attach to each process? Do you mean at production scale?
- geophile 4y ago
  Even in development. You could, I suppose, but I'm not sure why you would want to. Going back and forth to two debuggers (at least) seems like torture. And with multiple processes, timing and synchronization issues could make interactive debugging a real nightmare. Why? Why would you do this? Interactive debugging doesn't scale in any dimension.
  - jayd16 4y ago
    It's really not that hard, especially if you can run them all from the same IDE. I can see how it starts to get out hand but it's not at n=2.
    - geophile 4y ago
      Early in my software career, I was debugging a compiler problem. Version X worked, version X+1 did not. I debugged by tracing both step by step, in two gdb sessions, looking for when behavior diverged. Pure torture, and that was an extremely simple case -- the processes were not communicating in any way. All the usual problems, but doubled. Oops, I stepped too far, start over. Oops, the other one stepped too far, start over. Oops, start over, oops, start over. Never again.
  - mping 4y ago
    I used intellij to debug 3 processes over ssh, it worked seamlessly. The problem with logging is that you have to know where to log. Debuggers are excellent for exploration, where logging implies you already put the code. If you forget some log you need to do the whole build/deploy dance, with a debugger I just restart. There are even time travel debuggers that allow replays.
    - 4y ago
pradn 4y ago
At the level of single process, on one machine, I've found a mix of debug logs, trace points, conditional breakpoints, and the ability to execute arbitrary code when at a breakpoint to be super helpful in debugging complex code with many layers. I was working on a compiler of sorts. The only time I had access to all of these was when working on C# in Visual Studio. Being able to run arbitrary code at the breakpoint-paused stack is incredibly useful. With multiple processes or machines, it's helpful to pass around a trace ID and then have all the debug logs print the trace ID + timestamp at each event. This is a rather manual process; I wish there were automatic ways of doing this. At Google, the best we get is filtering logs by the RPC being processed at the time; but this ends at a particular process. Cross-process RPC traces are mostly sampled "stack traces"; you don't get to inject custom debug messages in those.
- pjmlp 4y ago
  On the Microsoft world you get that automated via ETW and in case of running on Azure, Application Insights. With debug symbols you can even pinpoint those events into code. The only competition I know to these workflows is using Java alongside Flight Recorder.
smrtinsert 4y ago
Add thread ids to your logs
- slaymaker1907 4y ago
  Context IDs are often even better since the physical thread executing some high-level operation might change (i.e. if you use a thread pool).
- saiya-jin 4y ago
  I would be so screwed without this 10 times a month in one messaging app I've developed to integrate our archaic pl/sql core banking system with 21st century reality. Luckily its not performance-oriented but more reliability-oriented so I can get away with more logging than usually necessary. It seems obsolete and over-the-top till it saves everyone's a*
  - sitkack 4y ago
    Sometimes the debug log is real product.
majormajor 4y ago
I've had luck with debugging locally, in a single-process context, not expecting to reproduce the problem when unfamiliar with the codebase to get a good understanding of what's happening when in the code. That tells me where I need to add metrics, logging, etc for the distributed deploy.
invalidname 4y ago
The problem is that in order to add a missing log in production you need to effectively redeploy. That isn't tenable for a polyglot production cluster. There's a rising field of developer observability which is specifically designed to provide polyglot cloud "debugging" capabilities. One of the capabilities is to inject new logs dynamically.
- geophile 4y ago
  And if you need to fix a bug in production you need to redeploy too. Bugs and logs and testing are all related. You obviously (should) do unit testing, integration testing, system testing before production. As you do these things, you will find and fix bugs. Ideally, you would add logging to help you diagnose the bugs you are fixing. Congratulations, you now have logging in code known to have been buggy, which will serve you well in production if bugs not caught by your tests turn up. But bugs in previously reliable code do slip through, and this gives rise to the need for previously unanticipated logging. If either of these things happens a lot, the problem is your testing leading up to production. In other words, if you often find yourself needing dynamically deployed logging, then the lack of this feature is not the problem you should be paying attention to. Also, dynamically deployed logging is basically shipping untested code, no?
  - invalidname 4y ago
    The difference between deploying once and deploying just to add more logs. And more logs is a huge difference. I don't know about your environment but I can't push something to production on my own schedule. I need approval, process, etc. This is painful to do just to add a log which might not include all the information I need.
    - geophile 4y ago
      The more constraints you have on pushing something to production, the less it makes sense to deploy logging dynamically! Those constraints are there to minimize risk, ensure quality and security, etc. Dynamically deployed logging seems to provide an extremely large loophole. For example, consider the recent Java logging fiasco.
roca 4y ago
> I have become less and less enamored of gdb (or IDE equivalent) debugging over my many years in software. Debugging is often very confusing with multithreaded code. It only works in one language at a time. If you're doing Java with JNI for example, the code called through JNI is a black box. If you are doing anything involving multiple processes, or distribution, debuggers are useless. These are true of plain-old-gdb but they're not inherent to interactive debugging. rr and Pernosco address most of these issues and with more work we could do even better. In particular: rr is great for debugging multithreaded code; record a race once and then debug the replay as much as you want until you've figured it out. rr will also record multiprocess code and let you debug whichever process you want; Pernosco goes further and supports seamless debugging across all processes. https://pernos.co/about/javascript/ https://pernos.co/about/javascript/ shows how Pernosco can support debugging high-level code (JS) and C++ code at the same time. rr can record processes on multiple machines; we don't have an integrated debugging experience for distributed systems yet, but it's not hard to see how that could be done. > But the real problem, and top-level issue, is that you are constantly rooting around the very lowest level of your code, and it is difficult and labor-intensive to get a higher level view of what's going on. That depends on the debugger. Interactive debuggers can do higher-level things if we want them to, e.g https://pernos.co/about/expressions/ https://pernos.co/about/expressions/

mirceal 4y ago

If you're at the point where you need to attach a debugger, in production, you have already lost. I don't care how much you like debuggers and debugging. Anything, but the most trivial, single-threaded code will be problematic to debug. So? What do you do? Logging at the right level + turn up logging. Metrics - proper metrics. Core dumps. Debugging turns into an exercise of grepping/inspecting the logs/metrics and/or working with the dump. Anyone that tells you differently is massaging the truth.

guhidalg 4y ago
I agree. In today's world of cloud services, good logging is the single most important way to debug issues.

10000truths 4y ago

Working with gdb is quite the learning curve. Luckily, it’s mostly unnecessary, except in the rare case where you actually need register-level debugging or hardware watchpoints. But it can still be done if you’re willing to suffer through the docs. For debugging multithreaded code, you can run a gdb script in batch/non-interactive mode, and direct the gdb output to some out-of-band channel, like a file or other terminal session. For debugging JNI, you can try running Java under gdb and set a breakpoint on the extension library entry function before it’s loaded.

geophile 4y ago
... or you could just log. I don't understand the extreme attachment to interactive debugging that I see in some of the comments here. It's like people are so attached to this idea that they will do anything to make it work as each complication is added. Furthermore, the amount of attention and manual labor needed to use a debugger effectively is just staggering. One mistake, e.g. you step over a function you should have stepped into, and you've got to start over. Or you could just log. Yes, writing logging code is no fun. But you don't even have to do it all at once. In fact, you probably shouldn't. I often found that I didn't know what to log until I had to investigate some problems turned up in integration testing. So I added logging to solve the problem, and then benefitted from it forever.
- GeorgeTirebiter 4y ago
  The lure of gdb is that you do not need to write the logging code. But I am reminded of the diddy: "If a system has a Powerful Debugger, what does that say about the apparent need for a Powerful Debugger?" --- I read that as: if you have a kick-a* debugger, you have a really complicated system. Maybe the answer is to make the system less complicated.
  - geophile 4y ago
    That doesn't make sense to me. Complication is mainly a property of the software you're writing, not the language in which the software is written.

jayd16 4y ago

> If you are doing anything involving multiple processes, or distribution, debuggers are useless How so? Can't you just attach to each process? Do you mean at production scale?

geophile 4y ago
Even in development. You could, I suppose, but I'm not sure why you would want to. Going back and forth to two debuggers (at least) seems like torture. And with multiple processes, timing and synchronization issues could make interactive debugging a real nightmare. Why? Why would you do this? Interactive debugging doesn't scale in any dimension.
- jayd16 4y ago
  It's really not that hard, especially if you can run them all from the same IDE. I can see how it starts to get out hand but it's not at n=2.
  - geophile 4y ago
    Early in my software career, I was debugging a compiler problem. Version X worked, version X+1 did not. I debugged by tracing both step by step, in two gdb sessions, looking for when behavior diverged. Pure torture, and that was an extremely simple case -- the processes were not communicating in any way. All the usual problems, but doubled. Oops, I stepped too far, start over. Oops, the other one stepped too far, start over. Oops, start over, oops, start over. Never again.
- mping 4y ago
  I used intellij to debug 3 processes over ssh, it worked seamlessly. The problem with logging is that you have to know where to log. Debuggers are excellent for exploration, where logging implies you already put the code. If you forget some log you need to do the whole build/deploy dance, with a debugger I just restart. There are even time travel debuggers that allow replays.
  - 4y ago

pradn 4y ago

At the level of single process, on one machine, I've found a mix of debug logs, trace points, conditional breakpoints, and the ability to execute arbitrary code when at a breakpoint to be super helpful in debugging complex code with many layers. I was working on a compiler of sorts. The only time I had access to all of these was when working on C# in Visual Studio. Being able to run arbitrary code at the breakpoint-paused stack is incredibly useful. With multiple processes or machines, it's helpful to pass around a trace ID and then have all the debug logs print the trace ID + timestamp at each event. This is a rather manual process; I wish there were automatic ways of doing this. At Google, the best we get is filtering logs by the RPC being processed at the time; but this ends at a particular process. Cross-process RPC traces are mostly sampled "stack traces"; you don't get to inject custom debug messages in those.

pjmlp 4y ago
On the Microsoft world you get that automated via ETW and in case of running on Azure, Application Insights. With debug symbols you can even pinpoint those events into code. The only competition I know to these workflows is using Java alongside Flight Recorder.

smrtinsert 4y ago

Add thread ids to your logs

slaymaker1907 4y ago
Context IDs are often even better since the physical thread executing some high-level operation might change (i.e. if you use a thread pool).
saiya-jin 4y ago
I would be so screwed without this 10 times a month in one messaging app I've developed to integrate our archaic pl/sql core banking system with 21st century reality. Luckily its not performance-oriented but more reliability-oriented so I can get away with more logging than usually necessary. It seems obsolete and over-the-top till it saves everyone's a*
- sitkack 4y ago
  Sometimes the debug log is real product.

majormajor 4y ago

I've had luck with debugging locally, in a single-process context, not expecting to reproduce the problem when unfamiliar with the codebase to get a good understanding of what's happening when in the code. That tells me where I need to add metrics, logging, etc for the distributed deploy.

invalidname 4y ago

The problem is that in order to add a missing log in production you need to effectively redeploy. That isn't tenable for a polyglot production cluster. There's a rising field of developer observability which is specifically designed to provide polyglot cloud "debugging" capabilities. One of the capabilities is to inject new logs dynamically.

geophile 4y ago
And if you need to fix a bug in production you need to redeploy too. Bugs and logs and testing are all related. You obviously (should) do unit testing, integration testing, system testing before production. As you do these things, you will find and fix bugs. Ideally, you would add logging to help you diagnose the bugs you are fixing. Congratulations, you now have logging in code known to have been buggy, which will serve you well in production if bugs not caught by your tests turn up. But bugs in previously reliable code do slip through, and this gives rise to the need for previously unanticipated logging. If either of these things happens a lot, the problem is your testing leading up to production. In other words, if you often find yourself needing dynamically deployed logging, then the lack of this feature is not the problem you should be paying attention to. Also, dynamically deployed logging is basically shipping untested code, no?
- invalidname 4y ago
  The difference between deploying once and deploying just to add more logs. And more logs is a huge difference. I don't know about your environment but I can't push something to production on my own schedule. I need approval, process, etc. This is painful to do just to add a log which might not include all the information I need.
  - geophile 4y ago
    The more constraints you have on pushing something to production, the less it makes sense to deploy logging dynamically! Those constraints are there to minimize risk, ensure quality and security, etc. Dynamically deployed logging seems to provide an extremely large loophole. For example, consider the recent Java logging fiasco.

roca 4y ago

> I have become less and less enamored of gdb (or IDE equivalent) debugging over my many years in software. Debugging is often very confusing with multithreaded code. It only works in one language at a time. If you're doing Java with JNI for example, the code called through JNI is a black box. If you are doing anything involving multiple processes, or distribution, debuggers are useless. These are true of plain-old-gdb but they're not inherent to interactive debugging. rr and Pernosco address most of these issues and with more work we could do even better. In particular: rr is great for debugging multithreaded code; record a race once and then debug the replay as much as you want until you've figured it out. rr will also record multiprocess code and let you debug whichever process you want; Pernosco goes further and supports seamless debugging across all processes. https://pernos.co/about/javascript/ https://pernos.co/about/javascript/ shows how Pernosco can support debugging high-level code (JS) and C++ code at the same time. rr can record processes on multiple machines; we don't have an integrated debugging experience for distributed systems yet, but it's not hard to see how that could be done. > But the real problem, and top-level issue, is that you are constantly rooting around the very lowest level of your code, and it is difficult and labor-intensive to get a higher level view of what's going on. That depends on the debugger. Interactive debuggers can do higher-level things if we want them to, e.g https://pernos.co/about/expressions/ https://pernos.co/about/expressions/