5 ms·

As someone who has dedicated their career to understanding and debugging software in production[1], it's hard to not be depressed about this -- not only because

As someone who has dedicated their career to understanding and debugging software in production[1], it's hard to not be depressed about this -- not only because it (emphatically) isn't "awesome", but also because of the failure to couch this in its proper context: this is -- at best -- a kludge to enable small programs to be debugged in development. While this is fine at some level (debugging is, after all, a good thing), small programs in development aren't where the most pernicious bugs are found; sophisticated tooling should (in my opinion) always focus on the production systems in which the nastiest, most undebuggable problems arise. Note that this needn't mean that the tools themselves run in production: for example, Valgrind is incredibly useful because it finds really nasty, production-grade issues -- even if it is only meaningfully usable in development. Similarly, postmortem debugging tooling is extraordinarily valuable for production problems -- and requires no tooling in the production environment itself. We have developed such tooling for node.js[2], and found it to be invaluable. There is nascent postmortem debugging support for Go[3], but there is much more to be done -- just hoping that some in the Go community see the value of debugging production problems! [1] http://www.infoq.com/presentations/Debugging-Production-Systems http://www.infoq.com/presentations/Debugging-Production-Syst... [2] http://www.slideshare.net/bcantrill/node-summit2013 http://www.slideshare.net/bcantrill/node-summit2013 [3] http://www.joyent.com/blog/mdb-support-for-go http://www.joyent.com/blog/mdb-support-for-go

4ad 11y ago
> just hoping that some in the Go community see the value of debugging production problems Some of us do. As I said in another post, Solaris support will only get better. As for Linux, some people obviously care, Go had some form of gdb support from quite early on. Unfortunately, it's not that people don't care about making the gdb support better, it's just that this is all that can be done with the gdb-python interface right now, due to the limitations of gdb... At least, now with frame pointers, dynamic instrumentation works better.
vruiz 11y ago
> just hoping that some in the Go community see the value of debugging production problems! I just can not remember where I heard it (some podcast maybe), but is my understanding that the Go Team hinted the possibility of something like JMX coming to the Go runtime.
FiloSottile 11y ago
Author here. I noticed before both mdb support and the Delve project that other have mentioned, and I find them great. The point of my excitement for godebug resides in the fact that it's an easy to work with, zero complexity added tool to replace most tedious println() debugging sessions. That is, the non-exciting stuff. Not all bugs only happen on production load, architecture, etc. Many times (with DNS servers even more so, I'd say) bugs are simple state or logic bugs. The Heisenbugs exist. The complex resource-constrained races exist. And I find it fun to track them down, but many bugs are much more mundane, and godebug is a great tool for catching those. Speaking instead about hard to trace production bugs, I think the Go design and library deserve their daily share of praise here, since they solve many of those problems. Take Valgrind (or ASAN). Amazing tools. Completely useless in Go -- the language is memory-safe and will make that kind of bugs both hard to introduce and immediately detectable. And about concurrency, first of all the simple design makes bugs harder to introduce and easier to reason about. But even when they make their way in I find provisions like net/http/pprof and its various profile a pleasure to work with. And finally there's obviously the race detector. Still, I can't wait to see more tools developed to cover different Go debugging needs. Today I'm just happy about making my println() session much more pleasant and elegant :)
- bcantrill 11y ago
  The enthusiasm is fine -- I just wish it were couched in the understanding of the limitations of the technique. I also think it's naive to say that Go's design solves "many [production] problems" -- memory-safe languages solve one set of problems, but often introduce or exacerbate another. (The most obvious example being memory exhaustion: postmortem heap analysis is often much easier on a C-based system than in memory-safe languages.[1][2]) More generally, do not fall into the trap of correctness from smugness: nasty production bugs can exist in any language, and time is much better spent on the production tooling to debug such problems rather than simply asserting that such bugs are tautologically impossible. [1] http://dtrace.org/blogs/ahl/2004/07/13/number-11-of-20-libumem/ http://dtrace.org/blogs/ahl/2004/07/13/number-11-of-20-libum... [2] https://queue.acm.org/detail.cfm?id=2039361 https://queue.acm.org/detail.cfm?id=2039361 Edit: added a much better reference.
pron 11y ago
See also Debugging at Full Speed http://medianetwork.oracle.com/video/player/3731019771001 http://medianetwork.oracle.com/video/player/3731019771001
rayiner 11y ago
Debugging software in production is a hard unsolved problem, the available tools are pretty primitive, and it doesn't help that people often don't plan it into their development and deployment cycle. We had a client once using our system, who would see anomalies only after long uptimes at sustained high loads. It took six months to find the cause (a race condition in the database code) because we had so few tools at our disposal to look at the post-mortem state of the program. Running the software at those loads for the weeks or months it took to reproduce the problem under a debugger was obviously not a tenable proposition, and reproducing the variability of the particular real-world traffic was difficult to do in a controlled environment. And the software wasn't written with any eye towards making state available for post-mortem debugging. I guess "don't write complicated multi-threaded software in C" is a pretty good response to the above too. That particular bug couldn't have happened in say Rust (though it could have in Java).
- TylerE 11y ago
  In languages like C it's an unsolved problem. Doesn't mean it's an unsolved problem everywhere (see, e.g. Erlang, which even allows for hot patching without even taking the system down).
  - rayiner 11y ago
    Relevant: http://www.flownet.com/gat/jpl-lisp.html http://www.flownet.com/gat/jpl-lisp.html (scroll to "Debugging a program running on a $100M piece of hardware that is 100 million miles away")
    - TylerE 11y ago
      Yep, old hat in lisp too. Industrial strength VMs tend to do well in this area.
mwcampbell 11y ago
In the QCon talk you referenced in [1], you said that it's impossible to debug a Python or Ruby application from a core file. Why is this? Would an mdb module for Python or Ruby, to delve into the interpreter state and expose it in the debugger, not help or not be feasible?
- bcantrill 11y ago
  "Impossible" is a bit strong, but yes, it requires deep and unholy knowledge of the interpreter in the debugger. For an example of what this looks like, see some of the monster block comments in mdb_v8.c[1]. [1] https://github.com/joyent/illumos-joyent/blob/master/usr/src/cmd/mdb/common/modules/v8/mdb_v8.c#L2003-L2117 https://github.com/joyent/illumos-joyent/blob/master/usr/src...
WalterBright 11y ago
My not-so-humble opinion is that valgrind has saved C (and C++) from oblivion, at least for large scale programs.
zamalek 11y ago
> postmortem debugging tooling is extraordinarily valuable for production problems When all else fails this is really all you are going to have to go by. Coming from the Microsoft side of the story I have come to a conclusion that any sufficiently well designed postmortem debugger is a few steps away from being a live/production debugger. Windbg, as a prime example of this, can be used to debug memory dumps, live processes and remote (TCP) processes using exactly the same debugging tools binaries (and scripts/plugins that you write). Debuggers have to be the biggest source of friction, for me, when adopting things like Go. If you are in charge of "production" you can throw in a few printlns, however, you can't just approach someone like the HP and say "can I put this debug binary on your production system?" No, the absolute best you can hope for is a memory dump and those memory dump had better count as you are only going to get maybe a few. Maybe it's because not many devs deal with this kind of stuff, which is probably why it isn't a top priority for things like e.g. Go. To some of us, though, the most important language of a feature is easily the quality of the debugger associated with it. Either way, great slides Bryan - it's nice to see at least one "modern" language taking these issues seriously.

4ad 11y ago

> just hoping that some in the Go community see the value of debugging production problems Some of us do. As I said in another post, Solaris support will only get better. As for Linux, some people obviously care, Go had some form of gdb support from quite early on. Unfortunately, it's not that people don't care about making the gdb support better, it's just that this is all that can be done with the gdb-python interface right now, due to the limitations of gdb... At least, now with frame pointers, dynamic instrumentation works better.

vruiz 11y ago

> just hoping that some in the Go community see the value of debugging production problems! I just can not remember where I heard it (some podcast maybe), but is my understanding that the Go Team hinted the possibility of something like JMX coming to the Go runtime.

FiloSottile 11y ago

Author here. I noticed before both mdb support and the Delve project that other have mentioned, and I find them great. The point of my excitement for godebug resides in the fact that it's an easy to work with, zero complexity added tool to replace most tedious println() debugging sessions. That is, the non-exciting stuff. Not all bugs only happen on production load, architecture, etc. Many times (with DNS servers even more so, I'd say) bugs are simple state or logic bugs. The Heisenbugs exist. The complex resource-constrained races exist. And I find it fun to track them down, but many bugs are much more mundane, and godebug is a great tool for catching those. Speaking instead about hard to trace production bugs, I think the Go design and library deserve their daily share of praise here, since they solve many of those problems. Take Valgrind (or ASAN). Amazing tools. Completely useless in Go -- the language is memory-safe and will make that kind of bugs both hard to introduce and immediately detectable. And about concurrency, first of all the simple design makes bugs harder to introduce and easier to reason about. But even when they make their way in I find provisions like net/http/pprof and its various profile a pleasure to work with. And finally there's obviously the race detector. Still, I can't wait to see more tools developed to cover different Go debugging needs. Today I'm just happy about making my println() session much more pleasant and elegant :)

bcantrill 11y ago
The enthusiasm is fine -- I just wish it were couched in the understanding of the limitations of the technique. I also think it's naive to say that Go's design solves "many [production] problems" -- memory-safe languages solve one set of problems, but often introduce or exacerbate another. (The most obvious example being memory exhaustion: postmortem heap analysis is often much easier on a C-based system than in memory-safe languages.[1][2]) More generally, do not fall into the trap of correctness from smugness: nasty production bugs can exist in any language, and time is much better spent on the production tooling to debug such problems rather than simply asserting that such bugs are tautologically impossible. [1] http://dtrace.org/blogs/ahl/2004/07/13/number-11-of-20-libumem/ http://dtrace.org/blogs/ahl/2004/07/13/number-11-of-20-libum... [2] https://queue.acm.org/detail.cfm?id=2039361 https://queue.acm.org/detail.cfm?id=2039361 Edit: added a much better reference.

pron 11y ago

See also Debugging at Full Speed http://medianetwork.oracle.com/video/player/3731019771001 http://medianetwork.oracle.com/video/player/3731019771001

rayiner 11y ago

Debugging software in production is a hard unsolved problem, the available tools are pretty primitive, and it doesn't help that people often don't plan it into their development and deployment cycle. We had a client once using our system, who would see anomalies only after long uptimes at sustained high loads. It took six months to find the cause (a race condition in the database code) because we had so few tools at our disposal to look at the post-mortem state of the program. Running the software at those loads for the weeks or months it took to reproduce the problem under a debugger was obviously not a tenable proposition, and reproducing the variability of the particular real-world traffic was difficult to do in a controlled environment. And the software wasn't written with any eye towards making state available for post-mortem debugging. I guess "don't write complicated multi-threaded software in C" is a pretty good response to the above too. That particular bug couldn't have happened in say Rust (though it could have in Java).

TylerE 11y ago
In languages like C it's an unsolved problem. Doesn't mean it's an unsolved problem everywhere (see, e.g. Erlang, which even allows for hot patching without even taking the system down).
- rayiner 11y ago
  Relevant: http://www.flownet.com/gat/jpl-lisp.html http://www.flownet.com/gat/jpl-lisp.html (scroll to "Debugging a program running on a $100M piece of hardware that is 100 million miles away")
  - TylerE 11y ago
    Yep, old hat in lisp too. Industrial strength VMs tend to do well in this area.

mwcampbell 11y ago

In the QCon talk you referenced in [1], you said that it's impossible to debug a Python or Ruby application from a core file. Why is this? Would an mdb module for Python or Ruby, to delve into the interpreter state and expose it in the debugger, not help or not be feasible?

bcantrill 11y ago
"Impossible" is a bit strong, but yes, it requires deep and unholy knowledge of the interpreter in the debugger. For an example of what this looks like, see some of the monster block comments in mdb_v8.c[1]. [1] https://github.com/joyent/illumos-joyent/blob/master/usr/src/cmd/mdb/common/modules/v8/mdb_v8.c#L2003-L2117 https://github.com/joyent/illumos-joyent/blob/master/usr/src...

WalterBright 11y ago

My not-so-humble opinion is that valgrind has saved C (and C++) from oblivion, at least for large scale programs.

zamalek 11y ago

> postmortem debugging tooling is extraordinarily valuable for production problems When all else fails this is really all you are going to have to go by. Coming from the Microsoft side of the story I have come to a conclusion that any sufficiently well designed postmortem debugger is a few steps away from being a live/production debugger. Windbg, as a prime example of this, can be used to debug memory dumps, live processes and remote (TCP) processes using exactly the same debugging tools binaries (and scripts/plugins that you write). Debuggers have to be the biggest source of friction, for me, when adopting things like Go. If you are in charge of "production" you can throw in a few printlns, however, you can't just approach someone like the HP and say "can I put this debug binary on your production system?" No, the absolute best you can hope for is a memory dump and those memory dump had better count as you are only going to get maybe a few. Maybe it's because not many devs deal with this kind of stuff, which is probably why it isn't a top priority for things like e.g. Go. To some of us, though, the most important language of a feature is easily the quality of the debugger associated with it. Either way, great slides Bryan - it's nice to see at least one "modern" language taking these issues seriously.