4 ms·

While it's great to see more attention brought to debugging, some of this is just (for lack of a better word) insane. e.g.: For searching in space, most progra

While it's great to see more attention brought to debugging, some of this is just (for lack of a better word) insane. e.g.: For searching in space, most programmers have done binary-search with commented code: comment out or bypass half of your codebase. Do that recursively until you nail down where (at which file, which function, which lines) the bug lives. No, most programmers actually haven't done this for the simple reason that it's highly unlikely to work: most codebases with half of their functionality removed don't actually function. That this kind of lunacy is being asserted as authoritative is galling enough, but it gets worse: When binary search doesn't work, you need more creative approaches. Consider brainstorming to enumerate even the wildest possibilities: for instance, maybe the bug is in some external resource like a library or a remote service; maybe there is version mismatch of libraries; maybe your tool (e.g. IDE) has a problem; maybe it is bit flipping in the hard disk; maybe the date and time library is sensitive to your computer's settings, etc. This is advocating debugging by superstition, and it represents toxic thinking. When we have defects in software, we need to be strictly empirical in approach: 1. Make observations. 2. Think of interesting questions. 3. Formulate a hypothesis. 4. Develop a testable prediction from the hypothesis. 5. Gather data to test the prediction. 6. Refine, alter or expand the hypothesis. 7. Go to step 4 until defect has been driven to root cause(s) If this sounds familiar, it is because it is the scientific method[1] to which we can credit much of modern knowledge and civilization. As to how this specifically relates to debugging, I touched on this in my recent DockerCon presentation[2][3]. Bringing attention to debugging is terrific -- but advocating superstition as a methodology is anathema to true understanding. [1] https://en.wikipedia.org/wiki/Scientific_method https://en.wikipedia.org/wiki/Scientific_method [2] https://www.youtube.com/watch?v=sYQ8j02wbCY https://www.youtube.com/watch?v=sYQ8j02wbCY [3] http://www.slideshare.net/bcantrill/running-aground-debugging-docker-in-production http://www.slideshare.net/bcantrill/running-aground-debuggin...

JoeAltmaier 11y ago
Doesn't all that stuff you call 'toxic' actually fit in step 5? Lets not get hyperbolic about it. Backing out changes; setting breakpoints and interval-halving through a procedure to find where a value goes wrong; comparing library versions with working build is all good practice.
- bcantrill 11y ago
  To be clear: the toxic bit here is the idea that you start with that first. Everything you describe is entirely appropriate when you have a hypothesis in hand, and you are using it as a technique to test it -- but I (strenuously) object to the idea that problems should be debugged by first randomly permuting the codebase.
  - ectoplasm 11y ago
    Binary search is not random permutation. The hypothesis is "problem lies in this half of the codebase". Hey, if your test passes after commenting out some stuff, great.
    - bcantrill 11y ago
      And if that hypothesis is based on data, great. And if there is no other alternative, fine -- but this technique should be viewed as a last resort (and an unusual one), not a first step.
      - ectoplasm 11y ago
        Wait a minute. First of all, the data is simply "there is a reproducible bug in the program". The real hypothesis is that a small, localized portion of the code is responsible for the defect. To find it, you can try binary search. If you have a better idea of the location, then by all means. Second, when you have a bug and you write a unit test, you are effectively commenting out the entire codebase except for the function under test. When you have a compiler error, whether it's a syntax/semantics bug in your code or a bug in the compiler, sometimes you need to produce a minimal example, so you have to cut cut cut until the bug is just barely provoked. When you have a pipeline of data transforms, and the end result is suddenly borked, it can work to chop off half the transforms and look at the result. When latex is crashing for some unintelligible reason, just comment out half of your document and see if the problem goes away. Sure it's really dumb if you're just excluding a* .c through m* .c (spaces due to HN formatting rules), but figuring out if the problem is in the first or second half of main is not outrageous. I don't think the guy was presenting it as the first step ("If you have no idea where your bug lives"), but I do agree that it comes across as a little naive, since he should have talked about all of the other techniques available first. So the problem isn't so much the lack of a hypothesis, but the inefficient experimental approach of using a brute force technique indiscriminately. I think the last resort is reached a little sooner for some types of bugs and some experience levels (language, environment, codebase, programming), and yes in many cases it won't even do anything for you. Personally I always liked dtrace. This guy gave a demo of it at my university once, I thought it was great, one of the best talks I've seen.
mwcampbell 11y ago
I listened to your DockerCon talk on debugging. So in a server that can handle multiple concurrent requests (e.g. a web application server), should each unhandled exception abort the whole process (after delivering something like an HTTP 500 to the client) so the developer can do postmortem debugging on a core dump? That would cause all other in-progress requests to be aborted too, unlike simply logging the exception and moving on. But then again, maybe that would increase the motivation to root-cause every failure.
- dap 11y ago
  I don't mean to speak for bcantrill, but yes, this is exactly how we (at Joyent) build programs. We wrote in detail about why we do it this way: https://www.joyent.com/developers/node/design/errors https://www.joyent.com/developers/node/design/errors It's not that this approach motivates root-causing failures (though that's true). It's that uncaught exceptions are programmer errors, and it's tautologically impossible to correctly deal with them. Attempting to do so can make things much, much worse. To make this concrete: I've dealt more than one production outage caused by the mere presence of an uncaughtException handler. If the program had merely crashed, a hundred requests may have been interrupted, but the program would have restarted and resumed servicing requests. Instead, the exception was thrown from a code path that had a database transaction open with database locks held. Because the uncaughtException handler just logged the exception and otherwise ignored it, that transaction stayed open (and the locks remained held) until a human got into the loop -- interfering with tens of thousands of subsequent requests. That's much, much worse. If the process had just exited, the db connection would have been closed, the transaction aborted, and the locks released. An unexpected error handler can't know the complete state that the program was in because by definition this wasn't a state that the programmer thought about.
  - mwcampbell 11y ago
    Thanks. Now I gotta figure out the best way to do that in Python. Most Python web frameworks and WSGI servers catch all exceptions and keep the process going.
- Licenser 11y ago

JoeAltmaier 11y ago

Doesn't all that stuff you call 'toxic' actually fit in step 5? Lets not get hyperbolic about it. Backing out changes; setting breakpoints and interval-halving through a procedure to find where a value goes wrong; comparing library versions with working build is all good practice.

bcantrill 11y ago
To be clear: the toxic bit here is the idea that you start with that first. Everything you describe is entirely appropriate when you have a hypothesis in hand, and you are using it as a technique to test it -- but I (strenuously) object to the idea that problems should be debugged by first randomly permuting the codebase.
- ectoplasm 11y ago
  Binary search is not random permutation. The hypothesis is "problem lies in this half of the codebase". Hey, if your test passes after commenting out some stuff, great.
  - bcantrill 11y ago
    And if that hypothesis is based on data, great. And if there is no other alternative, fine -- but this technique should be viewed as a last resort (and an unusual one), not a first step.
    - ectoplasm 11y ago
      Wait a minute. First of all, the data is simply "there is a reproducible bug in the program". The real hypothesis is that a small, localized portion of the code is responsible for the defect. To find it, you can try binary search. If you have a better idea of the location, then by all means. Second, when you have a bug and you write a unit test, you are effectively commenting out the entire codebase except for the function under test. When you have a compiler error, whether it's a syntax/semantics bug in your code or a bug in the compiler, sometimes you need to produce a minimal example, so you have to cut cut cut until the bug is just barely provoked. When you have a pipeline of data transforms, and the end result is suddenly borked, it can work to chop off half the transforms and look at the result. When latex is crashing for some unintelligible reason, just comment out half of your document and see if the problem goes away. Sure it's really dumb if you're just excluding a* .c through m* .c (spaces due to HN formatting rules), but figuring out if the problem is in the first or second half of main is not outrageous. I don't think the guy was presenting it as the first step ("If you have no idea where your bug lives"), but I do agree that it comes across as a little naive, since he should have talked about all of the other techniques available first. So the problem isn't so much the lack of a hypothesis, but the inefficient experimental approach of using a brute force technique indiscriminately. I think the last resort is reached a little sooner for some types of bugs and some experience levels (language, environment, codebase, programming), and yes in many cases it won't even do anything for you. Personally I always liked dtrace. This guy gave a demo of it at my university once, I thought it was great, one of the best talks I've seen.

mwcampbell 11y ago

I listened to your DockerCon talk on debugging. So in a server that can handle multiple concurrent requests (e.g. a web application server), should each unhandled exception abort the whole process (after delivering something like an HTTP 500 to the client) so the developer can do postmortem debugging on a core dump? That would cause all other in-progress requests to be aborted too, unlike simply logging the exception and moving on. But then again, maybe that would increase the motivation to root-cause every failure.

dap 11y ago
I don't mean to speak for bcantrill, but yes, this is exactly how we (at Joyent) build programs. We wrote in detail about why we do it this way: https://www.joyent.com/developers/node/design/errors https://www.joyent.com/developers/node/design/errors It's not that this approach motivates root-causing failures (though that's true). It's that uncaught exceptions are programmer errors, and it's tautologically impossible to correctly deal with them. Attempting to do so can make things much, much worse. To make this concrete: I've dealt more than one production outage caused by the mere presence of an uncaughtException handler. If the program had merely crashed, a hundred requests may have been interrupted, but the program would have restarted and resumed servicing requests. Instead, the exception was thrown from a code path that had a database transaction open with database locks held. Because the uncaughtException handler just logged the exception and otherwise ignored it, that transaction stayed open (and the locks remained held) until a human got into the loop -- interfering with tens of thousands of subsequent requests. That's much, much worse. If the process had just exited, the db connection would have been closed, the transaction aborted, and the locks released. An unexpected error handler can't know the complete state that the program was in because by definition this wasn't a state that the programmer thought about.
- mwcampbell 11y ago
  Thanks. Now I gotta figure out the best way to do that in Python. Most Python web frameworks and WSGI servers catch all exceptions and keep the process going.
Licenser 11y ago