8 ms·

I will corroborate one main point of the article: debugging with rr is so much better than a traditional debugger that I will only buy Intel hardware, even with

by sfink 7y ago

I will corroborate one main point of the article: debugging with rr is so much better than a traditional debugger that I will only buy Intel hardware, even with all of the security flaws and corresponding performance hits, even though recent AMD CPUs would halve my half-hour compile time. It really is that much better. Once you start using it, it's really hard to go back to a situation where you don't have it. It's like driving a car that can only turn left -- sure, if you need to go right you can always 270 degrees to the left (by rerunning over and over again to get closer to what you want to examine, assuming your problem is reproducible enough), but it feels burdensome and ridiculous. If AMD fixed their performance counters so that rr could work, I would switch immediately. I am also a fan of Pernosco. It is especially useful when hooked up to a continuous integration system that can hand you a ready-to-go recording of all failures. You can get assigned a bug with a recording to go along with it. Even if it doesn't look important or tractable to look into the ordinary way, having a recording makes it easy enough to take a glance and check it out that you'll actually do it. At shops that do a lot of system-level programming (as opposed to scripting), the productivity boost ought to be worth a serious amount to the bean counters.

blub 7y ago
I guess if it doesn't even work on AMD, ARM must be out of the question...
- fabrice_d 7y ago
  Technically possible, if someone wants to sponsor: https://twitter.com/rocallahan/status/1199477451602067456 https://twitter.com/rocallahan/status/1199477451602067456
- bzbarsky 7y ago
  https://github.com/mozilla/rr/issues/1373 https://github.com/mozilla/rr/issues/1373 is tracking that, but the short story is that ARM seems to not provide enough performance counters to make the current rr approach work at the moment. So it would take either CPU-side changes or a somewhat different approach.
archgoon 7y ago
What prevents you from having a build server that uses AMD cpus? Budget or physical space limitations? Power consumption concerns? Too complicated to have different build and test boxes? Large binary and debug symbols that would result in no effective gains after network transfer time was accounted for?
- sfink 7y ago
  In theory, nothing, and that's a good idea that I've considered. My past experience with distributed builds (distcc, icecc) hasn't been that great. They've worked, but they tended to break down or slow down, and I ended up spending more time maintaining my setup than I gained in compile times. Perhaps things have improved. I still have bad memories of trying to diagnose why the network transfers had slowed to a crawl again (on a local gigabit network.) The other demotivator is noise. I work from home, and the other family members who share my office aren't keen on the sound of a fully spun up build server. I could run ethernet cable under the house to another room, maybe. (Relying on wifi bandwidth hasn't worked out very well. If only debuginfo weren't so enormous...)
  - vardump 7y ago
    > network transfers had slowed to a crawl again (on a local gigabit network.) 10 gigabit is pretty cheap nowadays. Just in case your problem could simply be solved with higher bandwidth...
    - sfink 7y ago
      It very rarely hit the actual bandwidth limit. It would start out close to it for a while, then drop down. And down. And down. Until it was using like 2% of the full bandwidth, but never completely stalling.
glandium 7y ago
FWIW, rr support for AMD CPUs is getting close. See https://github.com/mozilla/rr/issues/2034#issuecomment-556494535 https://github.com/mozilla/rr/issues/2034#issuecomment-55649...
- garaetjjte 7y ago
  I don't see any signs here that it is close, just that it is still broken in the same way.
  - fulafel 7y ago
    Also from the issue: "the bug, whatever it is, could be in the kernel and not the hardware"
jchw 7y ago
I will concede that rr is likely extremely useful, but as a counterpoint to this and the entire article, getting bugs earlier and earlier in the development cycle beats better debugging any day. I will definitely try to add rr to my toolbelt if Ryzen support ever lands, but I still prefer catching problems by writing fast test cases instead. It's like debugging that happens automatically.
- olodus 7y ago
  I don't see these as separate solutions. Running your tests in a debugger usually makes it so much faster and easier to find where it went wrong. I would argue find bugs at compilation would be a separate solution from this and even earlier in the dev cycle.
  - jchw 7y ago
    Indeed, finding bugs at compilation is better, which is why I prefer languages with strong typing. I reread my comment to make sure, but I am being painfully explicit that good debug tools are still useful. But I prefer good tests over even good debug tools. I very rarely even printf debug as a result of automated testing. This isn’t really just lip service; I have a project right now where I literally can’t attach a debugger or run locally. Of course, that’s a flaw and unreasonable. But, with good test hygiene, you can still be pretty productive in the face of this. Not only that, pushing code feels safe, even without the ability to actually run the code locally. Do I think debugging is bad? No. I think it’s worse than testing, which when done right can save you from a hell of a lot of manual debugging, even in the face of complex software with networked dependencies. I get ire on HN every single time I mention the virtues of testing instead of debugging. I suspect people literally don’t believe you can exchange quantities of one for the other, but where I’m standing it’s obviously possible. Same thing comes up with strong typing frequently, where you have people making the claim that typings don’t prevent bugs - but anyone who’s transitioned a medium sized code base to TypeScript can attest to the improvements it can bring. In my opinion, automated tests are almost literally just debugging done by a script instead of a human. edit: It's also being pointed out to me that rr is being used to debug failing tests, which is not something I thought of and my ignorance may be contributing to some of the reason my comment was so poorly received here rather than the testing vs debugging comparison. (In many cases, for unit tests, I had never thought of using a debugger to try to find the problem, because the actual problem is often made obvious enough by the failed assertion output.)
de_watcher 7y ago
The wild thing is that scripting languages don't have that kind of a debugger.
- glandium 7y ago
  Javascript does, with caveats. https://developer.mozilla.org/en-US/docs/Mozilla/Projects/WebReplay https://developer.mozilla.org/en-US/docs/Mozilla/Projects/We...
  - The_rationalist 7y ago
    Is there any equivalent on chromium? I believe Firefox devtools have been ported to chromium.
    - sanxiyn 7y ago
      As far as I know, there isn't. This needs substantial support in the engine, so "devtools have been ported to chromium" is not enough.
- yvdriess 7y ago
  Depends. Image-based runtime languages such as Smalltalk and various Lisps will, crudely put, pause at the site of the error and give you options to fix it and continue. That already covers 99% of the OP's issues with gdb. edit: Implied is that image-based runtimes by definition produce reproducible snapshots of the error, with the full power of the Lisp or Smalltalk introspection tools. Edit-continue is the icing.
  - sanxiyn 7y ago
    Not at all. Edit and continue is only a small part of what debuggers can be.
  - vidarh 7y ago
    Doesn't need an image based runtime either. Ruby's "pry" is inspired by Smalltalk, in that it's providing functionality to manipulate the running environment, edit and reload code etc., and continue. During development I almost always have a make target to drop me in a pry session with all the app code loaded and initialized so I can play with the code as it is actually used.
srcmap 7y ago
I have use gdb and various debuggers for a long time. They are very useful but there are limitations. "gdb rr" - tried it with demo helloworld. That works. But when tried it with a more complex program (FBOSS - Faceboss' Switch/Router open source program), gdb rr function core dump immediately. Also there are certain categories of bugs what are very difficult to use gdb with: * multi threads / race condition bugs - gdb break point in affect the flow of the program and cause different behaviors. * Bugs can not be reliably reproduced. * Server programs or embedded system programs where the code can not be stopped. * This problem get much worst when combine with performance related issue - I work on one system design requirement that would switch over to backup system when system missing a heart beat for 50 milliseconds. Any gdb breakpoint in that app would automatically trigger a hard system fail over to backup. BTW, I do use gdb a lot and can script gdb to do conditional breakpoints with script that auto dump variables, info etc.
- rhinoceraptor 7y ago
  I would highly recommend looking into dtrace or bpftrace (depending on your platform). They're not as powerful as a debugger, but you can use them when nothing else will work. They're totally safe to use in production, because they can't modify memory, and they have a very small performance impact when in use.
- bzbarsky 7y ago
  I'm not sure what you mean by "gdb rr". "rr" is a separate debugger that uses the gdb frontend but a different backend. Importantly, it does not utilize breakpoints during the initial program execution, precisely to address your first, second, and fourth points. The third point is still a problem, since it requires the program being debugged to run under rr from the start.
- roca 7y ago
  If you really want to use rr on FBOSS, please file an rr issue and we'll look into it.

by sfink 7y ago

blub 7y ago

I guess if it doesn't even work on AMD, ARM must be out of the question...

fabrice_d 7y ago
Technically possible, if someone wants to sponsor: https://twitter.com/rocallahan/status/1199477451602067456 https://twitter.com/rocallahan/status/1199477451602067456
bzbarsky 7y ago
https://github.com/mozilla/rr/issues/1373 https://github.com/mozilla/rr/issues/1373 is tracking that, but the short story is that ARM seems to not provide enough performance counters to make the current rr approach work at the moment. So it would take either CPU-side changes or a somewhat different approach.

archgoon 7y ago

What prevents you from having a build server that uses AMD cpus? Budget or physical space limitations? Power consumption concerns? Too complicated to have different build and test boxes? Large binary and debug symbols that would result in no effective gains after network transfer time was accounted for?

sfink 7y ago
In theory, nothing, and that's a good idea that I've considered. My past experience with distributed builds (distcc, icecc) hasn't been that great. They've worked, but they tended to break down or slow down, and I ended up spending more time maintaining my setup than I gained in compile times. Perhaps things have improved. I still have bad memories of trying to diagnose why the network transfers had slowed to a crawl again (on a local gigabit network.) The other demotivator is noise. I work from home, and the other family members who share my office aren't keen on the sound of a fully spun up build server. I could run ethernet cable under the house to another room, maybe. (Relying on wifi bandwidth hasn't worked out very well. If only debuginfo weren't so enormous...)
- vardump 7y ago
  > network transfers had slowed to a crawl again (on a local gigabit network.) 10 gigabit is pretty cheap nowadays. Just in case your problem could simply be solved with higher bandwidth...
  - sfink 7y ago
    It very rarely hit the actual bandwidth limit. It would start out close to it for a while, then drop down. And down. And down. Until it was using like 2% of the full bandwidth, but never completely stalling.

glandium 7y ago

FWIW, rr support for AMD CPUs is getting close. See https://github.com/mozilla/rr/issues/2034#issuecomment-556494535 https://github.com/mozilla/rr/issues/2034#issuecomment-55649...

garaetjjte 7y ago
I don't see any signs here that it is close, just that it is still broken in the same way.
- fulafel 7y ago
  Also from the issue: "the bug, whatever it is, could be in the kernel and not the hardware"

jchw 7y ago

I will concede that rr is likely extremely useful, but as a counterpoint to this and the entire article, getting bugs earlier and earlier in the development cycle beats better debugging any day. I will definitely try to add rr to my toolbelt if Ryzen support ever lands, but I still prefer catching problems by writing fast test cases instead. It's like debugging that happens automatically.

olodus 7y ago
I don't see these as separate solutions. Running your tests in a debugger usually makes it so much faster and easier to find where it went wrong. I would argue find bugs at compilation would be a separate solution from this and even earlier in the dev cycle.
- jchw 7y ago
  Indeed, finding bugs at compilation is better, which is why I prefer languages with strong typing. I reread my comment to make sure, but I am being painfully explicit that good debug tools are still useful. But I prefer good tests over even good debug tools. I very rarely even printf debug as a result of automated testing. This isn’t really just lip service; I have a project right now where I literally can’t attach a debugger or run locally. Of course, that’s a flaw and unreasonable. But, with good test hygiene, you can still be pretty productive in the face of this. Not only that, pushing code feels safe, even without the ability to actually run the code locally. Do I think debugging is bad? No. I think it’s worse than testing, which when done right can save you from a hell of a lot of manual debugging, even in the face of complex software with networked dependencies. I get ire on HN every single time I mention the virtues of testing instead of debugging. I suspect people literally don’t believe you can exchange quantities of one for the other, but where I’m standing it’s obviously possible. Same thing comes up with strong typing frequently, where you have people making the claim that typings don’t prevent bugs - but anyone who’s transitioned a medium sized code base to TypeScript can attest to the improvements it can bring. In my opinion, automated tests are almost literally just debugging done by a script instead of a human. edit: It's also being pointed out to me that rr is being used to debug failing tests, which is not something I thought of and my ignorance may be contributing to some of the reason my comment was so poorly received here rather than the testing vs debugging comparison. (In many cases, for unit tests, I had never thought of using a debugger to try to find the problem, because the actual problem is often made obvious enough by the failed assertion output.)

de_watcher 7y ago

The wild thing is that scripting languages don't have that kind of a debugger.

glandium 7y ago
Javascript does, with caveats. https://developer.mozilla.org/en-US/docs/Mozilla/Projects/WebReplay https://developer.mozilla.org/en-US/docs/Mozilla/Projects/We...
- The_rationalist 7y ago
  Is there any equivalent on chromium? I believe Firefox devtools have been ported to chromium.
  - sanxiyn 7y ago
    As far as I know, there isn't. This needs substantial support in the engine, so "devtools have been ported to chromium" is not enough.
yvdriess 7y ago
Depends. Image-based runtime languages such as Smalltalk and various Lisps will, crudely put, pause at the site of the error and give you options to fix it and continue. That already covers 99% of the OP's issues with gdb. edit: Implied is that image-based runtimes by definition produce reproducible snapshots of the error, with the full power of the Lisp or Smalltalk introspection tools. Edit-continue is the icing.
- sanxiyn 7y ago
  Not at all. Edit and continue is only a small part of what debuggers can be.
- vidarh 7y ago
  Doesn't need an image based runtime either. Ruby's "pry" is inspired by Smalltalk, in that it's providing functionality to manipulate the running environment, edit and reload code etc., and continue. During development I almost always have a make target to drop me in a pry session with all the app code loaded and initialized so I can play with the code as it is actually used.

srcmap 7y ago

I have use gdb and various debuggers for a long time. They are very useful but there are limitations. "gdb rr" - tried it with demo helloworld. That works. But when tried it with a more complex program (FBOSS - Faceboss' Switch/Router open source program), gdb rr function core dump immediately. Also there are certain categories of bugs what are very difficult to use gdb with: * multi threads / race condition bugs - gdb break point in affect the flow of the program and cause different behaviors. * Bugs can not be reliably reproduced. * Server programs or embedded system programs where the code can not be stopped. * This problem get much worst when combine with performance related issue - I work on one system design requirement that would switch over to backup system when system missing a heart beat for 50 milliseconds. Any gdb breakpoint in that app would automatically trigger a hard system fail over to backup. BTW, I do use gdb a lot and can script gdb to do conditional breakpoints with script that auto dump variables, info etc.

rhinoceraptor 7y ago
I would highly recommend looking into dtrace or bpftrace (depending on your platform). They're not as powerful as a debugger, but you can use them when nothing else will work. They're totally safe to use in production, because they can't modify memory, and they have a very small performance impact when in use.
bzbarsky 7y ago
I'm not sure what you mean by "gdb rr". "rr" is a separate debugger that uses the gdb frontend but a different backend. Importantly, it does not utilize breakpoints during the initial program execution, precisely to address your first, second, and fourth points. The third point is still a problem, since it requires the program being debugged to run under rr from the start.
roca 7y ago
If you really want to use rr on FBOSS, please file an rr issue and we'll look into it.