3 ms·

rr is insanely useful. There are some debugging patterns that commonly come up for me when I use it. 1. Investigate a crash: run program until it crashes/shows

rr is insanely useful. There are some debugging patterns that commonly come up for me when I use it. 1. Investigate a crash: run program until it crashes/shows error dialog/other undesired state. If there was an exception involved then set up a catchpoint (`catch throw`) then reverse-continue. You are now in the vicinity of the crash, you can now debug it going backwards in time. 2. Some variable has garbage state, you want to know what caused it. Set up a hardware watchpoint to the variable, then reverse-continue. You can then see the line that wrote to the variable the last time. Sometimes it is a line that is writing to an entirely different variable, then you know you have variable lifetime issues. 3. The executable you debug is timing sensitive (network timeouts for example). Just record a run with rr then debug it later, don't worry about breakpoints in the middle of timing sensitive code. And I'm pretty sure that there are a ton of other use cases.

ynik 5y ago
What I like most: you can debug data flow! To see where a value came from, set a memory-breakpoint and reverse-continue. This is not just useful when you suspect memory corruption (your point 2). I use it more often with a large legacy C code base. It outputs a complex data structure. There's often several code paths computing values for a particular field. For someone unfamiliar with the code base, it can be tricky to tell which code path was responsible for computing the value in a particular instance of the struct (where I see incorrect output). By reverse debugging with memory breakpoints, I can trace the data flow backwards until I find where the computation went wrong. Note: Since I'm developing on Windows, I'm not using rr for that; but Microsoft's Time-Travel Debugger (WinDbg Preview).
db48x 5y ago
These are excellent examples. I’ll throw one of my stories into the pot: One time I had a client that was using a large and complicated C++ library to build a larger and yet more complicateder C++ program. It had an intermittent crash that they just couldn’t track down. The stack traces they showed me were always deep in the bowels of the C++ library, in places no crash should ever be. The library was open source and widely used; I _knew_ that it didn’t just crash in any of those places. I recorded it in rr a few times until I captured the crash. I set a watchpoint on the memory address of the crash, and ran the program backwards from the crash. (Memory addresses are stable in the replay, making this kind of thing super easy.) A few seconds later it stopped on the line that was responsible; turns out they were accidentally overwriting the method pointer to the deconstructor in the vtable of this one class. Eventually one of those objects would go out of scope on some random thread, and need to be deconstructed. BOOM. I checked a few of the recordings where it didn’t crash, and in those cases it had just overwritten something much less obvious, like a string containing HTML that it had downloaded. My advice? First, always make sure your pointers are initialized correctly before you go dereferencing them. You won’t like what happens otherwise. Second, learn rr. With rr, and a little brain–sweat, you can do in a couple of hours what an entire team of engineers couldn’t do in months. Admittedly they had a lot of things on their plate, but it’s definitely a superpower. I think this might have been the first time I ever ran rr, too; some of that time was getting rr built and installed. I think I even had to ask a question on the IRC channel, because there was a confusing error message. Third, learn Rust. This was a few years back, when Rust was little more than a crazy idea. If the program had been written in Rust, there would have been a lot fewer landmines for them to step on. That’s a rather different kind of superpower. Incidentally, you can combine these superpowers… but that’s a story for another day.
- ncmncm 5y ago
  Compare the time spent using rr to track down the occasional failure against the sum of time spent waiting for the Rust compiler, every build, every day. I know which number is bigger, at least for the sort of code I work on. In the last 10 years I have spent strictly more time preparing bug reports against compilers than on tracking down memory usage errors. So, while Rust solves a problem all C coders have, it is not a problem that modern C++ coders necessarily experience enough to justify the extra coding effort, build time, and tool maturity risk. Rust, at this stage of maturity, is fun in a puzzle-solving sense, and modern, and enlightening. No one who learns Rust will regret the effort spent. The only serious risk is that it may take many months to restore one's habit of putting a terminating semicolon where C++ demands one but Rust does not. A shift in preference against designs requiring mutex locks may improve the performance of your C++ code.
  - db48x 5y ago
    The real cost to my client was that for months or years their system had been crashing, and they could do nothing about it. They just had to live with it, rerunning jobs that had crashed. Maybe the compile times would have been unfortunate, but I think they would have come out ahead. On the other hand, they probably wouldn’t have had to hire me. Speed isn’t everything: https://www.youtube.com/watch?v=2wZ1pCpJUIM https://www.youtube.com/watch?v=2wZ1pCpJUIM
    - ncmncm 5y ago
      I watched it (at 2x, 17 mins): The enthusiasm of the convert. Values are in tension, but time is fungible. When you add up hundreds of hours waiting for very, very, very slow builds, you should wonder if maybe those hundreds of hours would be better spent elsewhere. If you have traded them for two hours of debugging, have you come out ahead? (Is there any objective reason for the Rust compiler to be two orders of magnitude slower than a normal compiler? Maybe an alternative implementation strategy would help?) It was telling when he posted his "values" of C++. Suffice to say, when you need to lie to make your case, the argument is already lost. Your client lived with crashes because they couldn't be bothered to use valgrind, to use the address sanitizer, to use the UB sanitizer, to set a watchpoint in gdb? They didn't need rr. There is no substitute for competence. If they had been competent to (re-?)write it in Rust, they would be more than competent to spend 0.1% as much time just fixing it, or (better) not coding the bug in the first place. But who could they hire to code it in Rust? There are orders of magnitude too few Rust coders for that to work. Rust might be a way not to code bugs in the first place, but modern C++ is another way. It doesn't pretend to make bugs impossible, as Rust pretends, but it does remove temptation to bugs: bug-prone code is ugly, and better ways are equally fast. And, modern C++ is mature, fun, fast building, and can "#include" C headers for essential libraries unchanged.

ynik 5y ago

What I like most: you can debug data flow! To see where a value came from, set a memory-breakpoint and reverse-continue. This is not just useful when you suspect memory corruption (your point 2). I use it more often with a large legacy C code base. It outputs a complex data structure. There's often several code paths computing values for a particular field. For someone unfamiliar with the code base, it can be tricky to tell which code path was responsible for computing the value in a particular instance of the struct (where I see incorrect output). By reverse debugging with memory breakpoints, I can trace the data flow backwards until I find where the computation went wrong. Note: Since I'm developing on Windows, I'm not using rr for that; but Microsoft's Time-Travel Debugger (WinDbg Preview).

db48x 5y ago

These are excellent examples. I’ll throw one of my stories into the pot: One time I had a client that was using a large and complicated C++ library to build a larger and yet more complicateder C++ program. It had an intermittent crash that they just couldn’t track down. The stack traces they showed me were always deep in the bowels of the C++ library, in places no crash should ever be. The library was open source and widely used; I _knew_ that it didn’t just crash in any of those places. I recorded it in rr a few times until I captured the crash. I set a watchpoint on the memory address of the crash, and ran the program backwards from the crash. (Memory addresses are stable in the replay, making this kind of thing super easy.) A few seconds later it stopped on the line that was responsible; turns out they were accidentally overwriting the method pointer to the deconstructor in the vtable of this one class. Eventually one of those objects would go out of scope on some random thread, and need to be deconstructed. BOOM. I checked a few of the recordings where it didn’t crash, and in those cases it had just overwritten something much less obvious, like a string containing HTML that it had downloaded. My advice? First, always make sure your pointers are initialized correctly before you go dereferencing them. You won’t like what happens otherwise. Second, learn rr. With rr, and a little brain–sweat, you can do in a couple of hours what an entire team of engineers couldn’t do in months. Admittedly they had a lot of things on their plate, but it’s definitely a superpower. I think this might have been the first time I ever ran rr, too; some of that time was getting rr built and installed. I think I even had to ask a question on the IRC channel, because there was a confusing error message. Third, learn Rust. This was a few years back, when Rust was little more than a crazy idea. If the program had been written in Rust, there would have been a lot fewer landmines for them to step on. That’s a rather different kind of superpower. Incidentally, you can combine these superpowers… but that’s a story for another day.

ncmncm 5y ago
Compare the time spent using rr to track down the occasional failure against the sum of time spent waiting for the Rust compiler, every build, every day. I know which number is bigger, at least for the sort of code I work on. In the last 10 years I have spent strictly more time preparing bug reports against compilers than on tracking down memory usage errors. So, while Rust solves a problem all C coders have, it is not a problem that modern C++ coders necessarily experience enough to justify the extra coding effort, build time, and tool maturity risk. Rust, at this stage of maturity, is fun in a puzzle-solving sense, and modern, and enlightening. No one who learns Rust will regret the effort spent. The only serious risk is that it may take many months to restore one's habit of putting a terminating semicolon where C++ demands one but Rust does not. A shift in preference against designs requiring mutex locks may improve the performance of your C++ code.
- db48x 5y ago
  The real cost to my client was that for months or years their system had been crashing, and they could do nothing about it. They just had to live with it, rerunning jobs that had crashed. Maybe the compile times would have been unfortunate, but I think they would have come out ahead. On the other hand, they probably wouldn’t have had to hire me. Speed isn’t everything: https://www.youtube.com/watch?v=2wZ1pCpJUIM https://www.youtube.com/watch?v=2wZ1pCpJUIM
  - ncmncm 5y ago
    I watched it (at 2x, 17 mins): The enthusiasm of the convert. Values are in tension, but time is fungible. When you add up hundreds of hours waiting for very, very, very slow builds, you should wonder if maybe those hundreds of hours would be better spent elsewhere. If you have traded them for two hours of debugging, have you come out ahead? (Is there any objective reason for the Rust compiler to be two orders of magnitude slower than a normal compiler? Maybe an alternative implementation strategy would help?) It was telling when he posted his "values" of C++. Suffice to say, when you need to lie to make your case, the argument is already lost. Your client lived with crashes because they couldn't be bothered to use valgrind, to use the address sanitizer, to use the UB sanitizer, to set a watchpoint in gdb? They didn't need rr. There is no substitute for competence. If they had been competent to (re-?)write it in Rust, they would be more than competent to spend 0.1% as much time just fixing it, or (better) not coding the bug in the first place. But who could they hire to code it in Rust? There are orders of magnitude too few Rust coders for that to work. Rust might be a way not to code bugs in the first place, but modern C++ is another way. It doesn't pretend to make bugs impossible, as Rust pretends, but it does remove temptation to bugs: bug-prone code is ugly, and better ways are equally fast. And, modern C++ is mature, fun, fast building, and can "#include" C headers for essential libraries unchanged.