3 ms·

GDBs built-in reverse debugging: https://www.sourceware.org/gdb/wiki/ProcessRecord/Tutorial https://www.sourceware.org/gdb/wiki/ProcessRecord/Tutorial I assu

GDBs built-in reverse debugging: https://www.sourceware.org/gdb/wiki/ProcessRecord/Tutorial https://www.sourceware.org/gdb/wiki/ProcessRecord/Tutorial I assume rr provides more features and flexibility. Anyway I want to mention that GDB itself can already reverse debug for some time now.

dima55 2y ago
If you want to mention this, then you very clearly haven't actually tried it. The implementation in GDB is more convenient than rr (you can start/stop recording at will), but it is also orders of magnitude less efficient. It's only usable for very small code snippets. Otherwise it takes effectively forever and/or runs out of resources.
- throwaway2037 2y ago
  > runs out of resources RAM? What kind of dev box runs out of RAM in 2024? I built a 64GB RAM dev box during COVID-19 crisis. I have never once come close to using all that RAM, even with a squillion Chrome tabs open. Still, thank you to share your first-hand experience. Did you ask the GDB Dev team for any feedback on the slow performance?
  - enedil 2y ago
    Uuh, any kind of dev box that requires more RAM than available? I promise you that storing that much data about the runtime can be really memory consumptive.
  - signa11 2y ago
    umm, embedded devices without active cooling f.e. those node-b’s sitting on cellphone towers come to mind here, there can be quite a few other similar examples i can think of.
  - kragen 2y ago
    gdb record and replay will absolutely eat up your piddly 64 gigs of ram in a few minutes if you just let it loose. it will eat your 12-terabyte hard disk too, it just takes a little longer. the gdb manual has many helpful tips for how to allocate your ram carefully to record and replay, as well as using a disk buffer instead of recording to ram, so the gdb dev team is already well aware of the problem
29athrowaway 2y ago
rr predates the one in gdb if I am not mistaken
- roca 2y ago
  Actually the gdb implementation predates rr, but (as an rr maintainer) I have to say that it is vastly inferior to rr. It's about 1000x slower than rr, and can't record across system calls or multiple threads or processes. It's so limited it's really a different feature.
  - ho_schi 2y ago
    Thanks. Can you explain why rr is so much more efficient?
    - roca 2y ago
      The two approaches are completely different. gdb singlesteps the program and, before each instruction, records the state of registers and memory that will be changed by that instruction --- an undo log. Then you can reverse-execute an instruction by restoring the state from the undo log. This is incredibly slow because singlestepping requires a full context switch for each instruction executed. rr, on the other hand, intercepts all system calls and other sources of nondeterminism but regular CPU instructions execute normally with no overhead. The details about rr are here: https://arxiv.org/abs/1705.05937 https://arxiv.org/abs/1705.05937
- Veserv 2y ago
  rr was introduced in 2014 [1]. gdb reverse debugging was introduced in 2009 [2]. You can see a fairly comprehensive history of time travel debugging here [3]. Not to say the built-in gdb reverse debugging was any good. It had (has?) like 1,000,000% overhead which is basically unusable. At least some implementations in the history that were introduced earlier only had ~1,000% overhead or less in general. Yes, a literal 1,000x overhead difference. [1] https://robert.ocallahan.org/2014/03/introducing-rr.html https://robert.ocallahan.org/2014/03/introducing-rr.html [2] https://www.sourceware.org/gdb/wiki/ReverseDebug https://www.sourceware.org/gdb/wiki/ReverseDebug [3] https://jakob.engbloms.se/archives/1564 https://jakob.engbloms.se/archives/1564
lambda 2y ago
I have successfully used GDB's build in reverse debugging once, on a platform that rr didn't yet support at the time. It worked, it helped me track down the bug, but it was painfully slow, I had to do things to limit the size of the input to make it possible to use at all (and thankfully was luckily able to still repro the problem after doing so).
kragen 2y ago
gdb's built-in replay implementation imposes a slowdown of about 10000× on your program, so if you can binary-search to the desired program state in less than 10000 restarts of the program, that will take less machine time than using reverse execution. in fact, the slowdown is large enough that even interactively navigating the debugger close to the right state repeatedly and then restarting the program is often enough i have been able to use gdb's replay functionality usefully because i had an input file which crashed the program within a fraction of a second after startup. this meant that i could navigate backward from "this variable is wrong" to "how did this variable get set to that wrong value?" in only several minutes of waiting on the computer

dima55 2y ago

If you want to mention this, then you very clearly haven't actually tried it. The implementation in GDB is more convenient than rr (you can start/stop recording at will), but it is also orders of magnitude less efficient. It's only usable for very small code snippets. Otherwise it takes effectively forever and/or runs out of resources.

throwaway2037 2y ago
> runs out of resources RAM? What kind of dev box runs out of RAM in 2024? I built a 64GB RAM dev box during COVID-19 crisis. I have never once come close to using all that RAM, even with a squillion Chrome tabs open. Still, thank you to share your first-hand experience. Did you ask the GDB Dev team for any feedback on the slow performance?
- enedil 2y ago
  Uuh, any kind of dev box that requires more RAM than available? I promise you that storing that much data about the runtime can be really memory consumptive.
- signa11 2y ago
  umm, embedded devices without active cooling f.e. those node-b’s sitting on cellphone towers come to mind here, there can be quite a few other similar examples i can think of.
- kragen 2y ago
  gdb record and replay will absolutely eat up your piddly 64 gigs of ram in a few minutes if you just let it loose. it will eat your 12-terabyte hard disk too, it just takes a little longer. the gdb manual has many helpful tips for how to allocate your ram carefully to record and replay, as well as using a disk buffer instead of recording to ram, so the gdb dev team is already well aware of the problem

29athrowaway 2y ago

rr predates the one in gdb if I am not mistaken

roca 2y ago
Actually the gdb implementation predates rr, but (as an rr maintainer) I have to say that it is vastly inferior to rr. It's about 1000x slower than rr, and can't record across system calls or multiple threads or processes. It's so limited it's really a different feature.
- ho_schi 2y ago
  Thanks. Can you explain why rr is so much more efficient?
  - roca 2y ago
    The two approaches are completely different. gdb singlesteps the program and, before each instruction, records the state of registers and memory that will be changed by that instruction --- an undo log. Then you can reverse-execute an instruction by restoring the state from the undo log. This is incredibly slow because singlestepping requires a full context switch for each instruction executed. rr, on the other hand, intercepts all system calls and other sources of nondeterminism but regular CPU instructions execute normally with no overhead. The details about rr are here: https://arxiv.org/abs/1705.05937 https://arxiv.org/abs/1705.05937
Veserv 2y ago
rr was introduced in 2014 [1]. gdb reverse debugging was introduced in 2009 [2]. You can see a fairly comprehensive history of time travel debugging here [3]. Not to say the built-in gdb reverse debugging was any good. It had (has?) like 1,000,000% overhead which is basically unusable. At least some implementations in the history that were introduced earlier only had ~1,000% overhead or less in general. Yes, a literal 1,000x overhead difference. [1] https://robert.ocallahan.org/2014/03/introducing-rr.html https://robert.ocallahan.org/2014/03/introducing-rr.html [2] https://www.sourceware.org/gdb/wiki/ReverseDebug https://www.sourceware.org/gdb/wiki/ReverseDebug [3] https://jakob.engbloms.se/archives/1564 https://jakob.engbloms.se/archives/1564

lambda 2y ago

I have successfully used GDB's build in reverse debugging once, on a platform that rr didn't yet support at the time. It worked, it helped me track down the bug, but it was painfully slow, I had to do things to limit the size of the input to make it possible to use at all (and thankfully was luckily able to still repro the problem after doing so).

kragen 2y ago

gdb's built-in replay implementation imposes a slowdown of about 10000× on your program, so if you can binary-search to the desired program state in less than 10000 restarts of the program, that will take less machine time than using reverse execution. in fact, the slowdown is large enough that even interactively navigating the debugger close to the right state repeatedly and then restarting the program is often enough i have been able to use gdb's replay functionality usefully because i had an input file which crashed the program within a fraction of a second after startup. this meant that i could navigate backward from "this variable is wrong" to "how did this variable get set to that wrong value?" in only several minutes of waiting on the computer