4 ms·

Interesting article. I'm curious how another debugging feature, reverse stepping works. Does anyone have a source on that?

khuey 5y ago
There are a lot of different techniques. In Pernosco we more or less do "SELECT program_state WHERE time=T_current-1". gdb's built in reverse debugging feature creates an undo stack when executing forwards, and executing backwards pops modifications off of that stack. rr uses forward execution to emulate reverse execution, backing up to a checkpoint, executing forwards once to figure out where to stop, and then executing a second time from that checkpoint to the stopping point.
- jraph 5y ago
  To add to that for people who don't know how rr works: rr records the entire state of the program by saving all the return values of syscalls. When replaying the execution, you can run the executable natively, only replacing syscalls by their recorded values. In GDB, in addition to reverse stepping, you can also use checkpoints (save the entire state of the program) and restore them later. GDB's checkpointing feature is based on fork. "Going back" is done by "exec()'ing" the checkpoint, replacing the current execution by the saved one. One big limitation is that forking does not work for debugging programs with multiple processes, and fork is not well defined when multi-threading too. As part of my PhD, I extended [1] gdb using its Python API and allowed checkpointing with CRIU (Checkpoint/Restore In Userspace) which does not have this limitation. The "small" difficulty is that both GDB and CRIU use ptrace to attach to the process, and it is not possible to attach to the same process twice. The trick was to send SIGSUSPEND to the program, then detach gdb, then run CRIU on the suspended program, then reattach to the process with GDB and let it continue. This is hacky but it does work, amazingly, including breakpoints and watchpoints which are restored. CRIU even restores the right PID number by using setpid. An advantage of checkpointing is that you can restore the execution and try another path (rr cannot change the execution, it can only replay what it recorded). You are not limited by the maximum size of an undo stack. But you need to save in advance. With rr, you can explore the whole execution, any state of the program is reachable. It's easier and more robust than checkpointing. I'd argue that's what is most useful most of the time. Checkpointing can be useful to explore multiple executions without having to run the program from the start, or to explore normally difficult to reproduce / unreachable states by modifying values with the debugger. [1] https://gricad-gitlab.univ-grenoble-alpes.fr/jakser/verde https://gricad-gitlab.univ-grenoble-alpes.fr/jakser/verde [2] https://criu.org/ https://criu.org/
  - ma2rten 5y ago
    How does rr work with multi-threading? I would imagine threads can mess with the order that syscalls are executed.
    - khuey 5y ago
      rr emulates a single core machine and only one thread in the process is making process at any given time
  - khuey 5y ago
    rr is capable of very limited diversion from the originally executed trace. This functionality exists to support things like `call myProgramObject->dump()` in gdb during replay. It will give up immediately on any non-trivial syscall though. Being able to change something and explore another execution path is interesting but it's extremely difficult to do in general. I think it would be very useful for verifying fixes, which is something that tools like rr are not particularly helpful with.

khuey 5y ago

There are a lot of different techniques. In Pernosco we more or less do "SELECT program_state WHERE time=T_current-1". gdb's built in reverse debugging feature creates an undo stack when executing forwards, and executing backwards pops modifications off of that stack. rr uses forward execution to emulate reverse execution, backing up to a checkpoint, executing forwards once to figure out where to stop, and then executing a second time from that checkpoint to the stopping point.

jraph 5y ago
To add to that for people who don't know how rr works: rr records the entire state of the program by saving all the return values of syscalls. When replaying the execution, you can run the executable natively, only replacing syscalls by their recorded values. In GDB, in addition to reverse stepping, you can also use checkpoints (save the entire state of the program) and restore them later. GDB's checkpointing feature is based on fork. "Going back" is done by "exec()'ing" the checkpoint, replacing the current execution by the saved one. One big limitation is that forking does not work for debugging programs with multiple processes, and fork is not well defined when multi-threading too. As part of my PhD, I extended [1] gdb using its Python API and allowed checkpointing with CRIU (Checkpoint/Restore In Userspace) which does not have this limitation. The "small" difficulty is that both GDB and CRIU use ptrace to attach to the process, and it is not possible to attach to the same process twice. The trick was to send SIGSUSPEND to the program, then detach gdb, then run CRIU on the suspended program, then reattach to the process with GDB and let it continue. This is hacky but it does work, amazingly, including breakpoints and watchpoints which are restored. CRIU even restores the right PID number by using setpid. An advantage of checkpointing is that you can restore the execution and try another path (rr cannot change the execution, it can only replay what it recorded). You are not limited by the maximum size of an undo stack. But you need to save in advance. With rr, you can explore the whole execution, any state of the program is reachable. It's easier and more robust than checkpointing. I'd argue that's what is most useful most of the time. Checkpointing can be useful to explore multiple executions without having to run the program from the start, or to explore normally difficult to reproduce / unreachable states by modifying values with the debugger. [1] https://gricad-gitlab.univ-grenoble-alpes.fr/jakser/verde https://gricad-gitlab.univ-grenoble-alpes.fr/jakser/verde [2] https://criu.org/ https://criu.org/
- ma2rten 5y ago
  How does rr work with multi-threading? I would imagine threads can mess with the order that syscalls are executed.
  - khuey 5y ago
    rr emulates a single core machine and only one thread in the process is making process at any given time
- khuey 5y ago
  rr is capable of very limited diversion from the originally executed trace. This functionality exists to support things like `call myProgramObject->dump()` in gdb during replay. It will give up immediately on any non-trivial syscall though. Being able to change something and explore another execution path is interesting but it's extremely difficult to do in general. I think it would be very useful for verifying fixes, which is something that tools like rr are not particularly helpful with.