11 ms·

What rr does

880 points by mmarq 4y ago

DoctorOW 4y ago
Previously: https://news.ycombinator.com/item?id=18388879 https://news.ycombinator.com/item?id=18388879
- mmarq 4y ago
  Sorry, I didn’t know that. I thought the site would recognise duplicated links.
  - mkl95 4y ago
    Reposting is OK. I once got an email from HN staff inviting me to repost a link.
  - speps 4y ago
    It's quite common for people to refer to old discussions (here 2+ years ago) for popular projects like rr.
    - PostOnce 4y ago
      Yes, because there may be something to learn from the old comments and the new. It's good. Different people comment in different eras.
  - deleted 4y ago
    [deleted]
  - jwilk 4y ago
    From the FAQ <https://news.ycombinator.com/newsfaq.html https://news.ycombinator.com/newsfaq.html>: > Are reposts ok? > If a story has not had significant attention in the last year or so, a small number of reposts is ok.
  - dang 4y ago
    As jwilk correctly says, reposts are fine after a year or so. Pointing to previous links with comments is just to satisfy users who might be curious for more. You did good!
- dang 4y ago
  Thanks! Macroexpanded: Instant replay: Debugging C and C++ programs with rr - https://news.ycombinator.com/item?id=27034588 https://news.ycombinator.com/item?id=27034588 - May 2021 (66 comments) Using time travel to remotely debug faulty DRAM - https://news.ycombinator.com/item?id=24589597 https://news.ycombinator.com/item?id=24589597 - Sept 2020 (62 comments) Time Traveling Linux Bug Reporting: Coming in Julia 1.5 - https://news.ycombinator.com/item?id=23069372 https://news.ycombinator.com/item?id=23069372 - May 2020 (21 comments) rr: lightweight recording and deterministic debugging - https://news.ycombinator.com/item?id=18388879 https://news.ycombinator.com/item?id=18388879 - Nov 2018 (52 comments) Rr 5.0 Released - https://news.ycombinator.com/item?id=15191445 https://news.ycombinator.com/item?id=15191445 - Sept 2017 (3 comments) Debugging Leaks with rr - https://news.ycombinator.com/item?id=10573308 https://news.ycombinator.com/item?id=10573308 - Nov 2015 (4 comments) Back to the Futu-Rr-e: Deterministic Debugging with Rr - https://news.ycombinator.com/item?id=10492664 https://news.ycombinator.com/item?id=10492664 - Nov 2015 (9 comments) Rr 4.0 Debugger Released with Reverse Execution - https://news.ycombinator.com/item?id=10441618 https://news.ycombinator.com/item?id=10441618 - Oct 2015 (11 comments) Rr records nondeterministic executions and debugs them deterministically - https://news.ycombinator.com/item?id=8817954 https://news.ycombinator.com/item?id=8817954 - Dec 2014 (9 comments) Rr 3.0 Released with x86-64 Support - https://news.ycombinator.com/item?id=8734502 https://news.ycombinator.com/item?id=8734502 - Dec 2014 (6 comments) Porting rr to x86-64 - https://news.ycombinator.com/item?id=8543624 https://news.ycombinator.com/item?id=8543624 - Nov 2014 (9 comments)
stefantalpalaru 4y ago
https://github.com/rr-debugger/rr#system-requirements https://github.com/rr-debugger/rr#system-requirements : "rr currently requires either: - An Intel CPU with Nehalem (2010) or later microarchitecture. - Certain AMD Zen or later processors (see https://github.com/rr-debugger/rr/wiki/Zen)"
ris58h 4y ago
Could you next time provide some small but meaningful description in the title? "Rr" is a little bit short in my opinion.
- asicsp 4y ago
  For those reading the above comment, but hadn't clicked the article link yet (like me): >rr aspires to be your primary C/C++ debugging tool for Linux, replacing — well, enhancing — gdb. You record a failure once, then debug the recording, deterministically, as many times as you want. The same execution is replayed every time. >rr also provides efficient reverse execution under gdb. Set breakpoints and data watchpoints and quickly reverse-execute to where they were hit.
- pvinis 4y ago
  maybe "rr, a gdb replacement"
  - bajsejohannes 4y ago
    That feels like it's underselling it a bit, since gdb does not have reverse execution, which is a pretty major contribution by rr.
    - ncmncm 4y ago
      AIUI, Gdb does claim reverse execution, for certain targets. So, there are differences, but I don't understand them.
      - ynik 4y ago
        gdb's reverse execution is incredibly slow. Performance overhead of reverse debuggers: * gdb: >1000x (note: I never tested this one myself; just heard about this overhead in a HN comment a long time ago) * Microsoft WinDbg TimeTravel Debugger: >40x * rr: 1.5x rr is the only one fast enough to be used on a regular basis -- the others are slow enough that they only make sense on particularly nasty bugs (usually memory corruption)
sph 4y ago
This is incredible! For those that have used it, how useful it is for debugging multithreading heisenbugs? Can I let a process run under rr for days, wait until it crashes to due a heisenbug, and replay the trace without rr having to go through days of recording? i.e. is it possible to fast forward the trace, somehow? (I nerd sniped myself a bit here, wondering how fast forwarding could be implemented. I think it might be achievable with periodic process memory snapshots and incremental traces.)
- fanf2 4y ago
  I have not used rr heavily, but I did use it to help find a multithreading heisenbug in BIND https://kb.isc.org/docs/aa-01606 https://kb.isc.org/docs/aa-01606 I could not reproduce the bug in less than an hour of run time, which meant that analysing the bug in gdb required an hour for it to run forward to the crash point, after which it was possible to skip back and forth.
- pm215 4y ago
  rr numbers each 'event' it records, and you can pass an event number to the gdb 'run' command to tell it to start from that event. Recent 'rr' now also supports the -e option to replay meaning 'start the debug session pointing at the last recorded event, whatever that was'. Details in the usage page: https://github.com/rr-debugger/rr/wiki/Usage https://github.com/rr-debugger/rr/wiki/Usage AIUI you get 'start at an event' basically for free, because 'step backward' is implemented as 'start at the preceding event and then step forward by N', so events are frequent in the trace and the machinery to get to that point without running all the way from the start of the debug session exists anyway. There's some stuff on the website about how this is all implemented, I think.
- roca 4y ago
  You probably could record a process running for days but it would also take days to replay to the end, which would not be much fun. We don't create checkpoints during the recording. You'd be better off restarting the recording periodically. Also, rr has a "chaos mode" which randomizes scheduling and often makes threading bugs easier to reproduce. https://robert.ocallahan.org/2016/02/introducing-rr-chaos-mode.html https://robert.ocallahan.org/2016/02/introducing-rr-chaos-mo...
stewbrew 4y ago
We already have https://r-project.org https://r-project.org. Now we have https://rr-project.org https://rr-project.org. So, https://rrr-project.org https://rrr-project.org is next?
- leoff 4y ago
  It is actually a geometric progression, so https://rrrr-project.org https://rrrr-project.org would be the next one.
  - throwamon 4y ago
    Common misconception. Actually it's a Fibonacci sequence, so the next one really is https://rrr-project.org https://rrr-project.org and then it's https://rrrrr-project.org https://rrrrr-project.org. This does also mean that there's https://-project.org https://-project.org, and that https://r-project.org https://r-project.org secretly disambiguates into two different projects.
- deleted 4y ago
  [deleted]
sirwhinesalot 4y ago
This sounds like a debugger I might actually enjoy using (unlike all the others).
- db48x 4y ago
  rr is a superpower, but pernosco is several superpowers (https://pernos.co/ https://pernos.co/; it’s built on rr). I recommend both!
  - Agingcoder 4y ago
    I second this. Pernosco is just unbelievable.
jng 4y ago
Eager for the day someone integrates this into VS Code.
- hsivonen 4y ago
  Already done: https://farre.github.io/midas/ https://farre.github.io/midas/
  - jng 4y ago
    Wow, thanks. I will give it a try!!
- jchw 4y ago
  FWIW, rr integrates into gdb, so it should be possible to use anything that integrates with gdb. https://github.com/rr-debugger/rr/wiki/Using-rr-in-an-IDE https://github.com/rr-debugger/rr/wiki/Using-rr-in-an-IDE
borodi 4y ago
rr is a life changing experience for debugging things. One underrated thing is being able to save and share rr traces. rr + CI makes finding and potentially fixing heisenbugs a lot easier.
jonnycomputer 4y ago
Looks like a cool debugging tool, but I clicked the link because I thought maybe it was related to R. Maybe modify the title to make it clearer?
- deleted 4y ago
  [deleted]
jchw 4y ago
Historically, rr has not worked on AMD processors, which is a bummer. However, I have been able to make good use of it on my 5950X now with the workaround script and newer versions of rr. This is good news. I've not read their extended technical report, but I am kind of curious exactly what performance counters AMD is implementing poorly and how that impacts rr.
- vchuravy 4y ago
  I think https://github.com/rr-debugger/rr/issues/2034#issuecomment-691339758 https://github.com/rr-debugger/rr/issues/2034#issuecomment-6... is the right synopsis.
- KenoFischer 4y ago
  vchuravy's link gives the details, but basically, there's a microarchitecture optimization in Zen that breaks determinism of the performance counters. Fortunately, there's a chicken bit that turns it off, which is what the script does. I've been trying to convince AMD to officially document the bit such that the kernel can set it automatically, but no luck so far. There is still one remaining annoyance, which is that AMD's NMI latency is super high, which directly tanks rr's reverse execution latency. There's probably some improvements that could be made to the replayer to be more aggressive about optimistic assumptions on NMI latency and retrying if those assumptions are off, but it'd be a fair bit of work. I don't really understand why AMD decided to use this kind of architecture. It also makes profiles much less accurate.
  - jchw 4y ago
    Thanks for the info. I was wondering what was going on in that script. It’s unfortunate that their architectural decisions had to impact rr, but I guess these days, every last bit of benchmark score really matters.
qumpis 4y ago
Is there a counterpart of this in Python?
- BiteCode_dev 4y ago
  There was an attempt for Python 2, but it didn't catch on.
- pizza 4y ago
  pytrace https://pytrace.com https://pytrace.com
InfiniteRand 4y ago
Does rr require debug builds? Like if I took a random executable on Linux and used rr record, would rr replay work?
- teaearlgraycold 4y ago
  You might be relegated to stepping through disassembled machine code. I was able to use rr with a home made JIT compiler, stepping through JIT’d instructions. So I see no reason why you can’t at least get that experience with a production binary.
- mstange 4y ago
  It works with optimized builds, and it works better with them than gdb does. When you debug an optimized build with debug info in gdb by stepping line by line, it is easy to accidentally step "too far" and completely lose your place. In rr, you can always step back and recover.
bornfreddy 4y ago
Now this is how the project documentation should look like! Explains what the project does, gives some short examples of the most common features, then goes into the details - while being easy to understand for the target audience. Kudos to whoever wrote this! (and rr sounds like a nice tool too ;) )
- logbiscuitswave 4y ago
  I fully agree. I wish more intro-level documentation had this kind of easy to follow and progressive level of detail. All too many things like this either dive straight into the deep end inundating you with superfluous details when all you want is a primer, or provide so little information as to be nearly useless. The writers on this did a great job.
omginternets 4y ago
How does this work, exactly? Is it recording every state change in the program?
- sfink 4y ago
  No, that has been done but is much slower. It records all communication with the external world. The full answer is well described in https://arxiv.org/pdf/1705.05937.pdf https://arxiv.org/pdf/1705.05937.pdf which I'll quote here: > We identify a boundary around state and computation, record all sources of nondeterminism within the boundary and all inputs crossing into the boundary, and reexecute the computation within the boundary by replaying the nondeterminism and inputs. If all inputs and nondeterminism have truly been captured, the state and computation within the boundary during replay will match that during recording. So for any chunk of time spent entirely in user space doing computation, the replay starts out in the same situation and executes in exactly the same way the original process did, with zero overhead. That's what enables rr to be so low overhead overall; most programs spend the bulk of their time computing stuff and reading/writing memory. The replayed process has no way of knowing that its file descriptors aren't actually open, since anything it does with them will be provided by the recording. Quoting again: > In particular, user-space memory and register values are preserved exactly, with a few exceptions noted later in the paper. This implies CPU-level control flow is identical between recording and replay, as is memory layout.
  - omginternets 4y ago
    That is very cool, and very clever! :)
adgjlsfhk1 4y ago
Breakpoint plus reverse watch is incredibly powerful. It makes it trivial to find the code that last modified a variable before a breakpoint.
pizza 4y ago
what I would give to have something like this that additionally worked on Mac and Windows too
sfink 4y ago
If you use https://pernos.co/ https://pernos.co/ then you don't need any of this, but I have a set of only slightly buggy gdbinit scripts that extend the rr debugging experience at <https://github.com/hotsphink/sfink-tools https://github.com/hotsphink/sfink-tools>. The main things it adds are: 1. a `log` command that just records whatever you give it into a plaintext file, together with its "point in time" according to rr. This is useful because when using rr, you tend to move forward and backward in time a lot, and it's hard to keep track of the actual sequence of events and where you are within them. It also creates a checkpoint so you can return to any one of your log points. It also has some niceties like replacing any expression enclosed in curly brackets with the results of executing the gdb expression given, so you can do things like log starting execution of Init() with v={v}. About to crash. 2. a `label` command that lets you assign names to random hex values. Then in the output of `p expr` or the above `log` with no arguments, which displays the full set of log messages you've recorded, it will replace known hex values with their labels. This is so much nicer than memorizing numeric values and matching them up. (rr) p obj $1 = (JSObject*) 0x7f606892a200 (rr) label OUTER_OBJ=obj (rr) p $OUTER_OBJ (JSObject*) $OUTER_OBJ (rr) log 701/31299795 [c4] starting processing with obj=(JSObject*) $OUTER_OBJ 983/31299 [c2] starting processing with obj=(JSObject*) 0x7f6068a2a200 2081/7382911 [c3] traversing to (JSObject*) 0x7f6069c2a7e8 3316/199 [c1] crashing while accessing field of object (JSObject*) $OUTER_OBJ The [c2] markers are the automatically-created checkpoints, numbered in order that you made those log entries in the debugger. It reorders the log messages to show them in execution order rather than debugging order. Pernosco has a very similar feature called the Notebook (where you only have to click on a log entry to view the state at that point in time.) The scripts are also intended for sharing log files and labels between multiple concurrent replays of the same execution, which I find useful to have separate windows each maintaining a different context (point in time, and portion of the execution that I'm examining.) That tends to be the buggier part of the scripts, though. ;-) If you're working with C or C++ (or Rust? haven't tried it), rr really is a superpower. I rarely bother using straight gdb anymore. It feels crippled.
logbiscuitswave 4y ago
This kind of replayable debugging can be wonderful - especially for hard to debug issues like heap corruption and such. Windows has something similar called Time Travel Debugging[1] but in my experience the dump files it creates can be enormous and be a pain to analyze as a result. (It also relies on WinDbg which while being extremely powerful and capable, has a huge learning and usability cliff. I’ve been using it for over a decade and I still need a cheat sheet from time to time. The revamped WinDbg Preview[2] improves the UI a lot, but ultimately it’s still WinDbg.) [1] https://docs.microsoft.com/en-us/windows-hardware/drivers/debugger/time-travel-debugging-overview https://docs.microsoft.com/en-us/windows-hardware/drivers/de... [2] https://docs.microsoft.com/en-us/windows-hardware/drivers/debugger/debugging-using-windbg-preview https://docs.microsoft.com/en-us/windows-hardware/drivers/de...
blacksqr 4y ago
Makes every day seem like Talk Like a Pirate Day.
raydiatian 4y ago
Man. I wish I had this for Typescript. Well done!

What rr does

880 points by mmarq 4y ago

DoctorOW 4y ago

Previously: https://news.ycombinator.com/item?id=18388879 https://news.ycombinator.com/item?id=18388879

mmarq 4y ago
Sorry, I didn’t know that. I thought the site would recognise duplicated links.
- mkl95 4y ago
  Reposting is OK. I once got an email from HN staff inviting me to repost a link.
- speps 4y ago
  It's quite common for people to refer to old discussions (here 2+ years ago) for popular projects like rr.
  - PostOnce 4y ago
    Yes, because there may be something to learn from the old comments and the new. It's good. Different people comment in different eras.
- deleted 4y ago
  [deleted]
- jwilk 4y ago
  From the FAQ <https://news.ycombinator.com/newsfaq.html https://news.ycombinator.com/newsfaq.html>: > Are reposts ok? > If a story has not had significant attention in the last year or so, a small number of reposts is ok.
- dang 4y ago
  As jwilk correctly says, reposts are fine after a year or so. Pointing to previous links with comments is just to satisfy users who might be curious for more. You did good!
dang 4y ago
Thanks! Macroexpanded: Instant replay: Debugging C and C++ programs with rr - https://news.ycombinator.com/item?id=27034588 https://news.ycombinator.com/item?id=27034588 - May 2021 (66 comments) Using time travel to remotely debug faulty DRAM - https://news.ycombinator.com/item?id=24589597 https://news.ycombinator.com/item?id=24589597 - Sept 2020 (62 comments) Time Traveling Linux Bug Reporting: Coming in Julia 1.5 - https://news.ycombinator.com/item?id=23069372 https://news.ycombinator.com/item?id=23069372 - May 2020 (21 comments) rr: lightweight recording and deterministic debugging - https://news.ycombinator.com/item?id=18388879 https://news.ycombinator.com/item?id=18388879 - Nov 2018 (52 comments) Rr 5.0 Released - https://news.ycombinator.com/item?id=15191445 https://news.ycombinator.com/item?id=15191445 - Sept 2017 (3 comments) Debugging Leaks with rr - https://news.ycombinator.com/item?id=10573308 https://news.ycombinator.com/item?id=10573308 - Nov 2015 (4 comments) Back to the Futu-Rr-e: Deterministic Debugging with Rr - https://news.ycombinator.com/item?id=10492664 https://news.ycombinator.com/item?id=10492664 - Nov 2015 (9 comments) Rr 4.0 Debugger Released with Reverse Execution - https://news.ycombinator.com/item?id=10441618 https://news.ycombinator.com/item?id=10441618 - Oct 2015 (11 comments) Rr records nondeterministic executions and debugs them deterministically - https://news.ycombinator.com/item?id=8817954 https://news.ycombinator.com/item?id=8817954 - Dec 2014 (9 comments) Rr 3.0 Released with x86-64 Support - https://news.ycombinator.com/item?id=8734502 https://news.ycombinator.com/item?id=8734502 - Dec 2014 (6 comments) Porting rr to x86-64 - https://news.ycombinator.com/item?id=8543624 https://news.ycombinator.com/item?id=8543624 - Nov 2014 (9 comments)

stefantalpalaru 4y ago

https://github.com/rr-debugger/rr#system-requirements https://github.com/rr-debugger/rr#system-requirements : "rr currently requires either: - An Intel CPU with Nehalem (2010) or later microarchitecture. - Certain AMD Zen or later processors (see https://github.com/rr-debugger/rr/wiki/Zen)"

ris58h 4y ago

Could you next time provide some small but meaningful description in the title? "Rr" is a little bit short in my opinion.

asicsp 4y ago
For those reading the above comment, but hadn't clicked the article link yet (like me): >rr aspires to be your primary C/C++ debugging tool for Linux, replacing — well, enhancing — gdb. You record a failure once, then debug the recording, deterministically, as many times as you want. The same execution is replayed every time. >rr also provides efficient reverse execution under gdb. Set breakpoints and data watchpoints and quickly reverse-execute to where they were hit.
pvinis 4y ago
maybe "rr, a gdb replacement"
- bajsejohannes 4y ago
  That feels like it's underselling it a bit, since gdb does not have reverse execution, which is a pretty major contribution by rr.
  - ncmncm 4y ago
    AIUI, Gdb does claim reverse execution, for certain targets. So, there are differences, but I don't understand them.
    - ynik 4y ago
      gdb's reverse execution is incredibly slow. Performance overhead of reverse debuggers: * gdb: >1000x (note: I never tested this one myself; just heard about this overhead in a HN comment a long time ago) * Microsoft WinDbg TimeTravel Debugger: >40x * rr: 1.5x rr is the only one fast enough to be used on a regular basis -- the others are slow enough that they only make sense on particularly nasty bugs (usually memory corruption)

sph 4y ago

This is incredible! For those that have used it, how useful it is for debugging multithreading heisenbugs? Can I let a process run under rr for days, wait until it crashes to due a heisenbug, and replay the trace without rr having to go through days of recording? i.e. is it possible to fast forward the trace, somehow? (I nerd sniped myself a bit here, wondering how fast forwarding could be implemented. I think it might be achievable with periodic process memory snapshots and incremental traces.)

fanf2 4y ago
I have not used rr heavily, but I did use it to help find a multithreading heisenbug in BIND https://kb.isc.org/docs/aa-01606 https://kb.isc.org/docs/aa-01606 I could not reproduce the bug in less than an hour of run time, which meant that analysing the bug in gdb required an hour for it to run forward to the crash point, after which it was possible to skip back and forth.
pm215 4y ago
rr numbers each 'event' it records, and you can pass an event number to the gdb 'run' command to tell it to start from that event. Recent 'rr' now also supports the -e option to replay meaning 'start the debug session pointing at the last recorded event, whatever that was'. Details in the usage page: https://github.com/rr-debugger/rr/wiki/Usage https://github.com/rr-debugger/rr/wiki/Usage AIUI you get 'start at an event' basically for free, because 'step backward' is implemented as 'start at the preceding event and then step forward by N', so events are frequent in the trace and the machinery to get to that point without running all the way from the start of the debug session exists anyway. There's some stuff on the website about how this is all implemented, I think.
roca 4y ago
You probably could record a process running for days but it would also take days to replay to the end, which would not be much fun. We don't create checkpoints during the recording. You'd be better off restarting the recording periodically. Also, rr has a "chaos mode" which randomizes scheduling and often makes threading bugs easier to reproduce. https://robert.ocallahan.org/2016/02/introducing-rr-chaos-mode.html https://robert.ocallahan.org/2016/02/introducing-rr-chaos-mo...

stewbrew 4y ago

We already have https://r-project.org https://r-project.org. Now we have https://rr-project.org https://rr-project.org. So, https://rrr-project.org https://rrr-project.org is next?

leoff 4y ago
It is actually a geometric progression, so https://rrrr-project.org https://rrrr-project.org would be the next one.
- throwamon 4y ago
  Common misconception. Actually it's a Fibonacci sequence, so the next one really is https://rrr-project.org https://rrr-project.org and then it's https://rrrrr-project.org https://rrrrr-project.org. This does also mean that there's https://-project.org https://-project.org, and that https://r-project.org https://r-project.org secretly disambiguates into two different projects.
deleted 4y ago
[deleted]

sirwhinesalot 4y ago

This sounds like a debugger I might actually enjoy using (unlike all the others).

db48x 4y ago
rr is a superpower, but pernosco is several superpowers (https://pernos.co/ https://pernos.co/; it’s built on rr). I recommend both!
- Agingcoder 4y ago
  I second this. Pernosco is just unbelievable.

jng 4y ago

Eager for the day someone integrates this into VS Code.

hsivonen 4y ago
Already done: https://farre.github.io/midas/ https://farre.github.io/midas/
- jng 4y ago
  Wow, thanks. I will give it a try!!
jchw 4y ago
FWIW, rr integrates into gdb, so it should be possible to use anything that integrates with gdb. https://github.com/rr-debugger/rr/wiki/Using-rr-in-an-IDE https://github.com/rr-debugger/rr/wiki/Using-rr-in-an-IDE

borodi 4y ago

rr is a life changing experience for debugging things. One underrated thing is being able to save and share rr traces. rr + CI makes finding and potentially fixing heisenbugs a lot easier.

jonnycomputer 4y ago

Looks like a cool debugging tool, but I clicked the link because I thought maybe it was related to R. Maybe modify the title to make it clearer?

deleted 4y ago
[deleted]

jchw 4y ago

Historically, rr has not worked on AMD processors, which is a bummer. However, I have been able to make good use of it on my 5950X now with the workaround script and newer versions of rr. This is good news. I've not read their extended technical report, but I am kind of curious exactly what performance counters AMD is implementing poorly and how that impacts rr.

vchuravy 4y ago
I think https://github.com/rr-debugger/rr/issues/2034#issuecomment-691339758 https://github.com/rr-debugger/rr/issues/2034#issuecomment-6... is the right synopsis.
KenoFischer 4y ago
vchuravy's link gives the details, but basically, there's a microarchitecture optimization in Zen that breaks determinism of the performance counters. Fortunately, there's a chicken bit that turns it off, which is what the script does. I've been trying to convince AMD to officially document the bit such that the kernel can set it automatically, but no luck so far. There is still one remaining annoyance, which is that AMD's NMI latency is super high, which directly tanks rr's reverse execution latency. There's probably some improvements that could be made to the replayer to be more aggressive about optimistic assumptions on NMI latency and retrying if those assumptions are off, but it'd be a fair bit of work. I don't really understand why AMD decided to use this kind of architecture. It also makes profiles much less accurate.
- jchw 4y ago
  Thanks for the info. I was wondering what was going on in that script. It’s unfortunate that their architectural decisions had to impact rr, but I guess these days, every last bit of benchmark score really matters.

qumpis 4y ago

Is there a counterpart of this in Python?

BiteCode_dev 4y ago
There was an attempt for Python 2, but it didn't catch on.
pizza 4y ago
pytrace https://pytrace.com https://pytrace.com

InfiniteRand 4y ago

Does rr require debug builds? Like if I took a random executable on Linux and used rr record, would rr replay work?

teaearlgraycold 4y ago
You might be relegated to stepping through disassembled machine code. I was able to use rr with a home made JIT compiler, stepping through JIT’d instructions. So I see no reason why you can’t at least get that experience with a production binary.
mstange 4y ago
It works with optimized builds, and it works better with them than gdb does. When you debug an optimized build with debug info in gdb by stepping line by line, it is easy to accidentally step "too far" and completely lose your place. In rr, you can always step back and recover.

bornfreddy 4y ago

Now this is how the project documentation should look like! Explains what the project does, gives some short examples of the most common features, then goes into the details - while being easy to understand for the target audience. Kudos to whoever wrote this! (and rr sounds like a nice tool too ;) )

logbiscuitswave 4y ago
I fully agree. I wish more intro-level documentation had this kind of easy to follow and progressive level of detail. All too many things like this either dive straight into the deep end inundating you with superfluous details when all you want is a primer, or provide so little information as to be nearly useless. The writers on this did a great job.

omginternets 4y ago

How does this work, exactly? Is it recording every state change in the program?

sfink 4y ago
No, that has been done but is much slower. It records all communication with the external world. The full answer is well described in https://arxiv.org/pdf/1705.05937.pdf https://arxiv.org/pdf/1705.05937.pdf which I'll quote here: > We identify a boundary around state and computation, record all sources of nondeterminism within the boundary and all inputs crossing into the boundary, and reexecute the computation within the boundary by replaying the nondeterminism and inputs. If all inputs and nondeterminism have truly been captured, the state and computation within the boundary during replay will match that during recording. So for any chunk of time spent entirely in user space doing computation, the replay starts out in the same situation and executes in exactly the same way the original process did, with zero overhead. That's what enables rr to be so low overhead overall; most programs spend the bulk of their time computing stuff and reading/writing memory. The replayed process has no way of knowing that its file descriptors aren't actually open, since anything it does with them will be provided by the recording. Quoting again: > In particular, user-space memory and register values are preserved exactly, with a few exceptions noted later in the paper. This implies CPU-level control flow is identical between recording and replay, as is memory layout.
- omginternets 4y ago
  That is very cool, and very clever! :)

adgjlsfhk1 4y ago

Breakpoint plus reverse watch is incredibly powerful. It makes it trivial to find the code that last modified a variable before a breakpoint.

pizza 4y ago

what I would give to have something like this that additionally worked on Mac and Windows too

sfink 4y ago

If you use https://pernos.co/ https://pernos.co/ then you don't need any of this, but I have a set of only slightly buggy gdbinit scripts that extend the rr debugging experience at <https://github.com/hotsphink/sfink-tools https://github.com/hotsphink/sfink-tools>. The main things it adds are: 1. a `log` command that just records whatever you give it into a plaintext file, together with its "point in time" according to rr. This is useful because when using rr, you tend to move forward and backward in time a lot, and it's hard to keep track of the actual sequence of events and where you are within them. It also creates a checkpoint so you can return to any one of your log points. It also has some niceties like replacing any expression enclosed in curly brackets with the results of executing the gdb expression given, so you can do things like log starting execution of Init() with v={v}. About to crash. 2. a `label` command that lets you assign names to random hex values. Then in the output of `p expr` or the above `log` with no arguments, which displays the full set of log messages you've recorded, it will replace known hex values with their labels. This is so much nicer than memorizing numeric values and matching them up. (rr) p obj $1 = (JSObject*) 0x7f606892a200 (rr) label OUTER_OBJ=obj (rr) p $OUTER_OBJ (JSObject*) $OUTER_OBJ (rr) log 701/31299795 [c4] starting processing with obj=(JSObject*) $OUTER_OBJ 983/31299 [c2] starting processing with obj=(JSObject*) 0x7f6068a2a200 2081/7382911 [c3] traversing to (JSObject*) 0x7f6069c2a7e8 3316/199 [c1] crashing while accessing field of object (JSObject*) $OUTER_OBJ The [c2] markers are the automatically-created checkpoints, numbered in order that you made those log entries in the debugger. It reorders the log messages to show them in execution order rather than debugging order. Pernosco has a very similar feature called the Notebook (where you only have to click on a log entry to view the state at that point in time.) The scripts are also intended for sharing log files and labels between multiple concurrent replays of the same execution, which I find useful to have separate windows each maintaining a different context (point in time, and portion of the execution that I'm examining.) That tends to be the buggier part of the scripts, though. ;-) If you're working with C or C++ (or Rust? haven't tried it), rr really is a superpower. I rarely bother using straight gdb anymore. It feels crippled.

logbiscuitswave 4y ago

This kind of replayable debugging can be wonderful - especially for hard to debug issues like heap corruption and such. Windows has something similar called Time Travel Debugging[1] but in my experience the dump files it creates can be enormous and be a pain to analyze as a result. (It also relies on WinDbg which while being extremely powerful and capable, has a huge learning and usability cliff. I’ve been using it for over a decade and I still need a cheat sheet from time to time. The revamped WinDbg Preview[2] improves the UI a lot, but ultimately it’s still WinDbg.) [1] https://docs.microsoft.com/en-us/windows-hardware/drivers/debugger/time-travel-debugging-overview https://docs.microsoft.com/en-us/windows-hardware/drivers/de... [2] https://docs.microsoft.com/en-us/windows-hardware/drivers/debugger/debugging-using-windbg-preview https://docs.microsoft.com/en-us/windows-hardware/drivers/de...

blacksqr 4y ago

Makes every day seem like Talk Like a Pirate Day.

raydiatian 4y ago

Man. I wish I had this for Typescript. Well done!