18 ms·
rr – record and replay debugger for C/C++
- rtpg 2y agorr is really cool, but almost every time I have decided to pull it out as one of the "big guns" it turns out that I have a concurrency bug and so rr is unable to reproduce it. Despite that, it would be very, very, very cool if some languages built rr directly into their tooling. Obviously you can always "just" use rr/gdb, but imagine if rr invocations were as easy to set up and do as pdb is in Python!
- angus-g 2y agoI actually had a concurrency bug that I was able to capture with rr: an MPI job where I only ran rr on rank 0 and managed to figure out where a different send/recv ordering was causing issues. In fact, it was also a Python model that ties in with a lot of native code generation, so quite a complex issue.
- IshKebab 2y agoYeah same for me. Actually the time I really wanted it was on Mac and unfortunately it only works on Linux. There is Undodb which works on Mac and maybe with multithreading (not sure about that), but unfortunately it costs about $50k.
- borodi 2y agoChaos mode is an option when invoking rr that can expose some concurrency issues. Basically it switches which thread is executing a bunch to try and simulate multiple cores executing. It has found some race conditions for me but it’s of course limited
- dzaima 2y agoUnfortunately that only works for large-scale races, and not, say, one instruction interleaving with another one on another thread without proper synchronization. -fsanitize=thread probably works for that though (and of course you could then combine said sanitizer with rr to some effect probably).
- borodi 2y agoI havent tried Tsan with rr but msan and asan work quite well with it (it’s quite slow when doing this) but seeing the sanitizer trigger then following back what caused it to trigger is very useful.
- bqmjjx0kac 2y agoYeah, the reason it only works for these coarser race conditions is that RR only has one thread executing at a time. Chaos mode randomizes the durations of time allotted to each thread before it is preempted. This may be out of date. I believe I read it in the Extended Technical Report from 2017: https://arxiv.org/pdf/1705.05937 https://arxiv.org/pdf/1705.05937
- roca 2y agoOne option would be to combine chaos mode with a dynamic race detector to try to focus chaos mode on specific fine-grained races. Someone should try that as a research project. Not really the same thing as rr + TSAN. There's still the fundamental limitation that rr won't help you with weak memory orderings.
- modeless 2y agoAnother thing that rr sadly doesn't support is GPUs. I'd love to use it but most of my stuff involves GPUs in some way or another.
- nsjolj 2y ago[flagged]
- tdullien 2y agoI've used it for years for vulnerability development (when you need to debug why a particular exploitation run went wrong etc.). It is indeed marvelous for many situations; except for concurrency bugs ;)
- i_am_a_peasant 2y agoWhy isn't it good for concurrency bugs? rr doesn't record the entire state up to the race condition?
- adgjlsfhk1 2y agoit serializes your problem because you can't atomically snapshot memory when there are multiple threads writing to it
- pm215 2y agoThis is true, but for at least some race conditions the rr "chaos mode" can help in tracking them down. Chaos mode basically randomly adjusts its thread scheduling so that it sometimes doesn't run a thread at all for a few seconds, so you run it a bunch of times until it hits the timing window for the race condition; and then when it does you debug the recording. IME it works at least sometimes but it does require that the thing you're testing doesn't take too long to fall over, since you need to do multiple recordings of it. https://robert.ocallahan.org/2016/02/introducing-rr-chaos-mode.html https://robert.ocallahan.org/2016/02/introducing-rr-chaos-mo...
- nemetroid 2y agoRR supports recording multithreaded programs, but only runs one thread at a time, so it cannot trigger data race bugs. It can find higher level concurrency bugs though, and has a "chaos mode" that schedules threads in an unfair random manner.
- suby 2y agoPerhaps worth mentioning is that someone attempted to port this to Rust and got about 60,000 lines of code into it before archiving the project. I feel like comparing these two efforts would be an interesting case study on the impacts / benefits / limitations or difficulties, etc involved in rewriting from C++ to Rust. https://github.com/sidkshatriya/rd/ https://github.com/sidkshatriya/rd/
- dwattttt 2y agoIt would be a good but difficult analysis; at a quick check, rr 1.0 took 3 years and signficant contributions from around 3 or 4 people (I saw at least 5 people contributing), and the rr we have today is 10 years further work on that.
- tarruda 2y agoI don't understand the "rewrite X/Y/Z in Rust" trend that has been going for a few years. I'm not familiar with Rust, but I'm almost sure it has a good C interoperability. If a certain piece of software is working well, what is the benefit of rewriting it in Rust?
- coldtea 2y ago>If a certain piece of software is working well, what is the benefit of rewriting it in Rust? Presumablly the idea here is to support Rust replay debugging, not just rewrite a C/C++ targetting replay debugger in Rust
- jfk13 2y agoA couple of previous discussions: https://news.ycombinator.com/item?id=31617600 https://news.ycombinator.com/item?id=31617600 (June 2022) https://news.ycombinator.com/item?id=18388879 https://news.ycombinator.com/item?id=18388879 (Nov 2018)
- laserbeam 2y agoIs it truly only for C/C++? My limited understanding says a debugger needs: a list of symbols (.pdb files on windows, can't remember what they are on linux), understanding of syscalls and a few other similar things. I thought they don't care too much what generated the binaries they are debugging (obviously as long as it's native code). Doesn't rr work with other languages like rust, zig, odin, nim, and similar ones? Obviously, I wouldn't expect it to work for python, js, c# and other languages with managed memory.
- fragmede 2y agofor js, there's http://replay.io http://replay.io
- leni536 2y agoInterestingly gdb (and in turn rr) has some limited support for debugging python. At least you can get a python backtrace, but I didn't have success in setting pyhton breakpoints.
- laserbeam 2y agoYeah, 'cause you're technically debugging the python interpreter. I've had some success with tracing tools designed for C/C++ for a python project. Was not easy to set up and will obviously include frames from the interpreter. Tho, it feels wrong to expect a tool designed for native binaries to work well with python in this context. And that's ok. It feels lucky when it works as much as it does.
- greg_law 2y agohttps://undo.io/resources/how-i-debug-python-code-with-a-time-travel-debugger https://undo.io/resources/how-i-debug-python-code-with-a-tim... "How I debug Python code with a Time Travel Debugger" Works surprisingly well.
- dzaima 2y agorr uses gdb as the actual debugger part, so anything that works in gdb will work in rr. (you won't get rr running on windows though, as it is very much linux-specific, having to wrap all of its syscalls. The linux symbol info thing is DWARF)
- whatsakandr 2y agoI've used rr very sucessfully for reverse engineering a large code base using a break on variable change combined with reverse-continue. Took the time to extract critical logic way down.
- rramadass 2y agoThat sounds very interesting; Do you have a write-up on this that you are willing to share?
- dima55 2y agoThis is the usual killer feature of something like rr. You debug, look at some variable: `p whatever`. You see that its value is wrong. You want to know where this wrong value came from, so you `watch -l whatever` and `rc`. Bam!
- vinnyvichy 2y agoMay I assume the large codebase was written in a language with (for lack of a better term) dynamic types?
- nesarkvechnep 2y ago
- whatsakandr 2y agoHas anyone gotten rr to work with opengl or vulkan? It seems to always crash for me after making an opengl call.
- arbruijn 2y agoVirGL might help, it redirects OpenGL calls over a socket. Start virgl_test_server and run your app with extra environment vars __GLX_VENDOR_LIBRARY_NAME=mesa LIBGL_ALWAYS_SOFTWARE=1 GALLIUM_DRIVER=virpipe
- boguscoder 2y agoCurious if anyone tried rr for/on Android? It seems possible to crosscompile it and it could be a good tool for native side debugging
- forrestthewoods 2y agoOn Windows you can use WinDbg for the same thing. It has better support for debugging multi-threaded issues. https://www.forrestthewoods.com/blog/windbg-time-travelling-debugger-is-amazing-magic/ https://www.forrestthewoods.com/blog/windbg-time-travelling-...
- Veserv 2y agoWinDbg uses a instruction-level emulation time travel implementation, so incurs the 10-20x slowdown associated with that technique. rr uses a replay-record time travel debugging implementation, which can incur far less overhead when done correctly. Last I saw, rr has overhead in the 2x slowdown range and, if I remember correctly, I have seen a different record-replay time travel debugger in the 10% range. 10% is 100x cheaper than WinDbg and cheap enough to leave on all the time in production. That is a game-changer.
- forrestthewoods 2y ago> 10% is 100x cheaper than WinDbg If you’re gonna throw around numbers like this you need to cite an actual tool not “if I remember correctly there exists a unicorn”.
- dzaima 2y agoMore precisely, rr records just system calls & interrupts, so its overhead is largely proportional to how syscall-heavy and multithreaded your code is. If it's single-threaded and mostly just pure computation, you can easily see ~0% slowdown, with replay at the exact same speed. (a quick simple test shows that 'mmap'+'munmap' of a file is 0.5ms, vs 0.006ms outside rr; but many syscalls are buffered in rr's userspace and thus way faster (e.g. a 'stat' on the same file path is only like 300ns slower than native))
- modeless 2y agoSee also https://pernos.co/ https://pernos.co/ which is based on rr but adds a queryable database of the whole program execution, which allows you to do things like this: > [...] just click on the incorrect value. With full program history Pernosco can immediately explain where this value came from. The value is tracked backwards through events such as memcpys or moves in and out of registers until we reach a point where the value "originated" from, and each step in this process is displayed in the "Dataflow" panel that opens automatically. There is no need to read or understand the code and think about what might have happened, we can simply ask the debugger what did happen. https://pernos.co/examples/race-condition https://pernos.co/examples/race-condition
- leoc 2y agoAre rr’s problems with Ryzen CPUs now firmly in the past or not?
- padenot 2y agoYes, I use rr all day every day (to record Firefox executions) on a rather recent Threaripper Pro 7950, and also with Pernosco. The rr wiki on GitHub explains how to make it work. Once the small workaround is in place it works very reliably.
- dmitrygr 2y agoLong ago, VMWare workstation supported doing this, but not just for userspace programs but also for kernels and even drivers, in a VM. The feature shipped and existed for a few versions before it was killed by internal politics.
- Arech 2y agoAnd, I guess, before that, there was AMD SymNow, which was plug-in extendable and gave a plugin full control over CPU being emulated. I wonder if something like that is available somewhere?
- ho_schi 2y agoGDBs built-in reverse debugging: https://www.sourceware.org/gdb/wiki/ProcessRecord/Tutorial https://www.sourceware.org/gdb/wiki/ProcessRecord/Tutorial I assume rr provides more features and flexibility. Anyway I want to mention that GDB itself can already reverse debug for some time now.
- dima55 2y agoIf you want to mention this, then you very clearly haven't actually tried it. The implementation in GDB is more convenient than rr (you can start/stop recording at will), but it is also orders of magnitude less efficient. It's only usable for very small code snippets. Otherwise it takes effectively forever and/or runs out of resources.
- throwaway2037 2y ago> runs out of resources RAM? What kind of dev box runs out of RAM in 2024? I built a 64GB RAM dev box during COVID-19 crisis. I have never once come close to using all that RAM, even with a squillion Chrome tabs open. Still, thank you to share your first-hand experience. Did you ask the GDB Dev team for any feedback on the slow performance?
- enedil 2y agoUuh, any kind of dev box that requires more RAM than available? I promise you that storing that much data about the runtime can be really memory consumptive.
- signa11 2y agoumm, embedded devices without active cooling f.e. those node-b’s sitting on cellphone towers come to mind here, there can be quite a few other similar examples i can think of.
- kragen 2y agogdb record and replay will absolutely eat up your piddly 64 gigs of ram in a few minutes if you just let it loose. it will eat your 12-terabyte hard disk too, it just takes a little longer. the gdb manual has many helpful tips for how to allocate your ram carefully to record and replay, as well as using a disk buffer instead of recording to ram, so the gdb dev team is already well aware of the problem
- teaearlgraycold 2y agoI used this to help make my toy JIT compiler: https://github.com/danthedaniel/BF-JIT https://github.com/danthedaniel/BF-JIT Super useful, especially considering I know barely anything about x86-64.
- iamcreasy 2y agoIs it possible to use this with C/C++ code compiled to dll/so and called by Python?
- throwaway2037 2y agoCan someone explain how it works?
- khuey 2y agohttps://arxiv.org/abs/1705.05937 https://arxiv.org/abs/1705.05937
- alfiedotwtf 2y ago[off-topic] does anyone here who regularly uses a debugger (even just breakpoints and watchers in their IDE) use it for async execution? I've never tried, but I'm just trying to think how all that jumping around the executor and any runtime would work (if at all).
- MaskRay 2y agoI almost use rr every day, along with a gdb frontend: cgdb. rr record /tmp/Debug/bin/llvm-mc a.s && rr replay -d cgdb I've have success story with some bugs only reproducible with LTO. Without rr it would be a significant challenge. It would be nice if Linux kernel could be debugged with rr. Does anyone have success with kernel under rr+qemu ? :)
- rapiz 2y agowhat's the benefit of using cgdb while you can use gdb layout src?