8 ms·
Instant replay: Debugging C and C++ programs with rr
- froh 6y agothis is how to use and set up rr on redhat, with a simple example. the rr project itself focusses on rr proper and delegates the os specifics to the "install" section. https://rr-project.org/ https://rr-project.org/ rr is supported by all mature binary distros, rh, suse, ubuntu, ... out of the box these days.
- dekken_ 6y agoAnd now also supports Ryzen CPUs (to some extent)
- chrismorgan 6y agoSee https://github.com/rr-debugger/rr/wiki/Zen https://github.com/rr-debugger/rr/wiki/Zen for details. (And since https://github.com/rr-debugger/rr/pull/2847 https://github.com/rr-debugger/rr/pull/2847, I think all 5000-series CPUs and APUs should be supported.)
- jackosdev 6y agoWasn't working on my 5950x
- chrismorgan 6y agoI’m guessing you’re using the stable release. All 5000-series support has been added since then; you’ll need to build from source.
- chrismorgan 6y agosudo sh -c 'echo 1 >/proc/sys/kernel/perf_event_paranoid' Here’s another way of doing this: echo 1 | sudo tee /proc/sys/kernel/perf_event_paranoid You can’t use > redirection to a different user, because that’s just a file descriptor and they don’t have a user associated with them; but pipes can run between different users like this (well, I guess it’s sudo handling the plumbing there), and so tee can run as root, receiving input from an unprivileged user and then doing what it does, writing the contents of stdin both to a file and to stdout. This leads to a particularly useful trick in Vim if you edit a file you don’t have permission to write (most likely because you forgot to run `sudo vim` but now you’ve made the changes and you just want to save it, not quit and do it again as root): :w !sudo tee % That is: “write the contents of this buffer to the stdin of a `sudo tee ‹buffer filename›` invocation”. (See also https://stackoverflow.com/questions/2600783/how-does-the-vim-write-with-sudo-trick-work https://stackoverflow.com/questions/2600783/how-does-the-vim... for more detailed explanation.)
- karottenreibe 6y agoThere's of course also a plugin called SudoWrite that adds a command for exactly this :-)
- 1vuio0pswjnm7 6y agoSomehow I never use tee because sed allows the same thing, with up to seven files. Sometimes I do not have a tee in the userland (yet), but in every toolchain I always have sed. echo 1 | sed w/proc/sys/kernel/perf_event_paranoid
- chrismorgan 6y agoHuh. Never come across a *nix machine without either. Both are part of POSIX since at least 2001 (I don’t have ready access to specs from older).
- carlmr 6y agoYeah, similarly surprised, since I always do sudo tee.
- 1vuio0pswjnm7 6y agoI build systems from source. Sometimes I do things before installing a full userland. (Sometimes I never do install a full userland, opting for a smaller customised one.) I have learnt to use what is available on BSD install media and in BSD toolchains. sed is popular choice utility for BSD toolchains. tee, not so much.
- chrismorgan 6y agoI see I was correct in guessing that you weren’t dealing with Linux, since that’s almost certain to use GNU userland, and there tee is part of coreutils, and good luck getting anything done without coreutils. (GNU sed is its own thing rather than being part of coreutils or any other *utils bundle. I don’t know why this is.) I’m still a little surprised to hear of tee not being present when sed is, but I’m not familiar with BSD. Maybe they care even less about POSIX than I would have expected? I’ve been a little bit embittered against sed because of the differences across platforms. As usual, the version that comes out of the box on macOS is ancient, for licensing reasons, and the distinction has caused me grief too many times. I seem to recall having to give up on using sed for in-place regular expression replacement in a file when I was only caring about Linux, Linux under WSL 1 and macOS, and ended up using a fairly convoluted Perl one-liner instead (something like -plni.bak followed by rm foo.bak, because -i sans extension broke under WSL 1). Cross-platform operation is trickier than I wish it were!
- Tepix 6y agoThis seems insanely powerful and huge time saver.
- dang 6y agoA partial list of related previous threads: Print Debugging Should Go Away - https://news.ycombinator.com/item?id=26955344 https://news.ycombinator.com/item?id=26955344 - April 2021 (244 comments) Using time travel to remotely debug faulty DRAM - https://news.ycombinator.com/item?id=24589597 https://news.ycombinator.com/item?id=24589597 - Sept 2020 (62 comments) Rr: Capture, Replay, and debug GDB traces - https://news.ycombinator.com/item?id=23510627 https://news.ycombinator.com/item?id=23510627 - June 2020 (1 comment) Time Traveling Linux Bug Reporting: Coming in Julia 1.5 - https://news.ycombinator.com/item?id=23069372 https://news.ycombinator.com/item?id=23069372 - May 2020 (21 comments) Debuggers Suck - https://news.ycombinator.com/item?id=21644866 https://news.ycombinator.com/item?id=21644866 - Nov 2019 (257 comments) rr: lightweight recording and deterministic debugging - https://news.ycombinator.com/item?id=18388879 https://news.ycombinator.com/item?id=18388879 - Nov 2018 (52 comments) Control Flow Visualizer (CFViz): an rr / gdb plugin - https://news.ycombinator.com/item?id=15993938 https://news.ycombinator.com/item?id=15993938 - Dec 2017 (3 comments) Rr 5.0 Released - https://news.ycombinator.com/item?id=15191445 https://news.ycombinator.com/item?id=15191445 - Sept 2017 (3 comments) Rr 4.5.0 Released - https://news.ycombinator.com/item?id=13572133 https://news.ycombinator.com/item?id=13572133 - Feb 2017 (2 comments) Rr 4.4.0 Released - https://news.ycombinator.com/item?id=12617332 https://news.ycombinator.com/item?id=12617332 - Oct 2016 (1 comment) How to Track Down Divergence Bugs in Rr - https://news.ycombinator.com/item?id=11846189 https://news.ycombinator.com/item?id=11846189 - June 2016 (1 comment) Debugging Leaks with rr - https://news.ycombinator.com/item?id=10573308 https://news.ycombinator.com/item?id=10573308 - Nov 2015 (4 comments) Rr 4.0 Debugger Released with Reverse Execution - https://news.ycombinator.com/item?id=10441618 https://news.ycombinator.com/item?id=10441618 - Oct 2015 (11 comments) Rr records nondeterministic executions and debugs them deterministically - https://news.ycombinator.com/item?id=8817954 https://news.ycombinator.com/item?id=8817954 - Dec 2014 (9 comments) Rr 3.0 Released with x86-64 Support - https://news.ycombinator.com/item?id=8734502 https://news.ycombinator.com/item?id=8734502 - Dec 2014 (6 comments) Porting rr to x86-64 - https://news.ycombinator.com/item?id=8543624 https://news.ycombinator.com/item?id=8543624 - Nov 2014 (9 comments) RR: lightweight execution recording and deterministic debugging - https://news.ycombinator.com/item?id=7458554 https://news.ycombinator.com/item?id=7458554 - March 2014 (2 comments) If any George RR Martin stories got mixed in there, let me know.
- Flex247A 6y agoThank you for posting this! As a budding C programmer, this will be very useful to me. Are there any more tools/tricks I should know to debug programs? Pitfalls I should avoid?
- dekken_ 6y agoyou could try use sanitizers to find problems so you don't need to debug so much ref: https://clang.llvm.org/docs/AddressSanitizer.html https://clang.llvm.org/docs/AddressSanitizer.html
- f00zz 6y agoValgrind is pretty useful: https://valgrind.org/ https://valgrind.org/
- bear8642 6y agoIndeed! Very useful for shell wrote as part of OS course last year. Compiling with debug (-g in gcc) lets valgrind report which function's allocating lost memory
- passivate 6y agoHonestly, the best tool is your own brain. The easiest way to develop that tool is through experience. In a previous role, I did a ton of system level coding in C++ [~10yrs], and no tool can match your own intuition and thinking about the problem at hand. The best way to develop this is to be mindful of what is happening as the code is executing, in terms of the CPU, the memory and any peripherals or other devices you may be using. So for e.g, be very very comfortable tracing through code in an end-end fashion and not just when it crashes. This way the code paths, intermediate variable values, and everything else is ingrained within you. Your brain will very quickly spot situations where something looks off. As you do this a bunch it will get easier and easier, and you will slowly develop an intuition for these things. If you're focusing on crash dumps and postmortems, you also need to be aware of what raw memory looks like at the bit level, understand the basics of the CPU architecture, etc. Again, the more you do this the easier it will get. With most things the big-picture view is always dark and murky when you start out. The knowledge and experience you gain will be the light that you shine to gain a clearer view. Good luck.
- steerablesafe 6y agorr is insanely useful. There are some debugging patterns that commonly come up for me when I use it. 1. Investigate a crash: run program until it crashes/shows error dialog/other undesired state. If there was an exception involved then set up a catchpoint (`catch throw`) then reverse-continue. You are now in the vicinity of the crash, you can now debug it going backwards in time. 2. Some variable has garbage state, you want to know what caused it. Set up a hardware watchpoint to the variable, then reverse-continue. You can then see the line that wrote to the variable the last time. Sometimes it is a line that is writing to an entirely different variable, then you know you have variable lifetime issues. 3. The executable you debug is timing sensitive (network timeouts for example). Just record a run with rr then debug it later, don't worry about breakpoints in the middle of timing sensitive code. And I'm pretty sure that there are a ton of other use cases.
- ynik 6y agoWhat I like most: you can debug data flow! To see where a value came from, set a memory-breakpoint and reverse-continue. This is not just useful when you suspect memory corruption (your point 2). I use it more often with a large legacy C code base. It outputs a complex data structure. There's often several code paths computing values for a particular field. For someone unfamiliar with the code base, it can be tricky to tell which code path was responsible for computing the value in a particular instance of the struct (where I see incorrect output). By reverse debugging with memory breakpoints, I can trace the data flow backwards until I find where the computation went wrong. Note: Since I'm developing on Windows, I'm not using rr for that; but Microsoft's Time-Travel Debugger (WinDbg Preview).
- db48x 6y agoThese are excellent examples. I’ll throw one of my stories into the pot: One time I had a client that was using a large and complicated C++ library to build a larger and yet more complicateder C++ program. It had an intermittent crash that they just couldn’t track down. The stack traces they showed me were always deep in the bowels of the C++ library, in places no crash should ever be. The library was open source and widely used; I _knew_ that it didn’t just crash in any of those places. I recorded it in rr a few times until I captured the crash. I set a watchpoint on the memory address of the crash, and ran the program backwards from the crash. (Memory addresses are stable in the replay, making this kind of thing super easy.) A few seconds later it stopped on the line that was responsible; turns out they were accidentally overwriting the method pointer to the deconstructor in the vtable of this one class. Eventually one of those objects would go out of scope on some random thread, and need to be deconstructed. BOOM. I checked a few of the recordings where it didn’t crash, and in those cases it had just overwritten something much less obvious, like a string containing HTML that it had downloaded. My advice? First, always make sure your pointers are initialized correctly before you go dereferencing them. You won’t like what happens otherwise. Second, learn rr. With rr, and a little brain–sweat, you can do in a couple of hours what an entire team of engineers couldn’t do in months. Admittedly they had a lot of things on their plate, but it’s definitely a superpower. I think this might have been the first time I ever ran rr, too; some of that time was getting rr built and installed. I think I even had to ask a question on the IRC channel, because there was a confusing error message. Third, learn Rust. This was a few years back, when Rust was little more than a crazy idea. If the program had been written in Rust, there would have been a lot fewer landmines for them to step on. That’s a rather different kind of superpower. Incidentally, you can combine these superpowers… but that’s a story for another day.
- kubb 6y agoDoes anyone know how rr implements the rewind capability? It seems that it has to keep a snapshot of the entire program memory at every step, or are there any techniques it uses to optimize this in specific cases?
- ynik 6y agoIt records the results of syscalls. In replay mode, the program is executed again, but rr will replay the recorded syscall effects instead of doing actual syscalls. As a result, the program will deterministically behave the same way it did the first time around. There's a bunch of more magic to make other things (incl. multi-threading) behave deterministically. rr uses hardware performance counters (telling the CPU things like "trigger an interrupt after executing N conditional branches") to seek to a particular time in the program's execution. So as a first approximation, "step back" can be implemented as "re-run from beginning, programming the perf counter break after N-1 steps". This is where the trouble with certain CPU models comes into play -- these perf counters aren't always perfectly reliable, and rr needs different workarounds for each CPU generation. And I guess rr has some full program snapshots as well at regular intervals so that the replay doesn't need to start all over from scratch on every "step back".
- DSingularity 6y agoIf you take a snapshot initially then all you need is the results of anything non-deterministic - ie anything which cannot be produced by program instructions themselves. For those you can pretty much get them all by logging the results of system calls. All that is left is things like RDTSC, RDRND, and thread interleaving of a multi threaded application.
- sanxiyn 6y agoThis works in theory, but if your program has been running for a minute, going back a second by re-executing from the beginning will take a minute. rr snapshots every second so that going back takes similar time to going forward, rounded to seconds, adjusted for overhead.
- dataflow 6y agoThis sounds very similar to Visual Studio's "historical debugger" (Enterprise feature) [1]... has anyone tried both? How do they compare? Either feature-wise or usability-wise. [1] https://docs.microsoft.com/en-us/visualstudio/debugger/historical-debugging?view=vs-2019 https://docs.microsoft.com/en-us/visualstudio/debugger/histo...
- roca 6y agoIntellitrace doesn't capture the full program history. rr reproduces everything in userspace memory (and registers) at any point in time. TTD is the closest thing Microsoft has to rr.
- dataflow 6y agoOh interesting, thanks! I'd heard of TTD but hadn't realized it's different from historical debugging at all. Would you have any thoughts on TTD vs. rr?
- roca 6y agoTTD uses code instrumentation so typically has a lot higher overhead than rr on single-threaded code. On the other hand, it can handle shared memory and use multicore, so for parallel programs on big machines TTD could be faster. TTD has some debugging features rr+gdb doesn't have, like some cool query-based debugging features. But it doesn't let you run application functions (e.g. prettyprinters) at debug time like rr does. Pernosco consumes rr recordings and blows TTD away IMHO :-).
- dataflow 6y agoAhh I see! Awesome, thanks!
- cesarb 6y ago> rr does not monitor processes outside the children of what it is recording, and misses any communication through shared memory to an outside process. The current tendency seems to be moving towards communication through ring buffers in shared memory, not only with outside processes but also with the kernel (io_uring), so unless they find a workaround, rr is going to become less useful as time goes on.
- Asooka 6y agoFor sure, the fact it cannot support shared memory is one of the biggest drawbacks. Maybe catching shared memory setup and marking the pages as not readable, then catching read accesses would work.
- tobyld 6y agoUDB [1] from Undo has support for recording processes which use shared memory with outside processes. Although, unlike rr it isn't open source. [1] https://undo.io/solutions/products/udb/ https://undo.io/solutions/products/udb/
- roca 6y agoI think we can support io_uring with some work. Basically we need to have rr manage the real ring buffers and give the application fake ring buffers, then have rr copy data between the fake and real ring buffers at defined times.
- souprock 6y agoI think it's easier to put the entire system into the reverse debugger. The downside is some complexity seeing into the system to view individual processes. Here is one such tool: https://www.raytheonintelligenceandspace.com/capabilities/products/dejavm https://www.raytheonintelligenceandspace.com/capabilities/pr... An extra benefit is that you can reverse debug the kernel.
- roca 6y agoThat is cool stuff but having to run your application in a VM is annoying for users (it was for VMWare). Also a problem in AWS where you don't have nested virtualization.
- neves 6y agorr is magic. Unfortunately I have no use for it since I'm not programming C/C++ anymore. Do you have good experiences with useful (non prototype) reverse debuggers for Python, or Java or Javascript? I would like not just links to projects that you can google for, but real world experience telling if something is useful or not.
- kevincox 6y agorr should work on Python, but you need to make sure that you have all of your GDB config set up to properly interact with python objects so that you aren't dealing with the raw CPython too much. GDB plugins are very powerful and if you find the right ones you can have an excellent experience in just about any language. (I don't know the current state of Python but a number of years ago it was pretty acceptable). However it probably will never be quite as convenient as IPDB on pure-python code. So in the past I have switched back and forth, using IPDB for the simpler stuff and breaking out rr+GDB when I needed advanced features (like time travel)
- iudqnolq 6y agoThis is just a link, sorry, but undo.io has "call for a quote" reverse debugging for Java. I've used pernos.co (~$4 per execution) a little on Rust. They claim to also support JavaScript running on v8, but are geared toward debugging javascript that interacts with native code. They don't have good support for debugging interacting with the dom.
- arronax 6y agoOur developers started using RR and had some good results. Not a silver bullet, but a nice addition to the toolbox. Some writeup here https://www.percona.com/blog/2021/04/12/replay-the-execution-of-mysql-with-rr-record-and-replay/ https://www.percona.com/blog/2021/04/12/replay-the-execution...
- iudqnolq 6y agoIf you're using Rust check out the light wrapper I wrote. It handles the minor tedium of running rr on the appropriate binary when you type something like `cargo rr test`. https://crates.io/crates/cargo-rr https://crates.io/crates/cargo-rr (I'm currently refactoring it to be a bit less hacky, you will run into issues with edge cases such as workspaces and tests in your main.rs)
- peter_d_sherman 6y ago>"rr records trace information about the execution of an application. This information allows you to repeatedly replay a particular recording of a failure and examine it in the GNU Debugger (GDB) to better investigate the cause. In addition to replaying the trace, rr lets you run the program in reverse, in essence allowing you “rewind the tape” to see what happened earlier in the execution of the program." PDS: An awesome set of ideas! A sorely lacking feature in most debuggers... Actually, now that I think about it... there's this notion of "machine state" associated with any point in time of any given program run... "Machine State" (of a program run, at a point in time) -- not only includes the set of processor registers at that point in time and the memory used by the program -- but also such things as OS entities, like files, sockets, critical sections, shared memory, etc., that are open and active (and their state!) -- at given point in time, during a program's run... It may be somewhat infeasible for today's Operating Systems to do this, but in the future, it might be possible to take a "snapshot" of a program (CPU, registers, memory, open OS objects, eveything) at a given point in time -- then be able to reload that snapshot into a debugger and move forward in time from there... On the one hand, OS'es have had for the longest time, the ability to write "Crash Dumps" of failed programs to disk, and debuggers can load them up and see machine state at the point of the crash -- but you typically can't reverse the program a series of steps from that point, nor can you re-create the now-closed OS objects that the program was holding on to -- nor reinstate their exact state at a given time, which would be necessary to run the point-in-time program snapshots as if nothing happened... But future OS's -- will/should -- be able to (if requested) -- store state information about a program's open OS objects at a given point in time (perhaps that should be a series of new OS API calls? Like perhaps SnapshotAllOSObjectsUsedByThisProgramNow(); and RestoreAllOSObjectsUsedByThisProgramAtTime(Time);...?). But that's just speculative at this point... Also -- wouldn't Plan 9 and other OS'es like it, ones that have the ability to move programs/processes from one physical machine to another while the program is running, have the preexisting infrastructure to snapshot a program's open OS objects at points in time, with very little code modification? It seems like they might... Anyway, rr is a great and awesome step forward for debugging!
- tracnar 6y agoAFAIK criu does something like what you describe, similarly to taking VM snapshots but for processes or containers: https://criu.org/Main_Page https://criu.org/Main_Page
- Teknoman117 6y agoDoes rr work on AMD (Zen 1/2/3) CPUs yet? I remember that being a major issue the last time I looked at it. edit - only very recently (sep. 2020), and it's still experimental: https://github.com/rr-debugger/rr/wiki/Zen https://github.com/rr-debugger/rr/wiki/Zen
- ncmncm 6y agoGDB is supposed to have reverse execution capability now. How does its facility compare to rr? Are they complementary, or one a pale shadow of the other, or something else?
- sanxiyn 6y agorr is approximately infinitely faster than GDB reverse debugging. rr's innovation is low overhead reverse debugging. rr's typical overhead is 20% -- 2x overhead is considered unacceptable for rr. You could reverse debug coreutils with GDB fine, but it won't work for Firefox. That's why rr was written.
- ncmncm 6y agoSo, pale shadow. Thank you, this is what I needed to know. Time spent learning rr looks like a good investment.
- khuey 6y agoYou shouldn't need to learn much. rr uses gdb's existing reverse-debugging UI commands, it just swaps out the backend implementation with a far more powerful version.