3 ms·
This is incredible! For those that have used it, how useful it is for debugging multithreading heisenbugs? Can I let a process run under rr for days, wait unti
by sph 4y ago
This is incredible!
For those that have used it, how useful it is for debugging multithreading heisenbugs? Can I let a process run under rr for days, wait until it crashes to due a heisenbug, and replay the trace without rr having to go through days of recording? i.e. is it possible to fast forward the trace, somehow?
(I nerd sniped myself a bit here, wondering how fast forwarding could be implemented. I think it might be achievable with periodic process memory snapshots and incremental traces.)
- fanf2 4y agoI have not used rr heavily, but I did use it to help find a multithreading heisenbug in BIND https://kb.isc.org/docs/aa-01606 https://kb.isc.org/docs/aa-01606 I could not reproduce the bug in less than an hour of run time, which meant that analysing the bug in gdb required an hour for it to run forward to the crash point, after which it was possible to skip back and forth.
- pm215 4y agorr numbers each 'event' it records, and you can pass an event number to the gdb 'run' command to tell it to start from that event. Recent 'rr' now also supports the -e option to replay meaning 'start the debug session pointing at the last recorded event, whatever that was'. Details in the usage page: https://github.com/rr-debugger/rr/wiki/Usage https://github.com/rr-debugger/rr/wiki/Usage AIUI you get 'start at an event' basically for free, because 'step backward' is implemented as 'start at the preceding event and then step forward by N', so events are frequent in the trace and the machinery to get to that point without running all the way from the start of the debug session exists anyway. There's some stuff on the website about how this is all implemented, I think.
- roca 4y agoYou probably could record a process running for days but it would also take days to replay to the end, which would not be much fun. We don't create checkpoints during the recording. You'd be better off restarting the recording periodically. Also, rr has a "chaos mode" which randomizes scheduling and often makes threading bugs easier to reproduce. https://robert.ocallahan.org/2016/02/introducing-rr-chaos-mode.html https://robert.ocallahan.org/2016/02/introducing-rr-chaos-mo...
- pm215 4y agoHmm, when I asked for 'replay -e' I thought it would be faster than 'type run and wait' -- is it not?
- roca 4y agoNo.
- Agingcoder 4y agoChaos mode works quite well in my experience, definitely worth trying if you don't want to wait for days. I had a heisenbug which would appear once a week, and that I couldn't trigger on my workstation. Chaos mode did the trick.
- anarazel 4y agoI've found that the scheduling quanta with chaos mode are too high to hit concurrency issues in a reasonable amount of time. And IIUC --num-cpu-ticks is not randomized. So if something happens below that tick quantum it's hard to hit. I wonder if a) rr could randomize the cpu ticks as well, at least in chaos mode, b) profiled code could somehow hint to rr that a certain instruction would be an "interesting" scheduling point.
- roca 4y agoChaos mode varies the scale of the tick quantum to try to catch stuff like that. It doesn't always work, especially if the window of vulnerability to the bug is incredibly small (e.g. a few instructions).
- amelius 4y agoNext question: can I do the same with a multi-node system?
- IshKebab 4y agoI haven't used it, but it might be quite useful. It does force code to run on a single core, so you won't get truly concurrent execution which I guess might hide some multithreading bugs. On the other hand it does come with "chaos mode" which is basically thread schedule fuzzing. You can "fast forward" the trace as you imagine. rr works by recording all non-deterministic input and output to the program so it can start from the core dump and step backwards. As I understand it anyway; I've never actually used it - the one time I really wanted something like rr was on a Mac.
- sfink 4y ago> You can "fast forward" the trace as you imagine. rr works by recording all non-deterministic input and output to the program so it can start from the core dump and step backwards. Not exactly. rr can't magically inflate a core dump into all the open file descriptors and other state accumulated during a process's execution. It needs to run from the beginning. So starting from the beginning, you can let it run to any arbitrary point. (And there are ways of knowing useful points, eg if you record with -M it will print out event counts with anything written to stdout/stderr, so you can quickly run with -g to start debugging at the point that message was emitted.) But it does still need to run from the beginning. And you're recording a whole process tree, you need to start from the initial process and let it go forward to your requested point in time. In practice, I usually use it by starting a replay, continuing forward to a crash (or a breakpoint at some line if it didn't crash), and only then starting to pay attention to what's going on. It's a simple, muscle-memory process to get to that point, and if it was a long recording you kind of start it up and wait until it's ready. (Which will take roughly as long as the initial run took to get to the same point. A little slower because of the overhead, a little faster because it doesn't actually have to wait for I/O, averaging out to a mostly unnoticeable amount slower.) I always have to mention: one of my favorite things about rr is something that doesn't even require all the sophisticated machinery. I often want to debug a single process within a whole process tree, and with most things there aren't --debugger flags (or they're broken). With rr, I can just record the whole tree, then pick out the process I care about after the fact. It's a small thing, but it saves me from my usual hairball of wrapper scripts. Random example: when debugging a gcc plugin, I record a call to gcc, but the actual compile I care about is done by a forked cc1plus process.
- eloff 4y agoOnly somewhat useful as it runs singlethreaded, last I checked. That will prevent some forms of heisenbugs from happening.
- patrec 4y ago> For those that have used it, how useful it is for debugging multithreading heisenbugs? Not that useful, because as it says on the page (under "Limitations"): > emulates a single-core machine.
- semiquaver 4y agoMultithreading does not require multiple processors, it has existed since long before SMP was a thing.
- db48x 4y agoBut rr does squash all your threads into a single virtual cpu core. It context–switches between them, but ultimately only one of them is running at a time. This makes it hard to capture some kinds of bugs. To compensate it also has a chaos mode that randomly stops switching between the threads fairly (starving some and giving others more than their fair share) in the hopes of triggering those same bugs. For most uses rr is a major win, but for race conditions it sometimes doesn’t help.
- patrec 4y agoAnd a lot of this multi-threaded software that previously worked fine on single-core machines, required urgent fixes as multi-core CPUs became more common.