4 ms·

rr is really cool, but almost every time I have decided to pull it out as one of the "big guns" it turns out that I have a concurrency bug and so rr is unable t

by rtpg 2y ago

rr is really cool, but almost every time I have decided to pull it out as one of the "big guns" it turns out that I have a concurrency bug and so rr is unable to reproduce it. Despite that, it would be very, very, very cool if some languages built rr directly into their tooling. Obviously you can always "just" use rr/gdb, but imagine if rr invocations were as easy to set up and do as pdb is in Python!

angus-g 2y ago
I actually had a concurrency bug that I was able to capture with rr: an MPI job where I only ran rr on rank 0 and managed to figure out where a different send/recv ordering was causing issues. In fact, it was also a Python model that ties in with a lot of native code generation, so quite a complex issue.
IshKebab 2y ago
Yeah same for me. Actually the time I really wanted it was on Mac and unfortunately it only works on Linux. There is Undodb which works on Mac and maybe with multithreading (not sure about that), but unfortunately it costs about $50k.
borodi 2y ago
Chaos mode is an option when invoking rr that can expose some concurrency issues. Basically it switches which thread is executing a bunch to try and simulate multiple cores executing. It has found some race conditions for me but it’s of course limited
- dzaima 2y ago
  Unfortunately that only works for large-scale races, and not, say, one instruction interleaving with another one on another thread without proper synchronization. -fsanitize=thread probably works for that though (and of course you could then combine said sanitizer with rr to some effect probably).
  - borodi 2y ago
    I havent tried Tsan with rr but msan and asan work quite well with it (it’s quite slow when doing this) but seeing the sanitizer trigger then following back what caused it to trigger is very useful.
  - bqmjjx0kac 2y ago
    Yeah, the reason it only works for these coarser race conditions is that RR only has one thread executing at a time. Chaos mode randomizes the durations of time allotted to each thread before it is preempted. This may be out of date. I believe I read it in the Extended Technical Report from 2017: https://arxiv.org/pdf/1705.05937 https://arxiv.org/pdf/1705.05937
  - roca 2y ago
    One option would be to combine chaos mode with a dynamic race detector to try to focus chaos mode on specific fine-grained races. Someone should try that as a research project. Not really the same thing as rr + TSAN. There's still the fundamental limitation that rr won't help you with weak memory orderings.
modeless 2y ago
Another thing that rr sadly doesn't support is GPUs. I'd love to use it but most of my stuff involves GPUs in some way or another.

by rtpg 2y ago

angus-g 2y ago

I actually had a concurrency bug that I was able to capture with rr: an MPI job where I only ran rr on rank 0 and managed to figure out where a different send/recv ordering was causing issues. In fact, it was also a Python model that ties in with a lot of native code generation, so quite a complex issue.

IshKebab 2y ago

Yeah same for me. Actually the time I really wanted it was on Mac and unfortunately it only works on Linux. There is Undodb which works on Mac and maybe with multithreading (not sure about that), but unfortunately it costs about $50k.

borodi 2y ago

Chaos mode is an option when invoking rr that can expose some concurrency issues. Basically it switches which thread is executing a bunch to try and simulate multiple cores executing. It has found some race conditions for me but it’s of course limited

dzaima 2y ago
Unfortunately that only works for large-scale races, and not, say, one instruction interleaving with another one on another thread without proper synchronization. -fsanitize=thread probably works for that though (and of course you could then combine said sanitizer with rr to some effect probably).
- borodi 2y ago
  I havent tried Tsan with rr but msan and asan work quite well with it (it’s quite slow when doing this) but seeing the sanitizer trigger then following back what caused it to trigger is very useful.
- bqmjjx0kac 2y ago
  Yeah, the reason it only works for these coarser race conditions is that RR only has one thread executing at a time. Chaos mode randomizes the durations of time allotted to each thread before it is preempted. This may be out of date. I believe I read it in the Extended Technical Report from 2017: https://arxiv.org/pdf/1705.05937 https://arxiv.org/pdf/1705.05937
- roca 2y ago
  One option would be to combine chaos mode with a dynamic race detector to try to focus chaos mode on specific fine-grained races. Someone should try that as a research project. Not really the same thing as rr + TSAN. There's still the fundamental limitation that rr won't help you with weak memory orderings.

modeless 2y ago

Another thing that rr sadly doesn't support is GPUs. I'd love to use it but most of my stuff involves GPUs in some way or another.