If you have a race condition in a multi threaded program, a simple printf is enough to screw the timings and make the bug disappear, it's not at all better than

If you have a race condition in a multi threaded program, a simple printf is enough to screw the timings and make the bug disappear, it's not at all better than breakpoints and stepping in a debugger. In my experience, the only way to produce safe multi threaded code is to isolate the threading parts and synchronization and stress test them early on. At this point I use a combination of prints, random delays and simple asserts on invariants that the code should hold. Most issues are reproduced within about 10 seconds of 100% CPU core utilization. Some nasty corner cases may require minutes of grinding away. Multi threaded debugging is an unsolved problem. There are no universal solutions to this problem. Conventional tools often fail so badly that it is best to write and test your multi threading / synchronization code in isolation, and then applying it to the actual workload which is tested in a single thread. Another option is trying to build a formally verified model ahead of time (e.g. using Spin - www.spinroot.com) and then write the actual program after the model has been verified.