Anecdote: Three weeks ago, I needed to debug what might have been a problem in a compiler. A small change to the packaging of some libraries had caused a test

Anecdote: Three weeks ago, I needed to debug what might have been a problem in a compiler. A small change to the packaging of some libraries had caused a test program to fail, and there was no obvious relation between the small change and the failure that was reported by the test. Experimentation showed that I could make the problem appear or vanish depending on the presence or absence of a small amount of additional read-only data to the link step of the test program. But I couldn't add any code to the test to try to isolate the root source of the failure by printing intermediate values or "hey I got here!" messages -- it was a true Heisenbug that vanished if I got too close to it. Long story short (if still possible): I ended up having to write a debugging tool that would drive the ptrace(2) API to fork and exec two binaries, let them run for a certain number of system calls, and then a certain number of breakpoint hits, and then single-step a certain number of instructions. Bisection on these three parameters allowed me to eventually find the point in execution where the PC registers of the two images first differed. Working backward from a trace found the conditional branch that first diverged, and then tracing register values helped nail down the address of a memory location whose load first yielded distinct values. (And yeah, it turned out to be a compiler bug that had caused the right value to be stored into the wrong address much earlier in the execution.) Anyway, thank goodness for ptrace(2). It's pretty easy to use and it's a very powerful tool.