But you had a theory you could test. The hardest bug I had to debug (over a month of constant work) was very difficult to reproduce on demand. The program (a

But you had a theory you could test. The hardest bug I had to debug (over a month of constant work) was very difficult to reproduce on demand. The program (a single threaded server) would crash after running a few hours in production. I was able to get it to crash on the development server only if I let it run handling requests for a few days. The core dumps (and yes, there were plenty of core dumps to look at) were inconsistent---each crash was in a different location. There was no reason I could find that caused the crashes, so no theory I could really test. I was able to locate the root cause, which was calling a non-async-signal safe function from a signal handler, but that came only after staring at the code and thinking hard for a long time.