The toughest bug I encountered took a month to find. The program (a Unix daemon) was single process, with no threads. The program would crash (seg fault) afte

The toughest bug I encountered took a month to find. The program (a Unix daemon) was single process, with no threads. The program would crash (seg fault) after a few hours (some system) or a few days (other systems). There did not seem to be any rhyme or reason for the crash. And running it under a debugger didn't help, as it kept crashing at different points. I finally found the issue, but only after close study of the code. I realized that one signal handlers was different than all the others, and that signal handler was calling some non-thread safe code. But because I wasn't using threads, the thought of a critical section of code being interrupted never occurred to me. Now I spend my work time finding bugs in a distributed system. Fun times. (latest possible bug---one component stoped logging but only after what I would call "ludicrous load" (say, 1000+ load average)---it was still running, but apparently got wedged on a mutex)