I worked on a program that ran large batch jobs, sometimes taking more than twenty-four hours. This was actually spectacular performance, since we used custom

I worked on a program that ran large batch jobs, sometimes taking more than twenty-four hours. This was actually spectacular performance, since we used custom hardware to do most of the computation. I wrote the code that interfaced with the hardware. When the code timed out trying to talk to the hardware, the only sensible thing to do was report the error and abort the program. Unfortunately, this seemed to happen quite often. Jobs would abort randomly, after about eight hours, sometimes much less and sometimes not at all. Overheating was the obvious first suspect, so that was investigated and ruled out. The hardware was running cool and was in perfect working order. The customer started splitting jobs into shorter batches and combining the results by hand. We wrote code to help them automate this workaround. But batches were still randomly aborting. And I couldn't replicate the bug, despite having identical hardware to the customer. Something in the customer's environment was essential. Eventually, somebody at the customer figured out the problem. My boss called me up and said, "The customer suspects your code is not time-travel compliant." It was true. My code assumed that time always goes forwards. If time ever went backwards while my code was waiting for hardware, it would immediately time out and abort the batch. And our customer encountered a bug where time did appear to go backwards occasionally. I was too stressed out over other tasks to ask for details of the bug. I just sent them a fix and breathed a sigh of relief when they accepted it. A Google search now reveals that there was an issue with time going backwards under Xen on dual-core Opterons, which is what the target platform was. They never told us they were using Xen. Maybe that's why they were much nicer to us after the problem was diagnosed!