The definitely most difficult bug I have ever worked on was a device that would have its flash memory erased from time to time. With couple of million of those

The definitely most difficult bug I have ever worked on was a device that would have its flash memory erased from time to time. With couple of million of those in the field only about one hundred would be affected, every day meaning the frequency was enough to cause huge financial issue for the company (bricked devices to be destroyed and replaced, unhappy customers, PR nightmare) yet not frequently enough to be observed in lab environment. We had to set up couple dozen of these doing operations 24/7 just to be able to note single occurrence of the problem maybe once a week or two. The device was built in a way that made it impossible to observe physical lines between CPU and flash chip. This was intended as a security feature but caused the whole debug procedure to be extremely difficult. The start of the problem could not be linked to any particular change in the software. In the end we have tracked the problem to a single decision made a year before the problem started showing up. The decision was to use UNLOCK BYPASS feature. UNLOCK is a special command sent to flash chip that kind of validates that the message was not garbled. UNLOCK BYPASS turns off the use of this feature. This was done to improve write performance as the UNLOCK command slows down writes. With the UNLOCK turned off, flash chip is more likely to interpret noise on its lines as a valid command. This change did not immediately cause problems. Only later, another chip on the board started being used in a bit different way which caused much more noise to be generated. With the higher noise floor the flash chip would occasionally execute a command that was not sent from the CPU. The irony here is that the rules for the construction of the device required that some of the signal lines are sandwiched in the inner layers of PCB between two layers of signal lines, to prevent easy access to the inner lines. With 4 layer PCB this prevented large ground planes that would help control induced noise that was causing the issue. Whenever we tried to replicate the problem on a set of development devices (devices with the communication lines available for probing), the problem would not show up due to a different layout of the PCB. In the end debugging the problem took about half a year.