I have seen a bug where the device came from clients completely wiped. Mind, that this device has been distributed to millions of people. We would see couple hu

I have seen a bug where the device came from clients completely wiped. Mind, that this device has been distributed to millions of people. We would see couple hundred of these came every day, with completely clean flash. The interesting part was that nowhere in the code was any sequence that could clear the flash. After very lengthy and expensive bug hunting we have determined the cause. The flash protocol had an unlock sequence that required a bunch of magic operations before the write could be performed. Somebody, at some point decided to use "unlock bypass" function to disable this magic sequence, ie. allow writes without magic bypass sequence. This was done to improve write performance when the software was upgraded in the field (with customers potentially impatiently waiting for it to complete). Unbeknownst, there was a coupling between a remote part of the circuit and the lines between MCU and the flash that caused noise to be introduced to the communication. The unlock sequence was the only thing that prevented noise to be interpreted as valid operation. The probability was still very low of this happening, but with millions of devices in the field it was enough to get about 100 failures a day.