Because we had insanely good crash reporting and we root caused every crash report we got in that had more than half a dozen hits in the wild. Hardware was at

Because we had insanely good crash reporting and we root caused every crash report we got in that had more than half a dozen hits in the wild. Hardware was at fault. Turns out suppliers push the truth a bit here and there about their chips, what is supposed to be an identical part # isn't always identical, parts from different manufacturers that are supposed to be interchangeable aren't, and technical manuals aren't always the best. We had (what we thought was) rigorous memory testing in place at the factory, but under certain extreme conditions there was a 100% reproducible flip of a few bits in RAM. It was almost always the same bits, thankfully, that made it possible to track down!