3 ms·
At work we make a hardware product, and among other things, it has a small microcontroller on it. We keep several hundred instances of our product to run autom
by repiret 3y ago
At work we make a hardware product, and among other things, it has a small microcontroller on it. We keep several hundred instances of our product to run automated integration tests. At that scale, we find SRAM bit-flips one of the microcontrollers every few weeks. We have a handful of software measures that are able to stop most instances of bit-flip from having a user-visible effect.
- bcrl 3y agoThat kind of rate smells like a hardware bug. Insufficient decoupling caps maybe?
- repiret 3y agoThat seems plausible. I wasn’t deeply involved in the investigation, but the silicon vendor supports the cosmic ray hypothesis.
- bcrl 3y agoThe rate of cosmic rays impacting hardware should be extremely low, unless there's some kind of environmental factor (like being at a very high altitude). Or maybe there was a special train going by as in: https://www.jakepoz.com/debugging-behind-the-iron-curtain/ https://www.jakepoz.com/debugging-behind-the-iron-curtain/