> Shoddy hardware here, not shoddy software. Mm-hmm. Story time! Let me tell you about my favorite chip bug that was resolved with firmware. :-) My first ass

> Shoddy hardware here, not shoddy software. Mm-hmm. Story time! Let me tell you about my favorite chip bug that was resolved with firmware. :-) My first assignments out of college were on firmware for complex enterprise-class servers. I wrote power/thermal/clock control code, and was quite proud of doing integration tests like overclocking or underclocking a 256-core computer, and having it not crash. :-) We got a bug, though, that the system would crash down on first boot when it had been left unpowered in a thermal chamber overnight at the minimum ambient operating temperature. This simulated it sitting in a loading dock or warehouse before getting "rolled into the lab" for install. We all assumed condensation was a problem, but were able to disprove it by controlling the humidity in the chamber. Someone thought that some capacitors or the power supply weren't producing a clean signal, but we also disproved that. Interestingly, the system would boot up just fine if the engineers immediately shut it down and restarted it. If we tried to repeat the test on the same day, it'd pass. It only failed if we waited overnight. Looking closer at the crash, the crash reason was due to a thermal limit being tripped. The hardware was detecting an overtemp on one of the main processors. This was strange, because the test was a "cold start" test, and it wasn't triggering any warnings. The problem processor also changed each day/night we repeated the test. We actually were forced to wait ~8 hours between test windows because timing was inexplicably tied to reproducing the fail. I checked everything, confirmed that we'd programmed the correct warning and limit values, and while talking it out with the hardware engineer and explaining my logic, my hardware colleague who wrote the VHDL realized what was going on. The thermal sensor we were using had a margin of error that turned out to be +/- 10 degrees C. When the system was left at 5 degrees C overnight and started, the sensor's first readings might be 5 C +/- 10... and for an 8-bit unsigned register value, would result in an underflow, immediately tripping the hardware checker which was implemented as an 8-bit unsigned comparison. Rather than change the VHDL, we turned off the built-in thermal protection which thankfully had its own dedicated mode bits, and implemented a firmware protection scheme where my code (which could mask off the high order bit) would check the temp and set warning/error bits. If the high order two bits were set, we'd ignore the sensor value and just log an informational event. If the sensor didn't report an in-range value after a certain amount of time, we'd report that the sensor was bad, and fall back to a "failsafe" operating mode, iirc. It wasn't as fast as the hardware, so we had to add more margin, but it saved having to re-spin the processor. That code's still running in a bunch of machines out in the world today. :-) I should also say that all of the above discusses registers and logic that are outside of the architected state of the computer that normal software has no access to.