I write code that is executed in rad environments. The fist thing I learned is that any plan that I come up with to help mitigate risks always ends up being pr

I write code that is executed in rad environments. The fist thing I learned is that any plan that I come up with to help mitigate risks always ends up being proven less effective than I initially thought. The idea that the hardware can flip bits or latch up behind your back is rarely a consideration for most software design work, so it really requires an entire new way of thinking. In the end, it's all about reducing probabilities, but sometimes your mitigation techniques can be borderline useless. This could be because of my relative lack of experience though. I'm sure there are people who have been doing this a while and have converged on a design process that works well. Couple this with the fact that code in the failure path very rarely gets actually tested with _real_ failures during design, and you end up with situations where your entire test/precaution/recovery technique is rendered useless by a single oversight. It kind of reminds me of how some bootloaders perform a DRAM test during power up. The only problem is that these tests are usually executing after the bootloader is already running from RAM itself. Sure, it's better than nothing and may help to find some obscure errors like a bad address line pin or something, but the issue here should be obvious - if there was a real problem with the DRAM, it's likely you wouldn't even get to that test, so now what? Another example: suppose you have a PI bus connecting to an FPGA, and you are relying on the PI_done line to pace transfers. Now, a common test to verify the bus is working would be to read some magic word from the FPGA or maybe use an XOR test register, but what happens when there's an actual failure like a bad connection to the bus or bad power to the FPGA? Well, the CPU throws an exception and resets because the timeout for the PI_done expires. This all happens before your test has a change to even print anything to the console. I run into these kinds of issues all the time when designing software routines to self-test the hardware on our devices. Everything is all good when the hardware works, but as soon as hardware starts actually failing, all of those diagnostic tests are suddenly useless. I've gotten kind of off topic here, but the thought process is what I'm getting at. It usually goes a little something like this: Possible solution: Always have a backup OS image. Caveat: There's only one copy of the bootloader, so if that area of the flash gets screwed, we're done. Possible solution: Scrub the contents of RAM to check for errors. Caveat: The scrub routine is running from RAM. Possible solution: Use a watchdog timer to force a reset if the CPU runs away. Caveat: What if the watchdog timer latches up and the CPU hard locks? (Unlikely, I know). In the end, you can do a lot in SW to make a system more rubust, but the ultimate solution is probably on the mechanical side. You're better off spending your efforts modifying the enclosure and submerging the thing under water than you are spending hours designing in software techniques to try to recover from HW errors. TLDR: Most people agree that software techniques for mitigating risks in a rad environment are all about reducing probabilities, but I feel that such techniques have a good chance of being less effective in practice than people may think. I don't know what the point of this comment really was though, other than being negative. :)