Software that can kill people if it malfunctions, has to be reliable. A monstrously complex OS that is prone to changing its behaviour through auto-updates isn

Software that can kill people if it malfunctions, has to be reliable. A monstrously complex OS that is prone to changing its behaviour through auto-updates isn't even remotely suitable. However, what works is if the complex system is monitored by a simple, auditable sanity check. In the case of the Therac-25, nothing bad would have happened if the simple mechanical interlocks ("if A isn't in place, don't turn on B") hadn't been optimized away due to misplaced faith in the software's reliability. Similarly, a trained medical person would look at the off-by-10x dosage number and say this isn't right, let's just check that. And a simple can't fail software check could probably flag this sort of thing as well.