4 ms·

I remember hearing about this case in college. My professor's point was don't trust software alone. Always include hardware safe guards when it comes to safety.

cr0sh 7y ago
Since I started really playing with hardware/software integration (hobby level) a couple of decades ago, and also knowing what I learned from various efforts I've been involved in at employers to "bandaid patch" in security (ie, logins, passwords, access control, etc) - and how it never works out, I've since advocated for the implementation of security controls first, whether implementing a hardware and/or software solution, and one of those controls, if hardware is involved, should be a physical "all-stop" cut-the-power control, manifestly separate from any "software" (whether implemented in hardware logic or otherwise). Design the security/safety system up front, for the task of the system being designed, for all potential failure modes - and then implement the first, and test the hell out of it, and document it thoroughly. Unfortunately, this tends to get shot down, until it is needed, and then it gets bandaid-patched in place, because usually at the point it is needed, a whole refactor of the code base is usually called for, and we can't have that, so "just fix it" becomes the call of duty. And it never works out properly. Things get missed, security is effectively out the window, until something happens, with the refrain of "why did this occur" and of course nobody wants to take the blame for "I told you so". Unfortunately, even implementing such a security system first will only get you so far, and it is still possible that something might be missed, and potentially be the cause of a safety critical issue. But hopefully, most if not all of those extreme events can be caught first in the system, by some part of it, and what's left will be relatively minor.
- Gibbon1 7y ago
  When I was doing industrial controls in the 80's, I read a paper on the topic. The most important point was safety is a primary design goal. You can't add it in later. Because security and safety concerns have pervasive effects on design decisions.
IfOnlyYouKnew 7y ago
> Always include hardware safe guards when it comes to safety. What does that actually mean? Maybe in the Therac-25 case the maximum dosage could somehow have been limited by a hardware dial in addition to software. But what form would hardware safeguards take for the millions of decisions software makes every second on a starting rocket? And considering hardware has far higher failure rates than no-moving-parts software, wouldn't any system that lets hardware dominate software decrease safety?
- ClassyJacket 7y ago
  I don't know exactly but the previous generation model had the hardware interlock that would have prevented this problem so it is possible. From Wikipedia: "One, when the operator incorrectly selected X-ray mode before quickly changing to electron mode, which allowed the electron beam to be set for X-ray mode without the X-ray target being in place. A second fault allowed the electron beam to activate during field-light mode, during which no beam scanner was active or target was in place. Previous models had hardware interlocks to prevent such faults, but the Therac-25 had removed them, depending instead on software checks for safety."
- ridgeguy 7y ago
  Perhaps just a dose integrator with an alarm limit? There's probably a total dose above which you could likely say "this is a mistake". A dose integrator would be a detector + an integrator independent of the therapeutic delivery software. If it goes over limit, it shuts the system down.

cr0sh 7y ago

Since I started really playing with hardware/software integration (hobby level) a couple of decades ago, and also knowing what I learned from various efforts I've been involved in at employers to "bandaid patch" in security (ie, logins, passwords, access control, etc) - and how it never works out, I've since advocated for the implementation of security controls first, whether implementing a hardware and/or software solution, and one of those controls, if hardware is involved, should be a physical "all-stop" cut-the-power control, manifestly separate from any "software" (whether implemented in hardware logic or otherwise). Design the security/safety system up front, for the task of the system being designed, for all potential failure modes - and then implement the first, and test the hell out of it, and document it thoroughly. Unfortunately, this tends to get shot down, until it is needed, and then it gets bandaid-patched in place, because usually at the point it is needed, a whole refactor of the code base is usually called for, and we can't have that, so "just fix it" becomes the call of duty. And it never works out properly. Things get missed, security is effectively out the window, until something happens, with the refrain of "why did this occur" and of course nobody wants to take the blame for "I told you so". Unfortunately, even implementing such a security system first will only get you so far, and it is still possible that something might be missed, and potentially be the cause of a safety critical issue. But hopefully, most if not all of those extreme events can be caught first in the system, by some part of it, and what's left will be relatively minor.

Gibbon1 7y ago
When I was doing industrial controls in the 80's, I read a paper on the topic. The most important point was safety is a primary design goal. You can't add it in later. Because security and safety concerns have pervasive effects on design decisions.

IfOnlyYouKnew 7y ago

> Always include hardware safe guards when it comes to safety. What does that actually mean? Maybe in the Therac-25 case the maximum dosage could somehow have been limited by a hardware dial in addition to software. But what form would hardware safeguards take for the millions of decisions software makes every second on a starting rocket? And considering hardware has far higher failure rates than no-moving-parts software, wouldn't any system that lets hardware dominate software decrease safety?

ClassyJacket 7y ago
I don't know exactly but the previous generation model had the hardware interlock that would have prevented this problem so it is possible. From Wikipedia: "One, when the operator incorrectly selected X-ray mode before quickly changing to electron mode, which allowed the electron beam to be set for X-ray mode without the X-ray target being in place. A second fault allowed the electron beam to activate during field-light mode, during which no beam scanner was active or target was in place. Previous models had hardware interlocks to prevent such faults, but the Therac-25 had removed them, depending instead on software checks for safety."
ridgeguy 7y ago
Perhaps just a dose integrator with an alarm limit? There's probably a total dose above which you could likely say "this is a mistake". A dose integrator would be a detector + an integrator independent of the therapeutic delivery software. If it goes over limit, it shuts the system down.