12 ms·

Killed by a Machine: The Therac-25

89 points by paraknight 10y ago

dsfyu404ed 10y ago
"It’s important to note that while the software was the lynch pin in the Therac-25, it wasn’t the root cause. The entire system design was the real problem. Safety-critical loads were placed upon a computer system that was not designed to control them" They wrote code that depended on hardware controls, didn't document their reliance on the hardware controls and killed a bunch of people. DOCUMENT ALL YOUR DEPENDANCES!!!
- wyldfire 10y ago
  > They wrote code that depended on hardware controls ...similar to Toyota's electronic throttle control system design. Also, if you rely on hardware features for safety, it's still ok/good to design the software as if it didn't depend on those features whereever possible.
  - dsfyu404ed 10y ago
    It all depends on the specifics. How much complexity will software checks add? In a lot of cases the lifetime cost of spec'ing a hard failsafe may be cheaper than designing, implementing and supporting a soft one. Then there's the whole issue of software false positives/negatives vs hardware failures At some level you can no longer abstract away the situation and the system has to perform as a system. You can write soft controls all you want but if hard controls are present in the spec and something that's not likely to change there becomes a point where chasing down every little problem and checking for every possible error is no longer cost effective. Nobody cares if you write Mars rover tier code for a bulldozer and spending the resources doing is wasteful if your competitors aren't also spending the resources. Obviously you can write total crap that's outside the acceptable range toward the other end. If you're designing software to control a widget that moves and does stuff has a hard switch to prevent your widget's equivalent of an out of battery detonation you have to strike a balance between relying on it and introducing complexity from soft controls. Every aspect of the system is involved in making that determination. If you've got a hard switch in most cases you may as well write a five line timeout controlled while loop that tries to perform the action and waits and tries again if there's no feedback indicating the action was performed. As long as nobody removes that switch and the code's dependency on it is very obviously documented then that simple unsafe code is probably better than more complex code that performs a redundant (redundant because you have a hard switch) check because the more complex code has more going on.
    - throwanem 10y ago
      I mean hey, if I'm writing code for a machine that might kill somebody if a hardware interlock is removed, I'm going to perform a software check for a valid state before I take the potentially fatal action, and if that's a problem for the organization that's paying me to write that code, I'm going to resign and find work elsewhere, because that outcome strikes me as strictly preferable to having someone's death on my conscience. I suppose it's arguable either way, though.
- 0xdeadbeefbabe 10y ago
  > DOCUMENT ALL YOUR DEPENDANCES!!! Still doesn't make a difference if there is no place for the documents to live.
kqr 10y ago
Are there any statistics on the safety/reliability of software controlled vs. hardware controlled devices? I work in software engineering, so I'm exposed daily to broken software controls, and I'm gradually becoming more of a "grumpy old man" longing for the good old days when machines (be it cameras, cars, watches, medical equipment or anything else, really) could be "debugged" by following levers, wires, physical stops and hoses. I feel much more comfortable with that than with complex computer systems. I would like to know if my fears can be founded in actual truths, or if I'm just riling myself up over nothing.
- HeyLaughingBoy 10y ago
  There are but I don't know where to point you :-( I spent over 10 years writing control software for medical devices. In classifying the criticality of a device, the FDA from the beginning put any device with software into a higher category. They have since refined their classifications, so you probably won't find that guidance on their site anymore, but 15 years ago it was pretty eye opening to me that two devices of the same type, one with and one without software, could be automatically in different safety categories. So, no, it's not just you :-)
  - jes 10y ago
    I don't remember the details at this point, but somewhere in the IEC 62304 regulation, there is language that suggests that you have to assume 100% failure for software components. I think it's in regard to risk controls, but I'm not certain of that. Edit: Just looked it up: "If the HAZARD could arise from the failure of the SOFTWARE SYSTEM to behave as specified, the probability of such failure shall be assumed to be 100%"
    - HeyLaughingBoy 10y ago
      Yes, this is to avoid the problem many an FMEA (Failure Modes Effects Analysis) meeting agonizes over: "it's possible for the software to fail in this way, but how do we quantify the probability of it happening so we can calculate the risk factor?" By assuming that if it can happen, then it will" you remove any handwaving from the equation.
platz 10y ago
These stories are horrific.
- jacquesm 10y ago
  Yes. They also should be required study material for anybody that writes software interacting with the real world. Humans are fragile. I wrote a cad/cam system for a lathe/mill combo, in all the years that that system was out there we found one bug that made it out past my desk and that only happened because some idiot decided to demo a new feature to a prospect to try to close a sale. The fact that a simple error could cause someone to lose a limb or die does wonders for your focus.
  - 0xdeadbeefbabe 10y ago
    Doesn't it also prevent focus?
    - jacquesm 10y ago
      For me it definitely doesn't. It became part of the release cycle, analyze each and every change made to make sure it was safe and have a whole bunch of code that was tested to the max but never ever changed that would shut down the machine if it ever got outside of it's expected envelope. This is actually quite a tricky thing to do right because to be able to jog the machine out of a shut-down like that you have to re-enable it in a potentially un-safe situation. For each and every little challenge like that we found a good solution but some of those were real head-scratchers. A really nasty one that I recall was that when you power up a bunch of latches they can come up in an undefined state, so the decision was made to include a detector for that undefined state which first would have to be cleared before the output of the latches was allowed to influence the motors. This worked well in practice but given the restrictions of the machine this was all done on it took a bit of thinking, the solution we settled on was a magic sequence output on the parallel port indicating the system had successfully reset after which the relay powering the motor drivers would engage. So until that relay triggered everything else was ignored. This was already important enough with stepper drivers, but once we switched to servos for some of the more demanding applications it became crucial to safe operation that the drivers would never ever be energized with faulty inputs. A servo driving a ball-screw will happily wreck itself, the machine it is bolted on to and anything standing in-between (including the operator) if it suddenly gets driven to -10 or +10 V and naturally, you'd always get one of those two, never the safe '0'.
chillydawg 10y ago
When you are literally a laser death cancer beam, it's probably best to assume that the hardware the software is controlling is actively trying to resist you and that the software controlling the hardware is actively trying to kill a patient.
triplesec 10y ago
Example story "On March 21, 1986, a patient in Tyler, Texas was scheduled to receive his 9th Therac-25 treatment. He was prescribed 180 rads to a small tumor on his back. When the machine turned on, he felt heat and pain, which was unexpected as radiation therapy is usually a painless process. The Therac-25 itself also started buzzing in an unusual way. The patient began to get up off the treatment table when he was hit by a second pulse of radiation. This time he did get up and began banging on the door for help. He received a massive overdose. He was hospitalized for radiation sickness, and died 5 months later."
83457 10y ago
"The VT-100 console used to enter Therac-25 prescriptions allowed cursor movement via cursor up and down keys. If the user selected X-ray mode, the machine would begin setting up the machine for high-powered X-rays. This process took about 8 seconds. If the user switched to Electron mode within those 8 seconds, the turntable would not switch over to the correct position, leaving the turntable in an unknown state. ... It’s important to note that all the testing to this date had been performed slowly and carefully, as one would expect. Due to the nature of this bug, that sort of testing would never have identified the culprit. It took someone who was familiar with the machine – who worked with the data entry system every day, before the error was found."
csours 10y ago
> It’s important to note that all the testing to this date had been performed slowly and carefully, as one would expect. Ah, so if I click these angular nav-pills very quickly in series without waiting for the page to load, something unexpected might happen? What is annoying in one context may be deadly in another. ---- If the "Sentinel Event" policy had been in place at the time, perhaps these deaths would have been prevented. A sentinel event is any event that either leads to death or serious permanent injury -OR- could have lead to death or serious permanent injury. 1. https://www.jointcommission.org/sentinel_event_policy_and_procedures/ https://www.jointcommission.org/sentinel_event_policy_and_pr...
blobbers 10y ago
This story was covered in the book 'Set Phasers On Stun' (https://www.amazon.com/Set-Phasers-Stun-Design-Technology/dp/0963617885 https://www.amazon.com/Set-Phasers-Stun-Design-Technology/dp...). It was required reading in our Introduction to Engineering class in first year. As an engineering student it was meant to imprint one thing upon us, but talk about a dark introduction to the word 'responsibility'...
- chiph 10y ago
  Before I switched to CS, I was in a EE program. Our intro to engineering class was filled with stuff like the outdoor decorative fountain that killed six people one at a time, and the transformer box that was used in a game of hide + seek (the lock was missing). In the case of the fountain, several couples had had a night out and decided to splash in the fountain. The first two knocked a power conduit for one of the pumps loose, electrocuting themselves. The others died as they each entered the fountain to rescue the others. The system lacked a GFCI protector.
- nathanyo 10y ago
  My introduction to computing course had us read about the Therac-25 and write an essay about it. While code we write doesn't always have such dire consequences, it was an eye opener to me as a freshman in college. It definitely made designing software/hardware to never fail have a much higher priority on my list of priorities.
gene-h 10y ago
As Therac-25 will perpetually tell, when you code your software, best do it well.
vkou 10y ago
Strangely enough, nobody is bringing up the argument of 'But how many people were saved by the existence of Therac-25?' Yet, when a poorly designed self-driving car kills people, many rush to point out (correctly or not) all the lives that were saved by the technology. The valley seems to have an entirely different level of regard for the consequences of their work then engineers and doctors do.
- galdosdi 10y ago
  I think that might merely be because self driving cars are in the future, and Therac-25 is in the past. Also, it's easy to imagine self driving cars that are pretty good, but still occasionally kill people, yet at a rate much lower than human drivers. For the Therac-25 on the other hand, it's not so complicated. Why it failed and how to fix it is already well known.
  - vkou 10y ago
    It is indeed in the past - unlike the claims about how self-driving technology is currently safer then human drivers (Which is not at all clear yet), we can definitely say that Therac-25 saved more lives then it ended. Yet, it is also a case study in how not to build systems. It's easy to imagine a medical device that's pretty good, but still occasionally kills people because of flaws in its design. The image is fairly terrifying, actually. We could also say the same about the fatal Tesla accident - we know why it failed, and how to fix it... And observe how quickly large parts of the tech community (I would expect no less of the vendor) was to blame the human.
lpage 10y ago
My uncle was a radiation oncologist who worked with machines like this one. As a young child in the early 90s he took me to the hospital where he worked to watch him and their staff physicist calibrate a linear accelerator - an experience I remember vividly. A very long checklist culminated in them irradiating a plate of acrylic, which I got to keep. The plate was about 20cm x 30cm x 4cm and aligned such that the beam would strike the 4cm face and travel down the 20cm side, fanning out in the process. The result looked like a thinner version of this: https://en.wikipedia.org/wiki/Lichtenberg_figure#/media/File:PlanePair2.jpg https://en.wikipedia.org/wiki/Lichtenberg_figure#/media/File... I asked him how they could possibly use such machines on humans, given what I just watched it do to a 1kg plate of acrylic. He told me that they hit the plate with way more energy than they ever would a human. That prompted my followup question: "uncle, what happens if you accidentally hit the wrong button?" He told me that accidents like that used to happen, but that the machines they used had special computers to keep the patient safe even if he made a mistake. "But uncle, what happens if the computer makes a mistake!?" I had no idea what a bug, or for that matter code was. He didn't have a good answer beyond "the computer can't make mistakes like we do." Having played with computers enough by then to know that his statement wasn't entirely true, I ended that outing wanting to know more. I started obsessing over what would happen to people if computers controlling things like that linear accelerator, or even the elevator in my dad's office building, made a mistake. Incidentally, my uncle was the one who got me interested in science, and that trip to the hospital got me using computers for something other than games. Fast forward 24 years and...well...part of what I do is work on provably correct systems.
Wile_E_Quixote 10y ago
If anyone is interested in how modern versions of these machines work, the annual conference for the American Association of Physicists in Medicine (AAPM) is currently proceeding at the Walter E. Washington Convention Center in Washington, DC, and will continue until Thursday, August 4th. It's the largest international gathering of medical physicists in the world. If anyone is in the area and interested, you shouldn't have much trouble sneaking in and taking a stroll through the vendor area where you can see examples of the newest technologies in imaging and therapy physics. Just dress business casual and you'll blend right in with the other couple thousand physicists.
ricardobeat 10y ago
"Killed by software" would've been a much more faithful title!

89 points by paraknight 10y ago

"It’s important to note that while the software was the lynch pin in the Therac-25, it wasn’t the root cause. The entire system design was the real problem. Safety-critical loads were placed upon a computer system that was not designed to control them" They wrote code that depended on hardware controls, didn't document their reliance on the hardware controls and killed a bunch of people. DOCUMENT ALL YOUR DEPENDANCES!!!

wyldfire 10y ago
> They wrote code that depended on hardware controls ...similar to Toyota's electronic throttle control system design. Also, if you rely on hardware features for safety, it's still ok/good to design the software as if it didn't depend on those features whereever possible.
- dsfyu404ed 10y ago
  It all depends on the specifics. How much complexity will software checks add? In a lot of cases the lifetime cost of spec'ing a hard failsafe may be cheaper than designing, implementing and supporting a soft one. Then there's the whole issue of software false positives/negatives vs hardware failures At some level you can no longer abstract away the situation and the system has to perform as a system. You can write soft controls all you want but if hard controls are present in the spec and something that's not likely to change there becomes a point where chasing down every little problem and checking for every possible error is no longer cost effective. Nobody cares if you write Mars rover tier code for a bulldozer and spending the resources doing is wasteful if your competitors aren't also spending the resources. Obviously you can write total crap that's outside the acceptable range toward the other end. If you're designing software to control a widget that moves and does stuff has a hard switch to prevent your widget's equivalent of an out of battery detonation you have to strike a balance between relying on it and introducing complexity from soft controls. Every aspect of the system is involved in making that determination. If you've got a hard switch in most cases you may as well write a five line timeout controlled while loop that tries to perform the action and waits and tries again if there's no feedback indicating the action was performed. As long as nobody removes that switch and the code's dependency on it is very obviously documented then that simple unsafe code is probably better than more complex code that performs a redundant (redundant because you have a hard switch) check because the more complex code has more going on.
  - throwanem 10y ago
    I mean hey, if I'm writing code for a machine that might kill somebody if a hardware interlock is removed, I'm going to perform a software check for a valid state before I take the potentially fatal action, and if that's a problem for the organization that's paying me to write that code, I'm going to resign and find work elsewhere, because that outcome strikes me as strictly preferable to having someone's death on my conscience. I suppose it's arguable either way, though.
0xdeadbeefbabe 10y ago
> DOCUMENT ALL YOUR DEPENDANCES!!! Still doesn't make a difference if there is no place for the documents to live.

kqr 10y ago

Are there any statistics on the safety/reliability of software controlled vs. hardware controlled devices? I work in software engineering, so I'm exposed daily to broken software controls, and I'm gradually becoming more of a "grumpy old man" longing for the good old days when machines (be it cameras, cars, watches, medical equipment or anything else, really) could be "debugged" by following levers, wires, physical stops and hoses. I feel much more comfortable with that than with complex computer systems. I would like to know if my fears can be founded in actual truths, or if I'm just riling myself up over nothing.

HeyLaughingBoy 10y ago
There are but I don't know where to point you :-( I spent over 10 years writing control software for medical devices. In classifying the criticality of a device, the FDA from the beginning put any device with software into a higher category. They have since refined their classifications, so you probably won't find that guidance on their site anymore, but 15 years ago it was pretty eye opening to me that two devices of the same type, one with and one without software, could be automatically in different safety categories. So, no, it's not just you :-)
- jes 10y ago
  I don't remember the details at this point, but somewhere in the IEC 62304 regulation, there is language that suggests that you have to assume 100% failure for software components. I think it's in regard to risk controls, but I'm not certain of that. Edit: Just looked it up: "If the HAZARD could arise from the failure of the SOFTWARE SYSTEM to behave as specified, the probability of such failure shall be assumed to be 100%"
  - HeyLaughingBoy 10y ago
    Yes, this is to avoid the problem many an FMEA (Failure Modes Effects Analysis) meeting agonizes over: "it's possible for the software to fail in this way, but how do we quantify the probability of it happening so we can calculate the risk factor?" By assuming that if it can happen, then it will" you remove any handwaving from the equation.

platz 10y ago

These stories are horrific.

jacquesm 10y ago
Yes. They also should be required study material for anybody that writes software interacting with the real world. Humans are fragile. I wrote a cad/cam system for a lathe/mill combo, in all the years that that system was out there we found one bug that made it out past my desk and that only happened because some idiot decided to demo a new feature to a prospect to try to close a sale. The fact that a simple error could cause someone to lose a limb or die does wonders for your focus.
- 0xdeadbeefbabe 10y ago
  Doesn't it also prevent focus?
  - jacquesm 10y ago
    For me it definitely doesn't. It became part of the release cycle, analyze each and every change made to make sure it was safe and have a whole bunch of code that was tested to the max but never ever changed that would shut down the machine if it ever got outside of it's expected envelope. This is actually quite a tricky thing to do right because to be able to jog the machine out of a shut-down like that you have to re-enable it in a potentially un-safe situation. For each and every little challenge like that we found a good solution but some of those were real head-scratchers. A really nasty one that I recall was that when you power up a bunch of latches they can come up in an undefined state, so the decision was made to include a detector for that undefined state which first would have to be cleared before the output of the latches was allowed to influence the motors. This worked well in practice but given the restrictions of the machine this was all done on it took a bit of thinking, the solution we settled on was a magic sequence output on the parallel port indicating the system had successfully reset after which the relay powering the motor drivers would engage. So until that relay triggered everything else was ignored. This was already important enough with stepper drivers, but once we switched to servos for some of the more demanding applications it became crucial to safe operation that the drivers would never ever be energized with faulty inputs. A servo driving a ball-screw will happily wreck itself, the machine it is bolted on to and anything standing in-between (including the operator) if it suddenly gets driven to -10 or +10 V and naturally, you'd always get one of those two, never the safe '0'.

chillydawg 10y ago

When you are literally a laser death cancer beam, it's probably best to assume that the hardware the software is controlling is actively trying to resist you and that the software controlling the hardware is actively trying to kill a patient.

triplesec 10y ago

Example story "On March 21, 1986, a patient in Tyler, Texas was scheduled to receive his 9th Therac-25 treatment. He was prescribed 180 rads to a small tumor on his back. When the machine turned on, he felt heat and pain, which was unexpected as radiation therapy is usually a painless process. The Therac-25 itself also started buzzing in an unusual way. The patient began to get up off the treatment table when he was hit by a second pulse of radiation. This time he did get up and began banging on the door for help. He received a massive overdose. He was hospitalized for radiation sickness, and died 5 months later."

83457 10y ago

"The VT-100 console used to enter Therac-25 prescriptions allowed cursor movement via cursor up and down keys. If the user selected X-ray mode, the machine would begin setting up the machine for high-powered X-rays. This process took about 8 seconds. If the user switched to Electron mode within those 8 seconds, the turntable would not switch over to the correct position, leaving the turntable in an unknown state. ... It’s important to note that all the testing to this date had been performed slowly and carefully, as one would expect. Due to the nature of this bug, that sort of testing would never have identified the culprit. It took someone who was familiar with the machine – who worked with the data entry system every day, before the error was found."

csours 10y ago

> It’s important to note that all the testing to this date had been performed slowly and carefully, as one would expect. Ah, so if I click these angular nav-pills very quickly in series without waiting for the page to load, something unexpected might happen? What is annoying in one context may be deadly in another. ---- If the "Sentinel Event" policy had been in place at the time, perhaps these deaths would have been prevented. A sentinel event is any event that either leads to death or serious permanent injury -OR- could have lead to death or serious permanent injury. 1. https://www.jointcommission.org/sentinel_event_policy_and_procedures/ https://www.jointcommission.org/sentinel_event_policy_and_pr...

blobbers 10y ago

This story was covered in the book 'Set Phasers On Stun' (https://www.amazon.com/Set-Phasers-Stun-Design-Technology/dp/0963617885 https://www.amazon.com/Set-Phasers-Stun-Design-Technology/dp...). It was required reading in our Introduction to Engineering class in first year. As an engineering student it was meant to imprint one thing upon us, but talk about a dark introduction to the word 'responsibility'...

chiph 10y ago
Before I switched to CS, I was in a EE program. Our intro to engineering class was filled with stuff like the outdoor decorative fountain that killed six people one at a time, and the transformer box that was used in a game of hide + seek (the lock was missing). In the case of the fountain, several couples had had a night out and decided to splash in the fountain. The first two knocked a power conduit for one of the pumps loose, electrocuting themselves. The others died as they each entered the fountain to rescue the others. The system lacked a GFCI protector.
nathanyo 10y ago
My introduction to computing course had us read about the Therac-25 and write an essay about it. While code we write doesn't always have such dire consequences, it was an eye opener to me as a freshman in college. It definitely made designing software/hardware to never fail have a much higher priority on my list of priorities.

gene-h 10y ago

As Therac-25 will perpetually tell, when you code your software, best do it well.

vkou 10y ago

Strangely enough, nobody is bringing up the argument of 'But how many people were saved by the existence of Therac-25?' Yet, when a poorly designed self-driving car kills people, many rush to point out (correctly or not) all the lives that were saved by the technology. The valley seems to have an entirely different level of regard for the consequences of their work then engineers and doctors do.

galdosdi 10y ago
I think that might merely be because self driving cars are in the future, and Therac-25 is in the past. Also, it's easy to imagine self driving cars that are pretty good, but still occasionally kill people, yet at a rate much lower than human drivers. For the Therac-25 on the other hand, it's not so complicated. Why it failed and how to fix it is already well known.
- vkou 10y ago
  It is indeed in the past - unlike the claims about how self-driving technology is currently safer then human drivers (Which is not at all clear yet), we can definitely say that Therac-25 saved more lives then it ended. Yet, it is also a case study in how not to build systems. It's easy to imagine a medical device that's pretty good, but still occasionally kills people because of flaws in its design. The image is fairly terrifying, actually. We could also say the same about the fatal Tesla accident - we know why it failed, and how to fix it... And observe how quickly large parts of the tech community (I would expect no less of the vendor) was to blame the human.

lpage 10y ago

My uncle was a radiation oncologist who worked with machines like this one. As a young child in the early 90s he took me to the hospital where he worked to watch him and their staff physicist calibrate a linear accelerator - an experience I remember vividly. A very long checklist culminated in them irradiating a plate of acrylic, which I got to keep. The plate was about 20cm x 30cm x 4cm and aligned such that the beam would strike the 4cm face and travel down the 20cm side, fanning out in the process. The result looked like a thinner version of this: https://en.wikipedia.org/wiki/Lichtenberg_figure#/media/File:PlanePair2.jpg https://en.wikipedia.org/wiki/Lichtenberg_figure#/media/File... I asked him how they could possibly use such machines on humans, given what I just watched it do to a 1kg plate of acrylic. He told me that they hit the plate with way more energy than they ever would a human. That prompted my followup question: "uncle, what happens if you accidentally hit the wrong button?" He told me that accidents like that used to happen, but that the machines they used had special computers to keep the patient safe even if he made a mistake. "But uncle, what happens if the computer makes a mistake!?" I had no idea what a bug, or for that matter code was. He didn't have a good answer beyond "the computer can't make mistakes like we do." Having played with computers enough by then to know that his statement wasn't entirely true, I ended that outing wanting to know more. I started obsessing over what would happen to people if computers controlling things like that linear accelerator, or even the elevator in my dad's office building, made a mistake. Incidentally, my uncle was the one who got me interested in science, and that trip to the hospital got me using computers for something other than games. Fast forward 24 years and...well...part of what I do is work on provably correct systems.

Wile_E_Quixote 10y ago

If anyone is interested in how modern versions of these machines work, the annual conference for the American Association of Physicists in Medicine (AAPM) is currently proceeding at the Walter E. Washington Convention Center in Washington, DC, and will continue until Thursday, August 4th. It's the largest international gathering of medical physicists in the world. If anyone is in the area and interested, you shouldn't have much trouble sneaking in and taking a stroll through the vendor area where you can see examples of the newest technologies in imaging and therapy physics. Just dress business casual and you'll blend right in with the other couple thousand physicists.

ricardobeat 10y ago

"Killed by software" would've been a much more faithful title!