18 ms·

The Therac-25 Incident

235 points by dagurp 6y ago

lordnacho 6y ago
How much money was AECL making selling these things? You'd think a second pair of eyes on the code would not cost too much. Do I blame the one person? Not really, who in this world hasn't written a race condition at some point? RCs are also one of those things where someone else might spot it a lot sooner than the original writer. I agree with the sentiment that they took the software for granted. I get the feeling that happens in a lot of settings, most of them less life-threatening than this one. I've come across it myself too, in finance. Somehow someone decides they have invented a brilliant money-making strategy, if they could only get the coders to implement it properly. Of course the coders come back to ask questions, and then depending on the environment it plays out to a resolution. I get the feeling the same thing happened here. Some scientist said "hey all it needs is to send this beam into the patient" and assumed their description was the only level of abstraction that needed to be understood.
ufmace 6y ago
> The Therac-25 was the first entirely software-controlled radiotherapy device. As that quote from Jacky above points out: most such systems use hardware interlocks to prevent the beam from firing when the targets are not properly configured. The Therac-25 did not. This makes me think - there was only one developer there, I guess, who was doing everything in assembly. This software, and the process to produce it, must have been designed in the early days of their devices, when there would be expected to be hardware interlocks to prevent any of the really bad failure modes. I bet they never did change much of the software, or their procedures for developing, testing, qualifying, and releasing it in light of the change from relying on hardware interlocks to the quality of the software being the only thing preventing something terrible from happening.
- bluGill 6y ago
  The software was working just fine for years before on earlier versions with the interlocks. They never checked to see how often or why the interlocks fired before removing them. Turns out those interlocks fired often because of the same bugs.
  - brians 6y ago
    They had two fuses, so they had a 2:1 safety margin! Just like the NASA managers who decided that 30% erosion in an O-ring designed for no erosion meant a 3:1 safety margin.
- Gare 6y ago
  A quote from the report: > Related problems were found in the Therac-20 software. These were not recognized until after the Therac-25 accidents because the Therac-20 included hardware safety interlocks and thus no injuries resulted. The safety fuses were occasionally blowing during the operation of Therac-20, but nobody asked why.
  - baobabKoodaa 6y ago
    > The safety fuses were occasionally blowing during the operation of Therac-20, but nobody asked why. Have you tried turning it off and on again?
omginternets 6y ago
The featured comment is great, for those who missed it: I am a physician who did a computer science degree before medical school. I frequently use the Therac-25 incident as an example of why we need dual experts who are trained in both fields. I must add two small points to this fantastic summary. 1. The shadow of the Therac-25 is much longer than those who remember it. In my opinion, this incident set medical informatics back 20 years. Throughout the 80s and 90s there was just a feeling in medicine that computers were dangerous, even if the individual physicians didn't know why. This is why, when I was a resident in 2002-2006 we still were writing all of our orders and notes on paper. It wasn't until the US federal government slammed down the hammer in the mid 2000's and said no payment unless you adopt electronic health records, that computers made real inroads into clinical medicine. 2. The medical profession, and the government agencies that regulate it, are accustomed to risk and have systems to manage it. The problem is that classical medicine is tuned to "continuous risks." If the Risk of 100 mg of aspirin is "1 risk unit" and the risk of 200 mg of aspirin is "2 risk units" then the risk of 150 mg of aspirin is strongly likely to be between 1 and 2, and it definitely won't be 1,000,000. The mechanisms we use to regulate medicine, with dosing trials, and pharmacokinetic studies, and so forth are based on this assumption that both benefit and harm are continuous functions of prescribed dose, and the physician's job is to find the sweet spot between them. When you let a computer handle a treatment you are exposed to a completely different kind of risk. Computers are inherently binary machines that we sometimes make simulate continuous functions. Because computers are binary, there is a potential for corner cases that expose erratic, and as this case shows, potentially fatal behavior. This is not new to computer science, but it is very foreign to medicine. Because of this, medicine has a built in blind spot in evaluating computer technology.
- deleted 6y ago
  [deleted]
- beerandt 6y ago
  It's so short-sighted that he doesn't see that medical records being forced so quickly to digital/computers is almost exactly the same problem being played out, just not as directly or dramatically, but with a much wider net, and way more short- and long-term problems (including the software/systems trust mentioned).
b3lvedere 6y ago
Interesting story. I've been on the testing site of medical hardware many many moons ago. It's quite amazing what you can and must test. For instance: We had to prove that if our equipment broke by falling damage that the potential debris flying off could not harm the patient. I always liked the testing philosophy institutes like for instance Underwriter Laboratories had: Your product will fail. This is stated as fact and is not debatable. What kind of fail safes and protection have you made so that when it fails (and it will) it cannot not harm the patient?
- buescher 6y ago
  Yes. It's amazing the number of engineers that resist doing that analysis - "oh that part won't ever break". Some safety standards do allow for "reliable components" (i.e. if the component is already been scrutinized for safe failure modes, you don't have to consider them) and for submitting reliability analysis or data. I've never seen reliability analysis or data submitted instead of single-point failure analysis though, myself. Single-point failure analysis techniques like fault trees, event trees, and especially the tabular "failure modes and effects analysis" (FMEA) are so powerful, especially for safety-critical hardware, that when people learn them they want to apply them to everything, including software. However, FMEA techniques actually have been found to not apply well to software below about the block diagram level. They don't find bugs that would not be found by other methods (static analysis, code review, requirements analysis etc) and they're extremely time and labor intensive. Here's an NRC report that goes into some detail: https://www.nrc.gov/reading-rm/doc-collections/nuregs/agreement/ia0254/index.html https://www.nrc.gov/reading-rm/doc-collections/nuregs/agreem...
  - b3lvedere 6y ago
    "Through analysis and examples of several real-life catastrophes, this report shows that FMEA could not have helped in the discovery of the underlying faults. The report concludes that the contribution of FMEA to regulatory assurance of Complex Logic, especially software, in a nuclear power plant safety system is marginal." Even more interesting! Thank you for this link. I appreciate it. Never too old to learn. :)
ed25519FUUU 6y ago
The very worst part of this story is that the manufacture vigorously defended their machine, threatening individuals and hospitals with lawsuits if they spoke out publicly. I have zero doubt this led to more deaths.
buescher 6y ago
Here's a bit more background, from Nancy Leveson (now at MIT): http://sunnyday.mit.edu/papers/therac.pdf http://sunnyday.mit.edu/papers/therac.pdf
- meristem 6y ago
  Leveson's Engineering a Safer World[1] is excellent, for those interested in safety engineering. [1] https://mitpress.mit.edu/books/engineering-safer-world https://mitpress.mit.edu/books/engineering-safer-world
  - buescher 6y ago
    The systems safety case studies in that are great. It's also available as "open access" (free-as-in-free-beer) at MIT press: https://direct.mit.edu/books/book/2908/Engineering-a-Safer-WorldSystems-Thinking-Applied https://direct.mit.edu/books/book/2908/Engineering-a-Safer-W...
jtchang 6y ago
I can say when I was doing my CS degree this was definitely covered. In fact it is one of the lectures that stood out in my mind at that time. My professor at the time (Bill Leahy) definitely drilled into us the importance of understanding the systems in which we were eventually going to work on. Not sure if this is still covered today.
- Jtsummers 6y ago
  Per younger, CS, colleagues who went through school in the last 6 years, it was still being taught at their smaller US colleges.
- phlyingpenguin 6y ago
  The book I use to instruct software engineering uses it, and I do use that chapter.
- lxgr 6y ago
  It was covered in at least one of my classes as well. (Graduated only a few years ago.)
- Mountain_Skies 6y ago
  When I was the graduate teaching assistant for a software engineering lab, the students got a week off to do research on software failures that harmed humans. For many of the students it was the first time they gave any thought to the concept of software causing actual physical harm. I'm glad we were able to expose them to this reality but also was a bit disheartened as they should have thought about it far before a fourth year course in their major.
- icelancer 6y ago
  Was definitely covered in my small school program in Embedded Systems in the early 2000s.
joncrane 6y ago
I feel like this makes it to HN once every few years or so. I know it well from it being the first and main case study in my software testing class as an undergraduate CS major in Washington DC in 1999. It will never not be interesting.
- siltpotato 6y ago
  Apparently this is the seventh one. I've never worked on a safety critical system but this is the story that makes me wonder what it's like to do so.
  - Jtsummers 6y ago
    It’s stressful, but often worthwhile. It requires diligence, deliberate action, and patience.
    - matthias509 6y ago
      I used to work on public safety radio systems. Things which seem like minor issues like clipping the beginning of a transmission every now and then are showstopper defects in that space. It’s because it can be the difference between “Shoot” and “Don’t shoot.”
dade_ 6y ago
The article says the developer was never named, as if that had anything to do with the actual problems. Everything about this project sounds insane & inept. Some questions in my mind while reading this article (but I couldn't find them quickly in a search): Who were the executives that were running the company? Sounds like something that should be taught to MBA as well as CS students. Further, the AECL was a crown corporation of the Canadian government. Who was the minister and bureaucrats in charge of the department? What role did they have in solving or covering up the issue?
- strken 6y ago
  The article has a very long explanation of why one developer is not to blame, and why it's entirely the fault of the company for having no testing procedure and no review process.
dang 6y ago
(For the curious) the Therac-25 stack on HN: 2019 https://news.ycombinator.com/item?id=21679287 https://news.ycombinator.com/item?id=21679287 2018 https://news.ycombinator.com/item?id=17740292 https://news.ycombinator.com/item?id=17740292 2016 https://news.ycombinator.com/item?id=12201147 https://news.ycombinator.com/item?id=12201147 2015 https://news.ycombinator.com/item?id=9643054 https://news.ycombinator.com/item?id=9643054 2014 https://news.ycombinator.com/item?id=7257005 https://news.ycombinator.com/item?id=7257005 2010 https://news.ycombinator.com/item?id=1143776 https://news.ycombinator.com/item?id=1143776 Others?
- kondro 6y ago
  It comes up a lot, but it’s an incredibly important story that bares repeating often. Especially with similar issues like the 737-MAX occurring pretty recently.
  - dang 6y ago
    Oh for sure, and the purpose of those links is not to imply that it's too often. Reposts are fine on HN if a story hasn't had a big discussion after a year or so (https://news.ycombinator.com/newsfaq.html https://news.ycombinator.com/newsfaq.html). I just list them because people often like to look at the past threads. That's what "For the curious" means :)
  - yummypaint 6y ago
    Yeah aparently we have to constantly relearn that software hotfixes are not an acceptible substitute for hardware interlocks and good overal design practices when lives are at stake.
- noefingway 6y ago
  As a long time subscriber to the Risks mailing list, I would have to note the extensive discussion of this incident: http://catless.ncl.ac.uk/Risks/search?query=therac http://catless.ncl.ac.uk/Risks/search?query=therac
time0ut 6y ago
Many years ago, I had an opportunity to work on a similar type of system (though more recent than this). In the final round of interviews, one of the executives asked if I would be comfortable working on a device that could deliver a potentially dangerous dose of radiation to a patient. In that moment, my mind flashed to this story. I try to be a careful engineer and I am sure there are many more safeguards in place now, but, in that moment, I realized I would not be able to live with myself if I harmed someone that way. I answered truthfully and he thanked me and we ended things there. I do not mean this as a judgement on those who do work on systems that can physically harm people. Obviously, we need good engineers to design potentially dangerous systems. It is just how I realized I really don't have the character to do it.
- Hydraulix989 6y ago
  I would have considered the fact that for a vast majority of people suffering from cancer, this device helps them instead of harms them. However, I can also imagine leadership at some places trying to move fast and pressure ICs into delivering something that isn't completely bulletproof in the name of bottom line. That is something I would have tried to discern from the executives. Similar tradeoffs have been made before with cars against expected legal costs. There are plenty of other high stakes software that involve human lives (Uber self driving cars, SpaceX, Patriot missiles) and many of them completely scare me and morally frustrate me as well to the point where I would not like to work on one, but I totally understand if you have a personal profile that is different than mine.
- sho_hn 6y ago
  > In the final round of interviews, one of the executives asked if I would be comfortable working on a device that could deliver a potentially dangerous dose of radiation to a patient Automotive software engineer here: I've asked the same in interviews. "We work on multi-ton machines that can kill people" is a frequently uttered statement at work.
- mcguire 6y ago
  Good for you to realize that, then. On the other hand, I'm not entirely sure it's appropriate to be comfortable with that kind of position in any case.
jancsika 6y ago
I found this comment from the article fascinating: > I am a physician who did a computer science degree before medical school. I frequently use the Therac-25 incident as an example of why we need dual experts who are trained in both fields. I must add two small points to this fantastic summary. > 1. The shadow of the Therac-25 is much longer than those who remember it. In my opinion, this incident set medical informatics back 20 years. Throughout the 80s and 90s there was just a feeling in medicine that computers were dangerous, even if the individual physicians didn't know why. This is why, when I was a resident in 2002-2006 we still were writing all of our orders and notes on paper. It wasn't until the US federal government slammed down the hammer in the mid 2000's and said no payment unless you adopt electronic health records, that computers made real inroads into clinical medicine. > 2. The medical profession, and the government agencies that regulate it, are accustomed to risk and have systems to manage it. The problem is that classical medicine is tuned to "continuous risks." If the Risk of 100 mg of aspirin is "1 risk unit" and the risk of 200 mg of aspirin is "2 risk units" then the risk of 150 mg of aspirin is strongly likely to be between 1 and 2, and it definitely won't be 1,000,000. The mechanisms we use to regulate medicine, with dosing trials, and pharmacokinetic studies, and so forth are based on this assumption that both benefit and harm are continuous functions of prescribed dose, and the physician's job is to find the sweet spot between them. > When you let a computer handle a treatment you are exposed to a completely different kind of risk. Computers are inherently binary machines that we sometimes make simulate continuous functions. Because computers are binary, there is a potential for corner cases that expose erratic, and as this case shows, potentially fatal behavior. This is not new to computer science, but it is very foreign to medicine. Because of this, medicine has a built in blind spot in evaluating computer technology.
- mcguire 6y ago
  I'm not sure I buy that. Or, well, I suppose that those in the medical field believe it, but I don't think they're right. Consider something like a surgeon nicking and artery while performing some routine surgery, the patient not responding normally to anesthesia or the anesthetist not getting the mixture right and the patient not coming back the way they went in. Or that subset of patients that have poor responses to a vaccine. Everybody likes to think of the world as a linear system, but it's not.
at_a_remove 6y ago
I rather randomly met a woman with a similar sort of background and trajectory as I have: trained in physics, got sucked into computers via the brain drain. She programmed the models for radiation dosing in the metaphorical descendants of Therac-25. I asked her just how often it was brought up in her work and she mentioned that she trained under someone who was in the original group of people brought in to analyze and understand just what happened with Therac-25. Fascinating stuff.
ZuLuuuuuu 6y ago
This is one of the infamous incidents where software failure caused harm on humans. I am kind of fascinated about such incidents (since I learn so much by reading about them), are there any other examples of such incidents that you guys know of? It doesn't have to result in harming human, but any software failure related incident which resulted in big consequences. One another example that comes to my mind is the Toyota "unintended acceleration" incident. Or the "Mars Climate Orbiter" incident.
- crocal 6y ago
  Google Ariane 501. Enjoy.
- probably_wrong 6y ago
  If big consequences is what you're after, I can think of three typical incidents: the "most expensive hyphen in history" (you can search it like that), it's companion piece, the Mars Climate Orbiter(which I see you added now), and the Denver Airport Baggage System fiasco [1] where bad software planning caused over $500M in delays. [1] http://calleam.com/WTPF/?page_id=2086 http://calleam.com/WTPF/?page_id=2086
FiatLuxDave 6y ago
I find it interesting how often the Therac 25 is mentioned on HN (thanks to Dan for the list), but nobody ever mentions that those kind of problems never entirely went away. Therac 25 is just the famous one. You don't have to go back to 1986, there are definitely examples from this century. The root causes are somewhat different, and somewhat the same. But no one seems to be teaching these more modern cases to aspiring programmers in school, at least not to the level where every programmer I know has heard of them. For example, the issue which caused this in 2007: https://www.heraldtribune.com/article/LK/20100124/News/605200351/SH https://www.heraldtribune.com/article/LK/20100124/News/60520... Or the process issues which caused this in 2001: https://www.fda.gov/radiation-emitting-products/alerts-and-notices/fda-statement-radiation-overexposures-panama https://www.fda.gov/radiation-emitting-products/alerts-and-n...
- sebmellen 6y ago
  I quote this from the article so people may take an interest in reading it. This is the opening paragraph: > As Scott Jerome-Parks lay dying, he clung to this wish: that his fatal radiation overdose -- which left him deaf, struggling to see, unable to swallow, burned, with his teeth falling out, with ulcers in his mouth and throat, nauseated, in severe pain and finally unable to breathe -- be studied and talked about publicly so that others might not have to live his nightmare.
- Shivetya 6y ago
  To this day I run into development managers who will dismiss an easily replicated issue on the grounds it won't happen enough, though of course with a big group of users it actually does, to warrant fixing. While this can be true at times and one excuse is always, we have support to fix the data, it tends to a standard by which all fixes are handled. which is, once you start blowing off testing, having exceptions to avoid every fix, you rarely if ever leave that mode and end up with a stressed support staff and users in the field who think your coders are incompetent.
- ahepp 6y ago
  The original NYT article for the 2007 incident has more details about the problematic software: https://www.nytimes.com/2010/01/24/health/24radiation.html https://www.nytimes.com/2010/01/24/health/24radiation.html
nickdothutton 6y ago
I mentioned this incident in passing on my post of a few years ago. We studied it many years ago in the early 90s at university. https://blog.eutopian.io/the-age-of-invisible-disasters/ https://blog.eutopian.io/the-age-of-invisible-disasters/
autoditype 6y ago
One of my university professors taught us this incidet, along with a number of other failures, as an ethics lesson. It shook me and I think of it whenever I see automated medical machines, self-driving cars, etc
ericsoderstrom 6y ago
This was (is?) used as a case study in MIT's system's engineering course (6.033)
- KMag 6y ago
  I was in Course 2 (Mechanical Engineering), but 6.004 and 6.033 were two of my favorite classes at MIT. As I remember, we spent 2 lectures covering many mistakes made in the Therac-25. We also covered the Ariane 5's first launch failure. We also covered a bunch of systems that made good design trade-offs and discussed their design spaces.
sn41 6y ago
The Therac-25 incident, and Bhopal Gas Tragedy were definitely routine lessons in courses on industrial safety and reliability. (Another reason not to turn your nose up against academic coursework - they do impart useful knowledge.) I feel that lot of manual errors and UI errors can be mitigated by careful design. Punishing natural careless errors is not very effective, as recent research shows [1]: [1] https://www.jstage.jst.go.jp/article/jjsca/32/7/32_954/_article/-char/en https://www.jstage.jst.go.jp/article/jjsca/32/7/32_954/_arti...
anotheraccount9 6y ago
I used to work in an ICU. The software that controlled the patient's cardiac monitor would not allow us to modify the alarm settings (sensitivity, volume, etc.) I'm not sure if the settings were locked as a default by the manufacturer, or a protected setting that no one knew how to change, or simply locked by management. Anyhow, from mild atrial fibrillation to irregular rhythm, the alarm was constantly beeping, which was to be expected, considering the acuity of our patients. Nurses and MD became really annoyed by this. Eventually, one of the staff covered the speaker from inside, to muffle the almost constant warnings. The staff repeatedly asked management to find a solution to this, and the associated risk. One day, a patient had a life threatening cardiac incident. The alarm was muffled enough that no one nearby could hear a thing.
sleepy_keita 6y ago
I recently watched this YouTube video about the Therac-25, it was similar to this article (albeit a little less in-depth). https://www.youtube.com/watch?v=-7gVqBY52MY https://www.youtube.com/watch?v=-7gVqBY52MY
- AdamJacobMuller 6y ago
  Hah! We both posted the EXACT same video at the exact same time! Plainly Difficult is an awesome channel.
  - sleepy_keita 6y ago
    Haha! It definitely is. I've been working through their backlog off and on for the past few weeks.
AdamJacobMuller 6y ago
Plainly Difficult has an episode on the Therac-25: https://www.youtube.com/watch?v=-7gVqBY52MY https://www.youtube.com/watch?v=-7gVqBY52MY Highly recommend all of his videos, actually.
- gww 6y ago
  Great channel. I found it a few months ago and have watched all of his videos. I like that he provides enough scientific detail without making it too hard to follow.
lmilcin 6y ago
As a professional developer interested in extreme reliability, I have studied this and similar stories. I have studied high profile software flops, project failures as well as NTSB reports for aircraft and train accidents, reports of buildings failing etc. and find all those reports very illuminating in the sense of a kind of mindset that is necessary to ensure reliability. I am not working on safety-critical systems but still, the failure of the type of backend systems I work on can bring huge damage to the company. You can come out with a set of rules that can really improve the reliability of your application. * Ensure the system always moves between known, valid states. Errors can still be valid states if you plan for them correctly. I like to compare this to logical induction: if you start with correct state and on every operation ensure that if it executes on correct state it must result in correct state, then you guarantee that the system is always in correct state. * Ensure incorrect operations are impossible to execute. As an example, if you want to guard against large data loss, you can choose to not expose ability to remove data. In my current project we took away everybody's write access to the database, we removed all functions that can remove the data from the database and exchanged them with functions that either mask data (setting flags so that application ignores it) or move to archive. If you need to run an operation on database, you need to submit a pull request and go through Code Review. There is additional review before merge where the team goes through checklist to verify the change meets certain standards. * Ensure anything the client can throw at the application does not have a chance to escalate to other transactions. All inputs are limited and validated mercilessly, all queries are constructed so that it is possible to predict resources necessary to execute the query. Every type of query has a limit on how many queries of that type can be executed in parallel or in a unit of time and how much results it can return. All functionality is constructed in such a way that the performance per transaction does not degrade when load increases (for example by amortizing costs and batching work). Then every functionality is load tested and based on testing results it is certified with maximum load that it can process. Everything over that limit is preemptively declined. * If an algorithm or a solution cannot be guaranteed to work correctly it cannot be used. For example, if library cannot be guaranteed to use at most X amount of memory for the operation, then it is not suitable. * Every single failure of the system is investigated. No system is without faults. The only way to build reliable system is to not ignore errors and always try to figure out underlying failure in the process that let the fault happen. Most projects deal with errors only after the error is urgent enough or only to fix the immediate cause of the error, rather than the failure in the process. If you are interested in very reliable software, when you have it fail for whatever reason you need to look at all stages of your process and identify all places that were supposed to prevent the problem but did not.

235 points by dagurp 6y ago

How much money was AECL making selling these things? You'd think a second pair of eyes on the code would not cost too much. Do I blame the one person? Not really, who in this world hasn't written a race condition at some point? RCs are also one of those things where someone else might spot it a lot sooner than the original writer. I agree with the sentiment that they took the software for granted. I get the feeling that happens in a lot of settings, most of them less life-threatening than this one. I've come across it myself too, in finance. Somehow someone decides they have invented a brilliant money-making strategy, if they could only get the coders to implement it properly. Of course the coders come back to ask questions, and then depending on the environment it plays out to a resolution. I get the feeling the same thing happened here. Some scientist said "hey all it needs is to send this beam into the patient" and assumed their description was the only level of abstraction that needed to be understood.

ufmace 6y ago

> The Therac-25 was the first entirely software-controlled radiotherapy device. As that quote from Jacky above points out: most such systems use hardware interlocks to prevent the beam from firing when the targets are not properly configured. The Therac-25 did not. This makes me think - there was only one developer there, I guess, who was doing everything in assembly. This software, and the process to produce it, must have been designed in the early days of their devices, when there would be expected to be hardware interlocks to prevent any of the really bad failure modes. I bet they never did change much of the software, or their procedures for developing, testing, qualifying, and releasing it in light of the change from relying on hardware interlocks to the quality of the software being the only thing preventing something terrible from happening.

bluGill 6y ago
The software was working just fine for years before on earlier versions with the interlocks. They never checked to see how often or why the interlocks fired before removing them. Turns out those interlocks fired often because of the same bugs.
- brians 6y ago
  They had two fuses, so they had a 2:1 safety margin! Just like the NASA managers who decided that 30% erosion in an O-ring designed for no erosion meant a 3:1 safety margin.
Gare 6y ago
A quote from the report: > Related problems were found in the Therac-20 software. These were not recognized until after the Therac-25 accidents because the Therac-20 included hardware safety interlocks and thus no injuries resulted. The safety fuses were occasionally blowing during the operation of Therac-20, but nobody asked why.
- baobabKoodaa 6y ago
  > The safety fuses were occasionally blowing during the operation of Therac-20, but nobody asked why. Have you tried turning it off and on again?

omginternets 6y ago

The featured comment is great, for those who missed it: I am a physician who did a computer science degree before medical school. I frequently use the Therac-25 incident as an example of why we need dual experts who are trained in both fields. I must add two small points to this fantastic summary. 1. The shadow of the Therac-25 is much longer than those who remember it. In my opinion, this incident set medical informatics back 20 years. Throughout the 80s and 90s there was just a feeling in medicine that computers were dangerous, even if the individual physicians didn't know why. This is why, when I was a resident in 2002-2006 we still were writing all of our orders and notes on paper. It wasn't until the US federal government slammed down the hammer in the mid 2000's and said no payment unless you adopt electronic health records, that computers made real inroads into clinical medicine. 2. The medical profession, and the government agencies that regulate it, are accustomed to risk and have systems to manage it. The problem is that classical medicine is tuned to "continuous risks." If the Risk of 100 mg of aspirin is "1 risk unit" and the risk of 200 mg of aspirin is "2 risk units" then the risk of 150 mg of aspirin is strongly likely to be between 1 and 2, and it definitely won't be 1,000,000. The mechanisms we use to regulate medicine, with dosing trials, and pharmacokinetic studies, and so forth are based on this assumption that both benefit and harm are continuous functions of prescribed dose, and the physician's job is to find the sweet spot between them. When you let a computer handle a treatment you are exposed to a completely different kind of risk. Computers are inherently binary machines that we sometimes make simulate continuous functions. Because computers are binary, there is a potential for corner cases that expose erratic, and as this case shows, potentially fatal behavior. This is not new to computer science, but it is very foreign to medicine. Because of this, medicine has a built in blind spot in evaluating computer technology.

deleted 6y ago
[deleted]
beerandt 6y ago
It's so short-sighted that he doesn't see that medical records being forced so quickly to digital/computers is almost exactly the same problem being played out, just not as directly or dramatically, but with a much wider net, and way more short- and long-term problems (including the software/systems trust mentioned).

b3lvedere 6y ago

Interesting story. I've been on the testing site of medical hardware many many moons ago. It's quite amazing what you can and must test. For instance: We had to prove that if our equipment broke by falling damage that the potential debris flying off could not harm the patient. I always liked the testing philosophy institutes like for instance Underwriter Laboratories had: Your product will fail. This is stated as fact and is not debatable. What kind of fail safes and protection have you made so that when it fails (and it will) it cannot not harm the patient?

buescher 6y ago
Yes. It's amazing the number of engineers that resist doing that analysis - "oh that part won't ever break". Some safety standards do allow for "reliable components" (i.e. if the component is already been scrutinized for safe failure modes, you don't have to consider them) and for submitting reliability analysis or data. I've never seen reliability analysis or data submitted instead of single-point failure analysis though, myself. Single-point failure analysis techniques like fault trees, event trees, and especially the tabular "failure modes and effects analysis" (FMEA) are so powerful, especially for safety-critical hardware, that when people learn them they want to apply them to everything, including software. However, FMEA techniques actually have been found to not apply well to software below about the block diagram level. They don't find bugs that would not be found by other methods (static analysis, code review, requirements analysis etc) and they're extremely time and labor intensive. Here's an NRC report that goes into some detail: https://www.nrc.gov/reading-rm/doc-collections/nuregs/agreement/ia0254/index.html https://www.nrc.gov/reading-rm/doc-collections/nuregs/agreem...
- b3lvedere 6y ago
  "Through analysis and examples of several real-life catastrophes, this report shows that FMEA could not have helped in the discovery of the underlying faults. The report concludes that the contribution of FMEA to regulatory assurance of Complex Logic, especially software, in a nuclear power plant safety system is marginal." Even more interesting! Thank you for this link. I appreciate it. Never too old to learn. :)

ed25519FUUU 6y ago

The very worst part of this story is that the manufacture vigorously defended their machine, threatening individuals and hospitals with lawsuits if they spoke out publicly. I have zero doubt this led to more deaths.

buescher 6y ago

Here's a bit more background, from Nancy Leveson (now at MIT): http://sunnyday.mit.edu/papers/therac.pdf http://sunnyday.mit.edu/papers/therac.pdf

meristem 6y ago
Leveson's Engineering a Safer World[1] is excellent, for those interested in safety engineering. [1] https://mitpress.mit.edu/books/engineering-safer-world https://mitpress.mit.edu/books/engineering-safer-world
- buescher 6y ago
  The systems safety case studies in that are great. It's also available as "open access" (free-as-in-free-beer) at MIT press: https://direct.mit.edu/books/book/2908/Engineering-a-Safer-WorldSystems-Thinking-Applied https://direct.mit.edu/books/book/2908/Engineering-a-Safer-W...

jtchang 6y ago

I can say when I was doing my CS degree this was definitely covered. In fact it is one of the lectures that stood out in my mind at that time. My professor at the time (Bill Leahy) definitely drilled into us the importance of understanding the systems in which we were eventually going to work on. Not sure if this is still covered today.

Jtsummers 6y ago
Per younger, CS, colleagues who went through school in the last 6 years, it was still being taught at their smaller US colleges.
phlyingpenguin 6y ago
The book I use to instruct software engineering uses it, and I do use that chapter.
lxgr 6y ago
It was covered in at least one of my classes as well. (Graduated only a few years ago.)
Mountain_Skies 6y ago
When I was the graduate teaching assistant for a software engineering lab, the students got a week off to do research on software failures that harmed humans. For many of the students it was the first time they gave any thought to the concept of software causing actual physical harm. I'm glad we were able to expose them to this reality but also was a bit disheartened as they should have thought about it far before a fourth year course in their major.
icelancer 6y ago
Was definitely covered in my small school program in Embedded Systems in the early 2000s.

joncrane 6y ago

I feel like this makes it to HN once every few years or so. I know it well from it being the first and main case study in my software testing class as an undergraduate CS major in Washington DC in 1999. It will never not be interesting.

siltpotato 6y ago
Apparently this is the seventh one. I've never worked on a safety critical system but this is the story that makes me wonder what it's like to do so.
- Jtsummers 6y ago
  It’s stressful, but often worthwhile. It requires diligence, deliberate action, and patience.
  - matthias509 6y ago
    I used to work on public safety radio systems. Things which seem like minor issues like clipping the beginning of a transmission every now and then are showstopper defects in that space. It’s because it can be the difference between “Shoot” and “Don’t shoot.”

dade_ 6y ago

The article says the developer was never named, as if that had anything to do with the actual problems. Everything about this project sounds insane & inept. Some questions in my mind while reading this article (but I couldn't find them quickly in a search): Who were the executives that were running the company? Sounds like something that should be taught to MBA as well as CS students. Further, the AECL was a crown corporation of the Canadian government. Who was the minister and bureaucrats in charge of the department? What role did they have in solving or covering up the issue?

strken 6y ago
The article has a very long explanation of why one developer is not to blame, and why it's entirely the fault of the company for having no testing procedure and no review process.

dang 6y ago

(For the curious) the Therac-25 stack on HN: 2019 https://news.ycombinator.com/item?id=21679287 https://news.ycombinator.com/item?id=21679287 2018 https://news.ycombinator.com/item?id=17740292 https://news.ycombinator.com/item?id=17740292 2016 https://news.ycombinator.com/item?id=12201147 https://news.ycombinator.com/item?id=12201147 2015 https://news.ycombinator.com/item?id=9643054 https://news.ycombinator.com/item?id=9643054 2014 https://news.ycombinator.com/item?id=7257005 https://news.ycombinator.com/item?id=7257005 2010 https://news.ycombinator.com/item?id=1143776 https://news.ycombinator.com/item?id=1143776 Others?

kondro 6y ago
It comes up a lot, but it’s an incredibly important story that bares repeating often. Especially with similar issues like the 737-MAX occurring pretty recently.
- dang 6y ago
  Oh for sure, and the purpose of those links is not to imply that it's too often. Reposts are fine on HN if a story hasn't had a big discussion after a year or so (https://news.ycombinator.com/newsfaq.html https://news.ycombinator.com/newsfaq.html). I just list them because people often like to look at the past threads. That's what "For the curious" means :)
- yummypaint 6y ago
  Yeah aparently we have to constantly relearn that software hotfixes are not an acceptible substitute for hardware interlocks and good overal design practices when lives are at stake.
noefingway 6y ago
As a long time subscriber to the Risks mailing list, I would have to note the extensive discussion of this incident: http://catless.ncl.ac.uk/Risks/search?query=therac http://catless.ncl.ac.uk/Risks/search?query=therac

time0ut 6y ago

Many years ago, I had an opportunity to work on a similar type of system (though more recent than this). In the final round of interviews, one of the executives asked if I would be comfortable working on a device that could deliver a potentially dangerous dose of radiation to a patient. In that moment, my mind flashed to this story. I try to be a careful engineer and I am sure there are many more safeguards in place now, but, in that moment, I realized I would not be able to live with myself if I harmed someone that way. I answered truthfully and he thanked me and we ended things there. I do not mean this as a judgement on those who do work on systems that can physically harm people. Obviously, we need good engineers to design potentially dangerous systems. It is just how I realized I really don't have the character to do it.

Hydraulix989 6y ago
I would have considered the fact that for a vast majority of people suffering from cancer, this device helps them instead of harms them. However, I can also imagine leadership at some places trying to move fast and pressure ICs into delivering something that isn't completely bulletproof in the name of bottom line. That is something I would have tried to discern from the executives. Similar tradeoffs have been made before with cars against expected legal costs. There are plenty of other high stakes software that involve human lives (Uber self driving cars, SpaceX, Patriot missiles) and many of them completely scare me and morally frustrate me as well to the point where I would not like to work on one, but I totally understand if you have a personal profile that is different than mine.
sho_hn 6y ago
> In the final round of interviews, one of the executives asked if I would be comfortable working on a device that could deliver a potentially dangerous dose of radiation to a patient Automotive software engineer here: I've asked the same in interviews. "We work on multi-ton machines that can kill people" is a frequently uttered statement at work.
mcguire 6y ago
Good for you to realize that, then. On the other hand, I'm not entirely sure it's appropriate to be comfortable with that kind of position in any case.

jancsika 6y ago

I found this comment from the article fascinating: > I am a physician who did a computer science degree before medical school. I frequently use the Therac-25 incident as an example of why we need dual experts who are trained in both fields. I must add two small points to this fantastic summary. > 1. The shadow of the Therac-25 is much longer than those who remember it. In my opinion, this incident set medical informatics back 20 years. Throughout the 80s and 90s there was just a feeling in medicine that computers were dangerous, even if the individual physicians didn't know why. This is why, when I was a resident in 2002-2006 we still were writing all of our orders and notes on paper. It wasn't until the US federal government slammed down the hammer in the mid 2000's and said no payment unless you adopt electronic health records, that computers made real inroads into clinical medicine. > 2. The medical profession, and the government agencies that regulate it, are accustomed to risk and have systems to manage it. The problem is that classical medicine is tuned to "continuous risks." If the Risk of 100 mg of aspirin is "1 risk unit" and the risk of 200 mg of aspirin is "2 risk units" then the risk of 150 mg of aspirin is strongly likely to be between 1 and 2, and it definitely won't be 1,000,000. The mechanisms we use to regulate medicine, with dosing trials, and pharmacokinetic studies, and so forth are based on this assumption that both benefit and harm are continuous functions of prescribed dose, and the physician's job is to find the sweet spot between them. > When you let a computer handle a treatment you are exposed to a completely different kind of risk. Computers are inherently binary machines that we sometimes make simulate continuous functions. Because computers are binary, there is a potential for corner cases that expose erratic, and as this case shows, potentially fatal behavior. This is not new to computer science, but it is very foreign to medicine. Because of this, medicine has a built in blind spot in evaluating computer technology.

mcguire 6y ago
I'm not sure I buy that. Or, well, I suppose that those in the medical field believe it, but I don't think they're right. Consider something like a surgeon nicking and artery while performing some routine surgery, the patient not responding normally to anesthesia or the anesthetist not getting the mixture right and the patient not coming back the way they went in. Or that subset of patients that have poor responses to a vaccine. Everybody likes to think of the world as a linear system, but it's not.

at_a_remove 6y ago

I rather randomly met a woman with a similar sort of background and trajectory as I have: trained in physics, got sucked into computers via the brain drain. She programmed the models for radiation dosing in the metaphorical descendants of Therac-25. I asked her just how often it was brought up in her work and she mentioned that she trained under someone who was in the original group of people brought in to analyze and understand just what happened with Therac-25. Fascinating stuff.

ZuLuuuuuu 6y ago

This is one of the infamous incidents where software failure caused harm on humans. I am kind of fascinated about such incidents (since I learn so much by reading about them), are there any other examples of such incidents that you guys know of? It doesn't have to result in harming human, but any software failure related incident which resulted in big consequences. One another example that comes to my mind is the Toyota "unintended acceleration" incident. Or the "Mars Climate Orbiter" incident.

crocal 6y ago
Google Ariane 501. Enjoy.
probably_wrong 6y ago
If big consequences is what you're after, I can think of three typical incidents: the "most expensive hyphen in history" (you can search it like that), it's companion piece, the Mars Climate Orbiter(which I see you added now), and the Denver Airport Baggage System fiasco [1] where bad software planning caused over $500M in delays. [1] http://calleam.com/WTPF/?page_id=2086 http://calleam.com/WTPF/?page_id=2086

FiatLuxDave 6y ago

I find it interesting how often the Therac 25 is mentioned on HN (thanks to Dan for the list), but nobody ever mentions that those kind of problems never entirely went away. Therac 25 is just the famous one. You don't have to go back to 1986, there are definitely examples from this century. The root causes are somewhat different, and somewhat the same. But no one seems to be teaching these more modern cases to aspiring programmers in school, at least not to the level where every programmer I know has heard of them. For example, the issue which caused this in 2007: https://www.heraldtribune.com/article/LK/20100124/News/605200351/SH https://www.heraldtribune.com/article/LK/20100124/News/60520... Or the process issues which caused this in 2001: https://www.fda.gov/radiation-emitting-products/alerts-and-notices/fda-statement-radiation-overexposures-panama https://www.fda.gov/radiation-emitting-products/alerts-and-n...

sebmellen 6y ago
I quote this from the article so people may take an interest in reading it. This is the opening paragraph: > As Scott Jerome-Parks lay dying, he clung to this wish: that his fatal radiation overdose -- which left him deaf, struggling to see, unable to swallow, burned, with his teeth falling out, with ulcers in his mouth and throat, nauseated, in severe pain and finally unable to breathe -- be studied and talked about publicly so that others might not have to live his nightmare.
Shivetya 6y ago
To this day I run into development managers who will dismiss an easily replicated issue on the grounds it won't happen enough, though of course with a big group of users it actually does, to warrant fixing. While this can be true at times and one excuse is always, we have support to fix the data, it tends to a standard by which all fixes are handled. which is, once you start blowing off testing, having exceptions to avoid every fix, you rarely if ever leave that mode and end up with a stressed support staff and users in the field who think your coders are incompetent.
ahepp 6y ago
The original NYT article for the 2007 incident has more details about the problematic software: https://www.nytimes.com/2010/01/24/health/24radiation.html https://www.nytimes.com/2010/01/24/health/24radiation.html

nickdothutton 6y ago

I mentioned this incident in passing on my post of a few years ago. We studied it many years ago in the early 90s at university. https://blog.eutopian.io/the-age-of-invisible-disasters/ https://blog.eutopian.io/the-age-of-invisible-disasters/

autoditype 6y ago

One of my university professors taught us this incidet, along with a number of other failures, as an ethics lesson. It shook me and I think of it whenever I see automated medical machines, self-driving cars, etc

ericsoderstrom 6y ago

This was (is?) used as a case study in MIT's system's engineering course (6.033)

KMag 6y ago
I was in Course 2 (Mechanical Engineering), but 6.004 and 6.033 were two of my favorite classes at MIT. As I remember, we spent 2 lectures covering many mistakes made in the Therac-25. We also covered the Ariane 5's first launch failure. We also covered a bunch of systems that made good design trade-offs and discussed their design spaces.

sn41 6y ago

The Therac-25 incident, and Bhopal Gas Tragedy were definitely routine lessons in courses on industrial safety and reliability. (Another reason not to turn your nose up against academic coursework - they do impart useful knowledge.) I feel that lot of manual errors and UI errors can be mitigated by careful design. Punishing natural careless errors is not very effective, as recent research shows [1]: [1] https://www.jstage.jst.go.jp/article/jjsca/32/7/32_954/_article/-char/en https://www.jstage.jst.go.jp/article/jjsca/32/7/32_954/_arti...

anotheraccount9 6y ago

I used to work in an ICU. The software that controlled the patient's cardiac monitor would not allow us to modify the alarm settings (sensitivity, volume, etc.) I'm not sure if the settings were locked as a default by the manufacturer, or a protected setting that no one knew how to change, or simply locked by management. Anyhow, from mild atrial fibrillation to irregular rhythm, the alarm was constantly beeping, which was to be expected, considering the acuity of our patients. Nurses and MD became really annoyed by this. Eventually, one of the staff covered the speaker from inside, to muffle the almost constant warnings. The staff repeatedly asked management to find a solution to this, and the associated risk. One day, a patient had a life threatening cardiac incident. The alarm was muffled enough that no one nearby could hear a thing.

sleepy_keita 6y ago

I recently watched this YouTube video about the Therac-25, it was similar to this article (albeit a little less in-depth). https://www.youtube.com/watch?v=-7gVqBY52MY https://www.youtube.com/watch?v=-7gVqBY52MY

AdamJacobMuller 6y ago
Hah! We both posted the EXACT same video at the exact same time! Plainly Difficult is an awesome channel.
- sleepy_keita 6y ago
  Haha! It definitely is. I've been working through their backlog off and on for the past few weeks.

AdamJacobMuller 6y ago

Plainly Difficult has an episode on the Therac-25: https://www.youtube.com/watch?v=-7gVqBY52MY https://www.youtube.com/watch?v=-7gVqBY52MY Highly recommend all of his videos, actually.

gww 6y ago
Great channel. I found it a few months ago and have watched all of his videos. I like that he provides enough scientific detail without making it too hard to follow.

lmilcin 6y ago

As a professional developer interested in extreme reliability, I have studied this and similar stories. I have studied high profile software flops, project failures as well as NTSB reports for aircraft and train accidents, reports of buildings failing etc. and find all those reports very illuminating in the sense of a kind of mindset that is necessary to ensure reliability. I am not working on safety-critical systems but still, the failure of the type of backend systems I work on can bring huge damage to the company. You can come out with a set of rules that can really improve the reliability of your application. * Ensure the system always moves between known, valid states. Errors can still be valid states if you plan for them correctly. I like to compare this to logical induction: if you start with correct state and on every operation ensure that if it executes on correct state it must result in correct state, then you guarantee that the system is always in correct state. * Ensure incorrect operations are impossible to execute. As an example, if you want to guard against large data loss, you can choose to not expose ability to remove data. In my current project we took away everybody's write access to the database, we removed all functions that can remove the data from the database and exchanged them with functions that either mask data (setting flags so that application ignores it) or move to archive. If you need to run an operation on database, you need to submit a pull request and go through Code Review. There is additional review before merge where the team goes through checklist to verify the change meets certain standards. * Ensure anything the client can throw at the application does not have a chance to escalate to other transactions. All inputs are limited and validated mercilessly, all queries are constructed so that it is possible to predict resources necessary to execute the query. Every type of query has a limit on how many queries of that type can be executed in parallel or in a unit of time and how much results it can return. All functionality is constructed in such a way that the performance per transaction does not degrade when load increases (for example by amortizing costs and batching work). Then every functionality is load tested and based on testing results it is certified with maximum load that it can process. Everything over that limit is preemptively declined. * If an algorithm or a solution cannot be guaranteed to work correctly it cannot be used. For example, if library cannot be guaranteed to use at most X amount of memory for the operation, then it is not suitable. * Every single failure of the system is investigated. No system is without faults. The only way to build reliable system is to not ignore errors and always try to figure out underlying failure in the process that let the fault happen. Most projects deal with errors only after the error is urgent enough or only to fix the immediate cause of the error, rather than the failure in the process. If you are interested in very reliable software, when you have it fail for whatever reason you need to look at all stages of your process and identify all places that were supposed to prevent the problem but did not.