10 ms·

Fatal Dose – Radiation Deaths linked to AECL Computer Errors (1994)

96 points by pionerkotik 7y ago

stuaxo 7y ago
This sort of thing is why I don't go through airport scanners.
- JudgeWapner 7y ago
  you don't go through airport scanners in 2019 because of crappy software written 30 years prior?
joering2 7y ago
This article/site keeps coming back to HN often and I cannot stop reading it. It reads like a novel - very sad but very good writing. I hope one day we will see the movie about the issue described.
Ensorceled 7y ago
I worked at a medical imaging company in Toronto in the 80's and 90's. Therac-25 was very much top of mind for all companies in our industry and changed everything from how we did Design, Development and QA to how the FDA reviewed and approved devices running our software.
- matt4077 7y ago
  It was one of the standard examples in my introductory computer science classes, along with the Ariane overflow and the inaccurate system clock on some missile defence system that got a bunch of soldiers killed in Iraq when someone missed the scheduled restart hack.
  - ceejayoz 7y ago
    The missile defense system issue: https://en.wikipedia.org/wiki/MIM-104_Patriot#Failure_at_Dhahran https://en.wikipedia.org/wiki/MIM-104_Patriot#Failure_at_Dha... > The Patriot missile battery at Dhahran had been in operation for 100 hours, by which time the system's internal clock had drifted by one-third of a second. Due to the missile's speed this was equivalent to a miss distance of 600 meters.
  - moioci 7y ago
    Brief summary of the Patriot missile failure in Saudia Arabia in 1991: http://www-users.math.umn.edu/~arnold//disasters/patriot.html http://www-users.math.umn.edu/~arnold//disasters/patriot.htm...
jonawesomegreen 7y ago
I went to an engineering school in Canada for Electrical/Computer Engineering and this was an example of engineering failure we studied in a course about software quality assurance and engineering ethics. Not sure if its widely used as a case study outside of Canada, but it really stuck with me. Especially when designing critical systems. There are two types of reactions to the Therac-25 story. 1) We need to put a lot of process in place to review and test and ensure we have proper interlocks in place. 2) They just didn't have enough talent on the team. It couldn't happen here. You want some people with reaction 1 working on a safety critical system.
- noir_lord 7y ago
  I'd say there are 3 with the third been 3) I'd never work on life critical medical (or anything else) software, I know I fuck up and even the best can (and I'm not). I simply wouldn't want it on my conscience that an easily preventable error (in hindsight) hurt someone.
  - carlmr 7y ago
    If you think like #3 you're one of the engineers that should work on safety critical software. Recognizing that you may produce errors I think is akin to the Dunning-Kruger effect. It shows me you know enough that you don't know enough, which is far preferable to people who don't know they don't know and thus think they're good.
    - LifeLiverTransp 7y ago
      The nightmare employee in this situation is the "rockstar".
  - URSpider94 7y ago
    This is why Quality is such an awesome field. It's all about how a group of smart, skilled, but fundamentally fallible people can build a product with an arbitrarily high level of safety. If it's done properly, no one individual should be able to create a bug that can cause a failure like this.
scandox 7y ago
> The truth is that any software program will probably contain one error for every 500 lines of code. The Therac-25's software program, relatively crude by today's standards, probably contained 101000 lines of code. At one error for every 500 lines, that works out to the possibility of twenty errors. 200 errors I think.
- cr0sh 7y ago
  Is it? Look at the rest of the document. It is almost clear from the errors you can find throughout the document that this "report" was posted from some original hard copy run through some form of OCR. I think this is a particular example of it. Why the number of LOC is 101000? Maybe the OCR interpreted a bad comma from scanning as a "1", in which case the number would have read 10,000 LOC - and thus "twenty errors". As another glaring example of this OCR issue, look for this line: > "There were [.4ECL] people sitting in our offices telling us it [the Therac-25] couldn't hurt anybody when they knew it could." "[.4ECL]"? It should read "[AECL]" - but the scan or the copy it was scanned from probably had a small break on the arm of the "A" - which the OCR interpreted as ".4" You can find other small anomalies like this elsewhere in the document if you look for them. They kinda bugged me, but I could tell from the pixels what was going on. ;) EDIT: My own mistakes fixed...sigh
  - scandox 7y ago
    Good point. The number certainly struck me as a strange approximation!
  - dredmorbius 7y ago
    The author, Barbara Rose Wade, has a website (a copy of story with the same artifacts is there), and a twitter account. Might ask if the has a copy or scan on the original. (I'm not on Twitter.) https://twitter.com/BarbaraWadeRose https://twitter.com/BarbaraWadeRose
ducttape12 7y ago
I remember hearing about this case in college. My professor's point was don't trust software alone. Always include hardware safe guards when it comes to safety.
- cr0sh 7y ago
  Since I started really playing with hardware/software integration (hobby level) a couple of decades ago, and also knowing what I learned from various efforts I've been involved in at employers to "bandaid patch" in security (ie, logins, passwords, access control, etc) - and how it never works out, I've since advocated for the implementation of security controls first, whether implementing a hardware and/or software solution, and one of those controls, if hardware is involved, should be a physical "all-stop" cut-the-power control, manifestly separate from any "software" (whether implemented in hardware logic or otherwise). Design the security/safety system up front, for the task of the system being designed, for all potential failure modes - and then implement the first, and test the hell out of it, and document it thoroughly. Unfortunately, this tends to get shot down, until it is needed, and then it gets bandaid-patched in place, because usually at the point it is needed, a whole refactor of the code base is usually called for, and we can't have that, so "just fix it" becomes the call of duty. And it never works out properly. Things get missed, security is effectively out the window, until something happens, with the refrain of "why did this occur" and of course nobody wants to take the blame for "I told you so". Unfortunately, even implementing such a security system first will only get you so far, and it is still possible that something might be missed, and potentially be the cause of a safety critical issue. But hopefully, most if not all of those extreme events can be caught first in the system, by some part of it, and what's left will be relatively minor.
  - Gibbon1 7y ago
    When I was doing industrial controls in the 80's, I read a paper on the topic. The most important point was safety is a primary design goal. You can't add it in later. Because security and safety concerns have pervasive effects on design decisions.
- IfOnlyYouKnew 7y ago
fitzoh 7y ago
Oh good, the submission title was updated to match the article title and become less useful. The previous title noted that this was bout the Therac-25.
- cr0sh 7y ago
  Hmm - that's not good. That said, once I read the article title, my first thought was "Therac-25"? A quick skim before I read it confirmed that suspicion. ...and I have to add that I'm one of those "informally educated software engineers" - though I've never touched or worked on any kind of safety-critical systems (and I would decline such a position if offered, to be honest). I've been doing SWE for over 25 years now, and the only "formal schooling" I've had in the field was a couple of community college classes I took to learn C/C++ 2+ decades ago. Everything else has either been learned thru my employment, or at home. Prior to my first "professional" position in 1992, all I had was a high school degree and an associates from a trade school in "computer electronics". But even I know about the Therac-25, and it's always in the back of my head. The fairly recent change in the industry surrounding automated testing, test case programming, CI, etc - all of that which more or less fell out of the whole "agile" movement - has led to a vast improvement in software, imho. Yet bugs still persist, despite all of that and more. We have to do better - I don't know how, or if it's even possible (from what I understand, it's not mathematically possible, except in maybe certain trivial examples, to validate all possible states of code - maybe I'm mistaken on that).
- dang 7y ago
  Readers are smart enough to figure that out. It's mentioned in the first paragraph, after all. It's good when not every title is completely obvious. It makes readers work a little and gets the brain out of internet reflex mode. https://hn.algolia.com/?sort=byDate&dateRange=all&type=comment&storyText=false&prefix=false&page=0&query=by:dang%20%22work%20a%20little%22 https://hn.algolia.com/?sort=byDate&dateRange=all&type=comme...
  - dredmorbius 7y ago
    Therac-25 is among the less clickbaity, knee-jerky clarifcations I can think of, FWIW.
dev_dull 7y ago
> The hospital staff, physicist Fritz Hager, and his technician, who had worked the machine in both accidents, stayed at the console long after everybody else had gone home for the weekend, typing and retyping the prescription into the computer console, determined to re-create Malfunction 54. Thank God for the curious minds of this world, the people who have he patience and dedication to reproduce bugs.
- mortb 7y ago
  Yes, it seems that this hospital staff were the testers that AECL should have had themselves. A great tester is curious, imaginative, dedicated, patient and worth a good salary.
lpage 7y ago
Previous discussion of the same article [1] Discussion of another good article on the Therac-25 (with pictures of the machine) [2] [1] https://news.ycombinator.com/item?id=7992548 https://news.ycombinator.com/item?id=7992548 [2] https://news.ycombinator.com/item?id=12201147 https://news.ycombinator.com/item?id=12201147
- dang 7y ago
  Also, 2018: https://news.ycombinator.com/item?id=17740292 https://news.ycombinator.com/item?id=17740292 2015: https://news.ycombinator.com/item?id=9643054 https://news.ycombinator.com/item?id=9643054 2014: https://news.ycombinator.com/item?id=7257005 https://news.ycombinator.com/item?id=7257005 2010: https://news.ycombinator.com/item?id=1143776 https://news.ycombinator.com/item?id=1143776
baybal2 7y ago
Quote: > there have been complaints in the high-tech community that software documentation is hampering competitiveness
scblzn 7y ago
Nearly the same happened in France years ago : https://www.thelocal.fr/20130130/doctors-jailed-over-cancer-radiation-scandal https://www.thelocal.fr/20130130/doctors-jailed-over-cancer-...
hammock 7y ago
I wonder about the increasing use of x-rays in non-medical applications. Industrial applications, or even with the public- like the backscatter machines that TSA and police forces use. Therac-25 seems to be the only story we have where something went wrong - but what are the chances a bad thing could happen, or already happened, in another machine? TSA employees are not permitted to wear dosimeters[1], which makes it hard to detect these things. Scientific American, the most established magazine in the US, is among respected bodies that have recommended dosimeters for accountability reasons.[2] There are also extremely infrequent (and sometimes rigged) studies done on the equipment, and they often find dangerous conditions. For example: >Doses for some of the baggage screeners exceeded the maximum dose for the public.... Some EDS machines were not well maintained (i.e., they had bent curtain rods and missing curtain flaps).... Most EDS machines emitted low levels of radiation; a few exceeded regulatory limits.[2] [1]https://www.tsa.gov/blog/2011/06/30/tsa-cancer-cluster-myth-buster https://www.tsa.gov/blog/2011/06/30/tsa-cancer-cluster-myth-... [2]https://www.scientificamerican.com/article/airport-screeners-to-be-monitored/?redirect=1 https://www.scientificamerican.com/article/airport-screeners... [3]https://www.cdc.gov/niosh/hhe/reports/pdfs/2003-0206-3067.pdf https://www.cdc.gov/niosh/hhe/reports/pdfs/2003-0206-3067.pd...
- toss1 7y ago
  TSA Officers aren't permitted to wear dosimeters on the job. PRetty damning, if you ask me. Here's their fationale from teh FAQ in the linked article [1] "Q: Why aren’t your officers permitted to wear dosimeters? A: There is a really good reason for this. The emissions from our X-ray technology are well below the requirements that would require their routine usage. To help reassure passengers and employees that the technology is safe, however,health physicists with the U.S. Army have been conducting area dosimeter surveys at multiple airports nationwide." This is at best massively flawed reasoning by the TSA, and at worst a deliberate misleading rationale. 1) While the NORMAL dosage is of course well below any level that would make a dosimeter useful, a dose from a malfunction or bent/broken/missing shield panel could easily exceed dosimeter detection thresholds. 2) Spot checks are nice, but unless they are very frequent, they could easily miss a malfunction and leave it overdosing agents and people for a long time. Even if inspections are every 6 months at every airport station, that will leave a malfunction overdosing people for on average, three months. 3) Detecting, identifying, and repairing a malfunctioning system could be done much sooner and at lower cost by using dosimeters, either on employee volunteers or just hanging them on the chairs & workstation surrounds. It is this kind of frankly stupid stick-your-head-in-the-sand reasoning that breeds distrust of governments and the companies that supply them. [1] https://www.tsa.gov/blog/2011/06/30/tsa-cancer-cluster-myth-buster https://www.tsa.gov/blog/2011/06/30/tsa-cancer-cluster-myth-...
tantalor 7y ago
the staff at Princess Margaret Hospital in Toronto had decided to... measure all doses of radiation in the beam and, in a fraction of a second, stop excessive doses before they could reach the patient Astonishing this basic safety precaution was not already in use. Do modern devices do this? I guess nobody thought about Murphy's Law.
- randlet 7y ago
  Modern linear accelerators (linacs) all have an ion chamber in the beam line (2 actually I think) to measure the instantaneous output of the machine.
gregmac 7y ago
> She verified everything else and turned on the beam. The machine stopped and the computer screen flashed "Malfunction 54," a mysterious message not even mentioned in the Therac-25 manual. > [...] the Therac-25 typically issued up to four error messages a day. It did so by displaying "Malfunction" plus a number, from 1 through 64. No explanation was offered by the computer nor was there any reference to the malfunction codes in the operator's manual. Technicians could, in most cases, bypass the irritating malfunctions simply by pressing the "p" key, for "proceed." Doing so became a matter of habit. Even if you're not doing safety-critical systems, there are many lessons to be learned just from this bit alone. Use human-readable error messages, and make sure they are understandable to the users/operators of your software (not just developers). Including a specific error code is fine -- it can disambiguate similar errors, and make it very easy to do a 'find all' in source to go directly to the offending spot -- but that doesn't help your user. If you're going to allow users to bypass an error, make it deliberate. The best way to do this is debatable, but through years of bad software, users have been trained to dismiss popup messages by pressing OK or 'yes' or 'proceed'. When you show an important error, make sure it's different from informational messages or warnings. Showing users routine warnings and messages is fatiguing and basically just trains them to always press the 'p' key. > According to a computer system's analysis of FDA documents, the computer would not accept new information on a particular phase of treatment (in the case of both Tyler accidents, changing the x-ray mode to electron mode) if the technician made the changes within eight seconds after reaching the end of the prescription data. That's what Malfunction 54 meant. Consider if they had actually tried to show an error message for this (or even document what it meant). Something like "Data on screen does not reflect actual parameters being used. Proceed?" Hopefully at some point during development, someone would question that: "why are we allowing them to proceed in this case?" "If we can detect that is the case, why even show this as an error, instead of just updating the parameters?"
- couchand 7y ago
  "why even show this as an error?" is one of my favorite questions. It's the UI equivalent of "make illegal states unrepresentable". > Use human-readable error messages, and make sure they are understandable to the users/operators of your software (not just developers). Including a specific error code is fine -- it can disambiguate similar errors, and make it very easy to do a 'find all' in source to go directly to the offending spot -- but that doesn't help your user. Yes. Remember, however, that much software needs to be localized, even if the initial developer didn't think to. Error codes are much better than hard-coded English-language error messages; map them to a message at the last possible moment.
FiatLuxDave 7y ago
A cobalt machine killed a patient just a few days ago, in Voronezh, Russia. The patient was crushed between the movable treatment table and the collimator. As far as I can tell from the limited information available on the web, the problem was not software but rather a stuck manual switch. Sorry for linking the daily fail, but the only articles I can find in English seem to be in UK tabloids: https://www.dailymail.co.uk/news/article-7066289/Cancer-patient-crushed-death-malfunctioning-radiation-treatment-machine-Russian-hospital.html https://www.dailymail.co.uk/news/article-7066289/Cancer-pati... по-русски: https://dni.ru/regions/2019/5/23/424405.html https://dni.ru/regions/2019/5/23/424405.html
- wolf550e 7y ago
  That's kinda like being killed by jacked up car crushing person working on the car underneath. Nothing to do with radiation or software. With a static device where operator does not position the jack, the cause would be lack of mechanical maintenance or material failure (unlikely).
  - jschwartzi 7y ago
    The failure mode of the switch contributed to this accident, and a switch which does not exhibit this failure mode should have been used if possible. If not possible, then the switch should have been moved through a test actuation before reaching a position where the failure mode causes the patient to be killed. This test actuation should be performed each time the table is moved. Finally, the pathway that the switch passes through should be periodically cleared of debris and the switch tested on a regular basis. If the test actuation fails(indicating switch failure) then the table should reset to a safe position. As an aside, every time I step on the treadmill at the gym I check that the emergency stop button works and that the incline down and speed down buttons work. I have been stuck on a treadmill which I couldn't stop or slow down before.
    - ryacko 7y ago
      That is what the sides of the treadmill are for.
Tistel 7y ago
When I did CS ~20 years ago in Canada this was part of the curriculum (SE class IIRC). I still reference it when I see reckless decisions being made.
deleted 7y ago
[deleted]
mortb 7y ago
I think this story is not as much a story about a software bug as it is a story about off hand accident management. If AECL had earlier been more humble and determined to find the errors maybe they would have? It seems jaw dropingly ignorantly handled. Why did it have to take so much time and several people dead before serious effort was put in to find the issues?

96 points by pionerkotik 7y ago

stuaxo 7y ago

This sort of thing is why I don't go through airport scanners.

JudgeWapner 7y ago
you don't go through airport scanners in 2019 because of crappy software written 30 years prior?

joering2 7y ago

This article/site keeps coming back to HN often and I cannot stop reading it. It reads like a novel - very sad but very good writing. I hope one day we will see the movie about the issue described.

Ensorceled 7y ago

I worked at a medical imaging company in Toronto in the 80's and 90's. Therac-25 was very much top of mind for all companies in our industry and changed everything from how we did Design, Development and QA to how the FDA reviewed and approved devices running our software.

matt4077 7y ago
It was one of the standard examples in my introductory computer science classes, along with the Ariane overflow and the inaccurate system clock on some missile defence system that got a bunch of soldiers killed in Iraq when someone missed the scheduled restart hack.
- ceejayoz 7y ago
  The missile defense system issue: https://en.wikipedia.org/wiki/MIM-104_Patriot#Failure_at_Dhahran https://en.wikipedia.org/wiki/MIM-104_Patriot#Failure_at_Dha... > The Patriot missile battery at Dhahran had been in operation for 100 hours, by which time the system's internal clock had drifted by one-third of a second. Due to the missile's speed this was equivalent to a miss distance of 600 meters.
- moioci 7y ago
  Brief summary of the Patriot missile failure in Saudia Arabia in 1991: http://www-users.math.umn.edu/~arnold//disasters/patriot.html http://www-users.math.umn.edu/~arnold//disasters/patriot.htm...

jonawesomegreen 7y ago

I went to an engineering school in Canada for Electrical/Computer Engineering and this was an example of engineering failure we studied in a course about software quality assurance and engineering ethics. Not sure if its widely used as a case study outside of Canada, but it really stuck with me. Especially when designing critical systems. There are two types of reactions to the Therac-25 story. 1) We need to put a lot of process in place to review and test and ensure we have proper interlocks in place. 2) They just didn't have enough talent on the team. It couldn't happen here. You want some people with reaction 1 working on a safety critical system.

noir_lord 7y ago
I'd say there are 3 with the third been 3) I'd never work on life critical medical (or anything else) software, I know I fuck up and even the best can (and I'm not). I simply wouldn't want it on my conscience that an easily preventable error (in hindsight) hurt someone.
- carlmr 7y ago
  If you think like #3 you're one of the engineers that should work on safety critical software. Recognizing that you may produce errors I think is akin to the Dunning-Kruger effect. It shows me you know enough that you don't know enough, which is far preferable to people who don't know they don't know and thus think they're good.
  - LifeLiverTransp 7y ago
    The nightmare employee in this situation is the "rockstar".
- URSpider94 7y ago
  This is why Quality is such an awesome field. It's all about how a group of smart, skilled, but fundamentally fallible people can build a product with an arbitrarily high level of safety. If it's done properly, no one individual should be able to create a bug that can cause a failure like this.

scandox 7y ago

> The truth is that any software program will probably contain one error for every 500 lines of code. The Therac-25's software program, relatively crude by today's standards, probably contained 101000 lines of code. At one error for every 500 lines, that works out to the possibility of twenty errors. 200 errors I think.

cr0sh 7y ago
Is it? Look at the rest of the document. It is almost clear from the errors you can find throughout the document that this "report" was posted from some original hard copy run through some form of OCR. I think this is a particular example of it. Why the number of LOC is 101000? Maybe the OCR interpreted a bad comma from scanning as a "1", in which case the number would have read 10,000 LOC - and thus "twenty errors". As another glaring example of this OCR issue, look for this line: > "There were [.4ECL] people sitting in our offices telling us it [the Therac-25] couldn't hurt anybody when they knew it could." "[.4ECL]"? It should read "[AECL]" - but the scan or the copy it was scanned from probably had a small break on the arm of the "A" - which the OCR interpreted as ".4" You can find other small anomalies like this elsewhere in the document if you look for them. They kinda bugged me, but I could tell from the pixels what was going on. ;) EDIT: My own mistakes fixed...sigh
- scandox 7y ago
  Good point. The number certainly struck me as a strange approximation!
- dredmorbius 7y ago
  The author, Barbara Rose Wade, has a website (a copy of story with the same artifacts is there), and a twitter account. Might ask if the has a copy or scan on the original. (I'm not on Twitter.) https://twitter.com/BarbaraWadeRose https://twitter.com/BarbaraWadeRose

ducttape12 7y ago

I remember hearing about this case in college. My professor's point was don't trust software alone. Always include hardware safe guards when it comes to safety.

cr0sh 7y ago
Since I started really playing with hardware/software integration (hobby level) a couple of decades ago, and also knowing what I learned from various efforts I've been involved in at employers to "bandaid patch" in security (ie, logins, passwords, access control, etc) - and how it never works out, I've since advocated for the implementation of security controls first, whether implementing a hardware and/or software solution, and one of those controls, if hardware is involved, should be a physical "all-stop" cut-the-power control, manifestly separate from any "software" (whether implemented in hardware logic or otherwise). Design the security/safety system up front, for the task of the system being designed, for all potential failure modes - and then implement the first, and test the hell out of it, and document it thoroughly. Unfortunately, this tends to get shot down, until it is needed, and then it gets bandaid-patched in place, because usually at the point it is needed, a whole refactor of the code base is usually called for, and we can't have that, so "just fix it" becomes the call of duty. And it never works out properly. Things get missed, security is effectively out the window, until something happens, with the refrain of "why did this occur" and of course nobody wants to take the blame for "I told you so". Unfortunately, even implementing such a security system first will only get you so far, and it is still possible that something might be missed, and potentially be the cause of a safety critical issue. But hopefully, most if not all of those extreme events can be caught first in the system, by some part of it, and what's left will be relatively minor.
- Gibbon1 7y ago
  When I was doing industrial controls in the 80's, I read a paper on the topic. The most important point was safety is a primary design goal. You can't add it in later. Because security and safety concerns have pervasive effects on design decisions.
IfOnlyYouKnew 7y ago

fitzoh 7y ago

Oh good, the submission title was updated to match the article title and become less useful. The previous title noted that this was bout the Therac-25.

cr0sh 7y ago
Hmm - that's not good. That said, once I read the article title, my first thought was "Therac-25"? A quick skim before I read it confirmed that suspicion. ...and I have to add that I'm one of those "informally educated software engineers" - though I've never touched or worked on any kind of safety-critical systems (and I would decline such a position if offered, to be honest). I've been doing SWE for over 25 years now, and the only "formal schooling" I've had in the field was a couple of community college classes I took to learn C/C++ 2+ decades ago. Everything else has either been learned thru my employment, or at home. Prior to my first "professional" position in 1992, all I had was a high school degree and an associates from a trade school in "computer electronics". But even I know about the Therac-25, and it's always in the back of my head. The fairly recent change in the industry surrounding automated testing, test case programming, CI, etc - all of that which more or less fell out of the whole "agile" movement - has led to a vast improvement in software, imho. Yet bugs still persist, despite all of that and more. We have to do better - I don't know how, or if it's even possible (from what I understand, it's not mathematically possible, except in maybe certain trivial examples, to validate all possible states of code - maybe I'm mistaken on that).
dang 7y ago
Readers are smart enough to figure that out. It's mentioned in the first paragraph, after all. It's good when not every title is completely obvious. It makes readers work a little and gets the brain out of internet reflex mode. https://hn.algolia.com/?sort=byDate&dateRange=all&type=comment&storyText=false&prefix=false&page=0&query=by:dang%20%22work%20a%20little%22 https://hn.algolia.com/?sort=byDate&dateRange=all&type=comme...
- dredmorbius 7y ago
  Therac-25 is among the less clickbaity, knee-jerky clarifcations I can think of, FWIW.

dev_dull 7y ago

> The hospital staff, physicist Fritz Hager, and his technician, who had worked the machine in both accidents, stayed at the console long after everybody else had gone home for the weekend, typing and retyping the prescription into the computer console, determined to re-create Malfunction 54. Thank God for the curious minds of this world, the people who have he patience and dedication to reproduce bugs.

mortb 7y ago
Yes, it seems that this hospital staff were the testers that AECL should have had themselves. A great tester is curious, imaginative, dedicated, patient and worth a good salary.

lpage 7y ago

Previous discussion of the same article [1] Discussion of another good article on the Therac-25 (with pictures of the machine) [2] [1] https://news.ycombinator.com/item?id=7992548 https://news.ycombinator.com/item?id=7992548 [2] https://news.ycombinator.com/item?id=12201147 https://news.ycombinator.com/item?id=12201147

dang 7y ago
Also, 2018: https://news.ycombinator.com/item?id=17740292 https://news.ycombinator.com/item?id=17740292 2015: https://news.ycombinator.com/item?id=9643054 https://news.ycombinator.com/item?id=9643054 2014: https://news.ycombinator.com/item?id=7257005 https://news.ycombinator.com/item?id=7257005 2010: https://news.ycombinator.com/item?id=1143776 https://news.ycombinator.com/item?id=1143776

baybal2 7y ago

Quote: > there have been complaints in the high-tech community that software documentation is hampering competitiveness

scblzn 7y ago

Nearly the same happened in France years ago : https://www.thelocal.fr/20130130/doctors-jailed-over-cancer-radiation-scandal https://www.thelocal.fr/20130130/doctors-jailed-over-cancer-...

hammock 7y ago

I wonder about the increasing use of x-rays in non-medical applications. Industrial applications, or even with the public- like the backscatter machines that TSA and police forces use. Therac-25 seems to be the only story we have where something went wrong - but what are the chances a bad thing could happen, or already happened, in another machine? TSA employees are not permitted to wear dosimeters[1], which makes it hard to detect these things. Scientific American, the most established magazine in the US, is among respected bodies that have recommended dosimeters for accountability reasons.[2] There are also extremely infrequent (and sometimes rigged) studies done on the equipment, and they often find dangerous conditions. For example: >Doses for some of the baggage screeners exceeded the maximum dose for the public.... Some EDS machines were not well maintained (i.e., they had bent curtain rods and missing curtain flaps).... Most EDS machines emitted low levels of radiation; a few exceeded regulatory limits.[2] [1]https://www.tsa.gov/blog/2011/06/30/tsa-cancer-cluster-myth-buster https://www.tsa.gov/blog/2011/06/30/tsa-cancer-cluster-myth-... [2]https://www.scientificamerican.com/article/airport-screeners-to-be-monitored/?redirect=1 https://www.scientificamerican.com/article/airport-screeners... [3]https://www.cdc.gov/niosh/hhe/reports/pdfs/2003-0206-3067.pdf https://www.cdc.gov/niosh/hhe/reports/pdfs/2003-0206-3067.pd...

toss1 7y ago
TSA Officers aren't permitted to wear dosimeters on the job. PRetty damning, if you ask me. Here's their fationale from teh FAQ in the linked article [1] "Q: Why aren’t your officers permitted to wear dosimeters? A: There is a really good reason for this. The emissions from our X-ray technology are well below the requirements that would require their routine usage. To help reassure passengers and employees that the technology is safe, however,health physicists with the U.S. Army have been conducting area dosimeter surveys at multiple airports nationwide." This is at best massively flawed reasoning by the TSA, and at worst a deliberate misleading rationale. 1) While the NORMAL dosage is of course well below any level that would make a dosimeter useful, a dose from a malfunction or bent/broken/missing shield panel could easily exceed dosimeter detection thresholds. 2) Spot checks are nice, but unless they are very frequent, they could easily miss a malfunction and leave it overdosing agents and people for a long time. Even if inspections are every 6 months at every airport station, that will leave a malfunction overdosing people for on average, three months. 3) Detecting, identifying, and repairing a malfunctioning system could be done much sooner and at lower cost by using dosimeters, either on employee volunteers or just hanging them on the chairs & workstation surrounds. It is this kind of frankly stupid stick-your-head-in-the-sand reasoning that breeds distrust of governments and the companies that supply them. [1] https://www.tsa.gov/blog/2011/06/30/tsa-cancer-cluster-myth-buster https://www.tsa.gov/blog/2011/06/30/tsa-cancer-cluster-myth-...

tantalor 7y ago

the staff at Princess Margaret Hospital in Toronto had decided to... measure all doses of radiation in the beam and, in a fraction of a second, stop excessive doses before they could reach the patient Astonishing this basic safety precaution was not already in use. Do modern devices do this? I guess nobody thought about Murphy's Law.

randlet 7y ago
Modern linear accelerators (linacs) all have an ion chamber in the beam line (2 actually I think) to measure the instantaneous output of the machine.

gregmac 7y ago

> She verified everything else and turned on the beam. The machine stopped and the computer screen flashed "Malfunction 54," a mysterious message not even mentioned in the Therac-25 manual. > [...] the Therac-25 typically issued up to four error messages a day. It did so by displaying "Malfunction" plus a number, from 1 through 64. No explanation was offered by the computer nor was there any reference to the malfunction codes in the operator's manual. Technicians could, in most cases, bypass the irritating malfunctions simply by pressing the "p" key, for "proceed." Doing so became a matter of habit. Even if you're not doing safety-critical systems, there are many lessons to be learned just from this bit alone. Use human-readable error messages, and make sure they are understandable to the users/operators of your software (not just developers). Including a specific error code is fine -- it can disambiguate similar errors, and make it very easy to do a 'find all' in source to go directly to the offending spot -- but that doesn't help your user. If you're going to allow users to bypass an error, make it deliberate. The best way to do this is debatable, but through years of bad software, users have been trained to dismiss popup messages by pressing OK or 'yes' or 'proceed'. When you show an important error, make sure it's different from informational messages or warnings. Showing users routine warnings and messages is fatiguing and basically just trains them to always press the 'p' key. > According to a computer system's analysis of FDA documents, the computer would not accept new information on a particular phase of treatment (in the case of both Tyler accidents, changing the x-ray mode to electron mode) if the technician made the changes within eight seconds after reaching the end of the prescription data. That's what Malfunction 54 meant. Consider if they had actually tried to show an error message for this (or even document what it meant). Something like "Data on screen does not reflect actual parameters being used. Proceed?" Hopefully at some point during development, someone would question that: "why are we allowing them to proceed in this case?" "If we can detect that is the case, why even show this as an error, instead of just updating the parameters?"

couchand 7y ago
"why even show this as an error?" is one of my favorite questions. It's the UI equivalent of "make illegal states unrepresentable". > Use human-readable error messages, and make sure they are understandable to the users/operators of your software (not just developers). Including a specific error code is fine -- it can disambiguate similar errors, and make it very easy to do a 'find all' in source to go directly to the offending spot -- but that doesn't help your user. Yes. Remember, however, that much software needs to be localized, even if the initial developer didn't think to. Error codes are much better than hard-coded English-language error messages; map them to a message at the last possible moment.

FiatLuxDave 7y ago

A cobalt machine killed a patient just a few days ago, in Voronezh, Russia. The patient was crushed between the movable treatment table and the collimator. As far as I can tell from the limited information available on the web, the problem was not software but rather a stuck manual switch. Sorry for linking the daily fail, but the only articles I can find in English seem to be in UK tabloids: https://www.dailymail.co.uk/news/article-7066289/Cancer-patient-crushed-death-malfunctioning-radiation-treatment-machine-Russian-hospital.html https://www.dailymail.co.uk/news/article-7066289/Cancer-pati... по-русски: https://dni.ru/regions/2019/5/23/424405.html https://dni.ru/regions/2019/5/23/424405.html

wolf550e 7y ago
That's kinda like being killed by jacked up car crushing person working on the car underneath. Nothing to do with radiation or software. With a static device where operator does not position the jack, the cause would be lack of mechanical maintenance or material failure (unlikely).
- jschwartzi 7y ago
  The failure mode of the switch contributed to this accident, and a switch which does not exhibit this failure mode should have been used if possible. If not possible, then the switch should have been moved through a test actuation before reaching a position where the failure mode causes the patient to be killed. This test actuation should be performed each time the table is moved. Finally, the pathway that the switch passes through should be periodically cleared of debris and the switch tested on a regular basis. If the test actuation fails(indicating switch failure) then the table should reset to a safe position. As an aside, every time I step on the treadmill at the gym I check that the emergency stop button works and that the incline down and speed down buttons work. I have been stuck on a treadmill which I couldn't stop or slow down before.
  - ryacko 7y ago
    That is what the sides of the treadmill are for.

Tistel 7y ago

When I did CS ~20 years ago in Canada this was part of the curriculum (SE class IIRC). I still reference it when I see reckless decisions being made.

deleted 7y ago

[deleted]

mortb 7y ago

I think this story is not as much a story about a software bug as it is a story about off hand accident management. If AECL had earlier been more humble and determined to find the errors maybe they would have? It seems jaw dropingly ignorantly handled. Why did it have to take so much time and several people dead before serious effort was put in to find the issues?