8 ms·

Worst Computer Bugs in History: Therac-25 (2017)

301 points by dangom 8y ago

dangom 8y ago
A descriptive video can be found here: https://www.youtube.com/watch?v=uEvu2PlDhO0 https://www.youtube.com/watch?v=uEvu2PlDhO0
- ysleepy 8y ago
  That video is terrible, read the article instead if possible.
  - dangom 8y ago
    Here's the link of (an updated version of) the original accident report: http://sunnyday.mit.edu/papers/therac.pdf http://sunnyday.mit.edu/papers/therac.pdf
LeoPanthera 8y ago
How horrible it must have been for the operator, to realize they had killed two patients, through no fault of their own.
- jsjohnst 8y ago
  Honestly, I disagree slightly. Reading the article as well as the original report years ago, I wasn’t left with the feeling the operator made “no fault of their own”. Are they to blame at all, no, but the operator certainly made mistakes. For example, assuming an error is innocuous when you are intentionally delivering radiation to a person is careless at best. Again, the machine is at fault solely, but that doesn’t mean the operator didn’t have a role in the death.
  - nv-vn 8y ago
    Yeah, I took away the same thing. As an example, in the aviation industry something like this would simply not be tolerated. When you are operating a potentially dangerous device, you have to do so with the utmost care. This isn't to say the technician should be punished, but one of the results of this investigation should have been a focus on making technicians aware of how disastrous the consequences could be if they don't respond appropriately to an error.
    - hexane360 8y ago
      Really? I don't think it's reasonable to expect nurses to wait an undocumented 8 seconds after changing modes to avoid a race condition. That goes far past "utmost care". Are pilots expected to never overlap command inputs? Are they allowed to engage the flaps and then activate the spoilers before the flaps are fully deployed? I'm basing my account on this report as well as the OP: https://hackaday.com/2015/10/26/killed-by-a-machine-the-therac-25/ https://hackaday.com/2015/10/26/killed-by-a-machine-the-ther...
      - AnssiH 8y ago
        I'm pretty sure the parent only meant that the "Malfunction 54" error should not have been ignored, not that the operator should have somehow avoided the race condition in the first place.
        Jtsummers 8y ago
        The operators had become conditioned to ignore those error/warning statements due to their pervasiveness and apparent lack of consequence. This is why, as a designer, you should use such warnings sparingly so that the operator/user doesn’t become “blind” to them.
        jsjohnst 8y ago
        Still doesn’t negate the fact the technician should’ve still checked. Again, not saying the technician is at fault at all, but they aren’t free of being involved in the death either. I’ll use a terrible analogy to make the point, if someone tells you “that’s bad” if you pull the trigger on a revolver playing Russian roulette and you pull the trigger five times without any apparent consequence despite being informed “don’t do that” each time, are you completely not responsible if you pull the trigger the sixth time and die?
- ergothus 8y ago
  > Due to the frequency at which other malfunctions occurred, and that “treatment pause” typically indicated a low-priority issue, the technician resumed treatment. I found this the most telling sentence...here the UX was so terrible the operator routinely chose to break the rules... but on some occasions it killed people
hedora 8y ago
The crazy thing about this classic story is that the industry has learned nothing from it: The lethal bugs were all in the frontend UI code. Today, companies build equally important UI logic in JS frameworks that target rapid prototyping and consumer-focused startups.
- pjmlp 8y ago
  Lawful punishment of bad quality software needs to be a thing, just like in other industries. Only then will most companies actually start to care about software quality in their development processes.
  - zzzcpan 8y ago
    I don't think it works, at least not within the current legal system. Where it becomes mostly about legal bureaucracy of avoiding responsibility, rather than truly focusing on reliability.
    - pjmlp 8y ago
      Sure it does, it is no different than when a company delivers spoiled goods or when one does returns at a shop because the product does not work as described on the box. The root problem is that society got used to turn off/on and hope for the best instead of going back to the shop and ask for their money back. Also every time that there is an bunch of black hat hackers that expose company internal data, if the security breach can be mapped into a CVE database entry, a good law firm could probably make something out of it. Not all jurisdictions are alike, but one needs to start somewhere.
      - anonuser123456 8y ago
        Probably not. As a company you just disclaim liability in your terms of service. Jurisdictions that try and override this, simply get excluded from the customer base. The market is still the ultimate decider for quality; if you build a crappy product, expect to get innovated out.
classichasclass 8y ago
Therac is one of the reasons I get nervous about "health hacking." Yes, people can verifiably benefit from some of the advancements made in this movement, like the DIY diabetic insulin pump, and yes, I prefer to see such advancements be open source than locked up in proprietary designs and trade secrets. And there probably is room in health regulation for trimming the red tape anyway even for innovations originating from the commercial sector. On the other hand, when corners are cut (no hardware interlocks, for example) and edge cases aren't considered, even innocently, then you get things like this. It makes products more expensive to design and more costly to buy and maintain to do the extra engineering. It is certainly a barrier to entry, too. But do we want another case like this because people said "this is good enough"?
- jacquesm 8y ago
  I posted a link about a insulin pump that could be hacked into remotely. Even if it isn't 'health hacking' because you've bought a product from what you thought was a reputable vendor there are no guarantees that it will be secure and bug free.
  - matheusmoreira 8y ago
    https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3262727/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3262727/ > An unauthorized third party can interfere with pump communication and undermine patient safety > we confirmed this through laboratory experiments by sending commands to an insulin pump using an unauthorized remote programmer at a distance of 100 ft > Thus, the specifically identified issues are a security breach that could result in: > (1) changing already-issued wireless pump commands; > (2) generating unauthorized wireless pump commands; > (3) remotely changing the software or settings on the device; > (4) denying communication with the pump device. People can also attack the blood glucose monitors and the data they report to the pump system. Scary.
- Digital-Citizen 8y ago
  I'd say Brad Kuhn got it right in http://ebb.org/bkuhn/blog/2016/08/13/does-not-kill.html http://ebb.org/bkuhn/blog/2016/08/13/does-not-kill.html including "[Y]our proprietary software has killed people, both quickly and slowly, and your attacks on GPLv3 and software freedom are not only unwarranted, they are clearly part of a political strategy to divert attention from your own industry's bad behavior and graft unfair blame onto FLOSS".
topkai22 8y ago
As terrible as it was, that Therac-25 remains one of the most frequently cited examples of software engineering flaws hurting people is somewhat encouraging for the profession. 3 deaths is a tragedy, but the Hyatt bridge collapse a year earlier was a couple of orders of magnitude worse (114 people, https://en.m.wikipedia.org/wiki/Hyatt_Regency_walkway_collapse https://en.m.wikipedia.org/wiki/Hyatt_Regency_walkway_collap...) from what was also a fairly subtle engineering failure. IMO, harm from software bugs (so far) have been vastly surpassed by explicit choices in system design. The various emission cheat scandals have almost certainly taken a real toll on human life going into the hundreds of person lives. More subtly, the choices to retain data inappropriately at Ashley Madison (probably) lead directly to suicides and serious emotional harm. Those are just the two recent examples that spring to mind as a practocing developer, not an ethicist. To somewhat over simplify but when discussing engineering ethics, the harm from software developers building things wrong is swamped by building the wrong things.
- btrettel 8y ago
  Mechanical engineer here. I don't think the Hyatt Regency bridge collapse was caused by a subtle problem. The design change should be obviously bad to any practicing civil engineer. Unfortunately far too many engineers don't perform even basic sanity checks. I'd say a better engineering culture would have caught the problem. Things like this are why I am becoming more and more into testing. Of course, as you have said, building the wrong thing swamps other harms. Unintended consequences are hard to predict, unfortunately, but I am interested in ways to improve this situation. Standards design also interests me, particularly standards which are hard to cheat.
  - azernik 8y ago
    The number one way to prevent building the wrong thing is a professional code of ethics, which software engineers (at least in the US) do not yet have.
    - btrettel 8y ago
      Do you say that to give poor software engineering legal consequences?
yc-kraln 8y ago
The Therac-25 is part of the core curriculum in computer engineering, but I wonder if it's actually (in the grand scheme of things) hat bad of an incident. Compared with Facebook fomenting ethnic cleansing in Asia, the people who were hurt or died were very limited. Are there any new(er) examples which can show the dangers of a widely distributed, connected horror?
- fpgaminer 8y ago
  I'm not familiar with the Facebook incident you are referring to, but it doesn't sound like something caused by a bug? The article is about bugs; i.e. unintentional tragedies. If what you're referring to is not the result of a bug then it's off-topic (though perhaps not unimportant).
- aaron695 8y ago
  The fact everyone knows it shows how safe computing is! Except how about the program that allocated shifts to doctors somewhere and needless put someone on who was tired.... so someone dies. Therac-25's biggest bug that kills is the fact it distracts morons from the real issues.
userbinator 8y ago
I read this article, and many years ago the full report, and one of the omissions on the list of causes that stood out to me was overcomplexity --- if you read about the possible functions of the machine, they really don't require multiple threads much less a full multitasking OS. None of these race conditions would've occurred if it was a simple single-threaded embedded controller. To paraphrase an old Hoare quote, software can either be so simple it obviously contains no bugs, or so complex that it contains no obvious bugs.
dangom 8y ago
According to the Wikipedia entry on the Therac-25, it was "In response to incidents like those that the IEC 62304 standard was created, which introduces development life cycle standards for medical device software and specific guidance on using software of unknown pedigree". For those working in safety and quality control of medical systems, how much does compliance to those specifications actually diminish the chances of another Therac-25 incident? Considering that automation continues to increase, from automatic patient table positioning up to diagnose-assisted AI, are there new challenges when it comes to designing medical systems in order to keep them safe and maintainable? How likely is it for the FDA or the equivalent agencies around the globe to authorize the use of open source systems?
- pjmlp 8y ago
  > How likely is it for the FDA or the equivalent agencies around the globe to authorize the use of open source systems? Actually they already authorize stuff like Qt. Computer systems where human lives are put in risk belong to what is called High Integrity Computing. There are very strict coding standards, where even C looks more like Ada than proper C. https://ldra.com/medical/ https://ldra.com/medical/ https://www.qt.io/qt-in-medical/ https://www.qt.io/qt-in-medical/ https://www.vectorcast.com/testing-solutions/software-testing-embedded-medical-devices-fda-iec-62304 https://www.vectorcast.com/testing-solutions/software-testin... Source code availability is not an issue, because it is part of the certification process to provide it. The problem is having the money to pay for a certification, which becomes invalid the moment anything gets changed, namely compiler being used, source code, or if any of the third party dependencies gets updated.
IronBacon 8y ago
I would guess the other infamous one is the Patriot missile timing bug.
Malic 8y ago
One of my favorite software horror stories is the one of $32 _billion_ overdraft by the Bank of New York. From "Computer-Related Risks" by Peter G. Neumann, published 1994 (REALLY recommended reading) "One of the most dramatic examples was the $32 billion overdraft experienced by the Bank of New York (BoNY) as the result of the overflow of a 16-bit counter that went unchecked. (Most of the other counters were 32-bits wide.) BoNY was unable to process the incoming credits from security transfers, while the New York Federal Reserve automatically debited BoNY's cash account. BoNY had to borrow $24 billion to cover itself for 1 day (until the software was fixed), the interest on which was about $5 million. Many customers were also affected by the delayed transaction completions." Additional reference: https://www.washingtonpost.com/archive/business/1985/12/13/computer-snarled-ny-bank/a707acbe-35bc-4a2e-bd80-180d131618c7/?noredirect=on&utm_term=.96220129dffd https://www.washingtonpost.com/archive/business/1985/12/13/c... Granted, no one died because of this but ... wow ... that was bad day for some developers somewhere.
- jobigoud 8y ago
  Imaging being called in. Ok guys we don't know what the problem is but it's costing the company $3500 per minute the bug stays unfixed just in interests. No pressure.
  - joering2 8y ago
    Also the follow up question should be ask: was an engineer who stopped the leak and fix the bug awarded to some reasonable point??
    - dasil003 8y ago
      No because it was also caused by an engineer, so this could lead to terrifying incentives.
      - sidlls 8y ago
        How is this different from rewarding a salesman who rescues a sale that a different salesman had botched?
dev_dull 8y ago
> The software consisted of several routines running concurrently. Both the Data Entry and Keyboard Handler routines shared a single variable, which recorded whether the technician had completed entering commands. Raise your hand if you've made this type of mistake many times in the past. Most of us have the luxury of not having our software bugs affect human lives. It's a shame they've removed the hardware safety controls. I don't think I'd even feel comfortable programming such a powerful tool without such circuit breakers.
matheusmoreira 8y ago
> users will ignore cryptic error messages, particularly if they occur often It's not just cryptic error messages. Pretty much anything that requires the attention of people will end up being ignored eventually. For example: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4894506/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4894506/ I've also read about an anesthesiologist who turned off the alarms because they annoyed him. One day he failed to secure the endotracheal tube during a surgery, it came off and nobody noticed. The result was cardiac arrest, brain damage, multiple organ failure, sepsis and death. Monitoring hardware is very sensitive so it will fire off alarms if anything changes, no matter how small. The more sensitive a test is, the more false positives you get. This is extremely demanding of a health care professional's attention, which in practice is multiplexed between countless patients. Up to 99% of these alarms and messages will do nothing but get in the way of people. These represent false positives, disconnected cables, and other minor failures that don't represent a real danger and can be easily fixed. People will get used to the alarms, and will learn to ignore them.
- jibal 8y ago
  > Pretty much anything that requires the attention of people will end up being ignored eventually. Perhaps, but your example is not supporting evidence. The PACU alarms were muted, precisely because they were so hard to ignore. > Monitoring hardware is very sensitive so it will fire off alarms if anything changes, no matter how small. The more sensitive a test is, the more false positives you get. This is fixable. The problem is the same as the one in the Therac-25 case ... severe and inconsequential alerts are indistinguishable. Here's an particularly enjoyable piece of literature crafted around an instance of alarm fatigue: https://gutenberg.ca/ebooks/smithcordwainer-deadladyofclowntown/smithcordwainer-deadladyofclowntown-00-h.html https://gutenberg.ca/ebooks/smithcordwainer-deadladyofclownt...
jibal 8y ago
In 1973 I programmed for the oncology department at L.A. County/US Medical Center. We had a Varian Clinac linear accelerator with computer-readable and -drivable motors. The clinicians would manually position the Clinac (with the patient on it) for the first treatment, and the position would be saved in the patient's computerized file and restored on subsequent treatments. For some treatments, a metal wedge would be placed within the beam to attenuate it more at the thick end of the wedge. Because of the non-linear attenuation along the length of the physical metal wedge, dosages were difficult to calculate. Someone got the bright idea of creating a software wedge by slowly moving the treatment couch at the same time as closing the beam aperture, so that there would be 100% exposure at one end of the "wedge" and 0% at the other, with a linearly decreasing distribution across the whole wedge. I was the programmer for this project, and we had just started testing it with a sheet of X-ray film on the couch when I received an offer I couldn't refuse to go work elsewhere. I'm glad that I departed before they started using this on live patients.
tfolbrecht 8y ago
A floating point bug [0] in the Patriot Missile Defense System killed 28 Americans and injured nearly 100 more when an Iraqi SCUD wasn't successfully countered during the Gulf War. [0] https://www.cs.drexel.edu/~introcs/Fa10/notes/07.1_FloatingPoint/Patriot.html?CurrentSlide=10 https://www.cs.drexel.edu/~introcs/Fa10/notes/07.1_FloatingP...
- deleted 8y ago
  [deleted]
binbag 8y ago
A lot of comments here seem to think having a hardware failsafe to backup the software failsafes is the key thing. In fact it doesn't matter about it being hardware. Hardware can fail too. The keys are 1) redundancy and 2) having multiple different failure modes. Adding hardware failsafe gives you both these, but you get the same level of safety by introducing any second failsafe that used a different method, including a separate software based technique as long as its failure mode is dissimilar and uncorrelated to the first. The best method is to add multiple different methods based on completely different technologies.

301 points by dangom 8y ago

A descriptive video can be found here: https://www.youtube.com/watch?v=uEvu2PlDhO0 https://www.youtube.com/watch?v=uEvu2PlDhO0

ysleepy 8y ago
That video is terrible, read the article instead if possible.
- dangom 8y ago
  Here's the link of (an updated version of) the original accident report: http://sunnyday.mit.edu/papers/therac.pdf http://sunnyday.mit.edu/papers/therac.pdf

LeoPanthera 8y ago

How horrible it must have been for the operator, to realize they had killed two patients, through no fault of their own.

jsjohnst 8y ago
Honestly, I disagree slightly. Reading the article as well as the original report years ago, I wasn’t left with the feeling the operator made “no fault of their own”. Are they to blame at all, no, but the operator certainly made mistakes. For example, assuming an error is innocuous when you are intentionally delivering radiation to a person is careless at best. Again, the machine is at fault solely, but that doesn’t mean the operator didn’t have a role in the death.
- nv-vn 8y ago
  Yeah, I took away the same thing. As an example, in the aviation industry something like this would simply not be tolerated. When you are operating a potentially dangerous device, you have to do so with the utmost care. This isn't to say the technician should be punished, but one of the results of this investigation should have been a focus on making technicians aware of how disastrous the consequences could be if they don't respond appropriately to an error.
  - hexane360 8y ago
    Really? I don't think it's reasonable to expect nurses to wait an undocumented 8 seconds after changing modes to avoid a race condition. That goes far past "utmost care". Are pilots expected to never overlap command inputs? Are they allowed to engage the flaps and then activate the spoilers before the flaps are fully deployed? I'm basing my account on this report as well as the OP: https://hackaday.com/2015/10/26/killed-by-a-machine-the-therac-25/ https://hackaday.com/2015/10/26/killed-by-a-machine-the-ther...
    - AnssiH 8y ago
      I'm pretty sure the parent only meant that the "Malfunction 54" error should not have been ignored, not that the operator should have somehow avoided the race condition in the first place.
      - Jtsummers 8y ago
        The operators had become conditioned to ignore those error/warning statements due to their pervasiveness and apparent lack of consequence. This is why, as a designer, you should use such warnings sparingly so that the operator/user doesn’t become “blind” to them.
        jsjohnst 8y ago
        Still doesn’t negate the fact the technician should’ve still checked. Again, not saying the technician is at fault at all, but they aren’t free of being involved in the death either. I’ll use a terrible analogy to make the point, if someone tells you “that’s bad” if you pull the trigger on a revolver playing Russian roulette and you pull the trigger five times without any apparent consequence despite being informed “don’t do that” each time, are you completely not responsible if you pull the trigger the sixth time and die?
ergothus 8y ago
> Due to the frequency at which other malfunctions occurred, and that “treatment pause” typically indicated a low-priority issue, the technician resumed treatment. I found this the most telling sentence...here the UX was so terrible the operator routinely chose to break the rules... but on some occasions it killed people

hedora 8y ago

The crazy thing about this classic story is that the industry has learned nothing from it: The lethal bugs were all in the frontend UI code. Today, companies build equally important UI logic in JS frameworks that target rapid prototyping and consumer-focused startups.

pjmlp 8y ago
Lawful punishment of bad quality software needs to be a thing, just like in other industries. Only then will most companies actually start to care about software quality in their development processes.
- zzzcpan 8y ago
  I don't think it works, at least not within the current legal system. Where it becomes mostly about legal bureaucracy of avoiding responsibility, rather than truly focusing on reliability.
  - pjmlp 8y ago
    Sure it does, it is no different than when a company delivers spoiled goods or when one does returns at a shop because the product does not work as described on the box. The root problem is that society got used to turn off/on and hope for the best instead of going back to the shop and ask for their money back. Also every time that there is an bunch of black hat hackers that expose company internal data, if the security breach can be mapped into a CVE database entry, a good law firm could probably make something out of it. Not all jurisdictions are alike, but one needs to start somewhere.
    - anonuser123456 8y ago
      Probably not. As a company you just disclaim liability in your terms of service. Jurisdictions that try and override this, simply get excluded from the customer base. The market is still the ultimate decider for quality; if you build a crappy product, expect to get innovated out.

classichasclass 8y ago

Therac is one of the reasons I get nervous about "health hacking." Yes, people can verifiably benefit from some of the advancements made in this movement, like the DIY diabetic insulin pump, and yes, I prefer to see such advancements be open source than locked up in proprietary designs and trade secrets. And there probably is room in health regulation for trimming the red tape anyway even for innovations originating from the commercial sector. On the other hand, when corners are cut (no hardware interlocks, for example) and edge cases aren't considered, even innocently, then you get things like this. It makes products more expensive to design and more costly to buy and maintain to do the extra engineering. It is certainly a barrier to entry, too. But do we want another case like this because people said "this is good enough"?

jacquesm 8y ago
I posted a link about a insulin pump that could be hacked into remotely. Even if it isn't 'health hacking' because you've bought a product from what you thought was a reputable vendor there are no guarantees that it will be secure and bug free.
- matheusmoreira 8y ago
  https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3262727/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3262727/ > An unauthorized third party can interfere with pump communication and undermine patient safety > we confirmed this through laboratory experiments by sending commands to an insulin pump using an unauthorized remote programmer at a distance of 100 ft > Thus, the specifically identified issues are a security breach that could result in: > (1) changing already-issued wireless pump commands; > (2) generating unauthorized wireless pump commands; > (3) remotely changing the software or settings on the device; > (4) denying communication with the pump device. People can also attack the blood glucose monitors and the data they report to the pump system. Scary.
Digital-Citizen 8y ago
I'd say Brad Kuhn got it right in http://ebb.org/bkuhn/blog/2016/08/13/does-not-kill.html http://ebb.org/bkuhn/blog/2016/08/13/does-not-kill.html including "[Y]our proprietary software has killed people, both quickly and slowly, and your attacks on GPLv3 and software freedom are not only unwarranted, they are clearly part of a political strategy to divert attention from your own industry's bad behavior and graft unfair blame onto FLOSS".

topkai22 8y ago

As terrible as it was, that Therac-25 remains one of the most frequently cited examples of software engineering flaws hurting people is somewhat encouraging for the profession. 3 deaths is a tragedy, but the Hyatt bridge collapse a year earlier was a couple of orders of magnitude worse (114 people, https://en.m.wikipedia.org/wiki/Hyatt_Regency_walkway_collapse https://en.m.wikipedia.org/wiki/Hyatt_Regency_walkway_collap...) from what was also a fairly subtle engineering failure. IMO, harm from software bugs (so far) have been vastly surpassed by explicit choices in system design. The various emission cheat scandals have almost certainly taken a real toll on human life going into the hundreds of person lives. More subtly, the choices to retain data inappropriately at Ashley Madison (probably) lead directly to suicides and serious emotional harm. Those are just the two recent examples that spring to mind as a practocing developer, not an ethicist. To somewhat over simplify but when discussing engineering ethics, the harm from software developers building things wrong is swamped by building the wrong things.

btrettel 8y ago
Mechanical engineer here. I don't think the Hyatt Regency bridge collapse was caused by a subtle problem. The design change should be obviously bad to any practicing civil engineer. Unfortunately far too many engineers don't perform even basic sanity checks. I'd say a better engineering culture would have caught the problem. Things like this are why I am becoming more and more into testing. Of course, as you have said, building the wrong thing swamps other harms. Unintended consequences are hard to predict, unfortunately, but I am interested in ways to improve this situation. Standards design also interests me, particularly standards which are hard to cheat.
- azernik 8y ago
  The number one way to prevent building the wrong thing is a professional code of ethics, which software engineers (at least in the US) do not yet have.
  - btrettel 8y ago
    Do you say that to give poor software engineering legal consequences?

yc-kraln 8y ago

The Therac-25 is part of the core curriculum in computer engineering, but I wonder if it's actually (in the grand scheme of things) hat bad of an incident. Compared with Facebook fomenting ethnic cleansing in Asia, the people who were hurt or died were very limited. Are there any new(er) examples which can show the dangers of a widely distributed, connected horror?

fpgaminer 8y ago
I'm not familiar with the Facebook incident you are referring to, but it doesn't sound like something caused by a bug? The article is about bugs; i.e. unintentional tragedies. If what you're referring to is not the result of a bug then it's off-topic (though perhaps not unimportant).
aaron695 8y ago
The fact everyone knows it shows how safe computing is! Except how about the program that allocated shifts to doctors somewhere and needless put someone on who was tired.... so someone dies. Therac-25's biggest bug that kills is the fact it distracts morons from the real issues.

userbinator 8y ago

I read this article, and many years ago the full report, and one of the omissions on the list of causes that stood out to me was overcomplexity --- if you read about the possible functions of the machine, they really don't require multiple threads much less a full multitasking OS. None of these race conditions would've occurred if it was a simple single-threaded embedded controller. To paraphrase an old Hoare quote, software can either be so simple it obviously contains no bugs, or so complex that it contains no obvious bugs.

dangom 8y ago

According to the Wikipedia entry on the Therac-25, it was "In response to incidents like those that the IEC 62304 standard was created, which introduces development life cycle standards for medical device software and specific guidance on using software of unknown pedigree". For those working in safety and quality control of medical systems, how much does compliance to those specifications actually diminish the chances of another Therac-25 incident? Considering that automation continues to increase, from automatic patient table positioning up to diagnose-assisted AI, are there new challenges when it comes to designing medical systems in order to keep them safe and maintainable? How likely is it for the FDA or the equivalent agencies around the globe to authorize the use of open source systems?

pjmlp 8y ago
> How likely is it for the FDA or the equivalent agencies around the globe to authorize the use of open source systems? Actually they already authorize stuff like Qt. Computer systems where human lives are put in risk belong to what is called High Integrity Computing. There are very strict coding standards, where even C looks more like Ada than proper C. https://ldra.com/medical/ https://ldra.com/medical/ https://www.qt.io/qt-in-medical/ https://www.qt.io/qt-in-medical/ https://www.vectorcast.com/testing-solutions/software-testing-embedded-medical-devices-fda-iec-62304 https://www.vectorcast.com/testing-solutions/software-testin... Source code availability is not an issue, because it is part of the certification process to provide it. The problem is having the money to pay for a certification, which becomes invalid the moment anything gets changed, namely compiler being used, source code, or if any of the third party dependencies gets updated.

IronBacon 8y ago

I would guess the other infamous one is the Patriot missile timing bug.

Malic 8y ago

One of my favorite software horror stories is the one of $32 _billion_ overdraft by the Bank of New York. From "Computer-Related Risks" by Peter G. Neumann, published 1994 (REALLY recommended reading) "One of the most dramatic examples was the $32 billion overdraft experienced by the Bank of New York (BoNY) as the result of the overflow of a 16-bit counter that went unchecked. (Most of the other counters were 32-bits wide.) BoNY was unable to process the incoming credits from security transfers, while the New York Federal Reserve automatically debited BoNY's cash account. BoNY had to borrow $24 billion to cover itself for 1 day (until the software was fixed), the interest on which was about $5 million. Many customers were also affected by the delayed transaction completions." Additional reference: https://www.washingtonpost.com/archive/business/1985/12/13/computer-snarled-ny-bank/a707acbe-35bc-4a2e-bd80-180d131618c7/?noredirect=on&utm_term=.96220129dffd https://www.washingtonpost.com/archive/business/1985/12/13/c... Granted, no one died because of this but ... wow ... that was bad day for some developers somewhere.

jobigoud 8y ago
Imaging being called in. Ok guys we don't know what the problem is but it's costing the company $3500 per minute the bug stays unfixed just in interests. No pressure.
- joering2 8y ago
  Also the follow up question should be ask: was an engineer who stopped the leak and fix the bug awarded to some reasonable point??
  - dasil003 8y ago
    No because it was also caused by an engineer, so this could lead to terrifying incentives.
    - sidlls 8y ago
      How is this different from rewarding a salesman who rescues a sale that a different salesman had botched?

dev_dull 8y ago

> The software consisted of several routines running concurrently. Both the Data Entry and Keyboard Handler routines shared a single variable, which recorded whether the technician had completed entering commands. Raise your hand if you've made this type of mistake many times in the past. Most of us have the luxury of not having our software bugs affect human lives. It's a shame they've removed the hardware safety controls. I don't think I'd even feel comfortable programming such a powerful tool without such circuit breakers.

matheusmoreira 8y ago

> users will ignore cryptic error messages, particularly if they occur often It's not just cryptic error messages. Pretty much anything that requires the attention of people will end up being ignored eventually. For example: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4894506/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4894506/ I've also read about an anesthesiologist who turned off the alarms because they annoyed him. One day he failed to secure the endotracheal tube during a surgery, it came off and nobody noticed. The result was cardiac arrest, brain damage, multiple organ failure, sepsis and death. Monitoring hardware is very sensitive so it will fire off alarms if anything changes, no matter how small. The more sensitive a test is, the more false positives you get. This is extremely demanding of a health care professional's attention, which in practice is multiplexed between countless patients. Up to 99% of these alarms and messages will do nothing but get in the way of people. These represent false positives, disconnected cables, and other minor failures that don't represent a real danger and can be easily fixed. People will get used to the alarms, and will learn to ignore them.

jibal 8y ago
> Pretty much anything that requires the attention of people will end up being ignored eventually. Perhaps, but your example is not supporting evidence. The PACU alarms were muted, precisely because they were so hard to ignore. > Monitoring hardware is very sensitive so it will fire off alarms if anything changes, no matter how small. The more sensitive a test is, the more false positives you get. This is fixable. The problem is the same as the one in the Therac-25 case ... severe and inconsequential alerts are indistinguishable. Here's an particularly enjoyable piece of literature crafted around an instance of alarm fatigue: https://gutenberg.ca/ebooks/smithcordwainer-deadladyofclowntown/smithcordwainer-deadladyofclowntown-00-h.html https://gutenberg.ca/ebooks/smithcordwainer-deadladyofclownt...

jibal 8y ago

In 1973 I programmed for the oncology department at L.A. County/US Medical Center. We had a Varian Clinac linear accelerator with computer-readable and -drivable motors. The clinicians would manually position the Clinac (with the patient on it) for the first treatment, and the position would be saved in the patient's computerized file and restored on subsequent treatments. For some treatments, a metal wedge would be placed within the beam to attenuate it more at the thick end of the wedge. Because of the non-linear attenuation along the length of the physical metal wedge, dosages were difficult to calculate. Someone got the bright idea of creating a software wedge by slowly moving the treatment couch at the same time as closing the beam aperture, so that there would be 100% exposure at one end of the "wedge" and 0% at the other, with a linearly decreasing distribution across the whole wedge. I was the programmer for this project, and we had just started testing it with a sheet of X-ray film on the couch when I received an offer I couldn't refuse to go work elsewhere. I'm glad that I departed before they started using this on live patients.

tfolbrecht 8y ago

A floating point bug [0] in the Patriot Missile Defense System killed 28 Americans and injured nearly 100 more when an Iraqi SCUD wasn't successfully countered during the Gulf War. [0] https://www.cs.drexel.edu/~introcs/Fa10/notes/07.1_FloatingPoint/Patriot.html?CurrentSlide=10 https://www.cs.drexel.edu/~introcs/Fa10/notes/07.1_FloatingP...

deleted 8y ago
[deleted]

binbag 8y ago

A lot of comments here seem to think having a hardware failsafe to backup the software failsafes is the key thing. In fact it doesn't matter about it being hardware. Hardware can fail too. The keys are 1) redundancy and 2) having multiple different failure modes. Adding hardware failsafe gives you both these, but you get the same level of safety by introducing any second failsafe that used a different method, including a separate software based technique as long as its failure mode is dissimilar and uncorrelated to the first. The best method is to add multiple different methods based on completely different technologies.