15 ms·

Therac-25

84 points by markmassie 13y ago

camperman 13y ago
Very sad story. This page introduced me to Nancy Leveson's work a few years back (her paper on the Therac is linked at the bottom of the Wiki page). She's written two excellent books (and a number of papers) on engineering, safety and complex systems which are well worth reading. http://sunnyday.mit.edu/ http://sunnyday.mit.edu/
ygra 13y ago
This was one example given in a very early lecture about software testing and failures in our uni. This (and others) make me hope that I never have that much responsibility over human lives as a developer.
- pjmlp 13y ago
  It is this type of failures that drives me to push for software quality and moving away from languages like C. As well as, consider that liability should be part of the industry as it is in other industries. Sadly the drive for profit speaks otherwise.
  - mortov 13y ago
    The issue is not because of the language used. You can have bugs - potentially fatal as this case underlines - in any language. When I studied the Therac case, there was also a study in the "Killer Robot" - see here : http://www.onlineethics.org/cms/5122.aspx http://www.onlineethics.org/cms/5122.aspx. Well worth a read to understand how software can become dangerous. Sadly nowadays, a search for Killer Robot turns up all sorts of stories of drones being used to kill people. When I first looked up killer robot a number of years ago, the results were pretty much all about the ethics scenario and not real life people being killed. How society has moved on...
    - evincarofautumn 13y ago
      > You can have bugs - potentially fatal as this case underlines - in any language. That is undoubtedly true. However, the commonest classes of bugs can be ruled out by languages with more expressive static verification tools—type systems, effect systems, and so on—leaving us with a stronger baseline from which to work.
    - pjmlp 13y ago
      > The issue is not because of the language used. You can have bugs - potentially fatal as this case underlines - in any language. This is true, however there are languages which allow for more programming errors than others. I just mentioned C as one possible example. The goal should be to have programming languages that reduce programming errors, not that make it easier to shot yourself. Of course this is only a small step towards better software, as whole system design also plays a big role.
    - Piskvorrr 13y ago
      Also relevant (and on HN newest as of now) - Toyota accelerator issue: http://embeddedgurus.com/state-space/2014/02/are-we-shooting-ourselves-in-the-foot-with-stack-overflow/ http://embeddedgurus.com/state-space/2014/02/are-we-shooting...
      - fnordfnordfnord 13y ago
        This one isn't merely an example of a failure of engineering, but also of organizational failure at Toyota, and organizational failure at both the NHTSA and NASA.
  - weland 13y ago
    This is the bug that occurred: > The defect was as follows: a one-byte counter in a testing routine frequently overflowed; if an operator provided manual input to the machine at the precise moment that this counter overflowed, the interlock would fail.[3] Can you describe how exactly a language other than C would have helped with this?
    - pjmlp 13y ago
      Safer system programming languages cause a panic on overflow, instead of happily keep on running. Unless the programmer has explicitly disabled them, that is. Crashing is better that corrupt data.
      - ars 13y ago
        Kinda curious - which languages panic on overflow?
        pjmlp 13y ago
        Ada, here is a small list of validations http://en.wikibooks.org/wiki/Ada_Programming/Pragmas/Suppress http://en.wikibooks.org/wiki/Ada_Programming/Pragmas/Suppres... Modula-2, a possible documentation link http://www.excelsior-usa.com/doc/xds/isom203.html#15 http://www.excelsior-usa.com/doc/xds/isom203.html#15 Just to cite two examples.
        weland 13y ago
        So what happens when the Ada program runs into a failed compiler check? Does it: * Raise an exception? If so, it's up to the programmer to treat it correctly. There are various ways to detect overflows outside Ada; it's still up to the programmer to treat them correctly. Runtime detection of the overflow (in a hopelessly untested firmware of the Therac-25!) would have brought the matter no closer to resolution. * Abort execution? If so, is the behaviour of the program defined if execution is aborted? Because undefined behaviour from a control system is quite worse.
  - troels 13y ago
    It's explicitly stated that the choice of language was not considered a root cause. And that was even asm - not C The software was written in assembly language that might require more attention for testing and good design. However the choice of language by itself is not listed as a primary cause in the report.
    - pjmlp 13y ago
      C was just as example, as it is nothing more than a portable macro assembler.
      - troels 13y ago
        As opposed to being written in actual assembler?
      - ColinWright 13y ago
        This is a common mis-conception invariably produced by people who've never written a C compiler. C is not just a portable macro assembler.
        pjmlp 13y ago
        > This is a common mis-conception invariably produced by people who've never written a C compiler. Wrong on that one. I do have a good background in compiler design. > C is not just a portable macro assembler. Given what it offers when compared with safer systems programming languages, it is one according to my own definition. :)
  - ars 13y ago
    > and moving away from languages like C If you look at the reports of serious failures like this once (and other famous ones), none of them are related to the use of C.
- chiph 13y ago
  Unless your code is the most trivial ever, there's still a tiny possibility that it could result in a death. Game: User goes into epileptic seizure, hits their head and dies. Social media: User calls for help with it - "Please send help #carcrash #bleeding" and the message is lost. As an engineer and a responsible human being, you don't ever want this to happen. But it will. You have to minimize the chances of it happening, with the realization that it will never go to 0.0% The world runs on software today. With all that that implies.
jrockway 13y ago
A couple lessons: Never underestimate the value of a good safety interlock. This is not the correct way to check for integer overflows: if x + 1 > INT_MAX { ... }
- angersock 13y ago
  Hardware interlocks are really important.
tinco 13y ago
In the Computer Science degree I did, every course that was in the Software Engineering or Formal Methods track started with "Why Software Engineering is important", and then a group of very bad software bugs, this one was always one of them. The slides would then be followed by how the professors believed their course would have prevented them. Especially funny was the formal verification course that mentioned the Ariane 5. Apparently all the new software in Ariane 5 was formally modelled and verified, but one part of the system was directly ported from the Ariane 4. Because the Ariane 4 mission had been successful they did not verify that (it's an expensive process). The bug that crashed the rocket involved the fact that the Ariane 4 was 16 bit, and the Ariane 5 was 64 bit, it resulted in an integer overflow somewhere leading to a crash. You can spend millions in painstaking formal verification, and pay for the small part that you did not verify.
- alceta 13y ago
  I opened the comments track to say exactly this. Every lecturer with any connection to software would bring this up. It felt like the bad (as in, people have died from Therac-25 malfunctions) running gag of my CS studies. Other popular bug choices were the Pentium Bug from 1994 or the Mars Climate Orbiter.
- tragomaskhalos 13y ago
  An erstwhile colleague of mine worked on the component that destroyed the Ariane 5; it performed exactly to spec, detecting that the vehicle was out of control - due to the integer overflow (in another, unrelated component nb) - and self-destructing it to prevent it crashing to earth. However explaining this subtlety to a rabid press was a different story, and the whole thing became a bit of a PR disaster
  - diminish 13y ago
    self-destroying components sound interesting. do you have any info/links to which type of technology is used in this component?
    - 13y ago
evincarofautumn 13y ago
This is a tragedy that bears remembering. Many descriptions of it are cold and clinical, and mention the deaths only statistically. The Therac-25 is regularly used as a soapbox for various software engineering disciplines and machine-checked program verification techniques. But it’s worth keeping this in mind: humans suffered and died because of our mistakes. Whether due to underpowered tooling or mere human error, as a field we are collectively at fault. This is not a fucking game.
- wldlyinaccurate 13y ago
  This is precisely why the current "hacker culture" scares the hell out of me. The vast majority of new programmers don't understand the basics of software design; we are entering an era where software is written by whoever will work for the lowest wage.
  - jrockway 13y ago
    I don't think that's necessarily true: software is as reliable as it needs to be. Yup, it's annoying when your favorite blog leaks your password. But since there are no legal consequences for doing that, why should they bother trying to prevent it?
    - joosters 13y ago
      Exactly the attitude that would be great to stamp out: "It won't hurt/cost me, so why should I care?" It's obviously on a far different scale from software causing deaths, I'm not trying to make comparisons between the two. But the motivation of the software developer needs to be a little more moral than just 'will I be in trouble with the law?'
      - jrockway 13y ago
        What I'm saying is that if you're the kind of person that cares, you'll find a higher-paying job using that knowledge for something more important. The kind of programmer that stores passwords in plaintext in a publicly-accessible database is not one that says, "I'm going to do this because it's faster." He or she doesn't even know that that could be a problem. Similarly, those hiring such a person are ignorant of what experience costs, and so they get what they paid for, even if they didn't make the conscious decision to make an insecure website. (You don't see Google or Microsoft or Apple posting ads on freelancer sites for $5/hour jobs. They know the results they want cost more than that.)
primitivesuave 13y ago
Another terrible tragedy caused by a software error: https://en.wikipedia.org/wiki/MIM-104_Patriot#Failure_at_Dhahran https://en.wikipedia.org/wiki/MIM-104_Patriot#Failure_at_Dha... Over 100 hours, the system's internal clock drifted by a third of a second, which was enough to introduce a significant error into the Patriot missile defense system. They updated the software the next day when the error caused 28 soldiers to lose their lives.
- agilebyte 13y ago
  The error contributed, SCUD missile caused.
  - spingsprong 13y ago
    The Patriot missile probably would have failed to destroy the Scud even without the error. The Patriot was designed to shoot down aircraft, but ballistic missiles fly much faster than an aircraft, and the warhead fusing system they had back then was too slow. Most Patriots detonated behind the Scuds they were targeting.
  - deleted 13y ago
    [deleted]
ksrm 13y ago
The investigation makes for fascinating reading: http://courses.cs.vt.edu/cs3604/lib/Therac_25/Therac_1.html http://courses.cs.vt.edu/cs3604/lib/Therac_25/Therac_1.html
bamdadd 13y ago
We have read this story as part of the Parallel and Distributed course in for the MSc. of CS in Manchester university. The whole story with details can be read here : http://sunnyday.mit.edu/papers/therac.pdf http://sunnyday.mit.edu/papers/therac.pdf Very sad story though.
deleted 13y ago
[deleted]
deleted 13y ago
[deleted]
tluyben2 13y ago
With outsourcing of medical software to overseas 'software factories' it still surprises me that we don't hear more about this. It does happen and you do read the stories if you search for them, but it's not big enough news. Anyone knows sources for this kind of news? I only know http://www.reddit.com/r/criticalsoftware/ http://www.reddit.com/r/criticalsoftware/
- josephlord 13y ago
  RISKs Digest http://catless.ncl.ac.uk/Risks http://catless.ncl.ac.uk/Risks
jbb555 13y ago
Formal verification is all very well, but I find in practice that a lot of the worst problems are specification problems "of course it wasn't meant to do that"
undoware 13y ago
band name. Called it.
- adultSwim 13y ago
  Too soon
joezydeco 13y ago
Just FYI, comp.risks (and Peter G. Neumann's moderation) live on at http://catless.ncl.ac.uk/Risks/ http://catless.ncl.ac.uk/Risks/
aaronem 13y ago
I don't disagree in the slightest that the software had a deadly bug, or that that bug shouldn't have been allowed to make it into production. But I am surprised no one in this thread has yet identified the real source of the problem. From Wikipedia: > The accidents occurred when the high-power electron beam was activated instead of the intended low power beam, and without the beam spreader plate rotated into place. Previous models had hardware interlocks in place to prevent this, but Therac-25 had removed them, depending instead on software interlocks for safety. I am just a self-taught programmer, and certainly not any kind of trained engineer. But when there's something your machine must not be able to do, and it's possible to design it such that it physically cannot do that thing, what possible excuse can there be for not doing so?
- NickNameNick 13y ago
  That's definitely poor engineering, the exact kind of thing we were told not to do when I took engineering classes.
perlgeek 13y ago
Here's a podcast about the Therac-25 and software safety: http://disastercast.co.uk/episode-13-therac-25-and-software-safety/ http://disastercast.co.uk/episode-13-therac-25-and-software-... I found it quite interesting.

84 points by markmassie 13y ago

Very sad story. This page introduced me to Nancy Leveson's work a few years back (her paper on the Therac is linked at the bottom of the Wiki page). She's written two excellent books (and a number of papers) on engineering, safety and complex systems which are well worth reading. http://sunnyday.mit.edu/ http://sunnyday.mit.edu/

ygra 13y ago

This was one example given in a very early lecture about software testing and failures in our uni. This (and others) make me hope that I never have that much responsibility over human lives as a developer.

pjmlp 13y ago
It is this type of failures that drives me to push for software quality and moving away from languages like C. As well as, consider that liability should be part of the industry as it is in other industries. Sadly the drive for profit speaks otherwise.
- mortov 13y ago
  The issue is not because of the language used. You can have bugs - potentially fatal as this case underlines - in any language. When I studied the Therac case, there was also a study in the "Killer Robot" - see here : http://www.onlineethics.org/cms/5122.aspx http://www.onlineethics.org/cms/5122.aspx. Well worth a read to understand how software can become dangerous. Sadly nowadays, a search for Killer Robot turns up all sorts of stories of drones being used to kill people. When I first looked up killer robot a number of years ago, the results were pretty much all about the ethics scenario and not real life people being killed. How society has moved on...
  - evincarofautumn 13y ago
    > You can have bugs - potentially fatal as this case underlines - in any language. That is undoubtedly true. However, the commonest classes of bugs can be ruled out by languages with more expressive static verification tools—type systems, effect systems, and so on—leaving us with a stronger baseline from which to work.
  - pjmlp 13y ago
    > The issue is not because of the language used. You can have bugs - potentially fatal as this case underlines - in any language. This is true, however there are languages which allow for more programming errors than others. I just mentioned C as one possible example. The goal should be to have programming languages that reduce programming errors, not that make it easier to shot yourself. Of course this is only a small step towards better software, as whole system design also plays a big role.
  - Piskvorrr 13y ago
    Also relevant (and on HN newest as of now) - Toyota accelerator issue: http://embeddedgurus.com/state-space/2014/02/are-we-shooting-ourselves-in-the-foot-with-stack-overflow/ http://embeddedgurus.com/state-space/2014/02/are-we-shooting...
    - fnordfnordfnord 13y ago
      This one isn't merely an example of a failure of engineering, but also of organizational failure at Toyota, and organizational failure at both the NHTSA and NASA.
- weland 13y ago
  This is the bug that occurred: > The defect was as follows: a one-byte counter in a testing routine frequently overflowed; if an operator provided manual input to the machine at the precise moment that this counter overflowed, the interlock would fail.[3] Can you describe how exactly a language other than C would have helped with this?
  - pjmlp 13y ago
    Safer system programming languages cause a panic on overflow, instead of happily keep on running. Unless the programmer has explicitly disabled them, that is. Crashing is better that corrupt data.
    - ars 13y ago
      Kinda curious - which languages panic on overflow?
      - pjmlp 13y ago
        Ada, here is a small list of validations http://en.wikibooks.org/wiki/Ada_Programming/Pragmas/Suppress http://en.wikibooks.org/wiki/Ada_Programming/Pragmas/Suppres... Modula-2, a possible documentation link http://www.excelsior-usa.com/doc/xds/isom203.html#15 http://www.excelsior-usa.com/doc/xds/isom203.html#15 Just to cite two examples.
        weland 13y ago
        So what happens when the Ada program runs into a failed compiler check? Does it: * Raise an exception? If so, it's up to the programmer to treat it correctly. There are various ways to detect overflows outside Ada; it's still up to the programmer to treat them correctly. Runtime detection of the overflow (in a hopelessly untested firmware of the Therac-25!) would have brought the matter no closer to resolution. * Abort execution? If so, is the behaviour of the program defined if execution is aborted? Because undefined behaviour from a control system is quite worse.
- troels 13y ago
  It's explicitly stated that the choice of language was not considered a root cause. And that was even asm - not C The software was written in assembly language that might require more attention for testing and good design. However the choice of language by itself is not listed as a primary cause in the report.
  - pjmlp 13y ago
    C was just as example, as it is nothing more than a portable macro assembler.
    - troels 13y ago
      As opposed to being written in actual assembler?
    - ColinWright 13y ago
      This is a common mis-conception invariably produced by people who've never written a C compiler. C is not just a portable macro assembler.
      - pjmlp 13y ago
        > This is a common mis-conception invariably produced by people who've never written a C compiler. Wrong on that one. I do have a good background in compiler design. > C is not just a portable macro assembler. Given what it offers when compared with safer systems programming languages, it is one according to my own definition. :)
- ars 13y ago
  > and moving away from languages like C If you look at the reports of serious failures like this once (and other famous ones), none of them are related to the use of C.
chiph 13y ago
Unless your code is the most trivial ever, there's still a tiny possibility that it could result in a death. Game: User goes into epileptic seizure, hits their head and dies. Social media: User calls for help with it - "Please send help #carcrash #bleeding" and the message is lost. As an engineer and a responsible human being, you don't ever want this to happen. But it will. You have to minimize the chances of it happening, with the realization that it will never go to 0.0% The world runs on software today. With all that that implies.

jrockway 13y ago

A couple lessons: Never underestimate the value of a good safety interlock. This is not the correct way to check for integer overflows: if x + 1 > INT_MAX { ... }

angersock 13y ago
Hardware interlocks are really important.

tinco 13y ago

In the Computer Science degree I did, every course that was in the Software Engineering or Formal Methods track started with "Why Software Engineering is important", and then a group of very bad software bugs, this one was always one of them. The slides would then be followed by how the professors believed their course would have prevented them. Especially funny was the formal verification course that mentioned the Ariane 5. Apparently all the new software in Ariane 5 was formally modelled and verified, but one part of the system was directly ported from the Ariane 4. Because the Ariane 4 mission had been successful they did not verify that (it's an expensive process). The bug that crashed the rocket involved the fact that the Ariane 4 was 16 bit, and the Ariane 5 was 64 bit, it resulted in an integer overflow somewhere leading to a crash. You can spend millions in painstaking formal verification, and pay for the small part that you did not verify.

alceta 13y ago
I opened the comments track to say exactly this. Every lecturer with any connection to software would bring this up. It felt like the bad (as in, people have died from Therac-25 malfunctions) running gag of my CS studies. Other popular bug choices were the Pentium Bug from 1994 or the Mars Climate Orbiter.
tragomaskhalos 13y ago
An erstwhile colleague of mine worked on the component that destroyed the Ariane 5; it performed exactly to spec, detecting that the vehicle was out of control - due to the integer overflow (in another, unrelated component nb) - and self-destructing it to prevent it crashing to earth. However explaining this subtlety to a rabid press was a different story, and the whole thing became a bit of a PR disaster
- diminish 13y ago
  self-destroying components sound interesting. do you have any info/links to which type of technology is used in this component?
  - 13y ago

evincarofautumn 13y ago

This is a tragedy that bears remembering. Many descriptions of it are cold and clinical, and mention the deaths only statistically. The Therac-25 is regularly used as a soapbox for various software engineering disciplines and machine-checked program verification techniques. But it’s worth keeping this in mind: humans suffered and died because of our mistakes. Whether due to underpowered tooling or mere human error, as a field we are collectively at fault. This is not a fucking game.

wldlyinaccurate 13y ago
This is precisely why the current "hacker culture" scares the hell out of me. The vast majority of new programmers don't understand the basics of software design; we are entering an era where software is written by whoever will work for the lowest wage.
- jrockway 13y ago
  I don't think that's necessarily true: software is as reliable as it needs to be. Yup, it's annoying when your favorite blog leaks your password. But since there are no legal consequences for doing that, why should they bother trying to prevent it?
  - joosters 13y ago
    Exactly the attitude that would be great to stamp out: "It won't hurt/cost me, so why should I care?" It's obviously on a far different scale from software causing deaths, I'm not trying to make comparisons between the two. But the motivation of the software developer needs to be a little more moral than just 'will I be in trouble with the law?'
    - jrockway 13y ago
      What I'm saying is that if you're the kind of person that cares, you'll find a higher-paying job using that knowledge for something more important. The kind of programmer that stores passwords in plaintext in a publicly-accessible database is not one that says, "I'm going to do this because it's faster." He or she doesn't even know that that could be a problem. Similarly, those hiring such a person are ignorant of what experience costs, and so they get what they paid for, even if they didn't make the conscious decision to make an insecure website. (You don't see Google or Microsoft or Apple posting ads on freelancer sites for $5/hour jobs. They know the results they want cost more than that.)

primitivesuave 13y ago

Another terrible tragedy caused by a software error: https://en.wikipedia.org/wiki/MIM-104_Patriot#Failure_at_Dhahran https://en.wikipedia.org/wiki/MIM-104_Patriot#Failure_at_Dha... Over 100 hours, the system's internal clock drifted by a third of a second, which was enough to introduce a significant error into the Patriot missile defense system. They updated the software the next day when the error caused 28 soldiers to lose their lives.

agilebyte 13y ago
The error contributed, SCUD missile caused.
- spingsprong 13y ago
  The Patriot missile probably would have failed to destroy the Scud even without the error. The Patriot was designed to shoot down aircraft, but ballistic missiles fly much faster than an aircraft, and the warhead fusing system they had back then was too slow. Most Patriots detonated behind the Scuds they were targeting.
- deleted 13y ago
  [deleted]

ksrm 13y ago

The investigation makes for fascinating reading: http://courses.cs.vt.edu/cs3604/lib/Therac_25/Therac_1.html http://courses.cs.vt.edu/cs3604/lib/Therac_25/Therac_1.html

bamdadd 13y ago

We have read this story as part of the Parallel and Distributed course in for the MSc. of CS in Manchester university. The whole story with details can be read here : http://sunnyday.mit.edu/papers/therac.pdf http://sunnyday.mit.edu/papers/therac.pdf Very sad story though.

deleted 13y ago

[deleted]

deleted 13y ago

[deleted]

tluyben2 13y ago

With outsourcing of medical software to overseas 'software factories' it still surprises me that we don't hear more about this. It does happen and you do read the stories if you search for them, but it's not big enough news. Anyone knows sources for this kind of news? I only know http://www.reddit.com/r/criticalsoftware/ http://www.reddit.com/r/criticalsoftware/

josephlord 13y ago
RISKs Digest http://catless.ncl.ac.uk/Risks http://catless.ncl.ac.uk/Risks

jbb555 13y ago

Formal verification is all very well, but I find in practice that a lot of the worst problems are specification problems "of course it wasn't meant to do that"

undoware 13y ago

band name. Called it.

adultSwim 13y ago
Too soon

joezydeco 13y ago

Just FYI, comp.risks (and Peter G. Neumann's moderation) live on at http://catless.ncl.ac.uk/Risks/ http://catless.ncl.ac.uk/Risks/

aaronem 13y ago

I don't disagree in the slightest that the software had a deadly bug, or that that bug shouldn't have been allowed to make it into production. But I am surprised no one in this thread has yet identified the real source of the problem. From Wikipedia: > The accidents occurred when the high-power electron beam was activated instead of the intended low power beam, and without the beam spreader plate rotated into place. Previous models had hardware interlocks in place to prevent this, but Therac-25 had removed them, depending instead on software interlocks for safety. I am just a self-taught programmer, and certainly not any kind of trained engineer. But when there's something your machine must not be able to do, and it's possible to design it such that it physically cannot do that thing, what possible excuse can there be for not doing so?

NickNameNick 13y ago
That's definitely poor engineering, the exact kind of thing we were told not to do when I took engineering classes.

perlgeek 13y ago

Here's a podcast about the Therac-25 and software safety: http://disastercast.co.uk/episode-13-therac-25-and-software-safety/ http://disastercast.co.uk/episode-13-therac-25-and-software-... I found it quite interesting.