6 ms·

> The Therac-25 X-ray machine killed several people ... The reason? A race condition. The problem with the Therac-25 was NOT the race condition. Complex device

by pdkl95 8y ago

> The Therac-25 X-ray machine killed several people ... The reason? A race condition. The problem with the Therac-25 was NOT the race condition. Complex devices will always have bugs, because humans haven't figured out a way to do perfect, bug-free engineering. The problem with the Therac-25 was that it wasn't designed to fail safe. The previous model had the same race condition bug, but it also had hardware monitors and interlocks which provided defense in depth. The lesson of the Therac-25 is not writing perfect software; the lesson is recognizing that humans make mistakes so anything remotely safety-critical needs to be designed to fail safely when - not if - mistakes/bugs happen.

_emacsomancer_ 8y ago
This is the Achilles heel of many pieces of software: lack of provisions to fail gracefully.
salawat 8y ago
Not being able to write perfect bug-free software is honestly a myth. Short of bit flipping due to environmental conditions, it's more than possibleto get an algorithm and state machine perfect. Look at the old landline networks. Look at vending machines. It's just HARD. No one wants to pay for HARD. Yes. Therac-25's problem did come from lack of defense-in-depth. That doesn't take away from the fact that Quality was sacrificed in the name of a cheaper production cost, and less time to market. The easiest way to get people to be okay with shoddy things? Tell them it's impossible to do better. Microsoft took that route, so everybody drinks the kool-aid. Even people who should know better.
- Aloha 8y ago
  the old landline network did have notable bugs - sometimes they caused service outages - we never noticed because the network as a whole was designed with enough redundancy and to not fail silently - you could loose many components of the network, and still have it remain functional, every switch for example deployed ran duplex (there were two computers running in lockstep), you could loose chunks of switching fabric within a switch and still have it be more or less functional, though in a degraded state. Almost every end office had more than one route out, to account for equipment or trunk failures. You are correct, bug free software is possible - but often designing stuff to fail safe, is cheaper than catching every bug, and a better use of time. There is a reason we include hardware watchdogs on stuff still, because the watchdog is cheaper than building bug free software, and still provides the same quality of service.
- pdkl95 8y ago
  > Short of bit flipping due to environmental conditions Cosmic rays do flip bits. If you're making anything that might be safety-critical, your design needs to fail safely when hit by a cosmic ray or any other hardware failure. If your device isn't controlling a 25 MeV linac, "failing safely" may be as simple as halting or otherwise indicating an error. If it is controlling something dangerous, multiple layers of defense are necessary. One layer might be error-correcting RAM. > Look at the old landline networks. As Aloha already described, the POTS network is one of the best examples of defense in depth. Every piece was designed to work around failures. They even guaranteed service for weeks after a major disaster that took out the entire electrical grid (they had backup generators and phones were self-powered by the phone line). The phone network is exactly the type of engineering we need for safety and reliability. > Look at vending machines. Vending machines have a simple enough problem that they don't actually need a full stored program computer. You can be (and have been) an electromechanical device implementing a finite state automata. Even if you use software, the necessary state machine is small enough that you can actually prove that the software is correct. Most problems are too complicated. Also, vending machines fail. They even fail safely; a locked up vending machine doesn't become a hazard to life and limb. [1] https://www.youtube.com/watch?v=vhiiia1_hC4 https://www.youtube.com/watch?v=vhiiia1_hC4 [2] https://en.wikipedia.org/wiki/Finite-state_machine https://en.wikipedia.org/wiki/Finite-state_machine
- InclinedPlane 8y ago
  It's impractical by a huge margin. Why? Well, not only because bit flipping does occur naturally due to radiation but also because there's no such thing as a formally proven hardware stack in the world. At the end of the day your software has to run on hardware that exists. Not only has that hardware not been formally proven, you can't even know exactly how the hardware will behave from one month to the next due to firmware patches (spectre/meltdown mitigations, for example). Those patches can affect behavior and performance by huge amounts. Additionally, formal proofs don't cover one of the biggest classes of bugs: insufficient specifications. If you formally prove your software to the wrong specification then you still end up with buggy software. So even with provably correct software methodology you still need classical QA processes. Moreover, it is not necessary to have formally correct algorithms and state machines in order to achieve the maximum quality of software.
- jodrellblank 8y ago
  Tell them it's impossible to do better. Microsoft took that route Citation needed; where and when did they state this? Look at vending machines. Like the one at work which tells me to use exact change only, then accepts more money and still gives me the right change?
  - Piskvorrr 8y ago
    That all-caps section in any Windows EULA since W95 saying essentially "if it breaks in any way it's all your fault"?
    - jodrellblank 8y ago
      The "NO WARRANTIES" and "LIMITATION OF LIABILITY" sections? The ones which appear (to a layperson) to match the MIT license "NO WARRANTY" section, or the Mozilla Public License "indemnify all Contributers" and "disclaimer of warranty", and sections 7, 8 of the Apache License 2, and sections 15, 16 of the GPL v3? If so, there's a lot of people stating "it's impossible to build better software than ours", and while I disagree that's what it communicates, if that is what it means, it's bad-faith to try and single Microsoft out on that count, isn't it?
      - Piskvorrr 8y ago
        I thought it was the canonical example - not that they're the only one.
  - salawat 8y ago
    You know what, give me a few days to do some research if you're genuinely interested. I heard about it from a co-worker who did work with the early telcos way back when. He's passed away, but I never actually looked it up considering that co-worker's role as my mentor and deep insight into computing history that checked out before.
    - jodrellblank 8y ago
      I am mildly interested, to the extent that it's any kind of specific, public Microsoft technical claim, rather than a piece of generic marketing hubris, or a rumour of something said in a closed-doors meeting of C-level people once upon a time. (Re: Marketing disclaimer, e.g. the slogan in the UK "You can't get better than a Kwik-Fit fitter", Kwik-Fit being a high-street chain of budget vehicle repair, I think analogous to something like Jiffy-Lube in the USA, it would be unreasonable for anyone to take this as a factual claim that "no better mechanics exist on the planet", excellent mechanics are more likely to work on racing car engines than routine oil and brake changes for near minimum wage, say)
- Piskvorrr 8y ago
  Bit flipping is low-probability, but coupled to massive amount of events it comes out, Fermi-style, to "quite common, actually": http://dinaburg.org/bitsquatting.html http://dinaburg.org/bitsquatting.html
cryptonector 8y ago
IIRC it wasn't a race condition. It was a poor design. It was not a software bug but a hardware bug: it lacked a sensor for the position of the window. I don't think software could make up for that.
- Jtsummers 8y ago
  http://sunnyday.mit.edu/papers/therac.pdf http://sunnyday.mit.edu/papers/therac.pdf Relevant section: It is clear from the AECL documentation on the modifications that the software allows concurrent access to shared memory, that there is no real synchronization aside from data that are stored in shared variables, and that the "test" and "set" for such variables are not indivisible operations. Race conditions resulting from this implementation of multitasking played an important part in the accidents. There's more in that paper.
Nomentatus 8y ago
Since John Stuart Mill, we've known that "cause" rarely refers to a sufficient condition, and that multiple necessary conditions always apply to real-world events (positive or negative.) We generally reserve "cause" for the necessary condition that is easiest to change. I think it's also legitimate to emphasise general solutions over particular ones, however; as you do.
kartan 8y ago
> The lesson of the Therac-25 is not writing perfect software I will guess that the author will agree with you. The goal is a safe solution and that implies better software, better hardware, better procedures, ... anything that you can learn from mistakes to make this not happen again. If you look at plane crashes reports they will highlight software and hardware malfunction but also pilot mistakes (one plane crash happened when the pilots were distracted talking during landing about another plane crash that happened because those pilots were distracted talking during landing). > The problem with the Therac-25 was NOT the race condition. Complex devices will always have bugs, That is the point of the article. " ... The reason? A race condition. An unplanned-for coincidence, made deadly by the conscious design decision to remove an old-fashioned safety interlock." The problem was the race condition. But if you solve that one, another one will pop-up. So what was deadly was the "decision to remove a safety interlock". We need to identify which one can be solved, and do it. If we can not write perfect software we should add hardware safety. If hardware safety is not enough we need to improve the procedures to handle it. If that is not good enough, we can add more software to help were humans are failing... it is a refining process that you learn studying past failures.

by pdkl95 8y ago

_emacsomancer_ 8y ago

This is the Achilles heel of many pieces of software: lack of provisions to fail gracefully.

salawat 8y ago

Not being able to write perfect bug-free software is honestly a myth. Short of bit flipping due to environmental conditions, it's more than possibleto get an algorithm and state machine perfect. Look at the old landline networks. Look at vending machines. It's just HARD. No one wants to pay for HARD. Yes. Therac-25's problem did come from lack of defense-in-depth. That doesn't take away from the fact that Quality was sacrificed in the name of a cheaper production cost, and less time to market. The easiest way to get people to be okay with shoddy things? Tell them it's impossible to do better. Microsoft took that route, so everybody drinks the kool-aid. Even people who should know better.

Aloha 8y ago
the old landline network did have notable bugs - sometimes they caused service outages - we never noticed because the network as a whole was designed with enough redundancy and to not fail silently - you could loose many components of the network, and still have it remain functional, every switch for example deployed ran duplex (there were two computers running in lockstep), you could loose chunks of switching fabric within a switch and still have it be more or less functional, though in a degraded state. Almost every end office had more than one route out, to account for equipment or trunk failures. You are correct, bug free software is possible - but often designing stuff to fail safe, is cheaper than catching every bug, and a better use of time. There is a reason we include hardware watchdogs on stuff still, because the watchdog is cheaper than building bug free software, and still provides the same quality of service.
pdkl95 8y ago
> Short of bit flipping due to environmental conditions Cosmic rays do flip bits. If you're making anything that might be safety-critical, your design needs to fail safely when hit by a cosmic ray or any other hardware failure. If your device isn't controlling a 25 MeV linac, "failing safely" may be as simple as halting or otherwise indicating an error. If it is controlling something dangerous, multiple layers of defense are necessary. One layer might be error-correcting RAM. > Look at the old landline networks. As Aloha already described, the POTS network is one of the best examples of defense in depth. Every piece was designed to work around failures. They even guaranteed service for weeks after a major disaster that took out the entire electrical grid (they had backup generators and phones were self-powered by the phone line). The phone network is exactly the type of engineering we need for safety and reliability. > Look at vending machines. Vending machines have a simple enough problem that they don't actually need a full stored program computer. You can be (and have been) an electromechanical device implementing a finite state automata. Even if you use software, the necessary state machine is small enough that you can actually prove that the software is correct. Most problems are too complicated. Also, vending machines fail. They even fail safely; a locked up vending machine doesn't become a hazard to life and limb. [1] https://www.youtube.com/watch?v=vhiiia1_hC4 https://www.youtube.com/watch?v=vhiiia1_hC4 [2] https://en.wikipedia.org/wiki/Finite-state_machine https://en.wikipedia.org/wiki/Finite-state_machine
InclinedPlane 8y ago
It's impractical by a huge margin. Why? Well, not only because bit flipping does occur naturally due to radiation but also because there's no such thing as a formally proven hardware stack in the world. At the end of the day your software has to run on hardware that exists. Not only has that hardware not been formally proven, you can't even know exactly how the hardware will behave from one month to the next due to firmware patches (spectre/meltdown mitigations, for example). Those patches can affect behavior and performance by huge amounts. Additionally, formal proofs don't cover one of the biggest classes of bugs: insufficient specifications. If you formally prove your software to the wrong specification then you still end up with buggy software. So even with provably correct software methodology you still need classical QA processes. Moreover, it is not necessary to have formally correct algorithms and state machines in order to achieve the maximum quality of software.
jodrellblank 8y ago
Tell them it's impossible to do better. Microsoft took that route Citation needed; where and when did they state this? Look at vending machines. Like the one at work which tells me to use exact change only, then accepts more money and still gives me the right change?
- Piskvorrr 8y ago
  That all-caps section in any Windows EULA since W95 saying essentially "if it breaks in any way it's all your fault"?
  - jodrellblank 8y ago
    The "NO WARRANTIES" and "LIMITATION OF LIABILITY" sections? The ones which appear (to a layperson) to match the MIT license "NO WARRANTY" section, or the Mozilla Public License "indemnify all Contributers" and "disclaimer of warranty", and sections 7, 8 of the Apache License 2, and sections 15, 16 of the GPL v3? If so, there's a lot of people stating "it's impossible to build better software than ours", and while I disagree that's what it communicates, if that is what it means, it's bad-faith to try and single Microsoft out on that count, isn't it?
    - Piskvorrr 8y ago
      I thought it was the canonical example - not that they're the only one.
- salawat 8y ago
  You know what, give me a few days to do some research if you're genuinely interested. I heard about it from a co-worker who did work with the early telcos way back when. He's passed away, but I never actually looked it up considering that co-worker's role as my mentor and deep insight into computing history that checked out before.
  - jodrellblank 8y ago
    I am mildly interested, to the extent that it's any kind of specific, public Microsoft technical claim, rather than a piece of generic marketing hubris, or a rumour of something said in a closed-doors meeting of C-level people once upon a time. (Re: Marketing disclaimer, e.g. the slogan in the UK "You can't get better than a Kwik-Fit fitter", Kwik-Fit being a high-street chain of budget vehicle repair, I think analogous to something like Jiffy-Lube in the USA, it would be unreasonable for anyone to take this as a factual claim that "no better mechanics exist on the planet", excellent mechanics are more likely to work on racing car engines than routine oil and brake changes for near minimum wage, say)
Piskvorrr 8y ago
Bit flipping is low-probability, but coupled to massive amount of events it comes out, Fermi-style, to "quite common, actually": http://dinaburg.org/bitsquatting.html http://dinaburg.org/bitsquatting.html

cryptonector 8y ago

IIRC it wasn't a race condition. It was a poor design. It was not a software bug but a hardware bug: it lacked a sensor for the position of the window. I don't think software could make up for that.

Jtsummers 8y ago
http://sunnyday.mit.edu/papers/therac.pdf http://sunnyday.mit.edu/papers/therac.pdf Relevant section: It is clear from the AECL documentation on the modifications that the software allows concurrent access to shared memory, that there is no real synchronization aside from data that are stored in shared variables, and that the "test" and "set" for such variables are not indivisible operations. Race conditions resulting from this implementation of multitasking played an important part in the accidents. There's more in that paper.

Nomentatus 8y ago

Since John Stuart Mill, we've known that "cause" rarely refers to a sufficient condition, and that multiple necessary conditions always apply to real-world events (positive or negative.) We generally reserve "cause" for the necessary condition that is easiest to change. I think it's also legitimate to emphasise general solutions over particular ones, however; as you do.

kartan 8y ago

> The lesson of the Therac-25 is not writing perfect software I will guess that the author will agree with you. The goal is a safe solution and that implies better software, better hardware, better procedures, ... anything that you can learn from mistakes to make this not happen again. If you look at plane crashes reports they will highlight software and hardware malfunction but also pilot mistakes (one plane crash happened when the pilots were distracted talking during landing about another plane crash that happened because those pilots were distracted talking during landing). > The problem with the Therac-25 was NOT the race condition. Complex devices will always have bugs, That is the point of the article. " ... The reason? A race condition. An unplanned-for coincidence, made deadly by the conscious design decision to remove an old-fashioned safety interlock." The problem was the race condition. But if you solve that one, another one will pop-up. So what was deadly was the "decision to remove a safety interlock". We need to identify which one can be solved, and do it. If we can not write perfect software we should add hardware safety. If hardware safety is not enough we need to improve the procedures to handle it. If that is not good enough, we can add more software to help were humans are failing... it is a refining process that you learn studying past failures.