5 ms·

Therac-25: When software reliability really does matter

23 points by almost 17y ago

kkowalcz 17y ago
That's exactly why you shouldn't employ "good enough" programmers - always try to recruit the best possible. It takes time, but tools like http://codility.com http://codility.com might help with the elimination of wrong candidates.
- jrockway 17y ago
  You should employ good programmers, but even good programmers make mistakes. It's even better when you give good programmers good tools. Computers don't make mistakes, so it's nice when one will catch yours.
- Silhouette 17y ago
  Are you connected with Codility in any way?
  - decode 17y ago
    He's only the CEO: http://uk.linkedin.com/in/krzysztofkowalczyk http://uk.linkedin.com/in/krzysztofkowalczyk
  - sshumaker 17y ago
    He's the CEO of Codility, according to his linkedin.
  - Silhouette 17y ago
    I just took their sample test to see what it was like. I was disappointed that they disallow the use of external libraries, but then criticise C++ code that is vulnerable to the obvious integer overflow. In a real interview, I would expect a good candidate to indicate that they would pull in a bignum library to avoid this, but there is no way to say that in this sort of test. Likewise, depending on the exact specifications, you might consider using a 64-bit integer type to hold the sum, but that's not portable yet. Presumably, if you're limited to standard C++ (which has no bignum library) and required to pass the overflow test to get a top score, the only way to do this is to reimplement the relevant parts of a bignum type. Of course, that is exactly the wrong thing to do if you have to solve this sort of problem in reality. It is particularly unfortunate that they would criticise a candidate's code for this, while at the same time making a basic mistake themselves in the specification for the equi function. (If, as the problem states, there may be many elements in the input, then the correct return type for an index into a std::vector<int> would be a std::vector<int>::size_type, not an int. A size_type is always unsigned, so returning -1 in the event of finding no match is a bad idea.)
  - csmeder 17y ago
    Hmmm it doesn't seem to understand the < symbol? Compiler output: wrapper.java:23: not a statement for(int i =0; i++; i<A.length()) ^
URSpider94 17y ago
I agree that good software engineering practices are hugely important. However, there are also supposed to be additional failsafes in the system, namely trained operators and supervisory personnel, many of whom are falling down on the job as well. Even worse, in many cases, the government does not compel public disclosure of accidents, which certainly hinders investigation of recurring failure modes. For more, see http://www.nytimes.com/2010/01/27/us/27radiation.html http://www.nytimes.com/2010/01/27/us/27radiation.html
jrockway 17y ago
The industry learned its lesson. Instead of continuing to use unsafe languages like C, they switched to safe, statically-typed languages. The tiny speed hit that checking memory bounds before accessing arrays required turned out to not be a big deal, and the improved programmer productivity more than made up for the need for slightly-more-expensive hardware. And, of course, the number of lives saved was more valuable than a few extra microseconds of execution time and the larger EEPROM needed to store the smarter runtime. Oh sorry, that was me daydreaming. Of course nothing has changed, people just "try to be more careful". Since people never make mistakes, even when pressed with tight deadlines, this has worked out fine. Asking the computer to double-check your code is just not necessary. (Hmm, I think I got that backwards. It's just not my day today, is it...)
- jordanb 17y ago
  While some of the newer, more exotic technologies may show some promise, reality is that the use of C is an Industry Best Practice that is Well Tested and has been Proven over many decades. So while perhaps, someday, some of the exotic technologies may become Standard, for now, when dealing with such important tasks, a conservative Wait-and-See approach must be taken. Translation: Nobody ever got fired for going with C, even when it leaves a stiff on the table.
- gamble 17y ago
  The Therac-25 was much more of a management and engineering failure than a technical problem, though. While the immediate cause of the deaths was a race condition in the software, it was only capable of causing harm because the hardware safety mechanism had been removed as a cost-saving measure, without proper verification that the software was capable of doing the same job. Worse, it was widely known among the Therac's users that the hardware safety would regularly trip during normal operation without apparent cause, yet the problem was never investigated until after the deaths. Ultimately, the Therac was a failure of software engineering more than a software failure. I don't blame the technology any more than I would blame a girder for buckling because a careless structural engineer didn't realize it was overloaded.
  - deleted 17y ago
RK 17y ago
These kinds of problems are still occurring: http://news.ycombinator.com/item?id=887406 http://news.ycombinator.com/item?id=887406 (Oct. 2009)
radiationboy 17y ago
Ho boy, I am due to start radiation treatment next week.
nirmal 17y ago
The Therac-25 is one of the first things Georgia Tech CS students study in our CS Ethics course. If you're interested in more such events see http://measure.cc.gt.atl.ga.us/classes/cs4001_fall_2009/ http://measure.cc.gt.atl.ga.us/classes/cs4001_fall_2009/
jonjacky 17y ago
News of these accidents motivated some of our approach on a more recent radiation therapy machine project. Here's a short paper I wrote about it. http://staff.washington.edu/jon/cnts/iccr.html http://staff.washington.edu/jon/cnts/iccr.html The software we wrote has been used without incident for more than ten years.
grinich 17y ago
Here's the infamous paper by Nancy Leveson: http://courses.cs.vt.edu/cs3604/lib/Therac_25/Therac_1.html http://courses.cs.vt.edu/cs3604/lib/Therac_25/Therac_1.html (original PDF if you have access to the IEEE: http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=274940 http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=...) And an updated PDF: http://sunnyday.mit.edu/papers/therac.pdf http://sunnyday.mit.edu/papers/therac.pdf It's one of the first paper's read in 6.033 (Computer Systems Engineering) at MIT, which is actually taught by rtm.

23 points by almost 17y ago

That's exactly why you shouldn't employ "good enough" programmers - always try to recruit the best possible. It takes time, but tools like http://codility.com http://codility.com might help with the elimination of wrong candidates.

jrockway 17y ago
You should employ good programmers, but even good programmers make mistakes. It's even better when you give good programmers good tools. Computers don't make mistakes, so it's nice when one will catch yours.
Silhouette 17y ago
Are you connected with Codility in any way?
- decode 17y ago
  He's only the CEO: http://uk.linkedin.com/in/krzysztofkowalczyk http://uk.linkedin.com/in/krzysztofkowalczyk
- sshumaker 17y ago
  He's the CEO of Codility, according to his linkedin.
- Silhouette 17y ago
  I just took their sample test to see what it was like. I was disappointed that they disallow the use of external libraries, but then criticise C++ code that is vulnerable to the obvious integer overflow. In a real interview, I would expect a good candidate to indicate that they would pull in a bignum library to avoid this, but there is no way to say that in this sort of test. Likewise, depending on the exact specifications, you might consider using a 64-bit integer type to hold the sum, but that's not portable yet. Presumably, if you're limited to standard C++ (which has no bignum library) and required to pass the overflow test to get a top score, the only way to do this is to reimplement the relevant parts of a bignum type. Of course, that is exactly the wrong thing to do if you have to solve this sort of problem in reality. It is particularly unfortunate that they would criticise a candidate's code for this, while at the same time making a basic mistake themselves in the specification for the equi function. (If, as the problem states, there may be many elements in the input, then the correct return type for an index into a std::vector<int> would be a std::vector<int>::size_type, not an int. A size_type is always unsigned, so returning -1 in the event of finding no match is a bad idea.)
- csmeder 17y ago
  Hmmm it doesn't seem to understand the < symbol? Compiler output: wrapper.java:23: not a statement for(int i =0; i++; i<A.length()) ^

URSpider94 17y ago

I agree that good software engineering practices are hugely important. However, there are also supposed to be additional failsafes in the system, namely trained operators and supervisory personnel, many of whom are falling down on the job as well. Even worse, in many cases, the government does not compel public disclosure of accidents, which certainly hinders investigation of recurring failure modes. For more, see http://www.nytimes.com/2010/01/27/us/27radiation.html http://www.nytimes.com/2010/01/27/us/27radiation.html

jrockway 17y ago

The industry learned its lesson. Instead of continuing to use unsafe languages like C, they switched to safe, statically-typed languages. The tiny speed hit that checking memory bounds before accessing arrays required turned out to not be a big deal, and the improved programmer productivity more than made up for the need for slightly-more-expensive hardware. And, of course, the number of lives saved was more valuable than a few extra microseconds of execution time and the larger EEPROM needed to store the smarter runtime. Oh sorry, that was me daydreaming. Of course nothing has changed, people just "try to be more careful". Since people never make mistakes, even when pressed with tight deadlines, this has worked out fine. Asking the computer to double-check your code is just not necessary. (Hmm, I think I got that backwards. It's just not my day today, is it...)

jordanb 17y ago
While some of the newer, more exotic technologies may show some promise, reality is that the use of C is an Industry Best Practice that is Well Tested and has been Proven over many decades. So while perhaps, someday, some of the exotic technologies may become Standard, for now, when dealing with such important tasks, a conservative Wait-and-See approach must be taken. Translation: Nobody ever got fired for going with C, even when it leaves a stiff on the table.
gamble 17y ago
The Therac-25 was much more of a management and engineering failure than a technical problem, though. While the immediate cause of the deaths was a race condition in the software, it was only capable of causing harm because the hardware safety mechanism had been removed as a cost-saving measure, without proper verification that the software was capable of doing the same job. Worse, it was widely known among the Therac's users that the hardware safety would regularly trip during normal operation without apparent cause, yet the problem was never investigated until after the deaths. Ultimately, the Therac was a failure of software engineering more than a software failure. I don't blame the technology any more than I would blame a girder for buckling because a careless structural engineer didn't realize it was overloaded.
- deleted 17y ago

RK 17y ago

These kinds of problems are still occurring: http://news.ycombinator.com/item?id=887406 http://news.ycombinator.com/item?id=887406 (Oct. 2009)

radiationboy 17y ago

Ho boy, I am due to start radiation treatment next week.

nirmal 17y ago

The Therac-25 is one of the first things Georgia Tech CS students study in our CS Ethics course. If you're interested in more such events see http://measure.cc.gt.atl.ga.us/classes/cs4001_fall_2009/ http://measure.cc.gt.atl.ga.us/classes/cs4001_fall_2009/

jonjacky 17y ago

News of these accidents motivated some of our approach on a more recent radiation therapy machine project. Here's a short paper I wrote about it. http://staff.washington.edu/jon/cnts/iccr.html http://staff.washington.edu/jon/cnts/iccr.html The software we wrote has been used without incident for more than ten years.

grinich 17y ago

Here's the infamous paper by Nancy Leveson: http://courses.cs.vt.edu/cs3604/lib/Therac_25/Therac_1.html http://courses.cs.vt.edu/cs3604/lib/Therac_25/Therac_1.html (original PDF if you have access to the IEEE: http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=274940 http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=...) And an updated PDF: http://sunnyday.mit.edu/papers/therac.pdf http://sunnyday.mit.edu/papers/therac.pdf It's one of the first paper's read in 6.033 (Computer Systems Engineering) at MIT, which is actually taught by rtm.