4 ms·

This is the bug that occurred: > The defect was as follows: a one-byte counter in a testing routine frequently overflowed; if an operator provided manual input

by weland 13y ago

This is the bug that occurred: > The defect was as follows: a one-byte counter in a testing routine frequently overflowed; if an operator provided manual input to the machine at the precise moment that this counter overflowed, the interlock would fail.[3] Can you describe how exactly a language other than C would have helped with this?

pjmlp 13y ago
Safer system programming languages cause a panic on overflow, instead of happily keep on running. Unless the programmer has explicitly disabled them, that is. Crashing is better that corrupt data.
- ars 13y ago
  Kinda curious - which languages panic on overflow?
  - pjmlp 13y ago
    Ada, here is a small list of validations http://en.wikibooks.org/wiki/Ada_Programming/Pragmas/Suppress http://en.wikibooks.org/wiki/Ada_Programming/Pragmas/Suppres... Modula-2, a possible documentation link http://www.excelsior-usa.com/doc/xds/isom203.html#15 http://www.excelsior-usa.com/doc/xds/isom203.html#15 Just to cite two examples.
    - weland 13y ago
      So what happens when the Ada program runs into a failed compiler check? Does it: * Raise an exception? If so, it's up to the programmer to treat it correctly. There are various ways to detect overflows outside Ada; it's still up to the programmer to treat them correctly. Runtime detection of the overflow (in a hopelessly untested firmware of the Therac-25!) would have brought the matter no closer to resolution. * Abort execution? If so, is the behaviour of the program defined if execution is aborted? Because undefined behaviour from a control system is quite worse.
      - jv2 13y ago
        Normally, raise an Exception; the outermost levels of the program should probably put the machine in a "safe" state if an exception is raised during operation e.g. turn off the radiation beam etc. I would have thought this is infinitely preferable to silently continuing after an error... SPARK Ada allows the use of source-code annotations to denote information flow in a program, and can vastly reduce the chances of these sorts of problems (at the expense of more programmer time). Standard Ada compilers can do a reasonable job of detecting some problems at compile-time, but SPARK seems to be the favoured method for safety-critical stuff.
        weland 13y ago
        > I would have thought this is infinitely preferable to silently continuing after an error... It definitely is! However, it still assumes that the exception is safely handled. The flaw in Therac 25's development process was that testing did not reveal this bug. Consequently, the error recovery process could have itself been incorrect and remain uncaught. It's tempting to think that, if error recovery is similar each time, testing it once is enough, but error recovery is as susceptible to being impeded by race conditions as any other part of the code. No uncertainty is removed this way. It's also worth noting that a value of 0 (to which the counter overflowed) was actually a legitimate value in the Therac-25 program.
    - ars 13y ago
      Any chance of an example from a slightly more recent language? If you are going to say we shouldn't use C because of overflow handling, at least suggest a language that actually does it. Otherwise you haven't really made your case.
      - pjmlp 13y ago
        Ada is a recent language, just last year got a new standard revision. Chances are, when you travel by train or take a plane, the control system was developed in Ada. When human lives are at risk, C is becoming less and less an option. And when the company still goes C, many countries have certifications in place like MISRA, that make C look like Ada with C syntax.
        ars 13y ago
        Fine, I'll grant you Ada, and it does seem like a good language for these kinds of things. But you have to acknowledge it's not going to become a popular language for regular software projects.
        pjmlp 13y ago
        Sure I won't see the a SV startup using it.
        ArkyBeagle 13y ago
        Language choice is secondary to methodology, but Ada does a darn fine job of checking an awful lot of methodologically useful things up front. It's less the language than the ecosystem and the infrastructure. It's eminently possible to develop 'C' that's exhaustively tested - I've done it. I am not sure that is true of C++ or Java.
- andyjohnson0 13y ago
  "Crashing is better that corrupt data" If the crashed program is controlling a device that is directing radiation at a person's body, then crashing is likely to be at least as undesirable as data corruption. I understand your point about programming language choice, and your comments elsewhere about the undesirability of the C language. But the root causes of the Therac-25 malfunction wasn't technical, they were organisational. Specifically poor code review and code reuse practices. Changing the programming language wouldn't fix them.
- jacquesm 13y ago
  > Crashing is better that corrupt data. In embedded control software of medical systems that are directly affecting the lives of the patients involved crashing is definitely not an option and it is not necessarily better than corrupt data. What is an option is to very carefully model the permitted states of the machine and then to exhaustively test those states and their transitions to make sure there are no undefined states the machine can be in. One way to do this is to have a watchdog process (or even an entirely different system) that monitors the state of the machine and that initiates a safe shutdown if any departure of the defined operation is detected.
hrjet 13y ago
A higher level language could have relieved the programmers from low-level details, thus giving them more cognitive space for avoiding logical bugs in other sections of their code. In general, the more DRY the code, the more mind-space for review / testing.
- weland 13y ago
  > A higher level language could have relieved the programmers from low-level details, thus giving them more cognitive space for avoiding logical bugs in other sections of their code. This can also be said of choosing simpler design, organizing the code in a better manner, or could have been avoided through more comprehensive testing -- all of which were pointed out by the official report.
  - hrjet 13y ago
    I am not sure I understand your argument. Using a high-level language doesn't mean one wouldn't follow other best-practices.
    - weland 13y ago
      Having the bug end up in production could have been avoided simply through adequate testing and verification. I fail to see how a higher-level language helps with that. The fact that the programmers might have had less stuff to worry about and, therefore, saved their concentration for more critical parts of the code, is hardly direct evidence that the bug could not have been introduced. It can equally well be argued that a higher-level language would have attracted programmers with less experience in mission-critical code, who would have failed to grasp even the possibility of bugs more subtle than this one.
- JoeAltmaier 13y ago
  Remember,this was a PDP11. Don't know what that was? Well, C WAS its high-level language.
- angersock 13y ago
  In general, the more DRY the code, the more mind-space for review / testing. It's something that would be considered firmware today, written in assembler. Trying to be DRY in assembly is a really great way of introducing subtle bugs.

by weland 13y ago

pjmlp 13y ago

Safer system programming languages cause a panic on overflow, instead of happily keep on running. Unless the programmer has explicitly disabled them, that is. Crashing is better that corrupt data.

ars 13y ago
Kinda curious - which languages panic on overflow?
- pjmlp 13y ago
  Ada, here is a small list of validations http://en.wikibooks.org/wiki/Ada_Programming/Pragmas/Suppress http://en.wikibooks.org/wiki/Ada_Programming/Pragmas/Suppres... Modula-2, a possible documentation link http://www.excelsior-usa.com/doc/xds/isom203.html#15 http://www.excelsior-usa.com/doc/xds/isom203.html#15 Just to cite two examples.
  - weland 13y ago
    So what happens when the Ada program runs into a failed compiler check? Does it: * Raise an exception? If so, it's up to the programmer to treat it correctly. There are various ways to detect overflows outside Ada; it's still up to the programmer to treat them correctly. Runtime detection of the overflow (in a hopelessly untested firmware of the Therac-25!) would have brought the matter no closer to resolution. * Abort execution? If so, is the behaviour of the program defined if execution is aborted? Because undefined behaviour from a control system is quite worse.
    - jv2 13y ago
      Normally, raise an Exception; the outermost levels of the program should probably put the machine in a "safe" state if an exception is raised during operation e.g. turn off the radiation beam etc. I would have thought this is infinitely preferable to silently continuing after an error... SPARK Ada allows the use of source-code annotations to denote information flow in a program, and can vastly reduce the chances of these sorts of problems (at the expense of more programmer time). Standard Ada compilers can do a reasonable job of detecting some problems at compile-time, but SPARK seems to be the favoured method for safety-critical stuff.
      - weland 13y ago
        > I would have thought this is infinitely preferable to silently continuing after an error... It definitely is! However, it still assumes that the exception is safely handled. The flaw in Therac 25's development process was that testing did not reveal this bug. Consequently, the error recovery process could have itself been incorrect and remain uncaught. It's tempting to think that, if error recovery is similar each time, testing it once is enough, but error recovery is as susceptible to being impeded by race conditions as any other part of the code. No uncertainty is removed this way. It's also worth noting that a value of 0 (to which the counter overflowed) was actually a legitimate value in the Therac-25 program.
  - ars 13y ago
    Any chance of an example from a slightly more recent language? If you are going to say we shouldn't use C because of overflow handling, at least suggest a language that actually does it. Otherwise you haven't really made your case.
    - pjmlp 13y ago
      Ada is a recent language, just last year got a new standard revision. Chances are, when you travel by train or take a plane, the control system was developed in Ada. When human lives are at risk, C is becoming less and less an option. And when the company still goes C, many countries have certifications in place like MISRA, that make C look like Ada with C syntax.
      - ars 13y ago
        Fine, I'll grant you Ada, and it does seem like a good language for these kinds of things. But you have to acknowledge it's not going to become a popular language for regular software projects.
        pjmlp 13y ago
        Sure I won't see the a SV startup using it.
      - ArkyBeagle 13y ago
        Language choice is secondary to methodology, but Ada does a darn fine job of checking an awful lot of methodologically useful things up front. It's less the language than the ecosystem and the infrastructure. It's eminently possible to develop 'C' that's exhaustively tested - I've done it. I am not sure that is true of C++ or Java.
andyjohnson0 13y ago
"Crashing is better that corrupt data" If the crashed program is controlling a device that is directing radiation at a person's body, then crashing is likely to be at least as undesirable as data corruption. I understand your point about programming language choice, and your comments elsewhere about the undesirability of the C language. But the root causes of the Therac-25 malfunction wasn't technical, they were organisational. Specifically poor code review and code reuse practices. Changing the programming language wouldn't fix them.
jacquesm 13y ago
> Crashing is better that corrupt data. In embedded control software of medical systems that are directly affecting the lives of the patients involved crashing is definitely not an option and it is not necessarily better than corrupt data. What is an option is to very carefully model the permitted states of the machine and then to exhaustively test those states and their transitions to make sure there are no undefined states the machine can be in. One way to do this is to have a watchdog process (or even an entirely different system) that monitors the state of the machine and that initiates a safe shutdown if any departure of the defined operation is detected.

hrjet 13y ago

A higher level language could have relieved the programmers from low-level details, thus giving them more cognitive space for avoiding logical bugs in other sections of their code. In general, the more DRY the code, the more mind-space for review / testing.

weland 13y ago
> A higher level language could have relieved the programmers from low-level details, thus giving them more cognitive space for avoiding logical bugs in other sections of their code. This can also be said of choosing simpler design, organizing the code in a better manner, or could have been avoided through more comprehensive testing -- all of which were pointed out by the official report.
- hrjet 13y ago
  I am not sure I understand your argument. Using a high-level language doesn't mean one wouldn't follow other best-practices.
  - weland 13y ago
    Having the bug end up in production could have been avoided simply through adequate testing and verification. I fail to see how a higher-level language helps with that. The fact that the programmers might have had less stuff to worry about and, therefore, saved their concentration for more critical parts of the code, is hardly direct evidence that the bug could not have been introduced. It can equally well be argued that a higher-level language would have attracted programmers with less experience in mission-critical code, who would have failed to grasp even the possibility of bugs more subtle than this one.
JoeAltmaier 13y ago
Remember,this was a PDP11. Don't know what that was? Well, C WAS its high-level language.
angersock 13y ago
In general, the more DRY the code, the more mind-space for review / testing. It's something that would be considered firmware today, written in assembler. Trying to be DRY in assembly is a really great way of introducing subtle bugs.