6 ms·

This was one example given in a very early lecture about software testing and failures in our uni. This (and others) make me hope that I never have that much re

by ygra 13y ago

This was one example given in a very early lecture about software testing and failures in our uni. This (and others) make me hope that I never have that much responsibility over human lives as a developer.

pjmlp 13y ago
It is this type of failures that drives me to push for software quality and moving away from languages like C. As well as, consider that liability should be part of the industry as it is in other industries. Sadly the drive for profit speaks otherwise.
- mortov 13y ago
  The issue is not because of the language used. You can have bugs - potentially fatal as this case underlines - in any language. When I studied the Therac case, there was also a study in the "Killer Robot" - see here : http://www.onlineethics.org/cms/5122.aspx http://www.onlineethics.org/cms/5122.aspx. Well worth a read to understand how software can become dangerous. Sadly nowadays, a search for Killer Robot turns up all sorts of stories of drones being used to kill people. When I first looked up killer robot a number of years ago, the results were pretty much all about the ethics scenario and not real life people being killed. How society has moved on...
  - evincarofautumn 13y ago
    > You can have bugs - potentially fatal as this case underlines - in any language. That is undoubtedly true. However, the commonest classes of bugs can be ruled out by languages with more expressive static verification tools—type systems, effect systems, and so on—leaving us with a stronger baseline from which to work.
  - pjmlp 13y ago
    > The issue is not because of the language used. You can have bugs - potentially fatal as this case underlines - in any language. This is true, however there are languages which allow for more programming errors than others. I just mentioned C as one possible example. The goal should be to have programming languages that reduce programming errors, not that make it easier to shot yourself. Of course this is only a small step towards better software, as whole system design also plays a big role.
  - Piskvorrr 13y ago
    Also relevant (and on HN newest as of now) - Toyota accelerator issue: http://embeddedgurus.com/state-space/2014/02/are-we-shooting-ourselves-in-the-foot-with-stack-overflow/ http://embeddedgurus.com/state-space/2014/02/are-we-shooting...
    - fnordfnordfnord 13y ago
      This one isn't merely an example of a failure of engineering, but also of organizational failure at Toyota, and organizational failure at both the NHTSA and NASA.
- weland 13y ago
  This is the bug that occurred: > The defect was as follows: a one-byte counter in a testing routine frequently overflowed; if an operator provided manual input to the machine at the precise moment that this counter overflowed, the interlock would fail.[3] Can you describe how exactly a language other than C would have helped with this?
  - pjmlp 13y ago
    Safer system programming languages cause a panic on overflow, instead of happily keep on running. Unless the programmer has explicitly disabled them, that is. Crashing is better that corrupt data.
    - ars 13y ago
      Kinda curious - which languages panic on overflow?
      - pjmlp 13y ago
        Ada, here is a small list of validations http://en.wikibooks.org/wiki/Ada_Programming/Pragmas/Suppress http://en.wikibooks.org/wiki/Ada_Programming/Pragmas/Suppres... Modula-2, a possible documentation link http://www.excelsior-usa.com/doc/xds/isom203.html#15 http://www.excelsior-usa.com/doc/xds/isom203.html#15 Just to cite two examples.
        weland 13y ago
        So what happens when the Ada program runs into a failed compiler check? Does it: * Raise an exception? If so, it's up to the programmer to treat it correctly. There are various ways to detect overflows outside Ada; it's still up to the programmer to treat them correctly. Runtime detection of the overflow (in a hopelessly untested firmware of the Therac-25!) would have brought the matter no closer to resolution. * Abort execution? If so, is the behaviour of the program defined if execution is aborted? Because undefined behaviour from a control system is quite worse.
- troels 13y ago
  It's explicitly stated that the choice of language was not considered a root cause. And that was even asm - not C The software was written in assembly language that might require more attention for testing and good design. However the choice of language by itself is not listed as a primary cause in the report.
  - pjmlp 13y ago
    C was just as example, as it is nothing more than a portable macro assembler.
    - troels 13y ago
      As opposed to being written in actual assembler?
    - ColinWright 13y ago
      This is a common mis-conception invariably produced by people who've never written a C compiler. C is not just a portable macro assembler.
      - pjmlp 13y ago
        > This is a common mis-conception invariably produced by people who've never written a C compiler. Wrong on that one. I do have a good background in compiler design. > C is not just a portable macro assembler. Given what it offers when compared with safer systems programming languages, it is one according to my own definition. :)
- ars 13y ago
  > and moving away from languages like C If you look at the reports of serious failures like this once (and other famous ones), none of them are related to the use of C.
chiph 13y ago
Unless your code is the most trivial ever, there's still a tiny possibility that it could result in a death. Game: User goes into epileptic seizure, hits their head and dies. Social media: User calls for help with it - "Please send help #carcrash #bleeding" and the message is lost. As an engineer and a responsible human being, you don't ever want this to happen. But it will. You have to minimize the chances of it happening, with the realization that it will never go to 0.0% The world runs on software today. With all that that implies.

by ygra 13y ago

pjmlp 13y ago

It is this type of failures that drives me to push for software quality and moving away from languages like C. As well as, consider that liability should be part of the industry as it is in other industries. Sadly the drive for profit speaks otherwise.

mortov 13y ago
The issue is not because of the language used. You can have bugs - potentially fatal as this case underlines - in any language. When I studied the Therac case, there was also a study in the "Killer Robot" - see here : http://www.onlineethics.org/cms/5122.aspx http://www.onlineethics.org/cms/5122.aspx. Well worth a read to understand how software can become dangerous. Sadly nowadays, a search for Killer Robot turns up all sorts of stories of drones being used to kill people. When I first looked up killer robot a number of years ago, the results were pretty much all about the ethics scenario and not real life people being killed. How society has moved on...
- evincarofautumn 13y ago
  > You can have bugs - potentially fatal as this case underlines - in any language. That is undoubtedly true. However, the commonest classes of bugs can be ruled out by languages with more expressive static verification tools—type systems, effect systems, and so on—leaving us with a stronger baseline from which to work.
- pjmlp 13y ago
  > The issue is not because of the language used. You can have bugs - potentially fatal as this case underlines - in any language. This is true, however there are languages which allow for more programming errors than others. I just mentioned C as one possible example. The goal should be to have programming languages that reduce programming errors, not that make it easier to shot yourself. Of course this is only a small step towards better software, as whole system design also plays a big role.
- Piskvorrr 13y ago
  Also relevant (and on HN newest as of now) - Toyota accelerator issue: http://embeddedgurus.com/state-space/2014/02/are-we-shooting-ourselves-in-the-foot-with-stack-overflow/ http://embeddedgurus.com/state-space/2014/02/are-we-shooting...
  - fnordfnordfnord 13y ago
    This one isn't merely an example of a failure of engineering, but also of organizational failure at Toyota, and organizational failure at both the NHTSA and NASA.
weland 13y ago
This is the bug that occurred: > The defect was as follows: a one-byte counter in a testing routine frequently overflowed; if an operator provided manual input to the machine at the precise moment that this counter overflowed, the interlock would fail.[3] Can you describe how exactly a language other than C would have helped with this?
- pjmlp 13y ago
  Safer system programming languages cause a panic on overflow, instead of happily keep on running. Unless the programmer has explicitly disabled them, that is. Crashing is better that corrupt data.
  - ars 13y ago
    Kinda curious - which languages panic on overflow?
    - pjmlp 13y ago
      Ada, here is a small list of validations http://en.wikibooks.org/wiki/Ada_Programming/Pragmas/Suppress http://en.wikibooks.org/wiki/Ada_Programming/Pragmas/Suppres... Modula-2, a possible documentation link http://www.excelsior-usa.com/doc/xds/isom203.html#15 http://www.excelsior-usa.com/doc/xds/isom203.html#15 Just to cite two examples.
      - weland 13y ago
        So what happens when the Ada program runs into a failed compiler check? Does it: * Raise an exception? If so, it's up to the programmer to treat it correctly. There are various ways to detect overflows outside Ada; it's still up to the programmer to treat them correctly. Runtime detection of the overflow (in a hopelessly untested firmware of the Therac-25!) would have brought the matter no closer to resolution. * Abort execution? If so, is the behaviour of the program defined if execution is aborted? Because undefined behaviour from a control system is quite worse.
troels 13y ago
It's explicitly stated that the choice of language was not considered a root cause. And that was even asm - not C The software was written in assembly language that might require more attention for testing and good design. However the choice of language by itself is not listed as a primary cause in the report.
- pjmlp 13y ago
  C was just as example, as it is nothing more than a portable macro assembler.
  - troels 13y ago
    As opposed to being written in actual assembler?
  - ColinWright 13y ago
    This is a common mis-conception invariably produced by people who've never written a C compiler. C is not just a portable macro assembler.
    - pjmlp 13y ago
      > This is a common mis-conception invariably produced by people who've never written a C compiler. Wrong on that one. I do have a good background in compiler design. > C is not just a portable macro assembler. Given what it offers when compared with safer systems programming languages, it is one according to my own definition. :)
ars 13y ago
> and moving away from languages like C If you look at the reports of serious failures like this once (and other famous ones), none of them are related to the use of C.

chiph 13y ago

Unless your code is the most trivial ever, there's still a tiny possibility that it could result in a death. Game: User goes into epileptic seizure, hits their head and dies. Social media: User calls for help with it - "Please send help #carcrash #bleeding" and the message is lost. As an engineer and a responsible human being, you don't ever want this to happen. But it will. You have to minimize the chances of it happening, with the realization that it will never go to 0.0% The world runs on software today. With all that that implies.