5 ms·

I think this is something that isnt that present in our profession as it should be. If you are an electrical engineer, it is clear and present, that your produc

by cf498 8y ago

I think this is something that isnt that present in our profession as it should be. If you are an electrical engineer, it is clear and present, that your products might kill someone if you screw up. The same goes for a mechanical engineer. And it will be your responsibility. If you screw up and should have known better, you might even go to jail for negligence. We on the other hand have the Therac-25. We claim to be on a level of engineering fields. We should also act that way.

Balgair 8y ago
Agreed. The Order of the Engineer is a good place to start and bringing that tradition down into comp-eng/sci would be a nice start. https://en.wikipedia.org/wiki/Order_of_the_Engineer https://en.wikipedia.org/wiki/Order_of_the_Engineer
- Latteland 8y ago
  Interesting, had not heard of that. What should we computer programmers, we flexible always trying new things without enough tried and true practices think of real engineers, what should they think of us?
  - ionised 8y ago
    Real engineers should probably look down on us from a great height.
    - Balgair 8y ago
      Here's a more humorous take: https://imgs.xkcd.com/comics/voting_software.png https://imgs.xkcd.com/comics/voting_software.png
Someone 8y ago
Therac-25 was thirty years ago, and still is the go to example. That made me wonder. If you compare, for example, https://en.wikipedia.org/wiki/List_of_bridge_failures#2000–present https://en.wikipedia.org/wiki/List_of_bridge_failures#2000–p... or https://en.wikipedia.org/wiki/List_of_structural_failures_and_collapses#2010–present https://en.wikipedia.org/wiki/List_of_structural_failures_an... with https://en.wikipedia.org/wiki/List_of_software_bugs https://en.wikipedia.org/wiki/List_of_software_bugs, software doesn’t look that deadly.
- jcranmer 8y ago
  People generally don't attribute failures to software bugs if there is anything else that contributes to failure. The list in particular is missing the loads of accidents caused by bad UI; incidents of "I was just following the GPS's/autopilot's directions" that result in death are missing (such as the KAL 007 airliner incident).
- cgdcraig 8y ago
  I'd imagine a lot of places where software is part of a safety critical system also come with physical fail safes. An example being burst plates on pressure vessels, if the software that controls pressurization spasms and causes an over pressure the physical fail safe will prevent a catastrophic accident.
- gruturo 8y ago
  Software doesn't have many opportunities to be deadly, and when said opportunities do exist, this is frequently in regulated fields where safeties are mandated by law (interlocks on a laser or a welding robot arm etc).
- tremon 8y ago
  The Therac-25 is the go-to example because it is the best documented and least-contested case. Also, software usually isn't in ultimate control, most systems have additional (mechanical) safety controls. Mars Climate Orbiter, 1999: http://articles.latimes.com/1999/oct/01/news/mn-17288 http://articles.latimes.com/1999/oct/01/news/mn-17288. Undetected metric conversion error (though unclear if this was a manual process or software). Toyota unintended acceleration, 2010: https://www.edmunds.com/car-safety/for-toyota-owners-unintended-acceleration-lawsuit-settlement.html https://www.edmunds.com/car-safety/for-toyota-owners-uninten.... Toyota maintains that it was not caused by software error. Now let's list some cases where the software did have ultimate control: Schiaparelli EDM lander, 2016: https://newatlas.com/esa-schiaparelli-mars-crash-inquiry/49696/ https://newatlas.com/esa-schiaparelli-mars-crash-inquiry/496.... Faulty decision-making in the automated descent system due to input saturation. Tesla auto-pilot crash, 2016: https://www.theguardian.com/technology/2016/jun/30/tesla-autopilot-death-self-driving-car-elon-musk https://www.theguardian.com/technology/2016/jun/30/tesla-aut.... Faulty decision-making due to incorrect image analysis. In this case, the additional safety controls (the driver himself) failed too. Uber car crash, 2018: https://money.cnn.com/2018/03/20/news/companies/self-driving-uber-death/index.html https://money.cnn.com/2018/03/20/news/companies/self-driving.... Faulty decision-making due to incorrect image analysis, though the investigation is still ongoing, so no definite cause determined (FAFAIK).
conanbatt 8y ago
Software engineering is more like witch-doctoring than to civil engineering. It is not remotely price-efficient to prove completeness on programs. If you expect your doctor's software to be perfect, be prepared to pay for 2 software engineers salaries per doctor.
pimmen 8y ago
I agree that we act recklessly sometimes, but I hope I'm not alone among software engineers in believing that this finding from the committee was just plain stupidity which should not be done by any engineer worth his or her salt: "AECL had never tested the Therac-25 with the combination of software and hardware until it was assembled at the hospital."

by cf498 8y ago

Balgair 8y ago

Agreed. The Order of the Engineer is a good place to start and bringing that tradition down into comp-eng/sci would be a nice start. https://en.wikipedia.org/wiki/Order_of_the_Engineer https://en.wikipedia.org/wiki/Order_of_the_Engineer

Latteland 8y ago
Interesting, had not heard of that. What should we computer programmers, we flexible always trying new things without enough tried and true practices think of real engineers, what should they think of us?
- ionised 8y ago
  Real engineers should probably look down on us from a great height.
  - Balgair 8y ago
    Here's a more humorous take: https://imgs.xkcd.com/comics/voting_software.png https://imgs.xkcd.com/comics/voting_software.png

Someone 8y ago

Therac-25 was thirty years ago, and still is the go to example. That made me wonder. If you compare, for example, https://en.wikipedia.org/wiki/List_of_bridge_failures#2000–present https://en.wikipedia.org/wiki/List_of_bridge_failures#2000–p... or https://en.wikipedia.org/wiki/List_of_structural_failures_and_collapses#2010–present https://en.wikipedia.org/wiki/List_of_structural_failures_an... with https://en.wikipedia.org/wiki/List_of_software_bugs https://en.wikipedia.org/wiki/List_of_software_bugs, software doesn’t look that deadly.

jcranmer 8y ago
People generally don't attribute failures to software bugs if there is anything else that contributes to failure. The list in particular is missing the loads of accidents caused by bad UI; incidents of "I was just following the GPS's/autopilot's directions" that result in death are missing (such as the KAL 007 airliner incident).
cgdcraig 8y ago
I'd imagine a lot of places where software is part of a safety critical system also come with physical fail safes. An example being burst plates on pressure vessels, if the software that controls pressurization spasms and causes an over pressure the physical fail safe will prevent a catastrophic accident.
gruturo 8y ago
Software doesn't have many opportunities to be deadly, and when said opportunities do exist, this is frequently in regulated fields where safeties are mandated by law (interlocks on a laser or a welding robot arm etc).
tremon 8y ago
The Therac-25 is the go-to example because it is the best documented and least-contested case. Also, software usually isn't in ultimate control, most systems have additional (mechanical) safety controls. Mars Climate Orbiter, 1999: http://articles.latimes.com/1999/oct/01/news/mn-17288 http://articles.latimes.com/1999/oct/01/news/mn-17288. Undetected metric conversion error (though unclear if this was a manual process or software). Toyota unintended acceleration, 2010: https://www.edmunds.com/car-safety/for-toyota-owners-unintended-acceleration-lawsuit-settlement.html https://www.edmunds.com/car-safety/for-toyota-owners-uninten.... Toyota maintains that it was not caused by software error. Now let's list some cases where the software did have ultimate control: Schiaparelli EDM lander, 2016: https://newatlas.com/esa-schiaparelli-mars-crash-inquiry/49696/ https://newatlas.com/esa-schiaparelli-mars-crash-inquiry/496.... Faulty decision-making in the automated descent system due to input saturation. Tesla auto-pilot crash, 2016: https://www.theguardian.com/technology/2016/jun/30/tesla-autopilot-death-self-driving-car-elon-musk https://www.theguardian.com/technology/2016/jun/30/tesla-aut.... Faulty decision-making due to incorrect image analysis. In this case, the additional safety controls (the driver himself) failed too. Uber car crash, 2018: https://money.cnn.com/2018/03/20/news/companies/self-driving-uber-death/index.html https://money.cnn.com/2018/03/20/news/companies/self-driving.... Faulty decision-making due to incorrect image analysis, though the investigation is still ongoing, so no definite cause determined (FAFAIK).

conanbatt 8y ago

Software engineering is more like witch-doctoring than to civil engineering. It is not remotely price-efficient to prove completeness on programs. If you expect your doctor's software to be perfect, be prepared to pay for 2 software engineers salaries per doctor.

pimmen 8y ago

I agree that we act recklessly sometimes, but I hope I'm not alone among software engineers in believing that this finding from the committee was just plain stupidity which should not be done by any engineer worth his or her salt: "AECL had never tested the Therac-25 with the combination of software and hardware until it was assembled at the hospital."