4 ms·
Therac-25 was thirty years ago, and still is the go to example. That made me wonder. If you compare, for example, https://en.wikipedia.org/wiki/List_of_bridge_f
by Someone 8y ago
Therac-25 was thirty years ago, and still is the go to example. That made me wonder. If you compare, for example, https://en.wikipedia.org/wiki/List_of_bridge_failures#2000–present https://en.wikipedia.org/wiki/List_of_bridge_failures#2000–p... or https://en.wikipedia.org/wiki/List_of_structural_failures_and_collapses#2010–present https://en.wikipedia.org/wiki/List_of_structural_failures_an... with https://en.wikipedia.org/wiki/List_of_software_bugs https://en.wikipedia.org/wiki/List_of_software_bugs, software doesn’t look that deadly.
- jcranmer 8y agoPeople generally don't attribute failures to software bugs if there is anything else that contributes to failure. The list in particular is missing the loads of accidents caused by bad UI; incidents of "I was just following the GPS's/autopilot's directions" that result in death are missing (such as the KAL 007 airliner incident).
- cgdcraig 8y agoI'd imagine a lot of places where software is part of a safety critical system also come with physical fail safes. An example being burst plates on pressure vessels, if the software that controls pressurization spasms and causes an over pressure the physical fail safe will prevent a catastrophic accident.
- gruturo 8y agoSoftware doesn't have many opportunities to be deadly, and when said opportunities do exist, this is frequently in regulated fields where safeties are mandated by law (interlocks on a laser or a welding robot arm etc).
- tremon 8y agoThe Therac-25 is the go-to example because it is the best documented and least-contested case. Also, software usually isn't in ultimate control, most systems have additional (mechanical) safety controls. Mars Climate Orbiter, 1999: http://articles.latimes.com/1999/oct/01/news/mn-17288 http://articles.latimes.com/1999/oct/01/news/mn-17288. Undetected metric conversion error (though unclear if this was a manual process or software). Toyota unintended acceleration, 2010: https://www.edmunds.com/car-safety/for-toyota-owners-unintended-acceleration-lawsuit-settlement.html https://www.edmunds.com/car-safety/for-toyota-owners-uninten.... Toyota maintains that it was not caused by software error. Now let's list some cases where the software did have ultimate control: Schiaparelli EDM lander, 2016: https://newatlas.com/esa-schiaparelli-mars-crash-inquiry/49696/ https://newatlas.com/esa-schiaparelli-mars-crash-inquiry/496.... Faulty decision-making in the automated descent system due to input saturation. Tesla auto-pilot crash, 2016: https://www.theguardian.com/technology/2016/jun/30/tesla-autopilot-death-self-driving-car-elon-musk https://www.theguardian.com/technology/2016/jun/30/tesla-aut.... Faulty decision-making due to incorrect image analysis. In this case, the additional safety controls (the driver himself) failed too. Uber car crash, 2018: https://money.cnn.com/2018/03/20/news/companies/self-driving-uber-death/index.html https://money.cnn.com/2018/03/20/news/companies/self-driving.... Faulty decision-making due to incorrect image analysis, though the investigation is still ongoing, so no definite cause determined (FAFAIK).