5 ms·

It's not "countless" it's "uncountable" because we couldn't agree on what a software caused death was. In my CS program we had an ethics class that included st

by moxious 10y ago

It's not "countless" it's "uncountable" because we couldn't agree on what a software caused death was. In my CS program we had an ethics class that included stories of bad X-ray machine software that overdosed people. Bad, bad bad. I don't think many people died as a direct result, but 10 years later there was probably a spiked cancer incidence. Did software kill people? Well yeah....kinda. In airplane systems, there have been a number of cases where bad alerting / warning systems basically either misled the pilot or lulled them into a false sense of security prior to events that caused crashes. Did software kill people? Uhm, yes, I think, sort of, but not directly? It's only when we get fully sentient AI that arms itself and decides to clean up the human pestilence that we'll be able to draw a straight line there. :)

hyperpape 10y ago
Three deaths from the Therac-25: https://en.wikipedia.org/wiki/Therac-25 https://en.wikipedia.org/wiki/Therac-25.
1wd 10y ago
Errors in the software of the radiation therapy machine Therac-25 directly lead to the death or serious injury of six patients. Three patients died within weeks or months. http://radonc.wdfiles.com/local--files/radiation-accident-therac25/Therac_UGuelph_TGall.pdf http://radonc.wdfiles.com/local--files/radiation-accident-th... Errors in the software of a MIM-104 Patriot resulted in failure to locate and intercept an incoming missile and the death of 28 soldiers. http://www.gao.gov/products/IMTEC-92-26 http://www.gao.gov/products/IMTEC-92-26 Errors in the software of a Chinook helicopter may have lead to a crash that killed 29 people. http://www.publications.parliament.uk/pa/ld200102/ldselect/ldchin/25/2504.htm http://www.publications.parliament.uk/pa/ld200102/ldselect/l... Errors in the software of Toyota's throttle control system may have lead to the death of 89 people. https://betterembsw.blogspot.ch/2014/09/a-case-study-of-toyota-unintended.html?m=1 https://betterembsw.blogspot.ch/2014/09/a-case-study-of-toyo... More: http://www.baselinemag.com/c/a/Projects-Processes/Eight-Fatal-SoftwareRelated-Accidents http://www.baselinemag.com/c/a/Projects-Processes/Eight-Fata...
- kbenson 10y ago
  There was no error in the Toyota throttle control system. It was a pedal error with floor mats, and a separate pedal design error for other models.[1], combined with operator error. If you are interested, I highly recommend Malcolm Gladwell's podcast Revisionist History, which did an episode on this[2]. Long story short, your brakes can easily stop your engine at full open throttle, and in not much more space than braking without any throttle. Unfortunately, a foible of human behavior seems to lead us to get flustered and often not do the right thing in situations like these. 1: https://en.wikipedia.org/wiki/Sudden_unintended_acceleration#Sudden_Acceleration_in_Toyota_Vehicles https://en.wikipedia.org/wiki/Sudden_unintended_acceleration... 2: http://revisionisthistory.com/episodes/08-blame-game http://revisionisthistory.com/episodes/08-blame-game
  - 1wd 10y ago
    I agree that the pedal design, human error and floor mat problems are much more likely causes. I don't claim it was a software error for sure. My understanding is that indeed no specific software error was identified, but it also was never ruled out for sure. Neither of the two links seem to contain anything to that effect either.
    - kbenson 10y ago
      To my memory, the podcast lays out a fairly comprehensive argument that it was just human error in the case of mechanical problems (as we'll as noting the car computers in all the cases show the brake wasn't pushed), and back it up with decades of research showing that this is a common problem, so that's about as close to definitive as you can get in this situation IMHO.
- smitherfield 10y ago
  > Errors in the software of the radiation therapy machine Therac-25… Caused by unsigned 8-bit integer overflow combined with (presumably) treating 0 as falsy. May or may not have been solved by using Rust (but probably wouldn't have because it sounds like a logic error in "clever" code). > Errors in the software of a MIM-104 Patriot… Caused by loss of precision in floating-point calculations. Would not have been solved by using Rust. The other two were more likely human or mechanical than (unspecified) software errors.
dwaltrip 10y ago
How many people have died from terrible UIs in apps that are commonly used while driving? Obviously, personal responsibility is primary here, but I don't think the app designers and builders should be absolved of all guilt. I was just thinking about this the other day when using Spotify in the car. Spotify's UI is pretty good, but after they lost all my saved songs, I've had to resort to only using playlists, which require 3x as many clicks to add a song.

by moxious 10y ago

hyperpape 10y ago

Three deaths from the Therac-25: https://en.wikipedia.org/wiki/Therac-25 https://en.wikipedia.org/wiki/Therac-25.

1wd 10y ago

Errors in the software of the radiation therapy machine Therac-25 directly lead to the death or serious injury of six patients. Three patients died within weeks or months. http://radonc.wdfiles.com/local--files/radiation-accident-therac25/Therac_UGuelph_TGall.pdf http://radonc.wdfiles.com/local--files/radiation-accident-th... Errors in the software of a MIM-104 Patriot resulted in failure to locate and intercept an incoming missile and the death of 28 soldiers. http://www.gao.gov/products/IMTEC-92-26 http://www.gao.gov/products/IMTEC-92-26 Errors in the software of a Chinook helicopter may have lead to a crash that killed 29 people. http://www.publications.parliament.uk/pa/ld200102/ldselect/ldchin/25/2504.htm http://www.publications.parliament.uk/pa/ld200102/ldselect/l... Errors in the software of Toyota's throttle control system may have lead to the death of 89 people. https://betterembsw.blogspot.ch/2014/09/a-case-study-of-toyota-unintended.html?m=1 https://betterembsw.blogspot.ch/2014/09/a-case-study-of-toyo... More: http://www.baselinemag.com/c/a/Projects-Processes/Eight-Fatal-SoftwareRelated-Accidents http://www.baselinemag.com/c/a/Projects-Processes/Eight-Fata...

kbenson 10y ago
There was no error in the Toyota throttle control system. It was a pedal error with floor mats, and a separate pedal design error for other models.[1], combined with operator error. If you are interested, I highly recommend Malcolm Gladwell's podcast Revisionist History, which did an episode on this[2]. Long story short, your brakes can easily stop your engine at full open throttle, and in not much more space than braking without any throttle. Unfortunately, a foible of human behavior seems to lead us to get flustered and often not do the right thing in situations like these. 1: https://en.wikipedia.org/wiki/Sudden_unintended_acceleration#Sudden_Acceleration_in_Toyota_Vehicles https://en.wikipedia.org/wiki/Sudden_unintended_acceleration... 2: http://revisionisthistory.com/episodes/08-blame-game http://revisionisthistory.com/episodes/08-blame-game
- 1wd 10y ago
  I agree that the pedal design, human error and floor mat problems are much more likely causes. I don't claim it was a software error for sure. My understanding is that indeed no specific software error was identified, but it also was never ruled out for sure. Neither of the two links seem to contain anything to that effect either.
  - kbenson 10y ago
    To my memory, the podcast lays out a fairly comprehensive argument that it was just human error in the case of mechanical problems (as we'll as noting the car computers in all the cases show the brake wasn't pushed), and back it up with decades of research showing that this is a common problem, so that's about as close to definitive as you can get in this situation IMHO.
smitherfield 10y ago
> Errors in the software of the radiation therapy machine Therac-25… Caused by unsigned 8-bit integer overflow combined with (presumably) treating 0 as falsy. May or may not have been solved by using Rust (but probably wouldn't have because it sounds like a logic error in "clever" code). > Errors in the software of a MIM-104 Patriot… Caused by loss of precision in floating-point calculations. Would not have been solved by using Rust. The other two were more likely human or mechanical than (unspecified) software errors.

dwaltrip 10y ago

How many people have died from terrible UIs in apps that are commonly used while driving? Obviously, personal responsibility is primary here, but I don't think the app designers and builders should be absolved of all guilt. I was just thinking about this the other day when using Spotify in the car. Spotify's UI is pretty good, but after they lost all my saved songs, I've had to resort to only using playlists, which require 3x as many clicks to add a song.