5 ms·

The most deadly bug in history. If you know any other deadly bug, please share! I love these stories!

by napolux 1y ago

NitpickLawyer 1y ago
The MCAS related bugs @ Boeing led to 300+ deaths, so it's probably a contender.
- solids 1y ago
  Was that a bug or a failure to inform pilots about a new system?
  - AdamN 1y ago
    Both - and really MCAS was fine but the issue was the metering systems (Pitot tubes) and the handling of conflicting data. That part of the puzzle was definitely a bug in the logic/software.
    - kijin 1y ago
      Remember the Airbus that crashed in the middle of the Atlantic because one of the pilots kept pulling on his yoke, and the computer decided to average his input with normal input from the other pilot? Conflict resolution in redundant systems seems to be one of the weakest spots in modern aircraft software.
      - sgerenser 1y ago
        Air France 447: https://en.m.wikipedia.org/wiki/Air_France_Flight_447 https://en.m.wikipedia.org/wiki/Air_France_Flight_447 Inputs were averaged, but supposedly there’s at least a warning: Confused, Bonin exclaimed, "I don't have control of the airplane any more now", and two seconds later, "I don't have control of the airplane at all!"[42] Robert responded to this by saying, "controls to the left", and took over control of the aircraft.[84][44] He pushed his side-stick forward to lower the nose and recover from the stall; however, Bonin was still pulling his side-stick back. The inputs cancelled each other out and triggered an audible "dual input" warning.
    - phire 1y ago
      That wasn't a bug. They deliberately designed it to only look at one of the Pitot tubes, because if they had designed it to look at both, then they would have had to implement a warning message for conflicting data. And if they had implemented a warning message, they would have had to tell the pilots about the new system, and train them how to deal with it. It wasn't a mistake in logic either. This design went through their internal safety certification, and passed. As far as I'm aware, MCAS functioned exactly as designed, zero bugs. It's just that the design was very bad.
    - mnw21cam 1y ago
      It wasn't pitot tubes that had the hardware problem, it was the angle of attack sensor. The software was poorly designed to believe the input from just one fallible angle of attack sensor.
  - thyristan 1y ago
    In the same vein one could argue that Therac-25 was not actually a software bug but a hardware problem. Interlocks, that could have prevented the accidents and that where present in earlier Therac models, were missing. The software was written with those interlocks in mind. Greedy management/hardware engineers skipped them for the -25 version. It's almost never just software. It's almost never just one cause.
    - actionfromafar 1y ago
      Just to point it out even clearer - there's almost never a root cause.
  - NitpickLawyer 1y ago
    I would say plenty of both. They obviously had to inform the pilots, but the way the system didn't reset permanently after 2-3 (whatever) sessions of "oh, the pilot trimmed manually, after 10 seconds we keep doing the same thing" was a major major logic blunder. Failure all across the board, if only from the perspective of end-to-end / integration testing if nothing else. Worryingly, e2e / full integration testing was also the main cause of other Boeing blunders, like the Starliner capsule.
  - fuckaj 1y ago
    Not a bug. A non airworthy plane they tried to patch up with software.
    - reorder9695 1y ago
      The plane was perfectly airworthy without MCAS, that was never the issue. The issue was it handled differently enough at high angles of attack to the 737NG that pilots would've needed additional training or possibly a new type rating without MCAS changing the trim in this situation. The competition (Airbus NEO family) did not need this kind of new training for existing pilots, so airlines being required to do this for new Boeing but not Airbus planes would've been a huge commercial disadvantage. [edit as I can't reply to the child comment]: The FAA and EASA both looked into the stall characteristics afterwards and concluded that the plane was stable enough to be certified without MCAS and while it did have more of a tenancy to pitch up at high angles of attack it was still an acceptable amount.
      - fuckaj 1y ago
        I may have understood wrong but thought is possible to get into an unrecoverable stall?
echelon 1y ago
The 737 Max MCAS is arguably a bug. That killed 346 people. Not a "bug" per se, but texting while driving kills ~400 people per year in the US. It's a bug at some level of granularity. To be tongue in cheek a bit, buggy JIRA latency has probably wasted 10,000 human years. Those are many whole human lives if you count them up.
- b_e_n_t_o_n 1y ago
  > To be tongue in cheek a bit, buggy JIRA latency has probably wasted 10,000 human years. Those are many whole human lives if you count them up. These kind of calculations always make me wonder...say someone wasted one minute of everybody's life, is the cost ~250 lives? One minute? Somewhere in between?
benrutter 1y ago
Probably many rather than a single bug, but the botched London Ambulance dispatch software from the 90s, is probably one of the most deadly software issues of all time, although there aren't any estimates I know of that try to quantify the number of lives lost as a result. http://www0.cs.ucl.ac.uk/staff/a.finkelstein/papers/lascase.pdf http://www0.cs.ucl.ac.uk/staff/a.finkelstein/papers/lascase....
- napolux 1y ago
  thanks for sharing!
kgwgk 1y ago
Several people killed themselves over this: https://www.wikipedia.org/wiki/British_Post_Office_scandal https://www.wikipedia.org/wiki/British_Post_Office_scandal https://www.theguardian.com/uk-news/2024/jan/09/how-the-post-offices-horizon-system-failed-a-technical-breakdown https://www.theguardian.com/uk-news/2024/jan/09/how-the-post... One member of the development team, David McDonnell, who had worked on the Epos system side of the project, told the inquiry that “of eight [people] in the development team, two were very good, another two were mediocre but we could work with them, and then there were probably three or four who just weren’t up to it and weren’t capable of producing professional code”. What sort of bugs resulted? As early as 2001, McDonnell’s team had found “hundreds” of bugs. A full list has never been produced, but successive vindications of post office operators have revealed the sort of problems that arose. One, named the “Dalmellington Bug”, after the village in Scotland where a post office operator first fell prey to it, would see the screen freeze as the user was attempting to confirm receipt of cash. Each time the user pressed “enter” on the frozen screen, it would silently update the record. In Dalmellington, that bug created a £24,000 discrepancy, which the Post Office tried to hold the post office operator responsible for. Another bug, called the Callendar Square bug – again named after the first branch found to have been affected by it – created duplicate transactions due to an error in the database underpinning the system: despite being clear duplicates, the post office operator was again held responsible for the errors.
- BoxOfRain 1y ago
  More heads should have rolled over this in my opinion, absolutely despicable that they cheerfully threw innocent people in prison rather than admit their software was a heap of crap. It makes me so angry this injustice was allowed to prevail for so long because nobody cared about the people being mistreated and tarred as thieves as long as they were 'little people' of no consequence, while senior management gleefully covered themselves in criminality to cover for their own uselessness. It's an archetypal example of 'one law for the connected, another law for the proles'.
A1kmm 1y ago
Not even close. Israel apparently has AI bombing target intel & selection systems called Gospel and Lavender - https://www.theguardian.com/world/2024/apr/03/israel-gaza-ai-database-hamas-airstrikes https://www.theguardian.com/world/2024/apr/03/israel-gaza-ai.... Claims are these systems have a selectivity of 90% per bombing, and they were willing to bomb up to 20 civilians per person classified by the system as a Hamas member. So assuming that is true, 90% of the time, they kill one Hamas member, and up to 20 innocents. 10% of the time, they kill up to 21 innocents and no Hamas members. Killing 20 innocents and one Hamas member is not a bug - it is callous, but that's a policy decision and the software working as intended. But when it is a false positive (10% of the time), due to inadequate / outdated data and inadequate models, that could reasonably classified as a bug - so all 21 deaths for each of those bombings would count as deaths caused by a bug. Apparently (at least earlier versions) of Gospel were trained on positive examples that mean someone is a member of Hamas, but not on negative examples; other problems could be due to, for example, insufficient data, and interpolation outside the valid range (e.g. using pre-war data about, e.g. how quickly cell phones are traded, or people movements, when behaviour is different post-war). I'd therefore estimate that deaths due to classification errors from those systems is likely in the thousands (out of the 60k+ Palestinian deaths in the conflict). Therac-25's bugs caused 6 deaths for comparison.
danadam 1y ago
Some Google Pixel phones couldn't dial emergency number (still can't?). I don't know if there were any deadly consequences of that. https://www.androidauthority.com/psa-google-pixel-911-emergency-calling-issues-3362990/ https://www.androidauthority.com/psa-google-pixel-911-emerge...
throwaway0261 1y ago
There was a news story from Norway last year where a car allegedly accelerated by itself, causing the car to fall off the second floor of a parking garage and kill the driver.
- mnw21cam 1y ago
  There are plenty of "car allegedly accelerated by itself" incidents, and usually the root cause is the driver mistakenly pressing the accelerator pedal when they think they're pressing the brake pedal. And then swearing blind afterwards that they were braking as hard as they possibly could but the car kept surging forwards.
  - bobmcnamara 1y ago
    Time and time again the introduction of electronic throttle control has spiked the number of reported unintended acceleration incidents. There's a chart here that shows it clearly for Toyota's rollout: https://www.embedded.com/unintended-acceleration-and-other-embedded-software-bugs/ https://www.embedded.com/unintended-acceleration-and-other-e...
    - mrguyorama 1y ago
      That chart is not clear. The Camry, the Solara, and the RAV4 are all the exact same engine hardware and software, at nearly all times. Especially the Solara, which is just a Camry with a shinier body. It uses an identical engine, throttle body, and ECU flash and is even considered a "Camry Solara". That Camry "Unintended accelerations" jumped while Solara didn't means that it isn't the hardware. Instead, they all started at the same time, about 2002. Similarly, the Scion TC is also literally the exact same hardware, software, and throttle body as a Camry. The entire Scion line is just Toyota software and hardware in a different body shell. Indeed, if you look at the Corolla, the jump in unintended acceleration cases start with mechanical throttle bodies still the norm, and do not change with the switch to electronic throttle control. IMO this graphic handily shows how the media affects average people. The media went on a large blitz about how Toyota was unsafe now that they had electronic throttle bodies, and so owners of those cars complained, but the average consumer does not realize that the Solara, Camry, and Scion TC are all identical vehicles under the body shell and don't realize that they "should" also be complaining about those cars if the problem was actually caused by the electronic throttle body code or design. Important note: People who report unintended acceleration events like this almost always say that the brakes didn't stop it. That seems.... hard to believe. The brakes on all Toyotas are fully hydraulic and cannot fail electronically. All toyota vehicles in that chart (maybe excluding some Tacomas and the top line Lexus model) have brake capability far exceeding their engine power. A V6 Camry can be at wide open throttle but hard application of the brakes would still overpower that engine with no problem. Interestingly the NHTSA's opinion is that the Toyota models in the graphic ARE defective: They allow faulty or improperly installed weather mats to interfere with the pedals. Toyota also believed this take, as they kept their handling of fixing this defect off the books, and changed the pedal positioning in their newer models. They eventually fined Toyota over a billion dollars for their handling of this situation, and claimed there was another "sticky pedals" problem that they were covering up.
bobmcnamara 1y ago
In Dhahran, Saudi Arabia, on February 25, 1991, a Patriot missile failed to intercept an Iraqi Scud causing the death of 28 American soldiers. The patriot missile system used floating point for time, so as uptime extended the clock became more and more granular, eventually to the point where time skipped so far that the range gate was tripped. The fix was being deployed earlier that year but this unit hadn't been updated yet. https://www.cs.unc.edu/~smp/COMP205/LECTURES/ERROR/lec23/node4.html https://www.cs.unc.edu/~smp/COMP205/LECTURES/ERROR/lec23/nod...

The most deadly bug in history. If you know any other deadly bug, please share! I love these stories!

NitpickLawyer 1y ago

The MCAS related bugs @ Boeing led to 300+ deaths, so it's probably a contender.

solids 1y ago
Was that a bug or a failure to inform pilots about a new system?
- AdamN 1y ago
  Both - and really MCAS was fine but the issue was the metering systems (Pitot tubes) and the handling of conflicting data. That part of the puzzle was definitely a bug in the logic/software.
  - kijin 1y ago
    Remember the Airbus that crashed in the middle of the Atlantic because one of the pilots kept pulling on his yoke, and the computer decided to average his input with normal input from the other pilot? Conflict resolution in redundant systems seems to be one of the weakest spots in modern aircraft software.
    - sgerenser 1y ago
      Air France 447: https://en.m.wikipedia.org/wiki/Air_France_Flight_447 https://en.m.wikipedia.org/wiki/Air_France_Flight_447 Inputs were averaged, but supposedly there’s at least a warning: Confused, Bonin exclaimed, "I don't have control of the airplane any more now", and two seconds later, "I don't have control of the airplane at all!"[42] Robert responded to this by saying, "controls to the left", and took over control of the aircraft.[84][44] He pushed his side-stick forward to lower the nose and recover from the stall; however, Bonin was still pulling his side-stick back. The inputs cancelled each other out and triggered an audible "dual input" warning.
  - phire 1y ago
    That wasn't a bug. They deliberately designed it to only look at one of the Pitot tubes, because if they had designed it to look at both, then they would have had to implement a warning message for conflicting data. And if they had implemented a warning message, they would have had to tell the pilots about the new system, and train them how to deal with it. It wasn't a mistake in logic either. This design went through their internal safety certification, and passed. As far as I'm aware, MCAS functioned exactly as designed, zero bugs. It's just that the design was very bad.
  - mnw21cam 1y ago
    It wasn't pitot tubes that had the hardware problem, it was the angle of attack sensor. The software was poorly designed to believe the input from just one fallible angle of attack sensor.
- thyristan 1y ago
  In the same vein one could argue that Therac-25 was not actually a software bug but a hardware problem. Interlocks, that could have prevented the accidents and that where present in earlier Therac models, were missing. The software was written with those interlocks in mind. Greedy management/hardware engineers skipped them for the -25 version. It's almost never just software. It's almost never just one cause.
  - actionfromafar 1y ago
    Just to point it out even clearer - there's almost never a root cause.
- NitpickLawyer 1y ago
  I would say plenty of both. They obviously had to inform the pilots, but the way the system didn't reset permanently after 2-3 (whatever) sessions of "oh, the pilot trimmed manually, after 10 seconds we keep doing the same thing" was a major major logic blunder. Failure all across the board, if only from the perspective of end-to-end / integration testing if nothing else. Worryingly, e2e / full integration testing was also the main cause of other Boeing blunders, like the Starliner capsule.
- fuckaj 1y ago
  Not a bug. A non airworthy plane they tried to patch up with software.
  - reorder9695 1y ago
    The plane was perfectly airworthy without MCAS, that was never the issue. The issue was it handled differently enough at high angles of attack to the 737NG that pilots would've needed additional training or possibly a new type rating without MCAS changing the trim in this situation. The competition (Airbus NEO family) did not need this kind of new training for existing pilots, so airlines being required to do this for new Boeing but not Airbus planes would've been a huge commercial disadvantage. [edit as I can't reply to the child comment]: The FAA and EASA both looked into the stall characteristics afterwards and concluded that the plane was stable enough to be certified without MCAS and while it did have more of a tenancy to pitch up at high angles of attack it was still an acceptable amount.
    - fuckaj 1y ago
      I may have understood wrong but thought is possible to get into an unrecoverable stall?

echelon 1y ago

The 737 Max MCAS is arguably a bug. That killed 346 people. Not a "bug" per se, but texting while driving kills ~400 people per year in the US. It's a bug at some level of granularity. To be tongue in cheek a bit, buggy JIRA latency has probably wasted 10,000 human years. Those are many whole human lives if you count them up.

b_e_n_t_o_n 1y ago
> To be tongue in cheek a bit, buggy JIRA latency has probably wasted 10,000 human years. Those are many whole human lives if you count them up. These kind of calculations always make me wonder...say someone wasted one minute of everybody's life, is the cost ~250 lives? One minute? Somewhere in between?

benrutter 1y ago

Probably many rather than a single bug, but the botched London Ambulance dispatch software from the 90s, is probably one of the most deadly software issues of all time, although there aren't any estimates I know of that try to quantify the number of lives lost as a result. http://www0.cs.ucl.ac.uk/staff/a.finkelstein/papers/lascase.pdf http://www0.cs.ucl.ac.uk/staff/a.finkelstein/papers/lascase....

napolux 1y ago
thanks for sharing!

kgwgk 1y ago

Several people killed themselves over this: https://www.wikipedia.org/wiki/British_Post_Office_scandal https://www.wikipedia.org/wiki/British_Post_Office_scandal https://www.theguardian.com/uk-news/2024/jan/09/how-the-post-offices-horizon-system-failed-a-technical-breakdown https://www.theguardian.com/uk-news/2024/jan/09/how-the-post... One member of the development team, David McDonnell, who had worked on the Epos system side of the project, told the inquiry that “of eight [people] in the development team, two were very good, another two were mediocre but we could work with them, and then there were probably three or four who just weren’t up to it and weren’t capable of producing professional code”. What sort of bugs resulted? As early as 2001, McDonnell’s team had found “hundreds” of bugs. A full list has never been produced, but successive vindications of post office operators have revealed the sort of problems that arose. One, named the “Dalmellington Bug”, after the village in Scotland where a post office operator first fell prey to it, would see the screen freeze as the user was attempting to confirm receipt of cash. Each time the user pressed “enter” on the frozen screen, it would silently update the record. In Dalmellington, that bug created a £24,000 discrepancy, which the Post Office tried to hold the post office operator responsible for. Another bug, called the Callendar Square bug – again named after the first branch found to have been affected by it – created duplicate transactions due to an error in the database underpinning the system: despite being clear duplicates, the post office operator was again held responsible for the errors.

BoxOfRain 1y ago
More heads should have rolled over this in my opinion, absolutely despicable that they cheerfully threw innocent people in prison rather than admit their software was a heap of crap. It makes me so angry this injustice was allowed to prevail for so long because nobody cared about the people being mistreated and tarred as thieves as long as they were 'little people' of no consequence, while senior management gleefully covered themselves in criminality to cover for their own uselessness. It's an archetypal example of 'one law for the connected, another law for the proles'.

A1kmm 1y ago

Not even close. Israel apparently has AI bombing target intel & selection systems called Gospel and Lavender - https://www.theguardian.com/world/2024/apr/03/israel-gaza-ai-database-hamas-airstrikes https://www.theguardian.com/world/2024/apr/03/israel-gaza-ai.... Claims are these systems have a selectivity of 90% per bombing, and they were willing to bomb up to 20 civilians per person classified by the system as a Hamas member. So assuming that is true, 90% of the time, they kill one Hamas member, and up to 20 innocents. 10% of the time, they kill up to 21 innocents and no Hamas members. Killing 20 innocents and one Hamas member is not a bug - it is callous, but that's a policy decision and the software working as intended. But when it is a false positive (10% of the time), due to inadequate / outdated data and inadequate models, that could reasonably classified as a bug - so all 21 deaths for each of those bombings would count as deaths caused by a bug. Apparently (at least earlier versions) of Gospel were trained on positive examples that mean someone is a member of Hamas, but not on negative examples; other problems could be due to, for example, insufficient data, and interpolation outside the valid range (e.g. using pre-war data about, e.g. how quickly cell phones are traded, or people movements, when behaviour is different post-war). I'd therefore estimate that deaths due to classification errors from those systems is likely in the thousands (out of the 60k+ Palestinian deaths in the conflict). Therac-25's bugs caused 6 deaths for comparison.

danadam 1y ago

Some Google Pixel phones couldn't dial emergency number (still can't?). I don't know if there were any deadly consequences of that. https://www.androidauthority.com/psa-google-pixel-911-emergency-calling-issues-3362990/ https://www.androidauthority.com/psa-google-pixel-911-emerge...

throwaway0261 1y ago

There was a news story from Norway last year where a car allegedly accelerated by itself, causing the car to fall off the second floor of a parking garage and kill the driver.

mnw21cam 1y ago
There are plenty of "car allegedly accelerated by itself" incidents, and usually the root cause is the driver mistakenly pressing the accelerator pedal when they think they're pressing the brake pedal. And then swearing blind afterwards that they were braking as hard as they possibly could but the car kept surging forwards.
- bobmcnamara 1y ago
  Time and time again the introduction of electronic throttle control has spiked the number of reported unintended acceleration incidents. There's a chart here that shows it clearly for Toyota's rollout: https://www.embedded.com/unintended-acceleration-and-other-embedded-software-bugs/ https://www.embedded.com/unintended-acceleration-and-other-e...
  - mrguyorama 1y ago
    That chart is not clear. The Camry, the Solara, and the RAV4 are all the exact same engine hardware and software, at nearly all times. Especially the Solara, which is just a Camry with a shinier body. It uses an identical engine, throttle body, and ECU flash and is even considered a "Camry Solara". That Camry "Unintended accelerations" jumped while Solara didn't means that it isn't the hardware. Instead, they all started at the same time, about 2002. Similarly, the Scion TC is also literally the exact same hardware, software, and throttle body as a Camry. The entire Scion line is just Toyota software and hardware in a different body shell. Indeed, if you look at the Corolla, the jump in unintended acceleration cases start with mechanical throttle bodies still the norm, and do not change with the switch to electronic throttle control. IMO this graphic handily shows how the media affects average people. The media went on a large blitz about how Toyota was unsafe now that they had electronic throttle bodies, and so owners of those cars complained, but the average consumer does not realize that the Solara, Camry, and Scion TC are all identical vehicles under the body shell and don't realize that they "should" also be complaining about those cars if the problem was actually caused by the electronic throttle body code or design. Important note: People who report unintended acceleration events like this almost always say that the brakes didn't stop it. That seems.... hard to believe. The brakes on all Toyotas are fully hydraulic and cannot fail electronically. All toyota vehicles in that chart (maybe excluding some Tacomas and the top line Lexus model) have brake capability far exceeding their engine power. A V6 Camry can be at wide open throttle but hard application of the brakes would still overpower that engine with no problem. Interestingly the NHTSA's opinion is that the Toyota models in the graphic ARE defective: They allow faulty or improperly installed weather mats to interfere with the pedals. Toyota also believed this take, as they kept their handling of fixing this defect off the books, and changed the pedal positioning in their newer models. They eventually fined Toyota over a billion dollars for their handling of this situation, and claimed there was another "sticky pedals" problem that they were covering up.

bobmcnamara 1y ago

In Dhahran, Saudi Arabia, on February 25, 1991, a Patriot missile failed to intercept an Iraqi Scud causing the death of 28 American soldiers. The patriot missile system used floating point for time, so as uptime extended the clock became more and more granular, eventually to the point where time skipped so far that the range gate was tripped. The fix was being deployed earlier that year but this unit hadn't been updated yet. https://www.cs.unc.edu/~smp/COMP205/LECTURES/ERROR/lec23/node4.html https://www.cs.unc.edu/~smp/COMP205/LECTURES/ERROR/lec23/nod...