12 ms·

Automated to Death (2009)

100 points by mcspecter 10y ago

exar0815 10y ago
Every automated System needs a very well trained and calm operator when it all goes south. Thats the difference between e.g. Chernobyl and Fukushima/Three Mile Island. While very bad accidents, the last fail-save, the humans, didnt fuck it completely and spectacularly in the latter two cases.
- throwanem 10y ago
  Chernobyl isn't a very good example of automation failure; responsibility for that disaster lies entirely with human beings from start to finish. Wikipedia's summary is solid, and rather than excerpt it here I'll just point you at https://en.wikipedia.org/wiki/Chernobyl_disaster#Accident https://en.wikipedia.org/wiki/Chernobyl_disaster#Accident .
  - exar0815 10y ago
    Yeah, my comment doesnt make that much sense, reading it again. That was just an example how badly trained personnel can make any accident worse, justifying a very well trained operator for crucial automated systems.
- dougk16 10y ago
  "Every automated System needs a very well trained and calm operator when it all goes south." I think that's sort of the paradox discussed in the article, or at least one that comes to my mind. The more automation you have in aggregate in society the less you'll have trained humans capable of responding when it goes south. Even individuals that keep up with regulated training and certification requirements will get lazy if 20 years pass without an incident. But then the flipside is that hey 20 years passed without incident. It will be interesting to see where the equilibrium is reached, especially considering that that one-in-twenty-year incident by a machine will probably be weighted just as high in public perception as 20 years of human screw ups.
  - digi_owl 10y ago
    Brings to mind Burke's Connections series. Specifically the first episode, where he presents all the technology needed to power New York City. Damn it, ever so often i sit in near awe that i can be typing this message and expect it to reach the server etc. The number of wires and circuits that need to work properly for that to happen is staggering.
userbinator 10y ago
It is notable that this article was published only months after AF447[1] which crashed also due to pilots' lack of experience in flying without automation. [1] https://en.wikipedia.org/wiki/Air_France_Flight_447 https://en.wikipedia.org/wiki/Air_France_Flight_447
munificent 10y ago
There's a weird vibe in this article I don't like. It (correctly) notes that as the number of anomalies that the automation passes on to the human operators goes down, the rate that humans successfully handle them also goes down. But it doesn't seem to do a good job of clarifying that the total number of incorrectly handled anomalies is still decreasing. Let's say your automation goes from handling 90% to 99% of the anomalies and that when it does handle one, it does so correctly. We'll say that the increased rarity of human interaction and the inattention and weakened training that causes makes the human pilots to go from being able to handle 90% of them correctly to only a terrifying 40%. Let's run the simulation. With the old automation: 1000 anomalies occur 100 (10%) make it past the automation 10 (10%) make it past the human operators So 10 catastrophes. Now with the new moderately better automation and much worse human performance: 1000 anomalies occur 10 (1%) make it past the automation 6 (60%) make it past the human operators 6 catastrophes. Even though the human performance was much worse, because they are the last stage in the pipeline, it has a lower effect. Now, I just pulled these numbers out of my ass, but I think it's important to focus on the total number of automation+human failures and not single out one stage or the other. From the passenger's perspective, they don't care who saved their ass, just that it got saved. If we can make one stage more failure proof at the expense of the other, it can still be a net win.
- galdosdi 10y ago
  Also, there are things organizations can do to combat the problem of human operators getting rusty. Practice. A lot of organizations just don't do it though because it's too tempting to view the automation's cost savings as "free" and just take them for granted, but it can help a lot.
  - thaumasiotes 10y ago
    People get rusty for a reason. Your solution suffers from a couple of problems: - Practice isn't the same thing as actual events. Being good at practice is more likely to diverge from being good at crisis response as actual crises become rare, because the criterion of matching what would happen in a crisis gets much less important. Thus, peacetime militaries often need radical overhauling before they can really get much accomplished when war breaks out. (Also consider - we have a lot of people who are really interested in how medieval combat (whether in a battle line or a duel) worked, what effective use of weapons looked like, and so on. But for all the discussion, we don't know, and we can't know unless we actually stage regular battles-to-the-death with period tooling. One form of combat practice, however, has been preserved as European fencing. How closely does it correspond? Again, we don't know, but consensus is "not well".) - The automation's cost savings are free -- in fact, in this example, they have a large negative cost, cutting catastrophes by 40%. Keeping everyone in shape to handle crises they're likely to never actually see is, arguably, an enormous waste of money. (In addition to actually being impossible much of the time, as in my first bullet point.)
Jtsummers 10y ago
However, when the second accelerometer failed, a latent software anomaly allowed inputs from the first faulty accelerometer to be used, resulting in the erroneous feed of acceleration information into the flight control systems. The anomaly, which lay hidden for a decade, wasn’t found in testing because the ADIRU’s designers had never considered that such an event might occur. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ An important lesson. Assumptions can kill.
- ArkyBeagle 10y ago
  Somehow, "never considered" seems a different thing than "assumed it would never happen".
  - Jtsummers 10y ago
    My reading is that the assumption was: There will never be two faulty sensors in flight.
    - ArkyBeagle 10y ago
      Understood. It's just ambiguous. Being nauseatingly pedantic :), it just seems that the probability of not thinking of it is higher than thinking of it and dismissing it. I just have a failure-fetish :) What ya gonna do?
      - Jtsummers 10y ago
        That's fair. And see my comment in the parallel thread. In that situation, I think the original tester just never conceived of the possibility that there would be a failure with reporting of fires when two systems which functioned individually were run together. So the procedures didn't have the situation defined in the test plan. It was in beefing up the test procedures that the error was discovered (and others).
        ArkyBeagle 10y ago
mschuster91 10y ago
For this reason, U-Bahn (subway) drivers in Munich have to randomly drive under signalling (i.e. total manual control), while "normal" operation is that the computer handles everything from acceleration over cruise to stopping at the station. S-Bahn (in Munich) is fully manual, too, but augmented.
- scotty79 10y ago
  > drive under signalling How would that help if signalling failed?
I_HALF_CATS 10y ago
Also check out the podcast by "99% Invisible" http://99percentinvisible.org/episode/children-of-the-magenta-automation-paradox-pt-1/ http://99percentinvisible.org/episode/children-of-the-magent...
pipio21 10y ago
I don't really understand. If one accelerometer fails for years and nothing is reported it is a big failure on the design team, probably with criminal responsibility. Depending on GPS for main navigation is also very bad idea. In the near future with the cost of today one fiber optic gyro and accelerometer you will be able to buy ten. Software will improve and redundancy like it has done tremendously in the past making airplanes the safest of transports precisely because it does not depend so much on fallible humans that get tired and need to rest,pee and other biological necessities, have ego(that blinds their judgments) or get in love with the air hostess,get bored(some flying could bore you to tears) have problems of vision or hearing with age, get distracted(and lose situational awareness) or ill or intoxicated by food. It is easy to forget that death was what we had when humans were in charge. We are talking about thousands of times more dangerous than today. So the title is yellow sensationalistic garbage. The only reason humans have not been completely replaced is because people naturally trust other people more than machines, landing on side winds automatically requires engineers taking responsibility for it(and nobody had sone so, so far), and someone needs to be in charge in the plane at all time(for example what to do if a person have an stroke).
lunchTime42 10y ago
Could the decay of abilitys be avoided if the supervisors where kept in constant uncertainty wether the system is working?
- Thriptic 10y ago
  Possibly but it would probably lead to people disregarding the data they are being provided with, effectively removing a lot of the benefits of the automation.
edem 10y ago
This is almost exactly the same as the Law of leaking abstractions don't you think?
ChoHag 10y ago
> “People, after all, are the backup systems, and they aren’t being exercised.” If it's not tested, you don't have a backup.
scotty79 10y ago
If you need operator to be ready to take over then allow him to play a game with the vehicle he drives. Give him points for how close his attempts at controlling vehicle are to what the software that actually contols the vehicle does. This way if emergency that can be handled by automation arrives he can train for it without risk but when you need to hand over the control to him he'll be ready and aware and do his best.
outworlder 10y ago
> As the plane passed 39 000 feet, the stall and overspeed warning indicators came on simultaneously—something that’s supposed to be impossible, and a situation the crew is not trained to handle. But it is not impossible at all! That's called the "coffin corner". All flight crews are aware of it.

100 points by mcspecter 10y ago

Every automated System needs a very well trained and calm operator when it all goes south. Thats the difference between e.g. Chernobyl and Fukushima/Three Mile Island. While very bad accidents, the last fail-save, the humans, didnt fuck it completely and spectacularly in the latter two cases.

throwanem 10y ago
Chernobyl isn't a very good example of automation failure; responsibility for that disaster lies entirely with human beings from start to finish. Wikipedia's summary is solid, and rather than excerpt it here I'll just point you at https://en.wikipedia.org/wiki/Chernobyl_disaster#Accident https://en.wikipedia.org/wiki/Chernobyl_disaster#Accident .
- exar0815 10y ago
  Yeah, my comment doesnt make that much sense, reading it again. That was just an example how badly trained personnel can make any accident worse, justifying a very well trained operator for crucial automated systems.
dougk16 10y ago
"Every automated System needs a very well trained and calm operator when it all goes south." I think that's sort of the paradox discussed in the article, or at least one that comes to my mind. The more automation you have in aggregate in society the less you'll have trained humans capable of responding when it goes south. Even individuals that keep up with regulated training and certification requirements will get lazy if 20 years pass without an incident. But then the flipside is that hey 20 years passed without incident. It will be interesting to see where the equilibrium is reached, especially considering that that one-in-twenty-year incident by a machine will probably be weighted just as high in public perception as 20 years of human screw ups.
- digi_owl 10y ago
  Brings to mind Burke's Connections series. Specifically the first episode, where he presents all the technology needed to power New York City. Damn it, ever so often i sit in near awe that i can be typing this message and expect it to reach the server etc. The number of wires and circuits that need to work properly for that to happen is staggering.

userbinator 10y ago

It is notable that this article was published only months after AF447[1] which crashed also due to pilots' lack of experience in flying without automation. [1] https://en.wikipedia.org/wiki/Air_France_Flight_447 https://en.wikipedia.org/wiki/Air_France_Flight_447

munificent 10y ago

There's a weird vibe in this article I don't like. It (correctly) notes that as the number of anomalies that the automation passes on to the human operators goes down, the rate that humans successfully handle them also goes down. But it doesn't seem to do a good job of clarifying that the total number of incorrectly handled anomalies is still decreasing. Let's say your automation goes from handling 90% to 99% of the anomalies and that when it does handle one, it does so correctly. We'll say that the increased rarity of human interaction and the inattention and weakened training that causes makes the human pilots to go from being able to handle 90% of them correctly to only a terrifying 40%. Let's run the simulation. With the old automation: 1000 anomalies occur 100 (10%) make it past the automation 10 (10%) make it past the human operators So 10 catastrophes. Now with the new moderately better automation and much worse human performance: 1000 anomalies occur 10 (1%) make it past the automation 6 (60%) make it past the human operators 6 catastrophes. Even though the human performance was much worse, because they are the last stage in the pipeline, it has a lower effect. Now, I just pulled these numbers out of my ass, but I think it's important to focus on the total number of automation+human failures and not single out one stage or the other. From the passenger's perspective, they don't care who saved their ass, just that it got saved. If we can make one stage more failure proof at the expense of the other, it can still be a net win.

galdosdi 10y ago
Also, there are things organizations can do to combat the problem of human operators getting rusty. Practice. A lot of organizations just don't do it though because it's too tempting to view the automation's cost savings as "free" and just take them for granted, but it can help a lot.
- thaumasiotes 10y ago
  People get rusty for a reason. Your solution suffers from a couple of problems: - Practice isn't the same thing as actual events. Being good at practice is more likely to diverge from being good at crisis response as actual crises become rare, because the criterion of matching what would happen in a crisis gets much less important. Thus, peacetime militaries often need radical overhauling before they can really get much accomplished when war breaks out. (Also consider - we have a lot of people who are really interested in how medieval combat (whether in a battle line or a duel) worked, what effective use of weapons looked like, and so on. But for all the discussion, we don't know, and we can't know unless we actually stage regular battles-to-the-death with period tooling. One form of combat practice, however, has been preserved as European fencing. How closely does it correspond? Again, we don't know, but consensus is "not well".) - The automation's cost savings are free -- in fact, in this example, they have a large negative cost, cutting catastrophes by 40%. Keeping everyone in shape to handle crises they're likely to never actually see is, arguably, an enormous waste of money. (In addition to actually being impossible much of the time, as in my first bullet point.)

Jtsummers 10y ago

However, when the second accelerometer failed, a latent software anomaly allowed inputs from the first faulty accelerometer to be used, resulting in the erroneous feed of acceleration information into the flight control systems. The anomaly, which lay hidden for a decade, wasn’t found in testing because the ADIRU’s designers had never considered that such an event might occur. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ An important lesson. Assumptions can kill.

ArkyBeagle 10y ago
Somehow, "never considered" seems a different thing than "assumed it would never happen".
- Jtsummers 10y ago
  My reading is that the assumption was: There will never be two faulty sensors in flight.
  - ArkyBeagle 10y ago
    Understood. It's just ambiguous. Being nauseatingly pedantic :), it just seems that the probability of not thinking of it is higher than thinking of it and dismissing it. I just have a failure-fetish :) What ya gonna do?
    - Jtsummers 10y ago
      That's fair. And see my comment in the parallel thread. In that situation, I think the original tester just never conceived of the possibility that there would be a failure with reporting of fires when two systems which functioned individually were run together. So the procedures didn't have the situation defined in the test plan. It was in beefing up the test procedures that the error was discovered (and others).
      - ArkyBeagle 10y ago

mschuster91 10y ago

For this reason, U-Bahn (subway) drivers in Munich have to randomly drive under signalling (i.e. total manual control), while "normal" operation is that the computer handles everything from acceleration over cruise to stopping at the station. S-Bahn (in Munich) is fully manual, too, but augmented.

scotty79 10y ago
> drive under signalling How would that help if signalling failed?

I_HALF_CATS 10y ago

Also check out the podcast by "99% Invisible" http://99percentinvisible.org/episode/children-of-the-magenta-automation-paradox-pt-1/ http://99percentinvisible.org/episode/children-of-the-magent...

pipio21 10y ago

I don't really understand. If one accelerometer fails for years and nothing is reported it is a big failure on the design team, probably with criminal responsibility. Depending on GPS for main navigation is also very bad idea. In the near future with the cost of today one fiber optic gyro and accelerometer you will be able to buy ten. Software will improve and redundancy like it has done tremendously in the past making airplanes the safest of transports precisely because it does not depend so much on fallible humans that get tired and need to rest,pee and other biological necessities, have ego(that blinds their judgments) or get in love with the air hostess,get bored(some flying could bore you to tears) have problems of vision or hearing with age, get distracted(and lose situational awareness) or ill or intoxicated by food. It is easy to forget that death was what we had when humans were in charge. We are talking about thousands of times more dangerous than today. So the title is yellow sensationalistic garbage. The only reason humans have not been completely replaced is because people naturally trust other people more than machines, landing on side winds automatically requires engineers taking responsibility for it(and nobody had sone so, so far), and someone needs to be in charge in the plane at all time(for example what to do if a person have an stroke).

lunchTime42 10y ago

Could the decay of abilitys be avoided if the supervisors where kept in constant uncertainty wether the system is working?

Thriptic 10y ago
Possibly but it would probably lead to people disregarding the data they are being provided with, effectively removing a lot of the benefits of the automation.

edem 10y ago

This is almost exactly the same as the Law of leaking abstractions don't you think?

ChoHag 10y ago

> “People, after all, are the backup systems, and they aren’t being exercised.” If it's not tested, you don't have a backup.

scotty79 10y ago

If you need operator to be ready to take over then allow him to play a game with the vehicle he drives. Give him points for how close his attempts at controlling vehicle are to what the software that actually contols the vehicle does. This way if emergency that can be handled by automation arrives he can train for it without risk but when you need to hand over the control to him he'll be ready and aware and do his best.

outworlder 10y ago

> As the plane passed 39 000 feet, the stall and overspeed warning indicators came on simultaneously—something that’s supposed to be impossible, and a situation the crew is not trained to handle. But it is not impossible at all! That's called the "coffin corner". All flight crews are aware of it.