7 ms·

The Hazards of Going on Autopilot

52 points by krallja 12y ago

ghjm 12y ago
I would like to object to the following statement from the article: > While lifeguards are taught all the possible signs of a person who is drowning, pilots don’t receive elaborate training on all the things that can go wrong, precisely because the many things that can go wrong so rarely do. In fact, pilots are given extensive training in the failure modes of their airplanes. When Casner got his private pilot license, he should have been told to memorize the emergency procedures for his airplane. De novo private pilots are not given the same level of training as airplane pilots, but failure modes and "interesting-looking instrument panels" are crucial components of instrument training. More generally, I don't disagree with his conclusions regarding the problems with cockpit automation, but they are hardly novel - the industry has been aware of this for years. However, the service life of an airliner is measured in decades, and the regulatory environment, for excellent reasons, doesn't allow us to change the avionics on existing airliners without re-certifying the new systems and re-training all the pilots. Last but not least, I'd like to point out that even if the airplane is on autopilot, there is still plenty for the pilots to do. For example, they are supposed to brief the instrument approach procedure and the missed approach procedure every single time they fly it. A lot of pilots relax this discipline (if you've flown into the same airport hundreds of times, it's hard to argue that re-reading the procedure yet again has much value). But relaxing it to the point of just idly chatting all the way down the glideslope? That's not a problem with the cockpit systems, and it's hard to see how better human factors would have magically turned these incredibly irresponsible people into good pilots.
- deleted 12y ago
  [deleted]
- the_af 12y ago
  Isn't the argument from the article that the pilots are just idly chatting all the way down because of excessive automation? If, like the article claims, human error is the leading cause of accidents (and I assume it is implied that distraction is a major factor in human error, though the article doesn't explicitly claim this), isn't mindless automation a major problem that should be addressed? To quote NASA's researcher: "Companies were introducing increasingly specialized automated functions to address particular errors without looking at their over-all effects [...] As it stood, increased automation hadn't reduced human errors on the whole; it had simply changed their form." I don't know if it's accurate to call these pilots "irresponsible people". Probably it is. But how does this help in reducing accidents?
  - gwern 12y ago
    > Isn't the argument from the article that the pilots are just idly chatting all the way down because of excessive automation? Yeah, but there seems to be an implicit argument appended: "and this causes more accidents than the automation prevents". I'm not sure about this one. Aren't jet planes ever safer to fly in? I was under the impression that the death rates kept going down. So the backfire effect from this automation can't be too bad or else net safety wouldn't increase. I think you could turn it into a weaker more valid argument: 'and the mind-wandering sets a bound on how safe air travel can ever get with human pilots because automation itself introduce human error'. But when you make it explicit like that, it starts to look like an argument for taking humans out of the loop entirely...
    - ekianjo 12y ago
      In the near/mid-term it would probably be a good course of action to have entirely automated airplanes, and "pilots" (or technicians or whatever you would call them) specialized in handling automation failures. That way, there would be no question about what their role is in the plane.
      - liotier 12y ago
        Pilots are all about handling failures - that is pretty much what their whole career is focused on. The normal operation of a modern aircraft is pretty boring - what makes a pilot worth his salary is all the continuous training and drilling that embedded emergency procedures into his mind at the reflex level... It is a lifelong process, regularly updated through new hardware and modified methods in response to incidents.
    - LnxPrgr3 12y ago
      It's definitely worth remembering how rare commercial airline crashes are: http://en.wikipedia.org/wiki/List_of_accidents_and_incidents_involving_commercial_aircraft#2014 http://en.wikipedia.org/wiki/List_of_accidents_and_incidents... There was also drama around a whistleblower being ignored and demoted after bringing up safety concerns about Colgan Air: http://www.nytimes.com/2009/06/04/nyregion/04colgan.html?_r=2&hp& http://www.nytimes.com/2009/06/04/nyregion/04colgan.html?_r=... Check ride stall recovery is now graded differently too, thanks to this incident: http://www.flyingmag.com/pilots-places/pilots-adventures-more/going-direct-push-or-pull http://www.flyingmag.com/pilots-places/pilots-adventures-mor..., under the belief they could've been making pilots instinctively afraid to lose altitude during stall recovery. It's also believed the pilots on this flight were fatigued, and performance was impaired. Given how bad humans are at even driving cars while exhausted, it's not surprising it's hard to fly a plane in that condition. Full disclosure: All but the first link are cited by the same Wikipedia article: http://en.wikipedia.org/wiki/Colgan_Air_Flight_3407 http://en.wikipedia.org/wiki/Colgan_Air_Flight_3407. I immediately recognized the story of this crash because I have a bit of a fascination with how things can go horribly wrong. This layman also cringed while watching a TV documentary about this flight the moment the reenactment pilot pulled up during a stall warning, and again when the co-pilot retracted the flaps. "Are you trying to crash‽" is what I wanted to yell at my monitor.
- frobozz 12y ago
  I'd also disagree with that same assertion with regard to lifeguards: Drownings (and near-drownings prevented by a rescue) are pretty uncommon, too. It is precisely because of the rarity that one needs to be trained to spot the problem. If they happened all the time, then it would merely be a matter of the journeyman pointing one out to the apprentice when it happens. Instead, one has to rely on a combination of book study and roleplay. This is even more the case in a pool situation (as described in the article). I was a lifeguard in a very busy pool for five years, in that time, I performed very few rescues, some of which would better be described as "assistances", only one would have been likely to turn into a drowning without a rescue. I might add, good lifeguarding is not about watching for signs that someone might be drowning, it's about watching for signs that might precede trouble. I'm no pilot, so I have no idea how many different things might go wrong on a plane, but I assume by the comparative duration and cost of training that it's a bit more complex than watching people swim. I'd be shocked if pilot training was strictly happy-path. I'd expect (as you state) that they would be assessed on their ability to interpret and react to things going wrong. Otherwise, what's the point of all the instrumentation?
Strilanc 12y ago
An off the wall idea: have the pilot duplicate work being done by the computer. Not to control the plane, but just to keep them alert. i.e. imagine a driverless car that only works when you tell it to do approximately what it was going to do anyway. Otherwise it beeps at you and pulls over.
- damian2000 12y ago
  I think pilots are already meant to do that to a certain degree - i.e. monitor their instruments, navigation, and understand what the autopilot is doing at all times. I think its called situational awareness.
ejdyksen 12y ago
See also, Children of the Magenta, a fantastic talk from American Airlines in the late 90s: http://vimeo.com/64502012 http://vimeo.com/64502012 (Magenta refers to the color on the displays of the Flight Directory/Autopilot's projected path)
sokoloff 12y ago
Colgan 3407 crashed because of a profound and astounding lack of basic airmanship and knowledge of the effects of icing on aircraft performance. Yes, the autopilot helped mask the knowledge problem and presented them with a situation where that demonstrated lack of knowledge (that it takes more power to fly an airplane with ice on the wings) contributed to a stick shaker event (a warning of impending aerodynamic stall due to excessive angle of attack) whereupon the captain (and it's hard to use that term given the airmanship circus that ensued) decided to hold the stick back, exacerbating the stall condition rather than doing what every 5-hour pilot is taught to do upon a stall warning: level the wings, lower the nose, and add power. The autopilot is a minuscule contributor to that accident, IMO. At most, the presence of autopilots allowed unqualified airman to not wash out of airline jobs, but to blame that crash on the autopilot is to misunderstand how non-complex that situation was that swallowed the crew and all aboard.
- ninkendo 12y ago
  In flight school one of the very first things they teach you in flight is what to do when the stall warning goes off. You always nose down. This story sounds like if I were driving a car and gradually started to drift into an oncoming lane, and upon the car's warning system alerting me to this, I jerked the steering wheel into the oncoming lane instead of out of it. At some point the pilot is just incompetent, you can't just blame the automation.
- dba7dba 12y ago
  > Colgan 3407 crashed because of a profound and astounding lack of basic airmanship I agree. I think the New Yorker just picked wrong incident to try to advance their theory. The particular pilot in Colgan 3407 flight FAILED a check ride in another airline. I understand this check ride is after you have already flown hundreds/thousands of flights and he somehow failed it.
  - ubernostrum 12y ago
    It's not just the Colgan incident; investigation into the Air France 447 crash concluded the crew did not recognize a stall, and may have been relying on an incorrect belief that the aircraft would not accept inputs that would produce a stall (which, for Airbus aircraft, is true except in alternate-law situations which occur when -- as with AF447 -- the flight systems notice a fault in data).
damian2000 12y ago
I don't know if it was mentioned in the article, but after the stick shaker there was also an automatic "stick pusher" event whereby the aircraft tried to automatically recover from an impending stall, when the pilots don't. The pilot overode it by pulling back on the stick thereby putting it into a stall.
phrogdriver 12y ago
The suggestion that complacency due to automated aircraft systems is somehow not acknowledged or well-known is simply false. It is a well documented occurence that we train for, both in terms of recognizing it in ourselves and others, and in combating it. Most of the "we didn't get to fly with all of this new fangled autopilot" is as misguided as the "we should add another computer to solve this complacency problem too."
chanon 12y ago
The main problem (quote from the article): "What we’re doing is using human beings as safety nets or backups to computers, and that’s completely backward" Made me stop and think a bit when I read this, very true.
- snowwrestler 12y ago
  I don't think it's backwards; humans are much more flexible and capable than computers, so it is right that they back up the computers not vice-versa.
  - LnxPrgr3 12y ago
    Computers overriding human instructions have also caused crashes. http://en.wikipedia.org/wiki/Scandinavian_Airlines_Flight_751 http://en.wikipedia.org/wiki/Scandinavian_Airlines_Flight_75... -- engines ingested ice and started surging. Pilots commanded reduced power, but the Automatic Thrust Restoration system no one told them was there countered that move, leading to a double engine failure and a crash landing. http://en.wikipedia.org/wiki/China_Airlines_Flight_140 http://en.wikipedia.org/wiki/China_Airlines_Flight_140 -- pilot accidentally flipped the takeoff/go-around switch, leading the autopilot to counter the crew's attempt to recover, causing a stall and crash.
laggyluke 12y ago
Makes me wonder, how often does accidents like Colgan happen that don't result in a crash? Are they even public? Investigated?

52 points by krallja 12y ago

ghjm 12y ago

I would like to object to the following statement from the article: > While lifeguards are taught all the possible signs of a person who is drowning, pilots don’t receive elaborate training on all the things that can go wrong, precisely because the many things that can go wrong so rarely do. In fact, pilots are given extensive training in the failure modes of their airplanes. When Casner got his private pilot license, he should have been told to memorize the emergency procedures for his airplane. De novo private pilots are not given the same level of training as airplane pilots, but failure modes and "interesting-looking instrument panels" are crucial components of instrument training. More generally, I don't disagree with his conclusions regarding the problems with cockpit automation, but they are hardly novel - the industry has been aware of this for years. However, the service life of an airliner is measured in decades, and the regulatory environment, for excellent reasons, doesn't allow us to change the avionics on existing airliners without re-certifying the new systems and re-training all the pilots. Last but not least, I'd like to point out that even if the airplane is on autopilot, there is still plenty for the pilots to do. For example, they are supposed to brief the instrument approach procedure and the missed approach procedure every single time they fly it. A lot of pilots relax this discipline (if you've flown into the same airport hundreds of times, it's hard to argue that re-reading the procedure yet again has much value). But relaxing it to the point of just idly chatting all the way down the glideslope? That's not a problem with the cockpit systems, and it's hard to see how better human factors would have magically turned these incredibly irresponsible people into good pilots.

deleted 12y ago
[deleted]
the_af 12y ago
Isn't the argument from the article that the pilots are just idly chatting all the way down because of excessive automation? If, like the article claims, human error is the leading cause of accidents (and I assume it is implied that distraction is a major factor in human error, though the article doesn't explicitly claim this), isn't mindless automation a major problem that should be addressed? To quote NASA's researcher: "Companies were introducing increasingly specialized automated functions to address particular errors without looking at their over-all effects [...] As it stood, increased automation hadn't reduced human errors on the whole; it had simply changed their form." I don't know if it's accurate to call these pilots "irresponsible people". Probably it is. But how does this help in reducing accidents?
- gwern 12y ago
  > Isn't the argument from the article that the pilots are just idly chatting all the way down because of excessive automation? Yeah, but there seems to be an implicit argument appended: "and this causes more accidents than the automation prevents". I'm not sure about this one. Aren't jet planes ever safer to fly in? I was under the impression that the death rates kept going down. So the backfire effect from this automation can't be too bad or else net safety wouldn't increase. I think you could turn it into a weaker more valid argument: 'and the mind-wandering sets a bound on how safe air travel can ever get with human pilots because automation itself introduce human error'. But when you make it explicit like that, it starts to look like an argument for taking humans out of the loop entirely...
  - ekianjo 12y ago
    In the near/mid-term it would probably be a good course of action to have entirely automated airplanes, and "pilots" (or technicians or whatever you would call them) specialized in handling automation failures. That way, there would be no question about what their role is in the plane.
    - liotier 12y ago
      Pilots are all about handling failures - that is pretty much what their whole career is focused on. The normal operation of a modern aircraft is pretty boring - what makes a pilot worth his salary is all the continuous training and drilling that embedded emergency procedures into his mind at the reflex level... It is a lifelong process, regularly updated through new hardware and modified methods in response to incidents.
  - LnxPrgr3 12y ago
    It's definitely worth remembering how rare commercial airline crashes are: http://en.wikipedia.org/wiki/List_of_accidents_and_incidents_involving_commercial_aircraft#2014 http://en.wikipedia.org/wiki/List_of_accidents_and_incidents... There was also drama around a whistleblower being ignored and demoted after bringing up safety concerns about Colgan Air: http://www.nytimes.com/2009/06/04/nyregion/04colgan.html?_r=2&hp& http://www.nytimes.com/2009/06/04/nyregion/04colgan.html?_r=... Check ride stall recovery is now graded differently too, thanks to this incident: http://www.flyingmag.com/pilots-places/pilots-adventures-more/going-direct-push-or-pull http://www.flyingmag.com/pilots-places/pilots-adventures-mor..., under the belief they could've been making pilots instinctively afraid to lose altitude during stall recovery. It's also believed the pilots on this flight were fatigued, and performance was impaired. Given how bad humans are at even driving cars while exhausted, it's not surprising it's hard to fly a plane in that condition. Full disclosure: All but the first link are cited by the same Wikipedia article: http://en.wikipedia.org/wiki/Colgan_Air_Flight_3407 http://en.wikipedia.org/wiki/Colgan_Air_Flight_3407. I immediately recognized the story of this crash because I have a bit of a fascination with how things can go horribly wrong. This layman also cringed while watching a TV documentary about this flight the moment the reenactment pilot pulled up during a stall warning, and again when the co-pilot retracted the flaps. "Are you trying to crash‽" is what I wanted to yell at my monitor.
frobozz 12y ago
I'd also disagree with that same assertion with regard to lifeguards: Drownings (and near-drownings prevented by a rescue) are pretty uncommon, too. It is precisely because of the rarity that one needs to be trained to spot the problem. If they happened all the time, then it would merely be a matter of the journeyman pointing one out to the apprentice when it happens. Instead, one has to rely on a combination of book study and roleplay. This is even more the case in a pool situation (as described in the article). I was a lifeguard in a very busy pool for five years, in that time, I performed very few rescues, some of which would better be described as "assistances", only one would have been likely to turn into a drowning without a rescue. I might add, good lifeguarding is not about watching for signs that someone might be drowning, it's about watching for signs that might precede trouble. I'm no pilot, so I have no idea how many different things might go wrong on a plane, but I assume by the comparative duration and cost of training that it's a bit more complex than watching people swim. I'd be shocked if pilot training was strictly happy-path. I'd expect (as you state) that they would be assessed on their ability to interpret and react to things going wrong. Otherwise, what's the point of all the instrumentation?

Strilanc 12y ago

An off the wall idea: have the pilot duplicate work being done by the computer. Not to control the plane, but just to keep them alert. i.e. imagine a driverless car that only works when you tell it to do approximately what it was going to do anyway. Otherwise it beeps at you and pulls over.

damian2000 12y ago
I think pilots are already meant to do that to a certain degree - i.e. monitor their instruments, navigation, and understand what the autopilot is doing at all times. I think its called situational awareness.

ejdyksen 12y ago

See also, Children of the Magenta, a fantastic talk from American Airlines in the late 90s: http://vimeo.com/64502012 http://vimeo.com/64502012 (Magenta refers to the color on the displays of the Flight Directory/Autopilot's projected path)

sokoloff 12y ago

Colgan 3407 crashed because of a profound and astounding lack of basic airmanship and knowledge of the effects of icing on aircraft performance. Yes, the autopilot helped mask the knowledge problem and presented them with a situation where that demonstrated lack of knowledge (that it takes more power to fly an airplane with ice on the wings) contributed to a stick shaker event (a warning of impending aerodynamic stall due to excessive angle of attack) whereupon the captain (and it's hard to use that term given the airmanship circus that ensued) decided to hold the stick back, exacerbating the stall condition rather than doing what every 5-hour pilot is taught to do upon a stall warning: level the wings, lower the nose, and add power. The autopilot is a minuscule contributor to that accident, IMO. At most, the presence of autopilots allowed unqualified airman to not wash out of airline jobs, but to blame that crash on the autopilot is to misunderstand how non-complex that situation was that swallowed the crew and all aboard.

ninkendo 12y ago
In flight school one of the very first things they teach you in flight is what to do when the stall warning goes off. You always nose down. This story sounds like if I were driving a car and gradually started to drift into an oncoming lane, and upon the car's warning system alerting me to this, I jerked the steering wheel into the oncoming lane instead of out of it. At some point the pilot is just incompetent, you can't just blame the automation.
dba7dba 12y ago
> Colgan 3407 crashed because of a profound and astounding lack of basic airmanship I agree. I think the New Yorker just picked wrong incident to try to advance their theory. The particular pilot in Colgan 3407 flight FAILED a check ride in another airline. I understand this check ride is after you have already flown hundreds/thousands of flights and he somehow failed it.
- ubernostrum 12y ago
  It's not just the Colgan incident; investigation into the Air France 447 crash concluded the crew did not recognize a stall, and may have been relying on an incorrect belief that the aircraft would not accept inputs that would produce a stall (which, for Airbus aircraft, is true except in alternate-law situations which occur when -- as with AF447 -- the flight systems notice a fault in data).

damian2000 12y ago

I don't know if it was mentioned in the article, but after the stick shaker there was also an automatic "stick pusher" event whereby the aircraft tried to automatically recover from an impending stall, when the pilots don't. The pilot overode it by pulling back on the stick thereby putting it into a stall.

phrogdriver 12y ago

The suggestion that complacency due to automated aircraft systems is somehow not acknowledged or well-known is simply false. It is a well documented occurence that we train for, both in terms of recognizing it in ourselves and others, and in combating it. Most of the "we didn't get to fly with all of this new fangled autopilot" is as misguided as the "we should add another computer to solve this complacency problem too."

chanon 12y ago

The main problem (quote from the article): "What we’re doing is using human beings as safety nets or backups to computers, and that’s completely backward" Made me stop and think a bit when I read this, very true.

snowwrestler 12y ago
I don't think it's backwards; humans are much more flexible and capable than computers, so it is right that they back up the computers not vice-versa.
- LnxPrgr3 12y ago
  Computers overriding human instructions have also caused crashes. http://en.wikipedia.org/wiki/Scandinavian_Airlines_Flight_751 http://en.wikipedia.org/wiki/Scandinavian_Airlines_Flight_75... -- engines ingested ice and started surging. Pilots commanded reduced power, but the Automatic Thrust Restoration system no one told them was there countered that move, leading to a double engine failure and a crash landing. http://en.wikipedia.org/wiki/China_Airlines_Flight_140 http://en.wikipedia.org/wiki/China_Airlines_Flight_140 -- pilot accidentally flipped the takeoff/go-around switch, leading the autopilot to counter the crew's attempt to recover, causing a stall and crash.

laggyluke 12y ago

Makes me wonder, how often does accidents like Colgan happen that don't result in a crash? Are they even public? Investigated?