6 ms·

My knee jerk reaction to this is "more automation after an automation failure? Huh?". But whenever I think someone is being that stupid, it's always worth a sec

My knee jerk reaction to this is "more automation after an automation failure? Huh?". But whenever I think someone is being that stupid, it's always worth a second look, and it turns out the reality is more complex. This article[1] linked off of the OP goes into detail about their crashes. It describes a design philosophy that pushes robustness fully onto the pilot, expecting him to flawlessly stitch together independent automated systems and unreasonably rapidly interpret and react to their error modes. In the case described in the article, separate automated systems had contrasting views of the world, and shouted their contradictory errors at the pilots, relying on them to be able to instantly reconcile these models and figure out what the root problem was. (In this case, a specific sensor failure made a single system think the plane was pointed up, and this system made the isolated decision to push the nose downwards while warning that the nose was too high. Naturally, other systems simultaneously complained about the nosedive). One solution, of course, is to remove automated systems from planes. But assuming they provide any value (I don't think this is in dispute?): another, more reasonable strategy is to commit more fully to an automated system design that is intentionally fault-tolerant, robust to system and sensor errors, and has a more complete view of the plane's world. Instead of requiring the overall state of the meta-system to be lossily communicated to humans in a time-critical emergency, the newer designs are intended to use all the information available to the plane's systems to reconcile subcompoments' contradictory view of the world. Note that this doesn't preclude human control at all: if anything, it makes pilot control _safer_, since the system can provide a coherent view to the pilot in a way that's currently impossible. I don't know enough about large aircraft operation to fully comment on the wisdom of this approach, but it's substantially more complex than "auto system failed, why would they expand use". [1] https://www.wsj.com/articles/the-four-second-catastrophe-how-boeing-doomed-the-737-max-11565966629 https://www.wsj.com/articles/the-four-second-catastrophe-how...

phkahler 7y ago
As a student pilot in a Cessna, I kept thinking how much even simple automation could help. Why do I need to deal with carb heat for example. The answer in almost every case is that things are simpler and probably safer the way they are. Once you start down the automation path, you also want fail-safe redundancy and systems that cooperate with each other when there is a problem. I can see how it can quickly get out of hand. The more complex something gets, the harder it is for human to intervene when there's a problem. I can now argue either way about it.
Traster 7y ago
I think there's a different way to explore this idea. I don't think automation had anything to do with the crashes. The problem was that the pilots weren't trained to know what tools the planes were equiped with and how to use them. Those crashes are like when I accidentally open a file with vim. There's nothing wrong with me, there's nothing wrong with vim but how the hell am I meant to know to hit esc :q? The lesson is orthogonal to automation. The lesson is that the pilots need to know what each system does and how to operate it. If the resolve of Boeing is to push forward with automation without taking the role of the pilot seriously, then this is doomed to failure, because automated systems will fail, and pilots will have to step in. It'll be rare, but that's what pilots are there for. Indeed, the more you automate things, the more liability you're inviting on yourself, which should be worrying for Boeing.
- paranoidrobot 7y ago
  > Those crashes are like when I accidentally open a file with vim. There's nothing wrong with me, there's nothing wrong with vim I disagree, strongly. The root cause of the issue isn't pilot training. The root fault is a design issue. Without the existence of MCAS, these crashes would not have occurred. Why? Because they would never have been built. Why? Because the 737 MAX 8 design would not have received approval to be incorporated into the 737 type certificate. Why? Because the MAX 8, without MCAS, has different handling characteristics during certain phases of flight than the original 737. Why does it handle differently? Because it has larger and more powerful engines, located differently on the wing, and these changes cause the aircraft to want to pull up more when climbing. I don't know that the MCAS's programmed behaviors have a good analogy, I can't think of one right now.
  - bkdbkd 7y ago
    Those crashes are like: You tell your machine to open the file with Emacs, but this new distro has unbeknownst to you (and you find out later, undocumented) symlinked the emacs command to a version of vim with a emacs-like overlay, because of some issue they had. The file crashes vim, but in a crazy memory overflow before he dies manages to take out the system, causing spurious disk writes which corrupts your boot drive. - The MCAS system. Got bad information and did a very bad thing. It was the entire time operating as it was designed. The fact that the astronauts were not given the information on how to turn HAL off in the event he goes berserk isn't really their fault. Just to be clear - The MCAS system was actively trying to fly the plane into the ground. It thought it was trying to save the plane from a stall. It was wrong. The pilots attempted to override the system. They could not. They attempted to outfly the system. They could not. They could have killed the system. Boeing did not teach them how.

phkahler 7y ago

As a student pilot in a Cessna, I kept thinking how much even simple automation could help. Why do I need to deal with carb heat for example. The answer in almost every case is that things are simpler and probably safer the way they are. Once you start down the automation path, you also want fail-safe redundancy and systems that cooperate with each other when there is a problem. I can see how it can quickly get out of hand. The more complex something gets, the harder it is for human to intervene when there's a problem. I can now argue either way about it.

Traster 7y ago

I think there's a different way to explore this idea. I don't think automation had anything to do with the crashes. The problem was that the pilots weren't trained to know what tools the planes were equiped with and how to use them. Those crashes are like when I accidentally open a file with vim. There's nothing wrong with me, there's nothing wrong with vim but how the hell am I meant to know to hit esc :q? The lesson is orthogonal to automation. The lesson is that the pilots need to know what each system does and how to operate it. If the resolve of Boeing is to push forward with automation without taking the role of the pilot seriously, then this is doomed to failure, because automated systems will fail, and pilots will have to step in. It'll be rare, but that's what pilots are there for. Indeed, the more you automate things, the more liability you're inviting on yourself, which should be worrying for Boeing.

paranoidrobot 7y ago
> Those crashes are like when I accidentally open a file with vim. There's nothing wrong with me, there's nothing wrong with vim I disagree, strongly. The root cause of the issue isn't pilot training. The root fault is a design issue. Without the existence of MCAS, these crashes would not have occurred. Why? Because they would never have been built. Why? Because the 737 MAX 8 design would not have received approval to be incorporated into the 737 type certificate. Why? Because the MAX 8, without MCAS, has different handling characteristics during certain phases of flight than the original 737. Why does it handle differently? Because it has larger and more powerful engines, located differently on the wing, and these changes cause the aircraft to want to pull up more when climbing. I don't know that the MCAS's programmed behaviors have a good analogy, I can't think of one right now.
- bkdbkd 7y ago
  Those crashes are like: You tell your machine to open the file with Emacs, but this new distro has unbeknownst to you (and you find out later, undocumented) symlinked the emacs command to a version of vim with a emacs-like overlay, because of some issue they had. The file crashes vim, but in a crazy memory overflow before he dies manages to take out the system, causing spurious disk writes which corrupts your boot drive. - The MCAS system. Got bad information and did a very bad thing. It was the entire time operating as it was designed. The fact that the astronauts were not given the information on how to turn HAL off in the event he goes berserk isn't really their fault. Just to be clear - The MCAS system was actively trying to fly the plane into the ground. It thought it was trying to save the plane from a stall. It was wrong. The pilots attempted to override the system. They could not. They attempted to outfly the system. They could not. They could have killed the system. Boeing did not teach them how.