3 ms·

Their automated systems caused two crashes and their solution is to automate more systems? Have they literally gone insane?

deleted 7y ago
[deleted]
wutbrodo 7y ago
This article[1] linked off of the OP goes into detail about their crashes. It describes a design philosophy that pushes robustness fully onto the pilot, expecting him to flawlessly stitch together independent automated systems and unreasonably rapidly interpret and react to their error modes. In the case described in the article, separate automated systems had contrasting views of the world, and shouted their contradictory errors at the pilots, relying on them to be able to instantly reconcile these models and figure out what the root problem was. (In this case, a specific sensor failure made a single system think the plane was pointed up, and this system made the isolated decision to push the nose downwards while warning that the nose was too high. Naturally, other systems simultaneously complained about the nosedive). One solution, of course, is to remove automated systems from planes. But assuming they provide any value (I don't think this is in dispute?): another, more reasonable strategy is to commit more fully to an automated system design that is intentionally fault-tolerant, robust to system and sensor errors, and has a more complete view of the plane's world. Instead of requiring the overall state of the meta-system to be lossily communicated to humans in a time-critical emergency, the newer designs are intended to use all the information available to the plane's systems to reconcile subcompoments' contradictory view of the world. Note that this doesn't preclude human control at all: if anything, it makes pilot control _safer_, since the system can provide a coherent view to the pilot in a way that's currently impossible. I don't know enough about large aircraft operation to fully comment on the wisdom of this approach, but it's substantially more complex than "auto system failed, why would they expand use". [1] https://www.wsj.com/articles/the-four-second-catastrophe-how-boeing-doomed-the-737-max-11565966629 https://www.wsj.com/articles/the-four-second-catastrophe-how...
- tehjoker 7y ago
  The auto system failed because they a) were band-aiding a flawed mechanical design b) under-designed the compensating software (only provided a single AOA sensor by default) to obscure the fact the plane was significantly different to avoid training costs. These are management problems, not technical problems. More automation would be subject to the same issues.
  - wutbrodo 7y ago
    Crap I meant to make the above comment as a top level comment, not a response.. Somehow it came through twice. Anyway, in response to your comment: perhaps I'm conflating these, but a commitment to robustness seems like it's included in the newer philosophy? It just extends that notion beyond the sensor-level to the system level. Sure, you can frame an inadequate emphasis on robustness as a "management problem" instead of a technical one, but that seems equally applicable to the "management solution" being put forth, no?
    - tehjoker 7y ago
      I would say that framing a business problem in engineering terms will lead to an unclear understanding of the problem. The executives were attempting to thread the needle of a potentially business killing competition with Airbus and were willing to make compromises to safety to save their skins. No engineering methodology can save you from motivated reasoning and systems level problems like that because management will force exceptions to whatever process is in place. The only thing that would be an appropriate counterweight is if the engineering staff refuses to put a defective design into production. A robust engineering process can make violations and exceptions clear, but if the staff still does what management says, it won't help.
- fgonzag 7y ago
  The problem is we don't trust Boeing to develop these automated systems, considering they developed a system with enough authority to crash the plane into the ground. essentially 0 sensor redundancy (alternating between sensors is non-redundant)

deleted 7y ago

[deleted]

wutbrodo 7y ago

This article[1] linked off of the OP goes into detail about their crashes. It describes a design philosophy that pushes robustness fully onto the pilot, expecting him to flawlessly stitch together independent automated systems and unreasonably rapidly interpret and react to their error modes. In the case described in the article, separate automated systems had contrasting views of the world, and shouted their contradictory errors at the pilots, relying on them to be able to instantly reconcile these models and figure out what the root problem was. (In this case, a specific sensor failure made a single system think the plane was pointed up, and this system made the isolated decision to push the nose downwards while warning that the nose was too high. Naturally, other systems simultaneously complained about the nosedive). One solution, of course, is to remove automated systems from planes. But assuming they provide any value (I don't think this is in dispute?): another, more reasonable strategy is to commit more fully to an automated system design that is intentionally fault-tolerant, robust to system and sensor errors, and has a more complete view of the plane's world. Instead of requiring the overall state of the meta-system to be lossily communicated to humans in a time-critical emergency, the newer designs are intended to use all the information available to the plane's systems to reconcile subcompoments' contradictory view of the world. Note that this doesn't preclude human control at all: if anything, it makes pilot control _safer_, since the system can provide a coherent view to the pilot in a way that's currently impossible. I don't know enough about large aircraft operation to fully comment on the wisdom of this approach, but it's substantially more complex than "auto system failed, why would they expand use". [1] https://www.wsj.com/articles/the-four-second-catastrophe-how-boeing-doomed-the-737-max-11565966629 https://www.wsj.com/articles/the-four-second-catastrophe-how...

tehjoker 7y ago
The auto system failed because they a) were band-aiding a flawed mechanical design b) under-designed the compensating software (only provided a single AOA sensor by default) to obscure the fact the plane was significantly different to avoid training costs. These are management problems, not technical problems. More automation would be subject to the same issues.
- wutbrodo 7y ago
  Crap I meant to make the above comment as a top level comment, not a response.. Somehow it came through twice. Anyway, in response to your comment: perhaps I'm conflating these, but a commitment to robustness seems like it's included in the newer philosophy? It just extends that notion beyond the sensor-level to the system level. Sure, you can frame an inadequate emphasis on robustness as a "management problem" instead of a technical one, but that seems equally applicable to the "management solution" being put forth, no?
  - tehjoker 7y ago
    I would say that framing a business problem in engineering terms will lead to an unclear understanding of the problem. The executives were attempting to thread the needle of a potentially business killing competition with Airbus and were willing to make compromises to safety to save their skins. No engineering methodology can save you from motivated reasoning and systems level problems like that because management will force exceptions to whatever process is in place. The only thing that would be an appropriate counterweight is if the engineering staff refuses to put a defective design into production. A robust engineering process can make violations and exceptions clear, but if the staff still does what management says, it won't help.
fgonzag 7y ago
The problem is we don't trust Boeing to develop these automated systems, considering they developed a system with enough authority to crash the plane into the ground. essentially 0 sensor redundancy (alternating between sensors is non-redundant)