11 ms·

MCAS: How 737 MAX system gained power and lost safeguards

97 points by eternalban 7y ago

thatswrong0 7y ago
Previous Seattle Times article about the certification of the 737MAX: https://www.seattletimes.com/business/boeing-aerospace/failed-certification-faa-missed-safety-issues-in-the-737-max-system-implicated-in-the-lion-air-crash/ https://www.seattletimes.com/business/boeing-aerospace/faile... And HN post: https://news.ycombinator.com/item?id=19421612 https://news.ycombinator.com/item?id=19421612
paulmd 7y ago
The thinking of "we don't want the system to not work if it's needed" thinking demonstrates a failure to understand the prior probabilities involved. Most of the time the flight conditions will not warrant activation of the MCAS, and apparently the system is sufficiently unreliable/easy to damage that the chances of the system inadvertently triggering is high. But Boeing wanted to be sure that MCAS would always activate if needed, even if that resulted in unintended activations normally. Boeing essentially optimized around a low probability of a false negative, while ignoring the high probability of a false positive, because they underestimated the consequences and confusion that would result from a false positive - in many cases, multiple alarms/system failures occurring simultaneously as other systems that rely on that sensor begin alarming as well. In that environment, designing around a 3 second response time (on a system that pilots weren't trained to know exists) is probably unreasonable.
- usrusr 7y ago
  Well, Boeing is multiple people. Business culture celebrates a manager who can push their project ahead of others. Maybe MCAS had a project manager who was a bit of an overachiever in that regard.
  - jdc 7y ago
    Surely I'm not the only one reading "overachiever" here as a wry euphemism.
    - usrusr 7y ago
      Not really: person A is tasked with remeditating risk X (dangerous pitch-up at high AoA), persons B, C are tasked with remediating some other risks Y, Z, or just generally keeping an eye on the big picture. Now if person A is better than the others at pushing the needs of their particular target metric (or just more stubborn, a particularly easy form of "strength"), that plane will surely never crash due pitch up at high AoA, but maybe more likely due to other dangers that became more likely for the sake of never crashing from that pitch up problem that A was sent out to solve. This cost will be seen as a consequence of the weakness of B, C, not of the "strength" of A. The high-resolve powerplayer who pushes their assigned task ahead of big-picture concerns is the hero of ambitious middle management culture. We see that happen many times, just rarely in domains as deadly as aviation.
- mantap 7y ago
  If it were designed correctly, it shouldn't matter if it were activated or not - it would always be safe. If it were designed correctly there would be at least 3 AoA sensors and the loss or disagreement of one would not affect the functioning of the system other than displaying a warning message to the pilots. If the two remaining disagreed then the system would be disabled. This is assuming that the system is necessary for certification or safety, if it is not necessary for certification or safety then it shouldn't be there at all. The fact that Boeing put a secondary autopilot in there with a single point of failure, can only be explained by organisational failures reminiscent of the Challenger disaster.
  - salawat 7y ago
    The airframe cannot be certified airworthy without MCAS according to existing regulations. It is quite literally the only thing enabling the airframe to pass basic design requirements that have been mandated for years.
    - xtof 7y ago
      Citation? Nothing in FAR 25 permits a computer system to make airworthy an unairworthy airplane design.
      - salawat 7y ago
        Equivalent safety exception. It's the blanket exception that makes most fly-by-wire controlled aircraft airworthy. The problem here of course, is that you never had enough scrutiny paid to the fly-by-wire implementation to make sure that under no circumstances can the system fail non-operational. This was the subject of much controversy back in the 60's when th 727 was being certified. It required additional workarounds (implementation of a stick pusher) to be certified airworthy in U.K. airspace. That certification went through despite the protest of Test pilots at the time that certifying an airframe with a widget to "plaster over" non-compliant behavior would simply start a slippery slope of normalization of deviance whereby engineers would over time be able to certify less "passively" airworthy frames. Those pilots were absolutely correct; as by definition that is exactly what most fly-by-wire aircraft are. Airframes that are only rendered airworthy with the addition of computer logic. See the Royal Aeronautics Society D.P. Davies Interview. Should be in this one. The 727 section starts at 51:00 Equivalent level of Safety discussion starts at 1:01:00 [1] https://www.aerosociety.com/news/audio-the-d-p-davies-interview-on-the-boeing-747-the-trident-vc10-one-eleven-the-boeing-727/ https://www.aerosociety.com/news/audio-the-d-p-davies-interv... Highly recommend to listening to all four episodes just because. It's informative, and for me was rather illuminative in terms of the constant conflict between aerodynamicists to leverage every trick possible to deliver more efficient designs, and pilots to ensure that planes don't become so complex to fly that they can't be reasonably saved even when things go wrong. While history forces us to accept the validity of the approach (no one reasonably wants to kill Airbus); it is obvious there are ways that poorly implemented/communicated fly-by-wire piloting aids render an aircraft less safe. The MAX has illustrated one of these corner cases. Furthermore, just read the article. They lay it out plain as day. The test pilots knew that the MAX's compliance and certifiability couldn't go forward in it's current configuration without being brought back into compliance with MCAS. Despite that, testing to ensure this lynchpin system would not hit any edge case behavior with catastrophic results wasn't done; despite the fact that any reasonable engineer should have realized it was necessary. Give me a bit to drill through and I'll try to find the exact line this "equivalent safety exception" maps to. I'm not sure I'll find it because from the way Davies talks about it it strikes me as one of those possibly "unwritten" corollaries that tends to live beside anything codified. My main point though, is that this cavalier behavior with regard to airworthyness isn't unprecedented, and has been called out before. I fully accept it was different people working there then, but time and again I find echoes of an organization's risk management culture tends to survive the changing of the guard without active attempts to remedy it.
- joshanderson 7y ago
  The real fix is to never fly on a 737 MAX
  - magduf 7y ago
    The plane needs to be refused certification, forcing Boeing to take them all back and scrap them. They need to go back to the drawing board and design a new jet to take its place. If that forces them into bankruptcy, then this should serve as a good lesson about what happens when your company that makes safety-critical products cuts corners and kills people.
- cmurf 7y ago
  If there is any decent probability it would fail danger (trigger a nose down as a result of bogus high angle of attack information from a single sensor), that means there is a need for pilots to be aware of both the true unaugmented aerodynamic behavior this (MCAS) system is meant to moderate as well as the function of that augmentation system. Put another way: Just put another (virtual) button in the airplane and don't tell the pilots about it or what it's masking. It's important we must have this feature to certify, but it's also really important no one knows about it! If what you're doing is so right, why lie about it? This is a lie of omission, at the very least.
olliej 7y ago
So my understanding following this: * they designed mcas with a g sensor and aoa sensor and a max pitch, to fix a problem in a high speed test. They provide safety assessments based on having two sensors, limited pitch changes, and it only triggering in specific rare situations. * they provided that to the FAA as justification no training was needed, and that there was no need to acknowledge the existence of the system. FAA agreed. * After FAA approval they discovered that the aircraft didn’t work correctly at low speed either, so they made it activate at low speed as well. Because it activated at low speed a g sensor wasn’t possible (?) so it got removed. Now you only have a single (notoriously unreliable) AoA sensor controlling it. To compensate for poor aero they also needed to increase the amount of pitch needed. So the FAA approved MCAS with a 0.6 degree pitch limit, occurring at high speed, triggered by two independent sensors. Boeing then made the system trigger at low speed, off a single sensor, and increase pitch adjustment to 2.6 degrees. * Boeing chose not to test what happened if the AoA sensor failed. * Boeing continued to insist that pilots not be made to retrain, because of contract penalties with southwest. That meant they did not want to mention the existence of the system. * there safety margins also included an assumption that pilots could respond to a problem in 3s. But when mcas triggered incorrectly they were given speed, stall, and altitude warnings. All of which distracted them from the non-functioning led that indicate activation of a system that Boeing did not tell the pilots existed. * the standard way to recover from the flight control computer triggering to much pitch was to flip a switch on the stick. Boeing made MCAS override that command, so the standard recovery could not work. * Boeing continued to claim the plane was airworthy and provided a work around for its hellbent desired to crash, which was to use the manual pitch wheel. Except MCAS was designed in such a way that as part of its crash into the ground logic it put so much stress on the system that it was not physically possible for a pilot to perform the workaround. In all: more than 300 people are dead because Boeing rushed to make a plane to compete with another company, and then hid information about an overriding control system so that they wouldn’t have to retrain pilots, and that completely changed the parameters of that system after FAA approval such that none of their approved safety claims were remotely true. All to make some money so executives could get their bonuses. Every executive who pushed for those changes should be put in jail.
- paulmd 7y ago
etaioinshrdlu 7y ago
All of this to help avoid stalls. Such a pity because avoiding a stall seems to be one of the first things a new pilot learns. Every pilot should know how to detect and correct a stall. Can an actual pilot conform or deny this?
- _croz 7y ago
  It was less about avoiding stalls and more about making sure the performance characteristics of the MAX were similar enough to the previous generations of 737s so that a new type rating wasn't required for the pilots which would have required expensive training.
- msbarnett 7y ago
  It’s not about avoiding stalls per se, it’s about the uncorrected aerodynamics of the 737 MAX being such that at a certain point it becomes easier to pull into a stall than push away from it. While pilots learn to avoid stalls, commercial planes are designed deliberately such that at any point short of a stall it always takes less force to avoid one than create one. Uncorrected, the 737 MAX’s behaviour is a dangerous and surprising deviation from norms. It runs counter to pilot training, and it makes the stall recovery they’re trained to perform significantly more challenging than it would otherwise be. That’s why the FAA rule about column forces exists.
  - salawat 7y ago
    The uncorrected behavior also violates a design directive encoded in the CFR's. The plane could not be certified as airworthy without MCAS active due to the non-compliant stick force response curves.
runciblespoon 7y ago
Retrospective ass covering, Boeing installed MCAS to save money, and in the process took control of the airplane away from the pilots. They should be looking at jailtime for this.
- fwip 7y ago
  Boeing still gives the pilots a lot of control over the airplane, compared to Airbus. Airbus has defaulted to fly-by-wire (i.e: the computer has control and the pilot makes suggestions) for planes made since the 1980's.
  - the_pwner224 7y ago
    Airbus doesn't try to hide that fact from its pilots.
    - fwip 7y ago
      Oh, for sure. The person I replied to made it sound like it was always a bad thing (to have a computer in charge), though, and that's certainly not the case.
      - magduf 7y ago
        Having a computer in charge works fine when the computer is designed correctly, with lots of redundancy and the pilots are trained to understand how it works. This isn't what Boeing did.
benatkin 7y ago
Props to the author or editor that came up with the title of this. To make it more embarrassing: > MCAS: How 737 MAX system gained power to compete with Airbus and lost safeguards It's really tragic. If only they'd seen how much of a blunder this would be before lives were lost. But it's hard to get a big picture view until after the fact, when it involves so many people. I think perhaps Southwest was the closest to having a big picture view, as they ordered additional equipment [1]. I wish they somehow blew the whistle on it. 1: https://theaircurrent.com/aviation-safety/southwest-airlines-is-adding-new-angle-of-attack-indicators-to-its-737-max-fleet/ https://theaircurrent.com/aviation-safety/southwest-airlines...
- CaptainZapp 7y ago
  It's really tragic. If only they'd seen how much of a blunder this would be before lives were lost. Or, at the very minimum grounded that damn death trap volountarily after the Lion Air crash. The first crash can be attributed to greed, short cuts, engineering blunders and immense pressure to rush the plane into production. The second crash is corporate mass murder.

97 points by eternalban 7y ago

Previous Seattle Times article about the certification of the 737MAX: https://www.seattletimes.com/business/boeing-aerospace/failed-certification-faa-missed-safety-issues-in-the-737-max-system-implicated-in-the-lion-air-crash/ https://www.seattletimes.com/business/boeing-aerospace/faile... And HN post: https://news.ycombinator.com/item?id=19421612 https://news.ycombinator.com/item?id=19421612

paulmd 7y ago

The thinking of "we don't want the system to not work if it's needed" thinking demonstrates a failure to understand the prior probabilities involved. Most of the time the flight conditions will not warrant activation of the MCAS, and apparently the system is sufficiently unreliable/easy to damage that the chances of the system inadvertently triggering is high. But Boeing wanted to be sure that MCAS would always activate if needed, even if that resulted in unintended activations normally. Boeing essentially optimized around a low probability of a false negative, while ignoring the high probability of a false positive, because they underestimated the consequences and confusion that would result from a false positive - in many cases, multiple alarms/system failures occurring simultaneously as other systems that rely on that sensor begin alarming as well. In that environment, designing around a 3 second response time (on a system that pilots weren't trained to know exists) is probably unreasonable.

usrusr 7y ago
Well, Boeing is multiple people. Business culture celebrates a manager who can push their project ahead of others. Maybe MCAS had a project manager who was a bit of an overachiever in that regard.
- jdc 7y ago
  Surely I'm not the only one reading "overachiever" here as a wry euphemism.
  - usrusr 7y ago
    Not really: person A is tasked with remeditating risk X (dangerous pitch-up at high AoA), persons B, C are tasked with remediating some other risks Y, Z, or just generally keeping an eye on the big picture. Now if person A is better than the others at pushing the needs of their particular target metric (or just more stubborn, a particularly easy form of "strength"), that plane will surely never crash due pitch up at high AoA, but maybe more likely due to other dangers that became more likely for the sake of never crashing from that pitch up problem that A was sent out to solve. This cost will be seen as a consequence of the weakness of B, C, not of the "strength" of A. The high-resolve powerplayer who pushes their assigned task ahead of big-picture concerns is the hero of ambitious middle management culture. We see that happen many times, just rarely in domains as deadly as aviation.
mantap 7y ago
If it were designed correctly, it shouldn't matter if it were activated or not - it would always be safe. If it were designed correctly there would be at least 3 AoA sensors and the loss or disagreement of one would not affect the functioning of the system other than displaying a warning message to the pilots. If the two remaining disagreed then the system would be disabled. This is assuming that the system is necessary for certification or safety, if it is not necessary for certification or safety then it shouldn't be there at all. The fact that Boeing put a secondary autopilot in there with a single point of failure, can only be explained by organisational failures reminiscent of the Challenger disaster.
- salawat 7y ago
  The airframe cannot be certified airworthy without MCAS according to existing regulations. It is quite literally the only thing enabling the airframe to pass basic design requirements that have been mandated for years.
  - xtof 7y ago
    Citation? Nothing in FAR 25 permits a computer system to make airworthy an unairworthy airplane design.
    - salawat 7y ago
      Equivalent safety exception. It's the blanket exception that makes most fly-by-wire controlled aircraft airworthy. The problem here of course, is that you never had enough scrutiny paid to the fly-by-wire implementation to make sure that under no circumstances can the system fail non-operational. This was the subject of much controversy back in the 60's when th 727 was being certified. It required additional workarounds (implementation of a stick pusher) to be certified airworthy in U.K. airspace. That certification went through despite the protest of Test pilots at the time that certifying an airframe with a widget to "plaster over" non-compliant behavior would simply start a slippery slope of normalization of deviance whereby engineers would over time be able to certify less "passively" airworthy frames. Those pilots were absolutely correct; as by definition that is exactly what most fly-by-wire aircraft are. Airframes that are only rendered airworthy with the addition of computer logic. See the Royal Aeronautics Society D.P. Davies Interview. Should be in this one. The 727 section starts at 51:00 Equivalent level of Safety discussion starts at 1:01:00 [1] https://www.aerosociety.com/news/audio-the-d-p-davies-interview-on-the-boeing-747-the-trident-vc10-one-eleven-the-boeing-727/ https://www.aerosociety.com/news/audio-the-d-p-davies-interv... Highly recommend to listening to all four episodes just because. It's informative, and for me was rather illuminative in terms of the constant conflict between aerodynamicists to leverage every trick possible to deliver more efficient designs, and pilots to ensure that planes don't become so complex to fly that they can't be reasonably saved even when things go wrong. While history forces us to accept the validity of the approach (no one reasonably wants to kill Airbus); it is obvious there are ways that poorly implemented/communicated fly-by-wire piloting aids render an aircraft less safe. The MAX has illustrated one of these corner cases. Furthermore, just read the article. They lay it out plain as day. The test pilots knew that the MAX's compliance and certifiability couldn't go forward in it's current configuration without being brought back into compliance with MCAS. Despite that, testing to ensure this lynchpin system would not hit any edge case behavior with catastrophic results wasn't done; despite the fact that any reasonable engineer should have realized it was necessary. Give me a bit to drill through and I'll try to find the exact line this "equivalent safety exception" maps to. I'm not sure I'll find it because from the way Davies talks about it it strikes me as one of those possibly "unwritten" corollaries that tends to live beside anything codified. My main point though, is that this cavalier behavior with regard to airworthyness isn't unprecedented, and has been called out before. I fully accept it was different people working there then, but time and again I find echoes of an organization's risk management culture tends to survive the changing of the guard without active attempts to remedy it.
joshanderson 7y ago
The real fix is to never fly on a 737 MAX
- magduf 7y ago
  The plane needs to be refused certification, forcing Boeing to take them all back and scrap them. They need to go back to the drawing board and design a new jet to take its place. If that forces them into bankruptcy, then this should serve as a good lesson about what happens when your company that makes safety-critical products cuts corners and kills people.
cmurf 7y ago
If there is any decent probability it would fail danger (trigger a nose down as a result of bogus high angle of attack information from a single sensor), that means there is a need for pilots to be aware of both the true unaugmented aerodynamic behavior this (MCAS) system is meant to moderate as well as the function of that augmentation system. Put another way: Just put another (virtual) button in the airplane and don't tell the pilots about it or what it's masking. It's important we must have this feature to certify, but it's also really important no one knows about it! If what you're doing is so right, why lie about it? This is a lie of omission, at the very least.

olliej 7y ago

So my understanding following this: * they designed mcas with a g sensor and aoa sensor and a max pitch, to fix a problem in a high speed test. They provide safety assessments based on having two sensors, limited pitch changes, and it only triggering in specific rare situations. * they provided that to the FAA as justification no training was needed, and that there was no need to acknowledge the existence of the system. FAA agreed. * After FAA approval they discovered that the aircraft didn’t work correctly at low speed either, so they made it activate at low speed as well. Because it activated at low speed a g sensor wasn’t possible (?) so it got removed. Now you only have a single (notoriously unreliable) AoA sensor controlling it. To compensate for poor aero they also needed to increase the amount of pitch needed. So the FAA approved MCAS with a 0.6 degree pitch limit, occurring at high speed, triggered by two independent sensors. Boeing then made the system trigger at low speed, off a single sensor, and increase pitch adjustment to 2.6 degrees. * Boeing chose not to test what happened if the AoA sensor failed. * Boeing continued to insist that pilots not be made to retrain, because of contract penalties with southwest. That meant they did not want to mention the existence of the system. * there safety margins also included an assumption that pilots could respond to a problem in 3s. But when mcas triggered incorrectly they were given speed, stall, and altitude warnings. All of which distracted them from the non-functioning led that indicate activation of a system that Boeing did not tell the pilots existed. * the standard way to recover from the flight control computer triggering to much pitch was to flip a switch on the stick. Boeing made MCAS override that command, so the standard recovery could not work. * Boeing continued to claim the plane was airworthy and provided a work around for its hellbent desired to crash, which was to use the manual pitch wheel. Except MCAS was designed in such a way that as part of its crash into the ground logic it put so much stress on the system that it was not physically possible for a pilot to perform the workaround. In all: more than 300 people are dead because Boeing rushed to make a plane to compete with another company, and then hid information about an overriding control system so that they wouldn’t have to retrain pilots, and that completely changed the parameters of that system after FAA approval such that none of their approved safety claims were remotely true. All to make some money so executives could get their bonuses. Every executive who pushed for those changes should be put in jail.

paulmd 7y ago

etaioinshrdlu 7y ago

All of this to help avoid stalls. Such a pity because avoiding a stall seems to be one of the first things a new pilot learns. Every pilot should know how to detect and correct a stall. Can an actual pilot conform or deny this?

_croz 7y ago
It was less about avoiding stalls and more about making sure the performance characteristics of the MAX were similar enough to the previous generations of 737s so that a new type rating wasn't required for the pilots which would have required expensive training.
msbarnett 7y ago
It’s not about avoiding stalls per se, it’s about the uncorrected aerodynamics of the 737 MAX being such that at a certain point it becomes easier to pull into a stall than push away from it. While pilots learn to avoid stalls, commercial planes are designed deliberately such that at any point short of a stall it always takes less force to avoid one than create one. Uncorrected, the 737 MAX’s behaviour is a dangerous and surprising deviation from norms. It runs counter to pilot training, and it makes the stall recovery they’re trained to perform significantly more challenging than it would otherwise be. That’s why the FAA rule about column forces exists.
- salawat 7y ago
  The uncorrected behavior also violates a design directive encoded in the CFR's. The plane could not be certified as airworthy without MCAS active due to the non-compliant stick force response curves.

runciblespoon 7y ago

Retrospective ass covering, Boeing installed MCAS to save money, and in the process took control of the airplane away from the pilots. They should be looking at jailtime for this.

fwip 7y ago
Boeing still gives the pilots a lot of control over the airplane, compared to Airbus. Airbus has defaulted to fly-by-wire (i.e: the computer has control and the pilot makes suggestions) for planes made since the 1980's.
- the_pwner224 7y ago
  Airbus doesn't try to hide that fact from its pilots.
  - fwip 7y ago
    Oh, for sure. The person I replied to made it sound like it was always a bad thing (to have a computer in charge), though, and that's certainly not the case.
    - magduf 7y ago
      Having a computer in charge works fine when the computer is designed correctly, with lots of redundancy and the pilots are trained to understand how it works. This isn't what Boeing did.

benatkin 7y ago

Props to the author or editor that came up with the title of this. To make it more embarrassing: > MCAS: How 737 MAX system gained power to compete with Airbus and lost safeguards It's really tragic. If only they'd seen how much of a blunder this would be before lives were lost. But it's hard to get a big picture view until after the fact, when it involves so many people. I think perhaps Southwest was the closest to having a big picture view, as they ordered additional equipment [1]. I wish they somehow blew the whistle on it. 1: https://theaircurrent.com/aviation-safety/southwest-airlines-is-adding-new-angle-of-attack-indicators-to-its-737-max-fleet/ https://theaircurrent.com/aviation-safety/southwest-airlines...

CaptainZapp 7y ago
It's really tragic. If only they'd seen how much of a blunder this would be before lives were lost. Or, at the very minimum grounded that damn death trap volountarily after the Lion Air crash. The first crash can be attributed to greed, short cuts, engineering blunders and immense pressure to rush the plane into production. The second crash is corporate mass murder.