18 ms·

Boeing Fought Lion Air on Proposed Max Simulator Training Requirement

357 points by berkut 7y ago

toantruong95 7y ago
This is so disgusting. I hope they will go bankrupt to learn a hard lesson.
_Codemonkeyism 7y ago
Key take away "There is absolutely no reason to require your pilots to require a MAX simulator to begin flying the MAX,” the Boeing employee replied. "Once the engines are started, there is only one difference between NG and MAX procedurally, and that is that there is no OFF position of the gear handle. Boeing does not understand what is to be gained by a three-hour simulator session, when the procedures are essentially the same.”
- Athas 7y ago
  I still think it's interesting to reflect on the mindset that leads to this conclusion. As far as I have been able to determine (although I'm not in the aerospace field), the 737 MAX is procedurally identical to the NG, except when something breaks. The failure modes are slightly different, with potentially lethal results. As a computer scientist, I'm not accustomed to thinking about functional equivalence in the presence of hardware failure, and maybe this Boeing employee was not sufficiently drilled on the need to consider such aspects for aeroplanes. It is of course the fault of Boeing corporate culture and internal procedures that this can be overlooked.
  - chopin 7y ago
    A great deal of procedures focus on failure scenarios. If these are different, then the MAX is procedurally different from the NG.
  - salawat 7y ago
    >As a computer scientist, I'm not accustomed to thinking about functional equivalence in the presence of hardware failure... How are you not? I mean, I get it 90% of the time we screw up the programming somehow, but as a computer scientist, I never ignore the possibility of hardware failure. Memory goes bad. Devices fail. Networks die. Semiconductors transiently in strange ways if you don't take the right precautions... It's the entire impetus behind GIGO. If you shove garbage into a perfectly working software system; (corrupt data from a malfunctioning input source), you still get out garbage. It's why life and safety critical automation is so fundamentally different from lower stakes programming tasks where "reboot the damn thing" is a viable option. If your sensor goes bad, and you're in the air, you can't do squat to fix it. You have to detect the error, and fail the system gracefully by taking it out of the loop, informing the operator of the system failure, and most importantly, never allow that system to do anything that could jeopardize the ability of the operator to continue operating. This is or at least I thought it was basic Control Systems 101...
    - Athas 7y ago
      > How are you not? I research compilers and type systems. If the RAM dies while the compiler is running, you rerun the compiler on a new machine. A lot of computer science abstracts away the notion of hardware failure, because otherwise it becomes enormously cumbersome to talk about anything. This is fine as long as you don't actually build real high-reliability systems with the same approach.
      - acqq 7y ago
        >> How are you not? > I research compilers and type systems. I hope it's obvious that the software you work on is not supposed to be run during the flight. The critical software is supposed to do as little as possible, and everything is expected to be in already compiled (and thoroughly verified) state. And even for the product of yours, as soon as it is not used only for the research but as a production compiler which produces a firmware for the plane, it would have to be proven much more than what is expected from it while it is just an artifact of a research. In short, even if you are lucky to just do the research, you should be aware (and thankful) that the critical software has other expectations. Including how it responds to failed sensors: different response to the external inputs is a fundamentally different software, even if you never thought about it before.
  - tragomaskhalos 7y ago
    Yes - this seems to me to be the same erroneous thinking that lead to the Ariane 5 maiden flight loss, caused by an integer overflow. There again the thinking was "this is effectively the same vehicle as its predecessor, therefore we do not need to test it thoroughly".
    - skissane 7y ago
      Similar thinking contributed to Therac-25 as well - the software worked safely on the old hardware, of course it will be safe on the updated hardware.
  - bumby 7y ago
    I suppose it depends on the field in which you work, but many safety-critical fields have an expectation that hardware failures are captured and mitigated and there are various tools to capture these design decisions and ensure they are tested. One example tool in this case would be a software fault analysis (FTA) or failure-mode-effects analysis (FMEA) that looks at a broken sensor input value as a failure mode.[1] It's been my experience, however, that these sorts of design tools are more unfamiliar to software groups than hardware bubbas. It's not uncommon to simply see "software fails" as a failure mode which isn't very helpful. I'd be curious what the HN community's experience is with software as it relates to design tools like FTAs, FMEAs, hazard analyses, etc. [1] https://standards.nasa.gov/standard/nasa/nasa-gb-871913 https://standards.nasa.gov/standard/nasa/nasa-gb-871913
- Angostura 7y ago
  The other key take away I had was: > "Boeing knew the approach might be questioned [Calling MCAS a simple addition to Speed Trim], so it sought input from its FAA-designated authorized representative (AR) "to ensure this strategy is acceptable” for certification. > "After speaking with the [AR], concurrence was provided that we can continue to use the MCAS nomenclature internally...while still considering MCAS to be an addition to the Speed Trim function,” the memo said. "This will allow us to maintain the MCAS nomenclature while not driving additional work due to training impacts and maintenance manual expansions I can imagine some Boeing employees being uncomfortable, but having it run past the FAA would have relieved that. Pretty shocking regulatory lapse. I know nothing about the AR system - is this a Boeing employee, or someone who works full-time for the FAA?
  - salawat 7y ago
    The FAA basically picks a Boeing employee and says "you represent and answer to us now." The employee however, is still managed, and reports first to Boeing management. They're a glorified liaison/paperwork interface. This was different than before as I understand it, because the FAA used to become the direct report for their Designated Engineering Representatives under the old system. This meant there was no management layer running interference between the rep and the regulator. I might be misremembering that though.
    - sseth 7y ago
      I believe with AR even the picking is done by Boeing. unlike DER. One huge downside of this is that the informal interactions between FAA and Boeing engineers are almost non-existent, leading to even greater lack of technical knowledge on the FAA side.
  - simion314 7y ago
    >I can imagine some Boeing employees being uncomfortable, but having it run past the FAA would have relieved that. There were some emails released where a Boeing employee boosted about he used "jedi mind tricks" on the FAA people.
  - passer_byer 7y ago
ummonk 7y ago
On the specific issue of simulator time though, if the simulator behaves basically the same as a Boeing 737 NG, what is the point of putting them through the MAX simulator? It does seem like a pointless exercise that wouldn't have provided the pilots any extra practice (or crucial knowledge of actual differences like MCAS). Obviously, now that they're adding checklist practice for emergency scenarios relevant to the 737 MAX, it makes sense to require simulator practice, but I don't think that would have previously made a difference.
- jacobush 7y ago
  Of course not. It was all part of the same ruse. If the MAX simulator behaved differently, it would have been that much more difficult to say that no training was needed.
- inferiorhuman 7y ago
  Well Boeing knew that the MAX did not behave like the NG so there's that. Even if the airlines and Boeing went along with that farce there's at least a couple of entries in ASRS where pilots complained that they didn't feel prepared to fly the MAX. One mentioned that the instrumentation was sufficiently different that their usual pattern of visually scanning the instruments did not work with the MAX and that monitoring the instruments in the MAX required significantly more effort. The implication was that additional training and familiarity with the specifics of the MAX would reduce the cognitive load.
- inferiorhuman 7y ago
  Here's the ASRS entry I was thinking of: I had my first flight on the Max [to] ZZZ1. We found out we were scheduled to fly the aircraft on the way to the airport in the limo. We had a little time [to] review the essentials in the car. Otherwise we would have walked onto the plane cold. My post flight evaluation is that we lacked the knowledge to operate the aircraft in all weather and aircraft states safely. The instrumentation is completely different - My scan was degraded, slow and labored having had no experience w/ the new ND (Navigation Display) and ADI (Attitude Director Indicator) presentations/format or functions (manipulation between the screens and systems pages were not provided in training materials. If they were, I had no recollection of that material). https://www.theatlantic.com/notes/2019/03/heres-what-was-on-the-record-about-problems-with-the-737-max/584791/ https://www.theatlantic.com/notes/2019/03/heres-what-was-on-...
justinclift 7y ago
This is why Boeing shouldn't be allowed to have anything to do with new Space contracts (ISS, etc) until this MAX issue is a solved, faint memory. eg for decades
- iforgotpassword 7y ago
  The company should just be shut down entirely and everyone who had a say in anything put behind bars.
- tzfld 7y ago
  Space hardware development is heavily supervised, and there are no issues induced by mass production.
  - justinclift 7y ago
    So, exactly like aviation, and it hasn't worked out there. :(
mcv 7y ago
Boeing was already plenty culpable for these disasters, but this underscores just how much they pressured they pressured their customers to avoid training that could have mitigated the problems and prevented the crash, just to keep up the illusion that they wanted to present to the world: that it was effectively identical to the NG when it really wasn't.
- rob74 7y ago
  On the other hand, Boeing was also under pressure from other customers (https://www.reuters.com/article/us-boeing-airplane-southwest/u-s-lawmakers-question-boeings-1-mln-rebate-clause-for-southwest-737-max-orders-idUSKBN1X92D4 https://www.reuters.com/article/us-boeing-airplane-southwest...) who wanted to avoid simulator training for their pilots. Which doesn't excuse them of course, but I think it's important to see the whole picture...
  - mcv 7y ago
    They could have taken the position that it's up to the airline to decide if they want simulator training. Recommended but not required. That puts the responsibility on the airline. Still not ideal, but better than actively discouraging the training.
    - salawat 7y ago
      They couldn't actually afford to have that happen because they then would have had to implement an accurate MAX specific simulator. Doing so would have revealed some of the other design flubs, such as the Flight Computer as a single point of failure. That's why they were so concerned about Lion Air setting a precedent for other customers with regard to simulators.
      - mcv 7y ago
        And that is the true damning point: they wanted to hide a problem they were aware of, and maintain an illusion that all was fine despite knowing it wasn't. Had they taken a more neutral stance, they could argue they didn't know. Now it's just reckless stupid endangerment.
konspence 7y ago
This whole thing looks like a disaster. As other commenters are pointing out, even if the pilots did the MAX simulators, would they have been trained to respond to the AOA failure? Unlikely. It seems like Boeing completely missed serious cases on the testing of the plane, and is hoping that the "UAT" phase (simulator training) would have uncovered the issues. But UAT never underwent the condition of an AOA sensor failing.
jakobegger 7y ago
Can someone explain how Airplane Simulators for pilots work? Do simulators have the same hardware as real planes, or do they have a software model of the airplane? If you simulated a broken AoA sensor, would the simulated plane behave similar to the real plane? Would the MCAS system have the same bugs in the simulator as in the one in the real aircraft? Can you try new scenarios in a simulator, or can you just try scenarios that the simulator was designed to run?
- Wamou 7y ago
  It varies wildly on the simulator. Boeing or Airbus are completly able to rehost their software on a simulator mostly made from real OEM parts. From that, it's simply a matter of designing new scenarios from what you're feeding the various sensors. On the other hand, if you're simply looking for some training hours on some specific basic scenario and/or aircraft, the simulators can be a lot rougher and still be certified. I "flew" on an airliner manufacturer designed simulator, and everything from the instrument panel to the hydraulics simulating the small impact when rolling between plates on a concrete runway felt pretty damn real. For more basic stuff, even X-Plane exists in an FAA-approved version.
  - HorkHunter 7y ago
    Out of curiosity, where can an average joe like me find and "fly" one of these?
    - joezydeco 7y ago
      https://www.x-plane.com/ https://www.x-plane.com/
      - robohoe 7y ago
        If OP does end up getting X-Plane, I suggest downloading a freeware 737 airplane called Zibo 737. It’s an extremely detailed version of 737-800.
    - 7y ago
tzfld 7y ago
The way Boeing seems more and more evil as the story is folded out, makes me think that all these article may be heavily one sided and never hearing Boeing's own version and motivations.
- cjslep 7y ago
  You can always read the prime source material yourself: Boeing's own internal emails and chats they released about the 737 MAX. And do note: those released items are Boeing putting its best foot forward. And it's a low bar. Imagine the emails and chat transcripts that were not released.
  - wahern 7y ago
    > And do note: those released items are Boeing putting its best foot forward. And it's a low bar. Imagine the emails and chat transcripts that were not released. My guess was that the new CEO was trying to get out ahead of developments by airing the remaining dirty laundry. It makes him and Boeing look sincere and starts the clock on the public forgetting the bad news. The constant drip of bad publicity made the previous CEO look like an idiot. I don't expect Boeing to actually clean up their act, but I bet they're going to be smarter about things going forward. I doubt there are going to be any more damning reveals about the 737 other than what can be mined from these latest disclosures. 777X is another matter, but I bet any revelations will occur more quickly and cleanly.
- bertjk 7y ago
  A good defense lawyer never puts their client on the stand. While this is not currently a criminal trial, the same principles surely apply. If Boeing starts talking, then the focus moves from debating whether more training was needed and who was responsible for deciding it wasn't (corrupt FAA? customer pressure? honest mistake?) to whether the company is hiding something. The first is a better scenario for Boeing than the second.
  - dlgeek 7y ago
    The DOJ is running a criminal investigation of Boeing related to the MAX.
redis_mlc 7y ago
To give you an idea how big a tragedy an airliner accident is in Indonesia ... The average Javanese person I've talked to knows 900 relatives by name.
toast0 7y ago
Even if Lion Air had done simulator training, it seems unlikely to have helped, unless there was specifically material on MCAS or at least stabilizer runaway by the electric trim system. One airline demanding and getting simulator time doesn't help if the decision has already been made to not inform pilots about MCAS. Boeing's flight rules model has always been around the pilot(s) having situational awareness and being in absolute control; the problem with MCAS is not what it does, or that it's needed to avoid stalling, the problem is that it was not known to pilots, that it operates without any clear indication that it's operating, and that it can only be disabled by disabling electric trim, and that manual trim is extremely difficult to impossible to use if the stabilizer is at full nose down (there was an old 737 procedure for this, but it was removed some number of redesigns ago, it was reportedly not in the 737 NG manual)
- willyt 7y ago
  Here is a video explaining how difficult the manual trim procedure is: https://www.youtube.com/watch?v=aoNOVlxJmow https://www.youtube.com/watch?v=aoNOVlxJmow
  - tzs 7y ago
    Note that the video is about manual trim in extreme conditions. If pilots knew about MCAS and that in the case of a runaway trim due to MCAS they need to disable electrical trim and switch to manual trim early and keep it manual for the rest of the flight, they would not get into the conditions where it is hard to turn the manual trim wheels.
    - willyt 7y ago
      But even if they had known about MCAS, I think I read somewhere that they had 30 seconds to diagnose the problem before it put the trim in this configuration.
epicgiga 7y ago
Lion: "we'd like to practice using this plane so we don't crash it". Boeing: "no". Lion: crashes it. Hundreds die. Boeing: "not our fault, their pilots were untrained". How is no one is prison again?
- mattacular 7y ago
  The Boeing CEO exited the company with just 60 million dollars, which is a lot like being sent to prison.
  - close04 7y ago
    It's because in the US it takes much more than knowingly allowing such issues to directly kill hundreds to go to prison. We're not talking about "calculated" deaths like the ones VW's CEO hopefully goes to prison for. Rather directly attributable ones like the 737MAX or GM's faulty ignition switch scandals.
    - sseveran 7y ago
      VW's former CEO isn't being charged with killing anyone one. He has been charged with committing fraud by knowingly lying to regulators.
      - close04 7y ago
        I wasn't suggesting that either VW's, GM's, or Boeing's CEOs actually go to prison for "killing" anyone but rather for "knowingly allowing such issues to directly kill". And as the GM example stands to show, it's not particularly hard to get away with this in the US if you are the CEO of a powerful company.
        ryanlol 7y ago
        Can you provide examples of CEOs of less powerful companies not getting away with this? Your comment seems to necessitate examples of such.
ameen 7y ago
In a way, Boeing’s plight underlines American work culture vs. European and other relaxed but stringent work cultures. American businesses are constantly churning output with not much to show for (relative to effort vs results) as the employees are burned out, investors and C-suite demand more of the middle-management and they in-turn pressure engineering and sales teams, etc. The results have spoken about which one works and which doesn’t. Toxic work culture starts from the top.
- ramblerman 7y ago
  I think the results have shown not all industries benefit from being purely profit driven. To extrapolate that further is a bit disingenuous. The US is still a leader in innovation over Europe, by a large margin. And I say that as a European.
  - Ragnarork 7y ago
    How do you measure "innovation" though? Is it in terms of patents, in terms of the number of new companies, new companies that succeed (and according to which metrics?), new companies using/developing new technologies? Do you consider modern implementations of old concepts as innovation or not (i.e. transitioning some activities into the digital world, that kind of things)? Do you normalize that "innovation" quantity with how much money is injected into the "innovation industry"?
    - jiqiren 7y ago
      yes.
- datenhorst 7y ago
  I believe Harvard Business School and its ilk deserve a lot of blame for the instilling of a profit-at-any-cost culture across the US.
  - strictfp 7y ago
    IMO the business school of though is the one which is currently hindering progression, holding on to values from the industrial revolution. It's as we all know a philosophy of top-down management of expendible resources. This way of thinking simply doesn't apply today. We need a breakthrough of the servant leadership way of thinking.
Entalpi 7y ago
What ever happened to ”safety first”.
- xiphias2 7y ago
  If you ask the Boeing CEO about it, safety was always the top priority.
toantruong95 7y ago
This is so disgusting. I hope they will go to bankruptcy for what they have done.
hhas01 7y ago
Jesus. Pardon my French, but seeing that photo makes me fully appreciate two things: 1. Just how incredibly low 737 sits to the ground (obviously a feature from back in the days when luggage was manually loaded), and 2. How anyone would think pushing those enormous engines forward like that could be any less disruptive of its proven design than lengthening the undercarriage to give them the clearance they so desperately need. Death of a thousand cuts. Indeed. Hope the monster never flies again.
- cedivad 7y ago
  2. is a great question I would like an answer to. Here is a side by side comparison with an a320: https://aviation.stackexchange.com/questions/61161/is-there-less-vertical-space-for-an-engine-under-the-wing-of-a-737-max-than-for https://aviation.stackexchange.com/questions/61161/is-there-...
  - inferiorhuman 7y ago
    FWIW That's a 737 Classic. The engines were already too big (and still smaller than the A320 engines) as evidenced by the flattened nacelle. The engines grew with each successive generation (NG, MAX). I've read (but not confirmed) one of the other things keeping the 737 so low to ground is the type of emergency exit used. Extending the gear so that the whole mess sits higher off the ground would require new exits and a ton of extra things to certify.
    - hhas01 7y ago
      Yeah, there’s a lot of knock-ons with “just” extending the gear. It would definitely cost a lot more (though still far less than a complete new airframe), which is obviously why Boeing and customers wanted to dodge it. On the other hand, building a plane so aerodynamically unsound that it cratered 346 customers the moment it went out the door is the very textbook definition of “false economy”. One that Boeing itself will be paying for for years to come; even as those who made all the actual decisions gently golden-parachute to the ground.
- amyjess 7y ago
GenericsMotors 7y ago
Around the time just before the MAX was grounded, I remember there being lots of unfounded speculations (here on HN as well!) that Lion Air's and Ethiopian's pilots were somehow substandard and lacking the training of their first world counterparts. Meanwhile it was Boeing actively keeping vital information and training from pilots. The lesson from this is still not learned, and can see at least one apologist on this thread repeating this same BS again...it's infuriating.
Brave-Steak 7y ago
I'd really be interested in knowing at what level these two employees were at. How deep does the rot go? Even if the CEO is gone, these people are still around.
vmchale 7y ago
What a shitshow. The entire thing is graft, and the US government turned a blind eye because it's deemed an important industry.
- mrtksn 7y ago
  Isn't it also unfaithfulness to capitalism to protect an industry in such a way? I would have assumed that if someone believes in the mechanics of capitalism would have let it be.
WalterBright 7y ago
> and that an erroneous MCAS activation would be quickly diagnosed as a runaway stabilizer. The 2013 memo casts doubt on the former, and the two MAX accident sequences disproved the latter. This does not mention that an Emergency Airworthiness Directive was sent to all 737MAX crews after the LA crash explaining exactly how to resolve the runaway trim issue, which is: 1. restore normal trim using the column trim switches 2. cut off the stabilizer trim with the console cutoff switch The text is: Boeing Emergency Airworthiness Directive "Initially, higher control forces may be needed to overcome any stabilizer nose down trim already applied. Electric stabilizer trim can be used to neutralize control column pitch forces before moving the STAB TRIM CUTOUT switches to CUTOUT. Manual stabilizer trim can be used before and after the STAB TRIM CUTOUT switches are moved to CUTOUT." https://theaircurrent.com/wp-content/uploads/2018/11/B737-MAX-AD-1107.pdf https://theaircurrent.com/wp-content/uploads/2018/11/B737-MA... Both the LA and EA crews repeatedly successfully countered the runaway trim with the electric trim switches. The LA crew never took the next step of cutting off the trim. The EA crew did cutoff the trim, but did not trim to normal first. Dealing with runaway trim is a "memory item" for the 737, meaning the pilots are supposed to know about the cutoff switches that are prominently placed on the center console in easy reach.
triceratops 7y ago
Can anyone experienced in aviation ballpark how much retraining a pilot on a new plane costs? How does it compare to a pilot's annual compensation or other figures of importance (e.g. annual airline revenue or profit, operating expenses of a plane etc)? Helping airlines avoid retraining seems like the driving factor behind all these apparently shortsighted decisions that led to the MAX but, assuming everything had gone well, what were the actual potential savings? I'm just trying to understand the stakes.
hurricanetc 7y ago
They wouldn’t have trained on an MCAS failure so this is irrelevant as far as the actual crashes are concerned. It’s just more evidence of Boeing being a corrupt organization.

357 points by berkut 7y ago

This is so disgusting. I hope they will go bankrupt to learn a hard lesson.

Key take away "There is absolutely no reason to require your pilots to require a MAX simulator to begin flying the MAX,” the Boeing employee replied. "Once the engines are started, there is only one difference between NG and MAX procedurally, and that is that there is no OFF position of the gear handle. Boeing does not understand what is to be gained by a three-hour simulator session, when the procedures are essentially the same.”

Athas 7y ago
I still think it's interesting to reflect on the mindset that leads to this conclusion. As far as I have been able to determine (although I'm not in the aerospace field), the 737 MAX is procedurally identical to the NG, except when something breaks. The failure modes are slightly different, with potentially lethal results. As a computer scientist, I'm not accustomed to thinking about functional equivalence in the presence of hardware failure, and maybe this Boeing employee was not sufficiently drilled on the need to consider such aspects for aeroplanes. It is of course the fault of Boeing corporate culture and internal procedures that this can be overlooked.
- chopin 7y ago
  A great deal of procedures focus on failure scenarios. If these are different, then the MAX is procedurally different from the NG.
- salawat 7y ago
  >As a computer scientist, I'm not accustomed to thinking about functional equivalence in the presence of hardware failure... How are you not? I mean, I get it 90% of the time we screw up the programming somehow, but as a computer scientist, I never ignore the possibility of hardware failure. Memory goes bad. Devices fail. Networks die. Semiconductors transiently in strange ways if you don't take the right precautions... It's the entire impetus behind GIGO. If you shove garbage into a perfectly working software system; (corrupt data from a malfunctioning input source), you still get out garbage. It's why life and safety critical automation is so fundamentally different from lower stakes programming tasks where "reboot the damn thing" is a viable option. If your sensor goes bad, and you're in the air, you can't do squat to fix it. You have to detect the error, and fail the system gracefully by taking it out of the loop, informing the operator of the system failure, and most importantly, never allow that system to do anything that could jeopardize the ability of the operator to continue operating. This is or at least I thought it was basic Control Systems 101...
  - Athas 7y ago
    > How are you not? I research compilers and type systems. If the RAM dies while the compiler is running, you rerun the compiler on a new machine. A lot of computer science abstracts away the notion of hardware failure, because otherwise it becomes enormously cumbersome to talk about anything. This is fine as long as you don't actually build real high-reliability systems with the same approach.
    - acqq 7y ago
      >> How are you not? > I research compilers and type systems. I hope it's obvious that the software you work on is not supposed to be run during the flight. The critical software is supposed to do as little as possible, and everything is expected to be in already compiled (and thoroughly verified) state. And even for the product of yours, as soon as it is not used only for the research but as a production compiler which produces a firmware for the plane, it would have to be proven much more than what is expected from it while it is just an artifact of a research. In short, even if you are lucky to just do the research, you should be aware (and thankful) that the critical software has other expectations. Including how it responds to failed sensors: different response to the external inputs is a fundamentally different software, even if you never thought about it before.
- tragomaskhalos 7y ago
  Yes - this seems to me to be the same erroneous thinking that lead to the Ariane 5 maiden flight loss, caused by an integer overflow. There again the thinking was "this is effectively the same vehicle as its predecessor, therefore we do not need to test it thoroughly".
  - skissane 7y ago
    Similar thinking contributed to Therac-25 as well - the software worked safely on the old hardware, of course it will be safe on the updated hardware.
- bumby 7y ago
  I suppose it depends on the field in which you work, but many safety-critical fields have an expectation that hardware failures are captured and mitigated and there are various tools to capture these design decisions and ensure they are tested. One example tool in this case would be a software fault analysis (FTA) or failure-mode-effects analysis (FMEA) that looks at a broken sensor input value as a failure mode.[1] It's been my experience, however, that these sorts of design tools are more unfamiliar to software groups than hardware bubbas. It's not uncommon to simply see "software fails" as a failure mode which isn't very helpful. I'd be curious what the HN community's experience is with software as it relates to design tools like FTAs, FMEAs, hazard analyses, etc. [1] https://standards.nasa.gov/standard/nasa/nasa-gb-871913 https://standards.nasa.gov/standard/nasa/nasa-gb-871913
Angostura 7y ago
The other key take away I had was: > "Boeing knew the approach might be questioned [Calling MCAS a simple addition to Speed Trim], so it sought input from its FAA-designated authorized representative (AR) "to ensure this strategy is acceptable” for certification. > "After speaking with the [AR], concurrence was provided that we can continue to use the MCAS nomenclature internally...while still considering MCAS to be an addition to the Speed Trim function,” the memo said. "This will allow us to maintain the MCAS nomenclature while not driving additional work due to training impacts and maintenance manual expansions I can imagine some Boeing employees being uncomfortable, but having it run past the FAA would have relieved that. Pretty shocking regulatory lapse. I know nothing about the AR system - is this a Boeing employee, or someone who works full-time for the FAA?
- salawat 7y ago
  The FAA basically picks a Boeing employee and says "you represent and answer to us now." The employee however, is still managed, and reports first to Boeing management. They're a glorified liaison/paperwork interface. This was different than before as I understand it, because the FAA used to become the direct report for their Designated Engineering Representatives under the old system. This meant there was no management layer running interference between the rep and the regulator. I might be misremembering that though.
  - sseth 7y ago
    I believe with AR even the picking is done by Boeing. unlike DER. One huge downside of this is that the informal interactions between FAA and Boeing engineers are almost non-existent, leading to even greater lack of technical knowledge on the FAA side.
- simion314 7y ago
  >I can imagine some Boeing employees being uncomfortable, but having it run past the FAA would have relieved that. There were some emails released where a Boeing employee boosted about he used "jedi mind tricks" on the FAA people.
- passer_byer 7y ago

ummonk 7y ago

On the specific issue of simulator time though, if the simulator behaves basically the same as a Boeing 737 NG, what is the point of putting them through the MAX simulator? It does seem like a pointless exercise that wouldn't have provided the pilots any extra practice (or crucial knowledge of actual differences like MCAS). Obviously, now that they're adding checklist practice for emergency scenarios relevant to the 737 MAX, it makes sense to require simulator practice, but I don't think that would have previously made a difference.

jacobush 7y ago
Of course not. It was all part of the same ruse. If the MAX simulator behaved differently, it would have been that much more difficult to say that no training was needed.
inferiorhuman 7y ago
Well Boeing knew that the MAX did not behave like the NG so there's that. Even if the airlines and Boeing went along with that farce there's at least a couple of entries in ASRS where pilots complained that they didn't feel prepared to fly the MAX. One mentioned that the instrumentation was sufficiently different that their usual pattern of visually scanning the instruments did not work with the MAX and that monitoring the instruments in the MAX required significantly more effort. The implication was that additional training and familiarity with the specifics of the MAX would reduce the cognitive load.
inferiorhuman 7y ago
Here's the ASRS entry I was thinking of: I had my first flight on the Max [to] ZZZ1. We found out we were scheduled to fly the aircraft on the way to the airport in the limo. We had a little time [to] review the essentials in the car. Otherwise we would have walked onto the plane cold. My post flight evaluation is that we lacked the knowledge to operate the aircraft in all weather and aircraft states safely. The instrumentation is completely different - My scan was degraded, slow and labored having had no experience w/ the new ND (Navigation Display) and ADI (Attitude Director Indicator) presentations/format or functions (manipulation between the screens and systems pages were not provided in training materials. If they were, I had no recollection of that material). https://www.theatlantic.com/notes/2019/03/heres-what-was-on-the-record-about-problems-with-the-737-max/584791/ https://www.theatlantic.com/notes/2019/03/heres-what-was-on-...

justinclift 7y ago

This is why Boeing shouldn't be allowed to have anything to do with new Space contracts (ISS, etc) until this MAX issue is a solved, faint memory. eg for decades

iforgotpassword 7y ago
The company should just be shut down entirely and everyone who had a say in anything put behind bars.
tzfld 7y ago
Space hardware development is heavily supervised, and there are no issues induced by mass production.
- justinclift 7y ago
  So, exactly like aviation, and it hasn't worked out there. :(

mcv 7y ago

Boeing was already plenty culpable for these disasters, but this underscores just how much they pressured they pressured their customers to avoid training that could have mitigated the problems and prevented the crash, just to keep up the illusion that they wanted to present to the world: that it was effectively identical to the NG when it really wasn't.

rob74 7y ago
On the other hand, Boeing was also under pressure from other customers (https://www.reuters.com/article/us-boeing-airplane-southwest/u-s-lawmakers-question-boeings-1-mln-rebate-clause-for-southwest-737-max-orders-idUSKBN1X92D4 https://www.reuters.com/article/us-boeing-airplane-southwest...) who wanted to avoid simulator training for their pilots. Which doesn't excuse them of course, but I think it's important to see the whole picture...
- mcv 7y ago
  They could have taken the position that it's up to the airline to decide if they want simulator training. Recommended but not required. That puts the responsibility on the airline. Still not ideal, but better than actively discouraging the training.
  - salawat 7y ago
    They couldn't actually afford to have that happen because they then would have had to implement an accurate MAX specific simulator. Doing so would have revealed some of the other design flubs, such as the Flight Computer as a single point of failure. That's why they were so concerned about Lion Air setting a precedent for other customers with regard to simulators.
    - mcv 7y ago
      And that is the true damning point: they wanted to hide a problem they were aware of, and maintain an illusion that all was fine despite knowing it wasn't. Had they taken a more neutral stance, they could argue they didn't know. Now it's just reckless stupid endangerment.

konspence 7y ago

This whole thing looks like a disaster. As other commenters are pointing out, even if the pilots did the MAX simulators, would they have been trained to respond to the AOA failure? Unlikely. It seems like Boeing completely missed serious cases on the testing of the plane, and is hoping that the "UAT" phase (simulator training) would have uncovered the issues. But UAT never underwent the condition of an AOA sensor failing.

jakobegger 7y ago

Can someone explain how Airplane Simulators for pilots work? Do simulators have the same hardware as real planes, or do they have a software model of the airplane? If you simulated a broken AoA sensor, would the simulated plane behave similar to the real plane? Would the MCAS system have the same bugs in the simulator as in the one in the real aircraft? Can you try new scenarios in a simulator, or can you just try scenarios that the simulator was designed to run?

Wamou 7y ago
It varies wildly on the simulator. Boeing or Airbus are completly able to rehost their software on a simulator mostly made from real OEM parts. From that, it's simply a matter of designing new scenarios from what you're feeding the various sensors. On the other hand, if you're simply looking for some training hours on some specific basic scenario and/or aircraft, the simulators can be a lot rougher and still be certified. I "flew" on an airliner manufacturer designed simulator, and everything from the instrument panel to the hydraulics simulating the small impact when rolling between plates on a concrete runway felt pretty damn real. For more basic stuff, even X-Plane exists in an FAA-approved version.
- HorkHunter 7y ago
  Out of curiosity, where can an average joe like me find and "fly" one of these?
  - joezydeco 7y ago
    https://www.x-plane.com/ https://www.x-plane.com/
    - robohoe 7y ago
      If OP does end up getting X-Plane, I suggest downloading a freeware 737 airplane called Zibo 737. It’s an extremely detailed version of 737-800.
  - 7y ago

tzfld 7y ago

The way Boeing seems more and more evil as the story is folded out, makes me think that all these article may be heavily one sided and never hearing Boeing's own version and motivations.

cjslep 7y ago
You can always read the prime source material yourself: Boeing's own internal emails and chats they released about the 737 MAX. And do note: those released items are Boeing putting its best foot forward. And it's a low bar. Imagine the emails and chat transcripts that were not released.
- wahern 7y ago
  > And do note: those released items are Boeing putting its best foot forward. And it's a low bar. Imagine the emails and chat transcripts that were not released. My guess was that the new CEO was trying to get out ahead of developments by airing the remaining dirty laundry. It makes him and Boeing look sincere and starts the clock on the public forgetting the bad news. The constant drip of bad publicity made the previous CEO look like an idiot. I don't expect Boeing to actually clean up their act, but I bet they're going to be smarter about things going forward. I doubt there are going to be any more damning reveals about the 737 other than what can be mined from these latest disclosures. 777X is another matter, but I bet any revelations will occur more quickly and cleanly.
bertjk 7y ago
A good defense lawyer never puts their client on the stand. While this is not currently a criminal trial, the same principles surely apply. If Boeing starts talking, then the focus moves from debating whether more training was needed and who was responsible for deciding it wasn't (corrupt FAA? customer pressure? honest mistake?) to whether the company is hiding something. The first is a better scenario for Boeing than the second.
- dlgeek 7y ago
  The DOJ is running a criminal investigation of Boeing related to the MAX.

redis_mlc 7y ago

To give you an idea how big a tragedy an airliner accident is in Indonesia ... The average Javanese person I've talked to knows 900 relatives by name.

toast0 7y ago

Even if Lion Air had done simulator training, it seems unlikely to have helped, unless there was specifically material on MCAS or at least stabilizer runaway by the electric trim system. One airline demanding and getting simulator time doesn't help if the decision has already been made to not inform pilots about MCAS. Boeing's flight rules model has always been around the pilot(s) having situational awareness and being in absolute control; the problem with MCAS is not what it does, or that it's needed to avoid stalling, the problem is that it was not known to pilots, that it operates without any clear indication that it's operating, and that it can only be disabled by disabling electric trim, and that manual trim is extremely difficult to impossible to use if the stabilizer is at full nose down (there was an old 737 procedure for this, but it was removed some number of redesigns ago, it was reportedly not in the 737 NG manual)

willyt 7y ago
Here is a video explaining how difficult the manual trim procedure is: https://www.youtube.com/watch?v=aoNOVlxJmow https://www.youtube.com/watch?v=aoNOVlxJmow
- tzs 7y ago
  Note that the video is about manual trim in extreme conditions. If pilots knew about MCAS and that in the case of a runaway trim due to MCAS they need to disable electrical trim and switch to manual trim early and keep it manual for the rest of the flight, they would not get into the conditions where it is hard to turn the manual trim wheels.
  - willyt 7y ago
    But even if they had known about MCAS, I think I read somewhere that they had 30 seconds to diagnose the problem before it put the trim in this configuration.

epicgiga 7y ago

Lion: "we'd like to practice using this plane so we don't crash it". Boeing: "no". Lion: crashes it. Hundreds die. Boeing: "not our fault, their pilots were untrained". How is no one is prison again?

mattacular 7y ago
The Boeing CEO exited the company with just 60 million dollars, which is a lot like being sent to prison.
- close04 7y ago
  It's because in the US it takes much more than knowingly allowing such issues to directly kill hundreds to go to prison. We're not talking about "calculated" deaths like the ones VW's CEO hopefully goes to prison for. Rather directly attributable ones like the 737MAX or GM's faulty ignition switch scandals.
  - sseveran 7y ago
    VW's former CEO isn't being charged with killing anyone one. He has been charged with committing fraud by knowingly lying to regulators.
    - close04 7y ago
      I wasn't suggesting that either VW's, GM's, or Boeing's CEOs actually go to prison for "killing" anyone but rather for "knowingly allowing such issues to directly kill". And as the GM example stands to show, it's not particularly hard to get away with this in the US if you are the CEO of a powerful company.
      - ryanlol 7y ago
        Can you provide examples of CEOs of less powerful companies not getting away with this? Your comment seems to necessitate examples of such.

ameen 7y ago

In a way, Boeing’s plight underlines American work culture vs. European and other relaxed but stringent work cultures. American businesses are constantly churning output with not much to show for (relative to effort vs results) as the employees are burned out, investors and C-suite demand more of the middle-management and they in-turn pressure engineering and sales teams, etc. The results have spoken about which one works and which doesn’t. Toxic work culture starts from the top.

ramblerman 7y ago
I think the results have shown not all industries benefit from being purely profit driven. To extrapolate that further is a bit disingenuous. The US is still a leader in innovation over Europe, by a large margin. And I say that as a European.
- Ragnarork 7y ago
  How do you measure "innovation" though? Is it in terms of patents, in terms of the number of new companies, new companies that succeed (and according to which metrics?), new companies using/developing new technologies? Do you consider modern implementations of old concepts as innovation or not (i.e. transitioning some activities into the digital world, that kind of things)? Do you normalize that "innovation" quantity with how much money is injected into the "innovation industry"?
  - jiqiren 7y ago
    yes.
datenhorst 7y ago
I believe Harvard Business School and its ilk deserve a lot of blame for the instilling of a profit-at-any-cost culture across the US.
- strictfp 7y ago
  IMO the business school of though is the one which is currently hindering progression, holding on to values from the industrial revolution. It's as we all know a philosophy of top-down management of expendible resources. This way of thinking simply doesn't apply today. We need a breakthrough of the servant leadership way of thinking.

Entalpi 7y ago

What ever happened to ”safety first”.

xiphias2 7y ago
If you ask the Boeing CEO about it, safety was always the top priority.

toantruong95 7y ago

This is so disgusting. I hope they will go to bankruptcy for what they have done.

hhas01 7y ago

Jesus. Pardon my French, but seeing that photo makes me fully appreciate two things: 1. Just how incredibly low 737 sits to the ground (obviously a feature from back in the days when luggage was manually loaded), and 2. How anyone would think pushing those enormous engines forward like that could be any less disruptive of its proven design than lengthening the undercarriage to give them the clearance they so desperately need. Death of a thousand cuts. Indeed. Hope the monster never flies again.

cedivad 7y ago
2. is a great question I would like an answer to. Here is a side by side comparison with an a320: https://aviation.stackexchange.com/questions/61161/is-there-less-vertical-space-for-an-engine-under-the-wing-of-a-737-max-than-for https://aviation.stackexchange.com/questions/61161/is-there-...
- inferiorhuman 7y ago
  FWIW That's a 737 Classic. The engines were already too big (and still smaller than the A320 engines) as evidenced by the flattened nacelle. The engines grew with each successive generation (NG, MAX). I've read (but not confirmed) one of the other things keeping the 737 so low to ground is the type of emergency exit used. Extending the gear so that the whole mess sits higher off the ground would require new exits and a ton of extra things to certify.
  - hhas01 7y ago
    Yeah, there’s a lot of knock-ons with “just” extending the gear. It would definitely cost a lot more (though still far less than a complete new airframe), which is obviously why Boeing and customers wanted to dodge it. On the other hand, building a plane so aerodynamically unsound that it cratered 346 customers the moment it went out the door is the very textbook definition of “false economy”. One that Boeing itself will be paying for for years to come; even as those who made all the actual decisions gently golden-parachute to the ground.
amyjess 7y ago

GenericsMotors 7y ago

Around the time just before the MAX was grounded, I remember there being lots of unfounded speculations (here on HN as well!) that Lion Air's and Ethiopian's pilots were somehow substandard and lacking the training of their first world counterparts. Meanwhile it was Boeing actively keeping vital information and training from pilots. The lesson from this is still not learned, and can see at least one apologist on this thread repeating this same BS again...it's infuriating.

Brave-Steak 7y ago

I'd really be interested in knowing at what level these two employees were at. How deep does the rot go? Even if the CEO is gone, these people are still around.

vmchale 7y ago

What a shitshow. The entire thing is graft, and the US government turned a blind eye because it's deemed an important industry.

mrtksn 7y ago
Isn't it also unfaithfulness to capitalism to protect an industry in such a way? I would have assumed that if someone believes in the mechanics of capitalism would have let it be.

WalterBright 7y ago

> and that an erroneous MCAS activation would be quickly diagnosed as a runaway stabilizer. The 2013 memo casts doubt on the former, and the two MAX accident sequences disproved the latter. This does not mention that an Emergency Airworthiness Directive was sent to all 737MAX crews after the LA crash explaining exactly how to resolve the runaway trim issue, which is: 1. restore normal trim using the column trim switches 2. cut off the stabilizer trim with the console cutoff switch The text is: Boeing Emergency Airworthiness Directive "Initially, higher control forces may be needed to overcome any stabilizer nose down trim already applied. Electric stabilizer trim can be used to neutralize control column pitch forces before moving the STAB TRIM CUTOUT switches to CUTOUT. Manual stabilizer trim can be used before and after the STAB TRIM CUTOUT switches are moved to CUTOUT." https://theaircurrent.com/wp-content/uploads/2018/11/B737-MAX-AD-1107.pdf https://theaircurrent.com/wp-content/uploads/2018/11/B737-MA... Both the LA and EA crews repeatedly successfully countered the runaway trim with the electric trim switches. The LA crew never took the next step of cutting off the trim. The EA crew did cutoff the trim, but did not trim to normal first. Dealing with runaway trim is a "memory item" for the 737, meaning the pilots are supposed to know about the cutoff switches that are prominently placed on the center console in easy reach.

triceratops 7y ago

Can anyone experienced in aviation ballpark how much retraining a pilot on a new plane costs? How does it compare to a pilot's annual compensation or other figures of importance (e.g. annual airline revenue or profit, operating expenses of a plane etc)? Helping airlines avoid retraining seems like the driving factor behind all these apparently shortsighted decisions that led to the MAX but, assuming everything had gone well, what were the actual potential savings? I'm just trying to understand the stakes.

hurricanetc 7y ago

They wouldn’t have trained on an MCAS failure so this is irrelevant as far as the actual crashes are concerned. It’s just more evidence of Boeing being a corrupt organization.