10 ms·

How the Boeing 737 Max Disaster Looks to a Software Developer (2019)

• sgt 6y ago
  Finally a comprehensive article about the 737 Max issues that I can understand.
• jiofih 6y ago
  (2019)
• raxxorrax 6y ago
  Very interesting article. Seems like Boeing used the Apple approach to computing here in the quest to make more money.
• pulse7 6y ago
  So they wanted to make a "backwards compatible airplane" with new engines so that pilots could use the existing certificates...
  • hn3333 6y ago
    let me finish your summary: ... but the aerodynamics were changed in the process and they fixed it with software, which was bad.
    • throwaway0a5e 6y ago
      Would have been fine if they hadn't fixed it with crap software that was incompatible with the pilots existing runaway trim checklist.
      • raxxorrax 6y ago
        I bet if we look at the requirements analysis of MCAS, the developers just met the specification. I think the article makes to much assumptions about the state of the industry and developer hubris, but the decision of system design of the plane was probably decided before it reached developers. They might also just have a perspective of this one system, so the result is as expected.
        • throwaway0a5e 6y ago
          Yup. The UX guys didn't apreciate how critical the software was and how half-baked it was going to be (and therefore require pilots to need to interact with it requiring good UX). The software guys didn't know that the pilots weren't gonna know how to turn off the software so they didn't know they couldn't half bake it. The engine guys didn't know the software guys were gonna half bake it. The AOA guys retired in the 1990s and had no idea the system was gonna be used for anything more than warning lights. Everyone did their job as it exists on paper but... https://youtu.be/452XjnaHr1A?t=20 https://youtu.be/452XjnaHr1A?t=20
      • jki275 6y ago
        It was perfectly compatible with existing runaway trim checklist. The problem was that the pilots didn't recognize it as runaway trim, in part due to pilot error and in part due to the fact they weren't told the capacity for this system to create a runaway trim situation existed.
    • MaxBarraclough 6y ago
      Fixing aerodynamics with software is nothing new, and isn't always a mistake. The F-16 (a multirole fighter from the 70's) does this. https://en.wikipedia.org/wiki/General_Dynamics_F-16_Fighting_Falcon#Negative_stability_and_fly-by-wire https://en.wikipedia.org/wiki/General_Dynamics_F-16_Fighting...
      • jki275 6y ago
        Every modern fighter jet does this, because they're all aerodynamically unstable.
        • rightbyte 6y ago
          And fighters fall out of the sky quite ofent for no good reason. Luckely the pilots can shoot them self out.
          • liability 6y ago
            > And fighters fall out of the sky quite ofent for no good reason What do you mean by that? That they're crashing for unknown reasons? Because that's generally not true. A lot of effort is put into figuring out the causes of crashes. If you're insinuating that the software is generally to blame, I'd like to see your sources for such a claim. The first F-35 to crash was due to a faulty fuel tube. Was that "for no good reason" or does that count as a "good reason"?
            • rightbyte 6y ago
              I mean "good reasons" as in storms, collisions with birds or trees etc. No I didn't mean software issues or the F-35 in specific, but fighters in general compared to passenger airplanes. I didn't find any specific statistics, but I remember reading it in a book about the Swedish airforce some years ago that fighters just crash alot. They seem to just be way more prone to crashing due to their combat agility need. E.g.: https://economictimes.indiatimes.com/news/defence/13-fighter-jets-crashed-from-2017-18-to-november-2019-government/articleshow/72475143.cms https://economictimes.indiatimes.com/news/defence/13-fighter...
• throwaway0a5e 6y ago
  The author should stick to software and stay out of airframes. >I’ll say it again: In the 737 Max, the engine nacelles themselves can, at high angles of attack, work as a wing and produce lift. And the lift they produce is well ahead of the wing’s center of lift, meaning the nacelles will cause the 737 Max at a high angle of attack to go to a higher angle of attack. This is aerodynamic malpractice of the worst kind. >Apparently the 737 Max pitched up a bit too much for comfort on power application as well as at already-high angles of attack. It violated that most ancient of aviation canons and probably violated the certification criteria of the U.S. Federal Aviation Administration. Pure bullshit. Many low wing passenger aircraft have this tendency somewhere in their flight envelope. You can tone it down with some aerodynamic trickery but it's just a fundamental trade-off you get when the engines are below and forward of the wing, the load is above the wing and you pin the throttle. The reason we got ht MAX debacle was because the new 737 had this tendency to a degree and in a portion of the flight envelope different than the old 737 which meant it couldn't ride on the old 737's cert without a bandage over that. The 737 MAX8 handles perfectly reasonably for an airborne bus. What it does not do is handle perfectly reasonably for a 737-800 hence the reason they overlaid half baked software taking inputs from less than adequate hardware systems (well they were adequate for the less critical systems they were designed to feed into). >Let’s review what the MCAS does: It pushes the nose of the plane down when the system thinks the plane might exceed its angle-of-attack limits; it does so to avoid an aerodynamic stall. No. It (when it's no malfunctioning and trying to put you in a dive) pushes the nose down when you're getting into the portions of the flight envelope where the MAX flies differently than the old 737 in order to fudge the same handling as the old 737 thereby skating by on the old cert. Big disasters require many small things to go just right. If the software was more robust this wouldn't have happened. If the hardware was more robust this wouldn't have happened. If the UX had been compatible with the pilots existing knowledge this wouldn't have happened. If the airlines were willing to pony up for type certs this wouldn't have happened. If the FAA had been hard asses this wouldn't have happened. To play it off as "Boeing tried to fix a bad plane with software and failed, whoopies" is exactly the kind of naive surface level postmortem that results in nothing being learned and these kinds of failures happening again. I know that opinions that deviate from "software engineers are angels that never do anything wrong and corporate is the devil also those hardware guys gave the software guys a crap situation to deal with" is not going to be appreciated on HN but the fact of the matter is that any one of the five-ish parties here (FAA, customers, and engineering, software and management within Boeing) could have broken this chain resulting in a different outcome and to prevent future events like this we need to understand the decisions that happened at the margin that prevented the chain of events from being broken.
• ppod 6y ago
  Great article. I just wonder if this bit is really true: >Most of those market and technical forces are on the side of economics, not safety. People seem to think about airline safety a lot, like, an irrational amount given the risk. I would guess that over the decades consumer pressure was a huge driver of safety improvements, and the increasing automation seems to have been a big safety benefit on balance.
  • nix23 6y ago
    >like, an irrational amount given the risk. The risk that 300 Peoples at once can die? I think that's very rational.
    • ppod 6y ago
      It seems to me that this danger receives a much larger degree of attention than other dangers which are more likely to happen, such as road traffic death.
      • rightbyte 6y ago
        Ye but imagine how dangerous airtravel would be if the safety routines were on level with road travel procedures (i.e. one checkup a year and pretravel routines always ignored).
        • nix23 6y ago
          It would be like that: https://edition.cnn.com/2020/06/25/business/pakistan-fake-pilot-intl-hnk/index.html https://edition.cnn.com/2020/06/25/business/pakistan-fake-pi... and the crash that led to the findings: https://www.bbc.com/news/world-asia-53162627 https://www.bbc.com/news/world-asia-53162627
          • rightbyte 6y ago
            Yes. I say when people e.g. compare paragliding to traffic dangers, "imagine taking those precautions you do before paragliding when walking over the road". It is just different things and the statistics doesn't tell you much about the danger from doing it.
• ppod 6y ago
  This article makes a really good case overall, but it must be deliberate that it chooses not to mention the stabiliser trim cut-out switch. In the part about the Cessna he says: >There are instructions on how to detect when the system malfunctions and how to disable the system, immediately. Disabling the system means pulling the autopilot circuit breaker; instructions on how to do that are strewn throughout the documentation, repeatedly. Every pilot who flies my plane becomes intimately aware that it is not the same as any other 172. This is a big difference between what pilots who want to fly my plane are told and what pilots stepping into a 737 Max are (or were) told. Which really implies that no such procedure existed for the Max, and that's just not true. There may have been problems with the procedure, as far as I know it's not fully known yet why that procedure didn't prevent the accidents, possibly inadequate pilot training. https://www.flightglobal.com/safety/trim-cut-out-puzzle-emerges-from-ethiopian-crash-probe/132175.article https://www.flightglobal.com/safety/trim-cut-out-puzzle-emer...
  • roelschroeven 6y ago
    > it's not fully known yet why that procedure didn't prevent the accidents The only way in the 737 Max to disable MCAS, is to disable electric trim altogether. That means only manual trim is available. When MCAS has trimmed all the way to one of the extremes (as happened in the Ethiopean flight), it requires many revolutions of the trim wheel and superhuman strength to return it to a more appropriate setting (see e.g. https://youtu.be/aoNOVlxJmow?t=750 https://youtu.be/aoNOVlxJmow?t=750). Previous generations of the 737 had two buttons related to the electric trim: one disabled the automatic trim system, the other one disabled electric trim altogether. The 737 Max still has two buttons, but they both have the same effect: disable the electric trim altogether. The old system could easily have prevented the crash by disabling MCAS but leaving the electrically assisted trim enabled. Boeing's procedure to make manual trim adjustments possible is to put the airplane in a dive, because that lowers the aerodynamic forces on the trim surface. When you're only a few hundred feet above the ground, that's not really an option. The pilots re-enabled the electric trim in the hope they could use it in their fight against MCAS. It's the only thing they could think of in the stress of the moment, faced with near-certain death and an airplane actively trying to crash. In hindsight, after reading accident reports and thinking about it in the comfort of chair that's not crashing, there were things they could have done. Lowering the throttle would have lessened the forces on the trim surface, activating the flaps once the speed was low enough would have disabled MCAS. I can't blame the pilots for not coming up with that in the heat of the moment.
• merricksb 6y ago
  If curious, see this big discussion from the time of publication in Apr 2019: https://news.ycombinator.com/item?id=19694207 https://news.ycombinator.com/item?id=19694207 (528 points/299 comments)
• commandlinefan 6y ago
  Isn't this the company that still distributes updates via floppy disks?
  • maccard 6y ago
    What's the issue with that?
    • linuxftw 6y ago
      Floppy disks seem relatively volatile compared to other storage medium. If you're using a floppy disk, are the computers they're loading the data from even capable of hashing the content and ensuring there is no corruption, or is the content signed?
      • nixass 6y ago
        There must be post-checks or checklist to perform after the update. I doubt they go airborne with Gremlins under the hood (737MAX issues aside..)
        • Jtsummers 6y ago
          Often, for software like this, you distribute a few pieces of data with the update. After it's installed the system runs and presents information like version number and the hash value of the binary (or binaries). You're supposed to verify these values as part of a checklist. Additionally, the primary host system can check that the hashed value presented is what it expects. How it handles it at that point may vary if it doesn't match, but at a minimum an error is displayed to the operators and stored in the logs. Since software isn't typically changed daily, this means verifying everything is updated once (update an LRU and update the corresponding portion of the host at the same time) and you're done for 6-18 months.
      • liability 6y ago
        Error detection and correction codes have existed for decades, about as long as digital computers have. Do you have any evidence that this hypothetical flaw has ever actually caused trouble for a Boeing airplane? > capable of hashing Using a modern fancy hash? Maybe not, I'm not sure. But you can bet your ass they can at least work with hamming codes. They're probably doing a lot more validation than you seem to believe possible.
• lqet 6y ago
  This is one of the best paragraphs of this great article: > I believe the relative ease—not to mention the lack of tangible cost—of software updates has created a cultural laziness within the software engineering community. Moreover, because more and more of the hardware that we create is monitored and controlled by software, that cultural laziness is now creeping into hardware engineering—like building airliners. Less thought is now given to getting a design correct and simple up front because it’s so easy to fix what you didn’t get right later. Personally, I have noticed this laziness creeping into areas that are not controlled by software at all after they are finished, like building or road construction.
  • dv_dt 6y ago
    This "laziness" is not really a function of the people doing software work, or as you point out any work in any area, it is imho, an expression of our broken leadership and management culture that puts short term concern for profits before quality.
    • lqet 6y ago
      Indeed, but with software, you can actually have both (short-time profits with low-quality software, and long term quality by fixing the hacks afterwards), so "move fast and break things" might actually be good leadership in software. It doesn't work so well with building planes or houses.
  • edgriebel 6y ago
    Exactly! Look at commercial game releases, it's getting to be where the game disc is merely a token to be able to download the "real" game code. Like laziness in movies today, "f-k it, we'll fix it in post"
• jnxx 6y ago
  So, if I read this right, Boeing's unique feature compared to Airbus was a more direct and manual mode of control, which they advertised as an advantage, and they removed this feature without telling clients and pilots?
• ncmncm 6y ago
  The following corrections to fundamental factual errors should not distract one from the meat of the article: 1. Jet engines, like car engines, are not described by the "Carnot cycle", and their efficiency is not governed by it. The Carnot cycle describes closed-system equipment like refrigerators and Nuke plants. Jet engines, like (our now obsolete) cars, start with new working fluid -- air -- at the intake, and discard it at the end -- exhaust. Their theoretical limit of efficiency is technically 100%, not the 30-40% typical of Carnot equipment. In practice, they don't get close, but those huge ship engines do achieve up to 55%: they all take in air much, much cooler than their exhaust. 2. The change to the thrust vector is not a consequence of moving the engines forward. It is a simple result of the bigger engine diameter making the center farther below the wing. Moving the engines forward and up reduced the size of the change to the thrust vector -- just not enough so to match the old one. But the move had other effects, besides... 3. The added aerodynamic lift of the engine nacelles has nothing to do with more power being produced by the bigger engines. It is purely a product of the angle relative to the direction of air moving past. Bigger barrels have more area, and more lift, and moving them forward gave them more lever arm to apply their lift to rotate the plane instead of just holding it up, as wing lift does. 4. The change to the thrust vector is what makes punching the throttle also apply a force to rotate the airframe to a higher pitch angle. So there are two independent forces -- off-center lift, and off-center thrust -- both causing the same effect: pitching the plane up more than the pilot asked for. How much you get of each is a very complicated product of all the details of how the plane is flying. 5. On all 737s of any vintage, the controls really are still directly, physically connected, by steel cables, to the control surfaces -- the hinged bits at the back edges of wings and wing-shaped things. But there are hydraulics that also yank on the cables, and other stuff yanks on the control column, to doctor the pilot's experience.