12 ms·
How the Boeing 737 Max disaster looks to a software developer (2019)
- remram 5y agoThat reminds me of openssl, which tends to release 1.0.2a, 1.0.2b, 1.0.1u and pretend it's the same version to not have to go through another audit. They fixed that and moved to semver, and the FAA should put limits on what can be "technically the same version" too.
- lamontcg 5y ago> Pitch changes with increasing angle of attack, however, are quite another thing. An airplane approaching an aerodynamic stall cannot, under any circumstances, have a tendency to go further into the stall. This is called “dynamic instability," and the only airplanes that exhibit that characteristic—fighter jets—are also fitted with ejection seats. I still don't get how this design was ever allowed to happen or continue to happen. Passenger jetliners shouldn't be dynamically unstable.
- adgjlsfhk1 5y agoAll planes are only stable within certain flight parameters. The problem wasn't that the plane was unstable, it was that the stability region was different than that of previous 737s. If the plane had been given a different name and required simulator training, none of this mess would have ever happened. (Or if they'd just correctly classified MCAS as a safety critical feature and made it have redundant inputs).
- tiahura 5y agoOr if they’d just done a sanity check on the AOA data.
- WalterBright 5y agoYou're right, but that's a different issue from requiring the airplane to be dynamically stable in all its flight modes.
- WalterBright 5y ago> Passenger jetliners shouldn't be dynamically unstable. Well, they are. The active yaw damper is required by the FAA. https://en.wikipedia.org/wiki/Yaw_damper https://en.wikipedia.org/wiki/Yaw_damper
- mulmen 5y agoWhere does that wiki say that the FAA requires the active yaw damper?
- WalterBright 5y ago"several airliners were deemed to be unsafe to fly without an active yaw damper."
- mulmen 5y agoThe string "FAA" does not appear in that wiki nor the cited article. The cited article just says that descent to FL260 is required in one model of 727 if one of the yaw dampers fails. The cited article doesn't even seem to support the line in the wiki. I don't see the word "unsafe" anywhere in the cited source and the only "requirement" is the descent to FL260 in case of a failure.
- WalterBright 5y agoAnything deemed unsafe has the FAA involved in it. You might also be interested in fatal crashes resulting from yaw damper failures: https://en.wikipedia.org/wiki/Dutch_roll#Accidents https://en.wikipedia.org/wiki/Dutch_roll#Accidents
- deleted 5y ago[deleted]
- VBprogrammer 5y agoAmusingly enough, the 737 is one of those few airframes which are dynamically stable in yaw. The yaw damper is not on the required equipment list. Others like the 757, the failure of one of the two fitted yaw dampers is enough to prevent dispatch.
- mulmen 5y agoThe 737 Max is not aerodynamically unstable to the point of causing these crashes. Any airplane can stall in any attitude or at any airspeed. The Max is not inherently unstable or aerodynamically unsuitable for flight. The Max could fly without MCAS just fine, but pilots would have to be re-trained. Airlines didn't want to foot the bill for such an expense so Boeing applied the Maneuvering Characteristics Augmentation System (MCAS) to literally augment the maneuvering characteristics of the Max to match the 737-NG, which pilots are already familiar with. Bugs in the system then led to runaway trim which caused a stall and crashes.
- WalterBright 5y agoThe runaway trim was driving the nose down into a dive, not up into a stall.
- sofixa 5y ago> Bugs in the system And most of all, and the truly inexcusable part, zero redundancy.
- lamontcg 5y agoThen the correct and simple solution is to rip out the MCAS and re-train all the pilots.
- VBprogrammer 5y ago
- gnabgib 5y agoPast discussion has been prolific... 3 years ago, 528pts, 297 comments[0], a year ago, 51pts, 72 comments[1] [0]: https://news.ycombinator.com/item?id=19694207 https://news.ycombinator.com/item?id=19694207 [1]: https://news.ycombinator.com/item?id=24233339 https://news.ycombinator.com/item?id=24233339
- eric4smith 5y agoWow! That's the best read of what's going on with the 737 Max system and the MCAS software that helps to drive it. Totally understandable now.
- WalterBright 5y agoThe author is a Cessna pilot. It's very, very different, from a jet airliner. High altitude flight with swept wings has little in common with a low-and-slow straight wing Cessna. http://www.gregorytravis.com/About/ http://www.gregorytravis.com/About/ He also writes about runaway trim: http://www.gregorytravis.com/WhyDoesNotPresent/ http://www.gregorytravis.com/WhyDoesNotPresent/ and assumes that Cessna training on stab trim is the same as for a jet airliner. It isn't. He also seems quite unaware that the flight previous to the LA flight that crashed also had MCAS activation, recognized it as runaway trim, and recovered by simply turning off the stab trim.
- WalterBright 5y agoP.S. My dad was in the Air Force during the transition from propellers to jets. He said there were a lot of crashes from expert prop pilots thinking a jet was just a faster and higher flying propeller machine. Jet pilots require substantially different flight training.
- xeromal 5y agoWhat an insightful article! Reading through this gave me a feeling of dread as I imagined the pilots pulling as hard as they could on the yoke (?) and being overpowered by the computer. That is incredibly tragic.
- WalterBright 5y agoOn both the LA flights, and the EA flight, the crews restored normal trim by simply using the electric trim switches, multiple times. One of the LA flights recognized runaway trim, and turned the trim system off after restoring normal trim, and landed without further incident. The other two flights did not. The EA flight received an Emergency Airworthiness Directive instructing them to (1) restore normal trim with the electric trim switches and then (2) turning off the trim system with the cutoff switch. They did not follow those directions. Runaway stabilizer trim is the trim motors coming on and driving the stabilizer in an adverse direction. This is exactly how the MCAS failure presented itself. The author tries to claim it isn't really runaway trim, an argument which doesn't make sense.
- henryrp 5y agoAre you trying to say the MAX was safe to fly with that EAD in place? If so, why did Boeing go to the trouble of redesigning MCAS? It looks like you are pushing a certain agenda. Anyway, just because the pilots "could have" recovered the plane(s) they were flying, does not make the design of MCAS sound. By all accounts the MCAS design was botched (single source of inputs, no sanity check on the inputs, etc.). This does not seem to be a point of controversy. Plus, as others will point out, two hull losses are a grim testament to this.
- WalterBright 5y agoThe agenda I'm pushing is the facts about the crashes that are consistently omitted from popular narratives about it. Other times this has come up, I faulted Boeing for making several design errors in the MCAS. But the pilots bear some responsibility, too, because they did not follow runaway trim emergency procedures. In order to make flying safe, all causes of an accident (and there are usually many for each accident) must be addressed.
- WalterBright 5y agoThe author makes the bizarre assertion that runaway trim is only caused by short circuits, and would only be continuous (not intermittent), and therefore the pilot would not recognize the MCAS runaway trim as runaway trim: http://www.gregorytravis.com/WhyDoesNotPresent/ http://www.gregorytravis.com/WhyDoesNotPresent/ First off, as an engineer who often has to find and fix electrical short circuits, they'd be easy to find if they were continuous. They're frustratingly intermittent. Secondly, I worked on the design of the 757 stab trim system. Runaway trim is the trim system coming on and trying to crash the airplane - it being intermittent is NOT part of the diagnosis. That intermittent vs continuous NEVER was part of the discussions about runaway trim. I don't know where that notion comes from. It doesn't make any sense. Anyhow, the author certainly knows a lot about Cessnas. He stretches this into knowing about 737 systems, a much more complex airplane in its systems and aerodynamics, and he stretches it past the breaking point.
- techie128 5y agoWeird part is that IEEE published this. Someone might actually take it seriously...
- tgsovlerkhgsel 5y agoThere is strong evidence against the assumption that this should be something that a typical flight crew can handle, in the form of the smoldering wreckage of not one but two planes, each with a trained, professional crew.
- WalterBright 5y agoYou're right that there's something wrong with the training. There were 3 flights that had MCAS failure. All three were able to restore normal trim with the electric trim switches, which override MCAS. #1 turned off the trim system and continued the flight without incident. (This flight is almost never mentioned, as it doesn't fit the narrative.) #2 never turned off the trim system and crashed. At this point, an Emergency Airworthiness Directive was distributed to all MAX pilots instructing them to restore trim with the electric trim switches and then turn it off. #3 turned off the trim system when it was severely nose down, and crashed.
- mastazi 5y ago> Everything about the design and manufacture of the Max was done to preserve the myth that 'it's just a 737.' Do we know what is next for Boeing in that same segment? Are they planning to come up with another 737 variant after the MAX, or starting from a completely different platform[1]? Did they disclose anything? [1](I mean from a technical standpoint - not marketing/naming)
- sofixa 5y agoThe 737 Max is already abusing the platform beyond what it could handle, and it's still less efficient than the A320neos, and even less so than the A220s. Boeing need a new short-haul platform, like a decade ago :)
- tgsovlerkhgsel 5y agoThe article mentioned something that I didn't know about before: That the yoke feel is computer-generated and that MCAS was messing with that. It also correctly points out that "with the 737 Max, Boeing has changed philosophies about human/machine interaction" - they didn't just hide a system in the plane that changed the planes behavior without telling the pilot, they fundamentally changed the basic assumptions pilots make, and still called it the same plane that doesn't require separate training. I'd consider that an even bigger crime than the other recklessness that ultimately led to the crashes.
- _ph_ 5y agoI think that part about the yoke feels is incorrect. MCAS only changes the trimming and that changes the yoke force required when trying to maintain the same angle of attack.
- coin 5y ago> Apparently the 737 Max pitched up a bit too much for comfort on power application as well as at already-high angles of attack. It violated that most ancient of aviation canons and probably violated the certification criteria of the U.S. Federal Aviation Administration. But instead of going back to the drawing board and getting the airframe hardware right (more on that below), Boeing relied on something called the “Maneuvering Characteristics Augmentation System," or MCAS. Nope, this is 100% false. The purpose of MCAS is to give the MAX the same aircraft handling characteristics so that it retains the original 737's type rating. Thus 737 type rated pilots can fly the MAX under their 737 type rating. A big part of selling the MAX is type rating commonality with the airlines existing 737-train pilots. Juan Browne, a real 777 airline pilot, mentions this in one of his excellent YouTube videos (https://www.youtube.com/user/blancolirio https://www.youtube.com/user/blancolirio). > I have been a pilot for 30 years I suspect this person is a private pilot with no actual airline aviation experience. Edit: This is the same guy that boosted his aviation credentials by citing that he's flown a 757 simulator (https://www.eetimes.com/software-wont-fix-boeings-faulty-airframe/ https://www.eetimes.com/software-wont-fix-boeings-faulty-air...). So yes, he's a GA pilot that has flow in some aircraft simulators. > Among Boeing’s critics is Gregory Travis, a veteran software engineer and experienced, instrument-rated pilot who has flown aircraft simulators as large as the Boeing 757.
- TedShiller 5y ago
- deleted 5y ago[deleted]
- WalterBright 5y agoI'm not a pilot either, but I worked for 3 years on the 757 stabilizer trim system and detail design, so know a lot about it. My lead engineer was a pilot and would take me flying and let me fly for a few seconds. A good chunk of the engineers at Boeing are also pilots. My dad and brother are pilots. Many of my friends are pilots. FWIW. But you don't have to take my internet credentials seriously, and you shouldn't. So here's some documents which you should, if you want to be informed about this: Boeing Emergency Airworthiness Directive "Initially, higher control forces may be needed to overcome any stabilizer nose down trim already applied. Electric stabilizer trim can be used to neutralize control column pitch forces before moving the STAB TRIM CUTOUT switches to CUTOUT. Manual stabilizer trim can be used before and after the STAB TRIM CUTOUT switches are moved to CUTOUT." https://theaircurrent.com/wp-content/uploads/2018/11/B737-MAX-AD-1107.pdf https://theaircurrent.com/wp-content/uploads/2018/11/B737-MA... 2018 - 035 - PK-LQP Final Report http://knkt.dephub.go.id/knkt/ntsc_aviation/baru/2018%20-%20035%20-%20PK-LQP%20Final%20Report.pdf http://knkt.dephub.go.id/knkt/ntsc_aviation/baru/2018%20-%20... Aviation Week 19-Sep-2019: "The [LA] pilots countered with main electric trim nose-up inputs. At least 25 automatic stabilizer nose-down, pilot-directed stabilizer nose-up exchanges took place and then several nose-down inputs were not countered." "The MCAS activated twice, and the [EA] crew countered with electric trim." https://www.bloomberg.com/news/articles/2020-03-09/why-did-boeing-s-warning-on-737-max-fail-to-stop-second-crash https://www.bloomberg.com/news/articles/2020-03-09/why-did-b...
- noughtme 5y ago> Those people have decades of institutional memory about what has worked in the past and what has not worked. Software people do not. > The people who wrote the code for the original MCAS system were obviously terribly far out of their league and did not know it. Hmmm... having met a few embedded aerospace software engineers, I'm not sure I would be pointing the finger in their direction, although I would love an insider account. All public information seems to indicate these were decisions made at the executive level.
- sofixa 5y agoI'm sorry, but no. Such criminal negligence isn't excusable by "decisions were made higher up". Human lives are at stake with aviation, the engineers implementing this should have known better and refused.
- salawat 5y agoI think you're really discounting the poisonous influence that management can have here. As a Quality Assurance person, my number one mental tic is the "I told you so" that inevitably follows an executive override.
- sofixa 5y agoI'm not discounting manager influence, I'm just saying that for such a matter the engineers should have refused.
- quickthrower2 5y agoMaybe they fear losing their jobs. As I understand it, with the healthcare system, this can itself be perilous.
- salawat 5y agoDon't forget with the consolidation in that space, there isn't exactly a large swathe of competitors to get hired by.
- WalterBright 5y agoI wrote about how my experience in airliner mechanical design influenced the D programming language: Origins of the D Programming Language https://dl.acm.org/doi/pdf/10.1145/3386323 https://dl.acm.org/doi/pdf/10.1145/3386323 Software development can learn a lot from aviation designs.
- salawat 5y agoLink appears to be broken for me. D is next on my GCC safari of programming languages to muck with.
- WalterBright 5y agoI just tried it and it works. I have no idea why it doesn't for you.
- salawat 5y agoNow it works... I will not question this.
- deleted 5y ago[deleted]
- bumblebritches5 5y ago
- atdrummond 5y agoI'm very surprised this article remains published. It directly contradicts the actual narrative of events (including the EAD that, if followed as by the first flights with recorded MCAS issues, offered a prevention for the subsequent crashes) and unfortunately perpetuates incorrect definitions and understandings of runaway trim and unstable airframes.
- sofixa 5y ago> including the EAD that, if followed as by the first flights with recorded MCAS issues, offered a prevention for the subsequent crashes The one that was followed by the Ethiopian pilots and the plane still crashed, resulting in the grounding of the 737 Max by all aviation authorities bar the FAA?
- WalterBright 5y agoIt wasn't followed by the Ethiopian pilots. For reference: Boeing Emergency Airworthiness Directive "Initially, higher control forces may be needed to overcome any stabilizer nose down trim already applied. Electric stabilizer trim can be used to neutralize control column pitch forces before moving the STAB TRIM CUTOUT switches to CUTOUT. Manual stabilizer trim can be used before and after the STAB TRIM CUTOUT switches are moved to CUTOUT." https://theaircurrent.com/wp-content/uploads/2018/11/B737-MAX-AD-1107.pdf https://theaircurrent.com/wp-content/uploads/2018/11/B737-MA... I highlighted the omitted action.
- ummonk 5y agoYou keep saying this every time the 737 Max comes up on HN. I don't know why you still can't understand that "can be" has a very different meaning from "must be", and the Ethiopian Airlines pilots followed the procedures as written. The Boeing airworthiness directive was incredibly poorly written, and Boeing is to blame for this, not the pilots who attempted to follow the directive as written.
- WalterBright 5y ago
- dotancohen 5y agoThis was a terrific article, I remember reading it maybe two years ago. As a self-taught software developer, I've noticed a critical difference between those with a degree fresh out of school and those with experience, a difference which is touched upon in the article. Those with experience know when to say "to implement that, I need to study". But those with a fresh degree rarely know to say "to implement that, I need experience". The author directly accuses software developers and software management inexperienced in the field of aviation as a cause of the failures. The author's name is Gregory Travis, and this abridged paragraph stands out to me: > Another difference is between the autopilots in my system and that in the 737 Max... > ...the system simply goes off-line and alerts the pilot that she is now flying manually. Why would the author refer the themself in the female? Is Gregory a female name too, or is this some literary device that I don't understand?
- salawat 5y ago>Why would the author refer the themself in the female? It's a thing that has become more popular to get away from assumptions that one sex or another is predominantly associated with some manner of activity. I've noticed it in several different authors writings. Or it could be that that by 'she', the author is referring to their aircraft, which I believe traditionally is considered a "she" just as a ship or boat is.
- dotancohen 5y agoYes, I agree in principal any hypothetical character in a narrative should be just as likely to be "she" as "he", but in this case the author specifically mentions his own personal aircraft. Maybe this was an attempt at such (justified) political correctness but being sloppy about where it was applied.
- quickthrower2 5y ago> For the life of me, I do not know why those two basic aviation design considerations, bedrocks of a mind-set that has served the industry so well until now, were not part of the original MCAS design. Because that would force retraining? “Sometimes the MCAS system will disagree, if that happens it will disengage and you are no longer flying a 737, good luck!”
- nicholast 5y agoThis article highlights a few fundamental flaws: - design certification treated as "just another 737" which ignored fundamental deviations in hardware and software, complicated by safety certification by manufacturer instead of public employee - hardware design including a "dynamic instability" in which airplane approaching an aerodynamic stall had a tendency to go further into the stall due to lift produced by the oversized engines at high angles of attack, which was intended to be mitigated with software - omission of using multiple inputs, including the opposite angle-of-attack sensor, in the computer's determination of an impending stall - a changed philosophy about human/machine interaction from humans winning a battle of the wills every time to computers winning a battle of wills in cases of envelope protection The final item is perhaps the one most fundamental to other cases of safety critical human machine interaction designs. If we are signing over agency to machines for envelope protection, that means we need in advance to understand every potential edge case scenario where that envelope may be mis-framed. Such comprehensive foresight in some environments may be intractable. For the 737 this was exasperated by the presence of an inherent source of instability originating from hardware design.
- 02020202 5y ago
- rurban 5y agoIf you click on past you'll see the previous huge discussions on this article
- appstorelottery 5y agoWow, what a fantastic article. Many thanks to Gregory Travis for writing this. It seems we're learning the same lessons over and over again about the nature of failure in complex systems. It seems reasonable to me that we'll observe more of these failures as our world becomes increasingly complicated, particularly in social and economic systems. Perhaps we should be teaching the fundamentals of complex systems in high-school just as we teach basic science? It seems to me there is a distinct lack of appreciation of the nature of complex systems in our culture at a general level. Just as some people think it would be better to learn statistics than calculus in high school - I wonder if it would be equally important to learn about complex systems? I'm interested to hear what others think.
- rramadass 5y agoThere is a bunch of FUD being thrown around in this thread and in the interest of Truth, here is a PSA; 1. Use the Wikipedia Page https://en.wikipedia.org/wiki/Boeing_737_MAX_groundings https://en.wikipedia.org/wiki/Boeing_737_MAX_groundings as your starting point for understanding the details. This article and more and better ones are all listed here. 2. This article from The Verge is the best for the layperson to get the overall picture of all the issues involved: https://www.theverge.com/2019/5/2/18518176/boeing-737-max-crash-problems-human-error-mcas-faa https://www.theverge.com/2019/5/2/18518176/boeing-737-max-cr... Note that "Human Error" mentioned here refers to everything from Boeing Management, Marketing, Sales and Training. 3. Article on Department of Transportation Inspector General's scathing report on how Boeing misled FAA and everybody else : https://www.flightglobal.com/airframers/inspector-general-slams-boeing-for-holding-back-information-on-737-max/139102.article https://www.flightglobal.com/airframers/inspector-general-sl... 4. Remarks from "Allied Pilots Association" : https://www.businessinsider.com/boeing-737-max-apa-head-questions-training-engineering-faa-oversight-2019-6?r=US&IR=T https://www.businessinsider.com/boeing-737-max-apa-head-ques... Boeing's previous suggestion that pilot error may have caused the crashes "offended" members of the APA, Carey said, as he criticized suggestions from some in the media that training standards may have been lower in the countries where the crashes occurred. "To make the claim that these accidents would not happen to US-trained pilots is presumptuous and not supported by fact. Vilifying non-US pilots is disrespectful and not solution-based, nor is it in line with a sorely needed global safety culture that delivers one standard of safety and training." He also said that Ethiopian Airlines has a simulator for the 737 Max, while no US airlines do. 5. Blaming Dead Pilots by Boeing and its lobby in Washington : https://www.corporatecrimereporter.com/news/200/blaming-dead-pilots-brought-to-you-by-boeing/ https://www.corporatecrimereporter.com/news/200/blaming-dead...
- thrower123 5y agoIt looks an awful lot like what I expect to happen when critical code is outsourced to lowest-rate offshore contractors like HCL.
- papito 5y agoThe background of this is covered in the new book, Flying Blind: https://www.amazon.com/Flying-Blind-Tragedy-Fall-Boeing/dp/0385546491 https://www.amazon.com/Flying-Blind-Tragedy-Fall-Boeing/dp/0... It talks about GE a lot as well, as both companies were destroyed by the Jack Welch "school of managers".
- FabHK 5y ago> The principle of Carnot efficiency dictates that the larger and hotter you can make any heat engine, the more efficient it becomes. While this is true, for air-breathing propulsion engines there's another factor: It is more efficient to accelerate a lot of air a bit, than a bit of air a lot. That's why the jet engine was replaced by the turbo fan with increasingly higher bypass ratio: they have a jet engine in the middle, but the fan also pushes through a lot of additional air that passes around the turbine itself entirely. In particular, the propulsive efficiency under some simplifying conditions is 2/(1+v_ex/v_in): the higher the exhaust velocity the worse the efficiency. https://en.wikipedia.org/wiki/Turbofan https://en.wikipedia.org/wiki/Turbofan https://en.wikipedia.org/wiki/Propulsive_efficiency#Jet_engines https://en.wikipedia.org/wiki/Propulsive_efficiency#Jet_engi...
- MilnerRoute 5y ago(2019)