13 ms·

I'm an assistant professor of aerospace engineering and I find this analysis quite spot on, in which this is representative of a much larger issue of economic a

I'm an assistant professor of aerospace engineering and I find this analysis quite spot on, in which this is representative of a much larger issue of economic and regulatory negative incentives, rather than just a "software issue" as some news outlets have reported. What I find downright criminal is this: > Boeing sells an option package that includes an extra AoA vane, and an AoA disagree light The fact that the redundancy of a sensor on which a system capable of sudden, large control inputs relies is an optional package to be purchased separately... I simply have no words. How was this package advertised in the brochure? Pay extra and when the airplane nosedives at high speed, this useful indicator will helpfully warn you it's because AoA reading disagreement?

ams6110 8y ago
This isn't exactly right. The aircraft has two sensors, but the system in question (MCAS) is only ever looking at one of them. That is part of what the software fix is reportedly going to address.
- JorgeGT 8y ago
  That... makes it even worse? Why would someone on Earth disregard an available extra sensor for a system that has unlimited authority to bring the horizontal stabilizer to full deflection?
  - ams6110 8y ago
    Yeah I don't disagree. I don't fly, but have sort of been following this out of curiosity. One comment I've seen from some pilots is basically "ehh, runaway trim isn't a new thing, we train for it in the sim, and there's a standard way to deal with it (disengage the automatic system and trim manually)." So perhaps Boeing felt that this didn't really change anything in that regard. They seem to have been proven wrong.
    - CaptainZapp 8y ago
      Key quote is training on a simulator That's a tall order when the manufacturer doesn't inform their customers out of commercial expediency.
    - salawat 8y ago
      No simulator accounted for a borked AoA sensor being able to nosedive the plane, or the altered aerodynamics from the different engine configuration. So while yes, runaway trim is a thing, the circumstances under which an error could happen are substantially different. For instance, a pilot could check the maintenance log book and see some work or inspection was done on the auto-trim subsystem. This would prime the pilot to be more on the alert for the possibility of trim misbehavior during the flight. Crisis averted, right? But no pilot would look at an entry for an AoA sensor being off or worked on, and would think, "Boy, better look out for that safety system I never knew existed that could cause a trim runaway because my previously airworthyness agnostic AoA sensor is unreliable." Flying and complex system diagnosis in time sensitive conditions requires extensive ahead of time mental model creation ahead of time.
      - lutorm 8y ago
        It is weird, though, that this was not caught in simulations (not training simulations, but in design simulations.) I would assume that running simulations to see how the design behaves when sensors fail would be an absolutely standard thing to do. I don't work on airplanes, but we routinely simulate sensor failures and their effect on software systems and you'd think this would not be optional for passenger aircraft certification.
        salawat 8y ago
        When all of your testing is in house, and you don't have people who are willing to put the project at jeopardy for the sake of doing it right, you'll be amazed what can get overlooked. If anything, the process of engineering is most difficult in that answering most questions tends to be straightforward (not easy, but straightforward once you know the right methods to apply), while figuring out if you've asked all the right questions is the thing that keeps me awake at night. If you don't ask and dig in until you've answered it fully, it's easy to get blind sided. I have difficulty believing it could have gone so horribly wrong now, but I can't deny that just from the information available, even if MCAS isn't the root cause of the Ethiopian Air crash, there is some egregious failures in sound practice going on with Boeing to have slipped up this badly. That's the thing with sound Engineering, when you do it, it just verks. When you don't...
  - inferiorhuman 8y ago
    Because there was no available extra sensor. Trevor gets a few other things wrong too. The elevator feel system (EFS) uses multiple sensors, in fact it has its own pitot probes[1] (two of them) on the elevator. It's not clear if it uses both, but I suspect it also uses both alpha vanes up front like MCAS should be doing. Input from the alpha vane (AoA sensor) is new on the MAX, IIRC. His tweets were a bit ambiguous, but the elevator feel system does not change the trim at all (also EFS was non-op on the Lion Air bird). It's almost certainly NOT a sensor problem. Look at the graphs from the black box on the Lion Air flight. The angle-of-attack was almost exactly offset by twenty degrees left to right[2]. That sensor was working just fine, but something else on the path was fucked up (bad wiring? bad ADIRU? both have happened before). The fact that the alpha vane was replaced and that didn't fix the angle-of-attack readings supports this. Also, the 737 NG and 737 MAX use identical alpha vanes. While the angle-of-attack data is less important on an NG, we haven't seen people come out of the woodwork talking about how common alpha vane failures are on an NG. I'm inclined to believe that they're not all that common. The Lion Air flight crew did write up the sensor problems[3]. I saw another one of these "I'm a software engineer so I've got great insight" posts[4] and chuckled. It's a bit of swing and a miss as well. MCAS is hardly silent or subtle. The same trim wheels his Cessna have are present on the 737 and they're noisy[5]. MCAS doesn't quite override the pilot either, and you can see that on the black box graphs from the Lion Air flight[2]. Twenty times the pilot flying entered opposite trim to counteract MCAS and twenty times MCAS paused for five seconds. On the twenty-first time something changed and the plane crashed. It's not quite that simple and that's why there are active investigations. 1: https://4.bp.blogspot.com/-ZsL1asAUBRU/W-pSdWs2dYI/AAAAAAAAFBg/RlymmfQCny8tUpNqZW6deaS-6GXWPL5KQCLcBGAs/s1600/Screen%2BShot%2B2018-11-12%2Bat%2B8.26.08%2BPM.png https://4.bp.blogspot.com/-ZsL1asAUBRU/W-pSdWs2dYI/AAAAAAAAF... 2: https://reports.aviation-safety.net/2018/20181029-0_B38M_PK-LQP_PRELIMINARY.pdf https://reports.aviation-safety.net/2018/20181029-0_B38M_PK-... 3: https://theaircurrent.com/wp-content/uploads/2018/10/jt43-jt610-lionair-mx-log-annotated.jpeg https://theaircurrent.com/wp-content/uploads/2018/10/jt43-jt... 4: https://medium.com/@jpaulreed/the-737max-and-why-software-engineers-should-pay-attention-a041290994bd https://medium.com/@jpaulreed/the-737max-and-why-software-en... 5: https://www.youtube.com/watch?v=3pPRuFHR1co https://www.youtube.com/watch?v=3pPRuFHR1co
    - 8y ago
  - m463 8y ago
    serious question: which sensor has failed? you need at least three sensors. (although the "sensors disagree" option you can purchase is better than nothing)
- im_down_w_otp 8y ago
  For a system with the responsibilities and impact on the safety-case that MCAS has, not even a 1oo2D design would be sufficient, since there's no clear way to determine which input is correct or incorrect if they diverge, but worse still is that MCAS doesn't seem to leverage even that basic, yet still insufficient, safety design from what I've been able to find on it. At the end of the day the fail-safe is to throw control back to the human. And, then of course pin all the liability onto the human operator when things go wrong. This was the problem I had with the Lion Air conclusion. Sure, it may be the case that a pilot can potentially override this system with situational awareness and training. However, the pilot didn't create this band-aid pile in the first place.
  - tzs 8y ago
    > For a system with the responsibilities and impact on the safety-case that MCAS has, not even a 1oo2D design would be sufficient, since there's no clear way to determine which input is correct or incorrect if they diverge Wouldn't that depend on how often MCAS is actually needed, and how it handles divergent sensors? My understanding as a layman is that there are situations where a 737MAX is more likely to stall than a "regular" 737 would, but I haven't seen anything on how often those occur. Are they expected to routinely occur, so a fully functioning MCAS is important for safe operation of the plane? Or are they supposed to be rare, one in several million events, and the MCAS is there to make handling those events the same as they would be handled in a "regular" 737, to minimize the need for retraining? If the later, then I'd expect two sensors would be fine, or even just one sensor IF there was a reliable way to tell if it had failed, as long as the MCAS response to a sensor problem is to disable itself and let the pilots know they have to stay out of those "more likely to stall" situations for the rest of the flight.
    - salawat 8y ago
      The problem comes in where the MCAS use should be infrequent, unless the sensor is borked. In computing, we have a principle stated Garbage in, Garbage out. Circuits and electronics don't think. They compute. There is no error checking except that which is specifically designed and implemented into the system. If you're getting data from a biased sensor with +/- 10 degrees AoA, a 10 degree actual AoA (well within the safe operating envelope) suddenly appears to MCAS as a 20 degree AoA (oh shit territory). The system therefore engages doing exactly what it was designed to do. Thus is the crux of the matter. The pilot was flying safely while his AoA sensor was telling a safety system he knew nothing about that he was flying dangerously. The AoA system on earlier models of aircraft that the MAX was based on were a functional luxury/situational awareness aid. The flyability was not impacted by a horked sensor. That changed once they had to add a software driven mechanism to keep the flight characteristics similar enough with the old airframe to be able to release the aircraft to 737 trained pilots, and not have to worry about retraining. Maintenance and pilot alike both needed to be aware that the AoA sensor became a safety critical component due to a failure or miscalibration jeopardizing the controllability of the airframe. If they had gone through a full recert of this airframe, and not an expedited self-certify/grandfathering, these tragedies would have had a much smaller likelihood of occurring due to the increased scrutiny. Props on Brazil for doing their own footwork, and not blindly trusting the FAA's delegation to the manufacturer. Personally, as a software engineer and quality assurance specialist, if I'd seen anything remotely like this come across my desk, I'd be raising hell, even if it meant yanking someone into a VP/C's office and giving them a dressing down for skimping on a cross-cutting safety critical concern, deadlines be damned.
- blattimwind 8y ago
  AIUI the entire FCC only uses one set of sensors and which set of sensors is used is switched together with the FCC?
- cameldrv 8y ago
  We will see. Part of the avionics design is that there is a “Brick Wall” between the two sets of sensors, air data computers, and flight control computers. If the FCC uses both for MCAS, that safety concept doesn’t exist anymore. That is certainly the reason they didn’t compare the AoA sensors in the MCAS logic. I think they could probably use pitot/static as a cross check on AoA, but it has some tricky edge cases.
- clavoie 8y ago
  So, a modern aircraft probably have a few thousands sensors (if not way, way more), not strictly redundant but overlapping in some fashion -- I have to believe there's been some modelling trying to check that sensor input is consistent; if only to have some kind of way for the computer to figure out that a sensor has gone crazy (say your compass suddenly detects a significant change in heading, but no sensor has detected even a degree of roll, a change in speed, heading computed by GPS over time is stable... compass might be borked, whaddya say?) Why isn't such modelling used to let sensors be disabled when they're clearly sending bogus input?
  - rocqua 8y ago
    My guess is that such systems are complex, hard to model for safety assurances, and hard to reason about in the moment. Yes, it will probably make the system safer, but your guarantees become a lot more fuzzy. This is not the field for fuzzy guarantees.
  - de_watcher 8y ago
    > Why isn't such modelling used to let sensors be disabled when they're clearly sending bogus input? You need to crash a lot of planes to train that NN.
  - m463 8y ago
    there should be at least 3 AOA sensors - so you can not only detect a mismatch, but figure out which sensor is most probably incorrect.
- cmurf 8y ago
  Reports say it alternates each flight between them; if that's correct, in the case of one good sensor and one bad, that you wouldn't see back to back flights with anomalous behavior related to a bad sensor. Otherwise, perhaps the problem isn't with the sensor itself, but with some part of how sensor information is communicated to the interpretive computer.
hn_throwaway_99 8y ago
Yes, was just about to post that exact section before I saw your comment. I've been following this story fairly closely, but this was news to me. I've seen other discussions that were along the lines of "We're not 100% sure why the planes crashed, so now that they've been grounded without a clear root-cause understanding it may take a long time before they are certified to fly again because we don't know exactly what to fix." I call BS on that line of thinking. At the very least it seems like an easy decision to require redundancy in the AoA vane before letting the planes fly again.
neuronic 8y ago
Might as well make the trim wheel or flap control optional packages as well at that point. Except they can't because all pilots know about those since day 1 of their career as opposed to the MCAS sensor-software system designed to deal with airframe engineering issues.
subcosmos 8y ago
Rate 1/5 stars. App is crippled without in-flight upgrade. SMH
Animats 8y ago
I was amazed at that. Boeing used to be known for overdesign for safety. The B-747 had four redundant hydraulic systems. Here's a 787 doing aerobatics at the Farnborough air show to show that it can operate way outside the normal passenger aircraft flight envelope.[1] Boeing used to be an engineering-first company. HQ was at Boeing's own airport near Seattle. Then they got new management and moved corporate HQ to Chicago. [1] https://www.youtube.com/watch?v=vzr313wSY_Y https://www.youtube.com/watch?v=vzr313wSY_Y
- tyingq 8y ago
  Similar video with a test pilot pushing a 707 pretty hard. Includes a barrel roll, which isn't hard on the aircraft, but unusual to see with a passenger aircraft. https://youtu.be/Ra_khhzuFlE https://youtu.be/Ra_khhzuFlE The 707 was pretty similar to the 737...same main fuselage dimensions, similar pax capacity, etc.
  - deleted 8y ago
    [deleted]
  - nickgrosvenor 8y ago
    Love the 4k footage.
    - tyingq 8y ago
      Ah, well, it is from 1955.
  - BuildTheRobots 8y ago
    Tex Johnston, the test pilot on that flight, did that off his own back and left president of Boeing fuming as it really wasn't part of the plan. I believe they fired him for it - though then hired him back after 24 hours and the realisation that it was a stunning piece of advertising. https://www.seattletimes.com/seattle-news/60-years-ago-the-famous-boeing-707-barrel-roll-over-lake-washington/ https://www.seattletimes.com/seattle-news/60-years-ago-the-f...
- 8y ago
tntn 8y ago
How is this different from extra safety features available as option packages for automobiles?
- simion314 8y ago
  If my airbag is missing my car won't crash into traffic by itself, if the airplane system fails the airplane points you into the ground.
  - therealx 8y ago
    No, but, say you don't pay for the automatic breaking system. You may rear end someone that the car might have otherwise saved you from.
    - acqq 8y ago
      But here it was: pay to know if the automatic braking system will speed up the car for himself every time you first press the brake and then stop pressing it as you see that it's seems to be better: https://www.nytimes.com/interactive/2019/03/13/world/boeing-737-crash-investigation.html https://www.nytimes.com/interactive/2019/03/13/world/boeing-... Look at the charts above. Both pilots "fought" with the controls, correcting the plane course, then believing "it's corrected now" and then the plane sunk again and again.
    - gulikoza 8y ago
      Again, yes, you may rear end someone, but this is not the correct analogy here. In this situation, the car rear ends someone while you are fully breaking because you didn't pay for the automatic breaking upgrade. This is a fail.
    - function_seven 8y ago
      The difference here is that MCAS is an automatic feature that Boeing needed to mimic the existing 737 flight characteristics. They Created the need for the automatic system in the first place, then sell the safety redundancy as an option. To extend (torture?) the automotive analogy, it would be as if the manufacturer substituted a new braking system that required preheating the rotors, installed software to automatically ride the brakes for the first 2 miles of driving in order to get that preheat done, then sold an optional safety feature to verify that the brakes were at the proper temperature. The old way worked just fine without need for “emulation”. The new airframe required some software to mimic the old one, but they decided to charge you for the add-on to make sure that their band-aid worked reliably.
Scoundreller 8y ago
Could it be an ETOPS requirement?
sitkack 8y ago
Not quite as egregious, but Amazon will sell you an Amazon Store Credit Card and then continually pressure you into buying the account protection. Giving away the liability and then up selling the cure is immoral.
- quickthrower2 8y ago
  Wait what? Aren’t there consumer credit laws that offer the protection you need?
  - ficklepickle 8y ago
    It's probably insurance that pays your outstanding balance if you are badly injured or get a terminal disease or something. It's usually nearly impossible to claim and therefore a scam.
bitminer 8y ago
What concerns me is the evidence of regulatory capture. Boeing was telling the FAA what the approval schedule was going to be. And the FAA sucked it up. Instead of saying "our review capacity is x pages per day. You will receive our reviews on such-and-such a date." (And yes I am oversimplifying.)
- bgorman 8y ago
  Boeing has spent more money lobbying the US federal government over the last 20 years than any other company. Last year they had the government kill Bombardier's attempt to break the Airbus/Boeing duopoly. Boeing also prevented the US military from buying an Airbus tanker a few years back. Boeing has definitely captured the US government.
  - spenczar5 8y ago
    > Boeing has spent more money lobbying the US federal government over the last 20 years than any other company. This is simply false. Boeing has spent a lot, but less than General Electric, or the National Association of Realtors, or many (many!) healthcare and pharmaceutical companies. Source: https://www.opensecrets.org/lobby/top.php?indexType=s https://www.opensecrets.org/lobby/top.php?indexType=s
    - bgorman 8y ago
      While your source seems to suggest that my statement is false, a closer looks shows that the only corporation (I.e. not an industry special interest group like the national association of realtors) that spent more money than Boeing was General Electric. So I would say your statement is even more misleading than mine. General Electric has fallen off a cliff since 1998. I suppose my "20" years statement is false according to "OpenSecrets", but if you change my statement to "15" years, it is probably correct, since General Electric did most of their lobbying closer to 1998 than 2018.
      - spenczar5 8y ago
        > I would say your statement is even more misleading than mine. Fair enough, I guess. I think of groups like Blue Cross/Blue Shield or the NAR as representing corporations, but I can see how you'd disagree on a strict reading. I didn't mean to mislead. > General Electric has fallen off a cliff since 1998. In 1998, GE spent $7.28m on lobbying. Nearly every year since then, they have spent more than that, generally staying north of $16m per year, with the exception of the last three years. Your statement is true only for the last three years. Same source: https://www.opensecrets.org/lobby/clientsum.php?id=D000000125&year=2018 https://www.opensecrets.org/lobby/clientsum.php?id=D00000012... for GE, https://www.opensecrets.org/lobby/clientsum.php?id=D000000100&year=2018 https://www.opensecrets.org/lobby/clientsum.php?id=D00000010... for Boeing.
michaelfeathers 8y ago
I'd go further. Use of software to make a plane airworthy when it isn't natively airworthy is a problem. Civilian aircraft should have a different standard than fighter aircraft: one that allows graceful degradation under failure.
- clavoie 8y ago
  Being a total layman in terms of aircraft design -- I've always wondered why they weren't built more akin to distributed systems. Instead of 1-3 large fuel tanks, have hundreds of small ones, so losing one is not a disaster. Similar with hydraulics and electronic wiring -- instead of a single cable running along wings, why not a redundant network that can survive everything short of a wing being sheared off?
  - BillinghamJ 8y ago
    The things you've mentioned are likely to have significant downsides specifically in terms of weight and complexity
    - inetsee 8y ago
      The A-10 Close Air Support aircraft manages to survive in exceptionally stressful circumstances. The flight control systems are triply redundant: two independent hydraulic systems, and a set of cables if both hydraulic systems fail. Having a second angle of attack sensor on the Boeing 737 MAX (which is critical to aircraft safety) be an extra cost option is unconscionable.
  - kenneth 8y ago
    That's pretty much the case. Fuel tanks are split into multiple smaller tanks in the wings, and most systems have one or multiple redundancy (multiple hydraulics systems, multiple flight computers, multiple black boxes, graceful failure protocols, etc.)
  - Aloha 8y ago
    How do you fill hundreds of small fuel tanks?
phkahler 8y ago
One would hope MCAS would not do anything if the AoA input was known bad. This looks really bad for the company. I'd be asking for FMEAs and/or FTAs on this, what the conclusions were, and how they intended to mitigate this exact failure - because it's got to be in there.
- zaphar 8y ago
  If you've only got 2 sensors and you aren't even reading from one of them then "bad" input is hard to know. You can do nothing if they both disagree but you can't actually know which sensor is bad and which is good without a tie breaker.
  - Gibbon1 8y ago
    Who knows but it the chatter makes it sound like the system doesn't have redundant sensors. You have two systems with one sensor each. One system is active and the other is inactive. And no voting logic.
  - ddalex 8y ago
    You don't need to automatically fix the disagreement. You just need to stop taking nose-down action and inform the human in charge . Problem is, if you need to inform the human, you also need to train the human, which leads to extra costs.
cameldrv 8y ago
(I originally wrote what my Brother in law posted on Twitter) I believe that the package is for Cat III hand flown landings. I think that to do that, you need AoA on the HUD, and in the event of an AoA failure you have to go around. It is strange for sure that you need AoA consistency checks for that, but not for MCAS.
ip26 8y ago
Read somewhere a few days ago that Southwest, operator of maybe the most 737's in the world, didn't like the MCAS on the MAX and that extra "package" was essentially customizations/modifications created specifically to make them happy.
Gibbon1 8y ago
I'll also point this out "Unlike the EFS system, MCAS can make huge nose down trim changes." That's a potential reason why an unaware pilot wouldn't disengage the auto trim system. Because he wouldn't think the trim system as being capable of causing an extreme pitch down. Also this response. > Interesting. So how did Boeing get a plane without redundancy in a critical sub system approved? Answered own question, by hiding it in what is normally a non-critical subsystem.
Alex3917 8y ago
> What I find downright criminal is this Not much different than how with most car companies, the only way to get headlights that don't blind oncoming cars is to pay to upgrade to leather seats. C.f.: https://www.iihs.org/iihs/news/desktopnews/night-vision-headlights-improve-but-base-models-leave-drivers-in-the-dark https://www.iihs.org/iihs/news/desktopnews/night-vision-head...
Reason077 8y ago
> "The fact that the redundancy of a sensor on which a system capable of sudden, large control inputs relies is an optional package to be purchased separately... I simply have no words." As far as I'm aware, all Boeing 737 aircraft have 2 AoA sensors. There isn't an option to add a 3rd redundant one like the Twitter poster suggests. From what I understand, what is optional is the display of the AoA on the main screen and/or HUD, and the "AOA DISAGREE" warning which cross-checks the data from the 2 sensors. (Airbus A320s have 3 AoA sensors. A380, A350, and Bombardier C-Series, aka A220, have 4.)
- gaius_baltar 8y ago
  I got this in the other way: there is no actual redundancy even with the optional package, the new sensor is only used to compare inputs with the one used by the control system and warn the pilots if they disagree. Pretty sure I'm misinterpreting things (I hope!). In the other hand, if there are two sensors, what the control system does when they disagree and there is no hardware to show this to the pilots?
  - Reason077 8y ago
    Right. The optional packages don't add additional sensors and don't add redundancy. But they may improve situational awareness by drawing pilot's attention to a sensor problem faster. The MCAS system, apparently, always takes its input from one of the two AoA sensors - it alternates each flight. Lots of details here: http://www.b737.org.uk/mcas.htm http://www.b737.org.uk/mcas.htm
m3kw9 8y ago
Stuff like this is usually triple redundant
gok 8y ago
Safety as an option is pretty standard in aviation, unfortunately. For example, collision avoidance systems have been in development since the 1950s, but TCAS remains only partially required to this day.
WalterBright 8y ago
> quite spot on > Nowhere in here is there a software problem. I worked on the design of the 757 stab trim system, am trained in aerospace engineering, and am a programmer. I'm not a pilot. I am not explicitly familiar with the 737 stab trim system. It is indeed at least partially a software problem. I had thought of two possible improvements: one is to limit the authority of the MCAS's commands to the trim, and the other is to not issue further commands if the pilot fights the trim commands. I read today that Boeing's proposed software fix includes both of those.
- sgustard 8y ago
  I have to agree. If you're saying "the software is working to spec, but the specs are bad," then you're still describing a software problem.
  - speedplane 8y ago
    > If you're saying "the software is working to spec, but the specs are bad," then you're still describing a software problem. This is a pretty bold statement. The spec includes many non-software related features.
    - cashsterling 8y ago
      In aerospace, lack of proper requirements in the flight software would be a system engineering problem... a semantic difference really, but important nonetheless. System engineering and validation has a metric ton of 'tools' to catch bad system design and improper algorithm implementation... I really question what the hell happened here: - Did they just pencil whip the FMEA on this or what? (failure mode and effects analysis) - What happened with the Hardware-in-the-loop flight simulation when they tested the scenario where the AoA sensor givesg spurious data, both high and low (but especially high)? I mean... they did test this, right?
- JorgeGT 8y ago
  In that sense, I agree with you. I meant to say that this wasn't a software bug or programmer's fault like "oh, someone wrote max_authority = 6º instead of 0.5º in line 8745679", but something bigger: the very design and specs of the system and a certification process that didn't challenge those aspects.
neurotech1 8y ago
Airbus A320 used to have GPWS (Ground Proximity Warning System) as an optional extra. A budget airline didn't order it, and quite possibly encouraged pilots to make excessively fast approaches, and led to a CFIT (Controlled Flight Into Terrain) crash of a perfectly good jet.[0] Another crash related to the 737 MAX crash, is Scandinavian 751 [1]. On a MD-81 airliner, with "Automatic Thrust Restoration", A system the pilots were not trained on, caused the thrust to be increased, when the pilot tried to throttle back the damaged engines and clear the compressor stall. "Pilot Betrayed" was the Air Crash Investigations episode. [0] https://en.wikipedia.org/wiki/Air_Inter_Flight_148 https://en.wikipedia.org/wiki/Air_Inter_Flight_148 [1] https://en.wikipedia.org/wiki/Scandinavian_Airlines_Flight_751 https://en.wikipedia.org/wiki/Scandinavian_Airlines_Flight_7... [2] https://www.imdb.com/title/tt1862850/ https://www.imdb.com/title/tt1862850/
jacurtis 8y ago
For anyone who is old enough to remember when airbags first came onto the scene in cars, they were sold as optional packages. Most car manufacturers called these "safety packages". These optional packages were sold basically in the same way as you jokingly mentioned above. "Buy this optional package and when you hit a tree at 50mph, this will keep you from flying through the windshield and will probably save your life." I actually remember a conversation that my parents had with eachother when I was young about whether it was "worth it" to buy airbags in their new car. Yes, that was a REAL conversation that Millions of people had and a decision they had to make for nearly 25 years before airbags became standard in cars in the late 90s. Lots of people felt that the optional saftey package weren't worth the money and many people died through the '80s and '90s from car crashes that they would have survived had they splurged for the "safety package" when they purchased their car from the dealer. It turns out that when people are buying cars, they aren't planning on crashing them. So spending extra money that only helps them when they crash felt unnecessary. These packages could cost in the neighborhood of $2,500 extra. Which on a $20,000 car is over 10% extra! Would you pay 10% extra for a safety package if cars today didn't come with standard safety equipment? This is what led to the USA making airbags mandatory in 1998, forcing this safety feature on manufacturers in order to save lives. I imagine the 737 AoA sensors were sold as the same thing. Boeing sold this as a "additional redundancy package" or "safety package" and probably included many other features as part of a larger package. Just like thrifty car buyers of the 70s, 80s, and 90s who figured they didn't need airbags in their cars and opted out of the safety package. The budget airlines of Lion Air and Ethiopian Airlines had a good history with the normal 737 planes and figured this additional safety package wasn't worth the money.
- heavenlyblue 8y ago
  You do understand that a single light in an aircraft doesn’t cost 10% of it’s total budget? That’s like saying that car manufacturers decided to add a light saying their ABS is disengaged depending on whether you paid for it. That’s more like it.
cashsterling 8y ago
I, too, have no words to communicate my astonishment. A phrase comes to mind: "To screw up this bad you need an MBA." Which is not really intended as a dig on MBA's... but rather to indicate that this debacle has the feeling of business decisions overruling sound engineering design. This seems poor system engineering on multiple levels: - the whole airframe / engine thrust composite flight characteristic requiring an agressive AoA limiting safety... on a commercial airliner? what? - The default MCAS relying on a single sensor, which seems to be occasionally quite unreliable, without sensor truth'ing based on other measurements (i.e. dividing ground speed by rate of altitude change to estimate AoA, then also figuring for control surface position feedback. - The MCAS not having proper situational behavior: i.e. "I'm only 500 feet of the deck... maybe this not a great time to try to tip the nose down too hard... you know, in case that stupid sensor is wrong. Oh... and the pilot is also pulling back on the yoke trying to correct something I'm doing. yeah... my bad, MCAS will chill right now and assume the pilot is NOT trying to fly this thing like a stunt plane 500 feet off the deck".
- chipsa 8y ago
  Angle of Attack cannot be inferred from ground speed and vertical speed. It's quite possible to have a massively negative vertical speed (10000 feet per minute descent), and have an Angle of Attack greater than 45 degrees. Angle of Attack is strictly the angle of the airflow compared to the aircraft.
  - cashsterling 8y ago
    We could argue about this (technically, you are not wrong) but the point is, the aircraft MCAS should be able to do this robustly with available sensor data.* Without going into a lengthy analysis... you can infer AoA from groundspeed, rate of climb/descent, pitch/yaw/roll, and position feedback of control surfaces. Recent information on these data points can be used to calculate present and near future flight dynamics. The fields of Kalman filtering, adaptive control, and optimal control as applied to aerospace engineering are decades old. There are many examples where adaptive flight controls can fly an aircraft that has inherently unstable flight dynamics... In fact most modern jet fighter require computer controlled flight controls to maintain stable flight. Applying these algorithms to an airliner for sensor truth'ing is a cake walk. * I think you actually can infer situational AoA from ground speed and altitude data: An example you could build a regression model of standard take-off scenarios and using recent groundspeed, altitude change, and the rate of change of (groundspeed/altitude). You could build this off of flight test data and simulation. In practice, there may be no point to doing this because a wealth of additional flight sensor information is available.
engire 8y ago
Can I first correct you on two major errors in your statement. TWO AOA vanes are NOT an option on any aircraft. Every aircraft have two vanes as standard. the option is for a small AOA position indicator icon on the top corner on the main screen for the captains side. also there is no AOA disagree light. This is an amber AOA disagree text that appears on the screen. I can only say that on my airline this is standard as it was on the 737 NG in our fleet.

ams6110 8y ago

This isn't exactly right. The aircraft has two sensors, but the system in question (MCAS) is only ever looking at one of them. That is part of what the software fix is reportedly going to address.

JorgeGT 8y ago
That... makes it even worse? Why would someone on Earth disregard an available extra sensor for a system that has unlimited authority to bring the horizontal stabilizer to full deflection?
- ams6110 8y ago
  Yeah I don't disagree. I don't fly, but have sort of been following this out of curiosity. One comment I've seen from some pilots is basically "ehh, runaway trim isn't a new thing, we train for it in the sim, and there's a standard way to deal with it (disengage the automatic system and trim manually)." So perhaps Boeing felt that this didn't really change anything in that regard. They seem to have been proven wrong.
  - CaptainZapp 8y ago
    Key quote is training on a simulator That's a tall order when the manufacturer doesn't inform their customers out of commercial expediency.
  - salawat 8y ago
    No simulator accounted for a borked AoA sensor being able to nosedive the plane, or the altered aerodynamics from the different engine configuration. So while yes, runaway trim is a thing, the circumstances under which an error could happen are substantially different. For instance, a pilot could check the maintenance log book and see some work or inspection was done on the auto-trim subsystem. This would prime the pilot to be more on the alert for the possibility of trim misbehavior during the flight. Crisis averted, right? But no pilot would look at an entry for an AoA sensor being off or worked on, and would think, "Boy, better look out for that safety system I never knew existed that could cause a trim runaway because my previously airworthyness agnostic AoA sensor is unreliable." Flying and complex system diagnosis in time sensitive conditions requires extensive ahead of time mental model creation ahead of time.
    - lutorm 8y ago
      It is weird, though, that this was not caught in simulations (not training simulations, but in design simulations.) I would assume that running simulations to see how the design behaves when sensors fail would be an absolutely standard thing to do. I don't work on airplanes, but we routinely simulate sensor failures and their effect on software systems and you'd think this would not be optional for passenger aircraft certification.
      - salawat 8y ago
        When all of your testing is in house, and you don't have people who are willing to put the project at jeopardy for the sake of doing it right, you'll be amazed what can get overlooked. If anything, the process of engineering is most difficult in that answering most questions tends to be straightforward (not easy, but straightforward once you know the right methods to apply), while figuring out if you've asked all the right questions is the thing that keeps me awake at night. If you don't ask and dig in until you've answered it fully, it's easy to get blind sided. I have difficulty believing it could have gone so horribly wrong now, but I can't deny that just from the information available, even if MCAS isn't the root cause of the Ethiopian Air crash, there is some egregious failures in sound practice going on with Boeing to have slipped up this badly. That's the thing with sound Engineering, when you do it, it just verks. When you don't...
- inferiorhuman 8y ago
  Because there was no available extra sensor. Trevor gets a few other things wrong too. The elevator feel system (EFS) uses multiple sensors, in fact it has its own pitot probes[1] (two of them) on the elevator. It's not clear if it uses both, but I suspect it also uses both alpha vanes up front like MCAS should be doing. Input from the alpha vane (AoA sensor) is new on the MAX, IIRC. His tweets were a bit ambiguous, but the elevator feel system does not change the trim at all (also EFS was non-op on the Lion Air bird). It's almost certainly NOT a sensor problem. Look at the graphs from the black box on the Lion Air flight. The angle-of-attack was almost exactly offset by twenty degrees left to right[2]. That sensor was working just fine, but something else on the path was fucked up (bad wiring? bad ADIRU? both have happened before). The fact that the alpha vane was replaced and that didn't fix the angle-of-attack readings supports this. Also, the 737 NG and 737 MAX use identical alpha vanes. While the angle-of-attack data is less important on an NG, we haven't seen people come out of the woodwork talking about how common alpha vane failures are on an NG. I'm inclined to believe that they're not all that common. The Lion Air flight crew did write up the sensor problems[3]. I saw another one of these "I'm a software engineer so I've got great insight" posts[4] and chuckled. It's a bit of swing and a miss as well. MCAS is hardly silent or subtle. The same trim wheels his Cessna have are present on the 737 and they're noisy[5]. MCAS doesn't quite override the pilot either, and you can see that on the black box graphs from the Lion Air flight[2]. Twenty times the pilot flying entered opposite trim to counteract MCAS and twenty times MCAS paused for five seconds. On the twenty-first time something changed and the plane crashed. It's not quite that simple and that's why there are active investigations. 1: https://4.bp.blogspot.com/-ZsL1asAUBRU/W-pSdWs2dYI/AAAAAAAAFBg/RlymmfQCny8tUpNqZW6deaS-6GXWPL5KQCLcBGAs/s1600/Screen%2BShot%2B2018-11-12%2Bat%2B8.26.08%2BPM.png https://4.bp.blogspot.com/-ZsL1asAUBRU/W-pSdWs2dYI/AAAAAAAAF... 2: https://reports.aviation-safety.net/2018/20181029-0_B38M_PK-LQP_PRELIMINARY.pdf https://reports.aviation-safety.net/2018/20181029-0_B38M_PK-... 3: https://theaircurrent.com/wp-content/uploads/2018/10/jt43-jt610-lionair-mx-log-annotated.jpeg https://theaircurrent.com/wp-content/uploads/2018/10/jt43-jt... 4: https://medium.com/@jpaulreed/the-737max-and-why-software-engineers-should-pay-attention-a041290994bd https://medium.com/@jpaulreed/the-737max-and-why-software-en... 5: https://www.youtube.com/watch?v=3pPRuFHR1co https://www.youtube.com/watch?v=3pPRuFHR1co
  - 8y ago
- m463 8y ago
  serious question: which sensor has failed? you need at least three sensors. (although the "sensors disagree" option you can purchase is better than nothing)
im_down_w_otp 8y ago
For a system with the responsibilities and impact on the safety-case that MCAS has, not even a 1oo2D design would be sufficient, since there's no clear way to determine which input is correct or incorrect if they diverge, but worse still is that MCAS doesn't seem to leverage even that basic, yet still insufficient, safety design from what I've been able to find on it. At the end of the day the fail-safe is to throw control back to the human. And, then of course pin all the liability onto the human operator when things go wrong. This was the problem I had with the Lion Air conclusion. Sure, it may be the case that a pilot can potentially override this system with situational awareness and training. However, the pilot didn't create this band-aid pile in the first place.
- tzs 8y ago
  > For a system with the responsibilities and impact on the safety-case that MCAS has, not even a 1oo2D design would be sufficient, since there's no clear way to determine which input is correct or incorrect if they diverge Wouldn't that depend on how often MCAS is actually needed, and how it handles divergent sensors? My understanding as a layman is that there are situations where a 737MAX is more likely to stall than a "regular" 737 would, but I haven't seen anything on how often those occur. Are they expected to routinely occur, so a fully functioning MCAS is important for safe operation of the plane? Or are they supposed to be rare, one in several million events, and the MCAS is there to make handling those events the same as they would be handled in a "regular" 737, to minimize the need for retraining? If the later, then I'd expect two sensors would be fine, or even just one sensor IF there was a reliable way to tell if it had failed, as long as the MCAS response to a sensor problem is to disable itself and let the pilots know they have to stay out of those "more likely to stall" situations for the rest of the flight.
  - salawat 8y ago
    The problem comes in where the MCAS use should be infrequent, unless the sensor is borked. In computing, we have a principle stated Garbage in, Garbage out. Circuits and electronics don't think. They compute. There is no error checking except that which is specifically designed and implemented into the system. If you're getting data from a biased sensor with +/- 10 degrees AoA, a 10 degree actual AoA (well within the safe operating envelope) suddenly appears to MCAS as a 20 degree AoA (oh shit territory). The system therefore engages doing exactly what it was designed to do. Thus is the crux of the matter. The pilot was flying safely while his AoA sensor was telling a safety system he knew nothing about that he was flying dangerously. The AoA system on earlier models of aircraft that the MAX was based on were a functional luxury/situational awareness aid. The flyability was not impacted by a horked sensor. That changed once they had to add a software driven mechanism to keep the flight characteristics similar enough with the old airframe to be able to release the aircraft to 737 trained pilots, and not have to worry about retraining. Maintenance and pilot alike both needed to be aware that the AoA sensor became a safety critical component due to a failure or miscalibration jeopardizing the controllability of the airframe. If they had gone through a full recert of this airframe, and not an expedited self-certify/grandfathering, these tragedies would have had a much smaller likelihood of occurring due to the increased scrutiny. Props on Brazil for doing their own footwork, and not blindly trusting the FAA's delegation to the manufacturer. Personally, as a software engineer and quality assurance specialist, if I'd seen anything remotely like this come across my desk, I'd be raising hell, even if it meant yanking someone into a VP/C's office and giving them a dressing down for skimping on a cross-cutting safety critical concern, deadlines be damned.
blattimwind 8y ago
AIUI the entire FCC only uses one set of sensors and which set of sensors is used is switched together with the FCC?
cameldrv 8y ago
We will see. Part of the avionics design is that there is a “Brick Wall” between the two sets of sensors, air data computers, and flight control computers. If the FCC uses both for MCAS, that safety concept doesn’t exist anymore. That is certainly the reason they didn’t compare the AoA sensors in the MCAS logic. I think they could probably use pitot/static as a cross check on AoA, but it has some tricky edge cases.
clavoie 8y ago
So, a modern aircraft probably have a few thousands sensors (if not way, way more), not strictly redundant but overlapping in some fashion -- I have to believe there's been some modelling trying to check that sensor input is consistent; if only to have some kind of way for the computer to figure out that a sensor has gone crazy (say your compass suddenly detects a significant change in heading, but no sensor has detected even a degree of roll, a change in speed, heading computed by GPS over time is stable... compass might be borked, whaddya say?) Why isn't such modelling used to let sensors be disabled when they're clearly sending bogus input?
- rocqua 8y ago
  My guess is that such systems are complex, hard to model for safety assurances, and hard to reason about in the moment. Yes, it will probably make the system safer, but your guarantees become a lot more fuzzy. This is not the field for fuzzy guarantees.
- de_watcher 8y ago
  > Why isn't such modelling used to let sensors be disabled when they're clearly sending bogus input? You need to crash a lot of planes to train that NN.
- m463 8y ago
  there should be at least 3 AOA sensors - so you can not only detect a mismatch, but figure out which sensor is most probably incorrect.
cmurf 8y ago
Reports say it alternates each flight between them; if that's correct, in the case of one good sensor and one bad, that you wouldn't see back to back flights with anomalous behavior related to a bad sensor. Otherwise, perhaps the problem isn't with the sensor itself, but with some part of how sensor information is communicated to the interpretive computer.

hn_throwaway_99 8y ago

Yes, was just about to post that exact section before I saw your comment. I've been following this story fairly closely, but this was news to me. I've seen other discussions that were along the lines of "We're not 100% sure why the planes crashed, so now that they've been grounded without a clear root-cause understanding it may take a long time before they are certified to fly again because we don't know exactly what to fix." I call BS on that line of thinking. At the very least it seems like an easy decision to require redundancy in the AoA vane before letting the planes fly again.

neuronic 8y ago

Might as well make the trim wheel or flap control optional packages as well at that point. Except they can't because all pilots know about those since day 1 of their career as opposed to the MCAS sensor-software system designed to deal with airframe engineering issues.

subcosmos 8y ago

Rate 1/5 stars. App is crippled without in-flight upgrade. SMH

Animats 8y ago

I was amazed at that. Boeing used to be known for overdesign for safety. The B-747 had four redundant hydraulic systems. Here's a 787 doing aerobatics at the Farnborough air show to show that it can operate way outside the normal passenger aircraft flight envelope.[1] Boeing used to be an engineering-first company. HQ was at Boeing's own airport near Seattle. Then they got new management and moved corporate HQ to Chicago. [1] https://www.youtube.com/watch?v=vzr313wSY_Y https://www.youtube.com/watch?v=vzr313wSY_Y

tyingq 8y ago
Similar video with a test pilot pushing a 707 pretty hard. Includes a barrel roll, which isn't hard on the aircraft, but unusual to see with a passenger aircraft. https://youtu.be/Ra_khhzuFlE https://youtu.be/Ra_khhzuFlE The 707 was pretty similar to the 737...same main fuselage dimensions, similar pax capacity, etc.
- deleted 8y ago
  [deleted]
- nickgrosvenor 8y ago
  Love the 4k footage.
  - tyingq 8y ago
    Ah, well, it is from 1955.
- BuildTheRobots 8y ago
  Tex Johnston, the test pilot on that flight, did that off his own back and left president of Boeing fuming as it really wasn't part of the plan. I believe they fired him for it - though then hired him back after 24 hours and the realisation that it was a stunning piece of advertising. https://www.seattletimes.com/seattle-news/60-years-ago-the-famous-boeing-707-barrel-roll-over-lake-washington/ https://www.seattletimes.com/seattle-news/60-years-ago-the-f...
8y ago

tntn 8y ago

How is this different from extra safety features available as option packages for automobiles?

simion314 8y ago
If my airbag is missing my car won't crash into traffic by itself, if the airplane system fails the airplane points you into the ground.
- therealx 8y ago
  No, but, say you don't pay for the automatic breaking system. You may rear end someone that the car might have otherwise saved you from.
  - acqq 8y ago
    But here it was: pay to know if the automatic braking system will speed up the car for himself every time you first press the brake and then stop pressing it as you see that it's seems to be better: https://www.nytimes.com/interactive/2019/03/13/world/boeing-737-crash-investigation.html https://www.nytimes.com/interactive/2019/03/13/world/boeing-... Look at the charts above. Both pilots "fought" with the controls, correcting the plane course, then believing "it's corrected now" and then the plane sunk again and again.
  - gulikoza 8y ago
    Again, yes, you may rear end someone, but this is not the correct analogy here. In this situation, the car rear ends someone while you are fully breaking because you didn't pay for the automatic breaking upgrade. This is a fail.
  - function_seven 8y ago
    The difference here is that MCAS is an automatic feature that Boeing needed to mimic the existing 737 flight characteristics. They Created the need for the automatic system in the first place, then sell the safety redundancy as an option. To extend (torture?) the automotive analogy, it would be as if the manufacturer substituted a new braking system that required preheating the rotors, installed software to automatically ride the brakes for the first 2 miles of driving in order to get that preheat done, then sold an optional safety feature to verify that the brakes were at the proper temperature. The old way worked just fine without need for “emulation”. The new airframe required some software to mimic the old one, but they decided to charge you for the add-on to make sure that their band-aid worked reliably.

Scoundreller 8y ago

Could it be an ETOPS requirement?

sitkack 8y ago

Not quite as egregious, but Amazon will sell you an Amazon Store Credit Card and then continually pressure you into buying the account protection. Giving away the liability and then up selling the cure is immoral.

quickthrower2 8y ago
Wait what? Aren’t there consumer credit laws that offer the protection you need?
- ficklepickle 8y ago
  It's probably insurance that pays your outstanding balance if you are badly injured or get a terminal disease or something. It's usually nearly impossible to claim and therefore a scam.

bitminer 8y ago

What concerns me is the evidence of regulatory capture. Boeing was telling the FAA what the approval schedule was going to be. And the FAA sucked it up. Instead of saying "our review capacity is x pages per day. You will receive our reviews on such-and-such a date." (And yes I am oversimplifying.)

bgorman 8y ago
Boeing has spent more money lobbying the US federal government over the last 20 years than any other company. Last year they had the government kill Bombardier's attempt to break the Airbus/Boeing duopoly. Boeing also prevented the US military from buying an Airbus tanker a few years back. Boeing has definitely captured the US government.
- spenczar5 8y ago
  > Boeing has spent more money lobbying the US federal government over the last 20 years than any other company. This is simply false. Boeing has spent a lot, but less than General Electric, or the National Association of Realtors, or many (many!) healthcare and pharmaceutical companies. Source: https://www.opensecrets.org/lobby/top.php?indexType=s https://www.opensecrets.org/lobby/top.php?indexType=s
  - bgorman 8y ago
    While your source seems to suggest that my statement is false, a closer looks shows that the only corporation (I.e. not an industry special interest group like the national association of realtors) that spent more money than Boeing was General Electric. So I would say your statement is even more misleading than mine. General Electric has fallen off a cliff since 1998. I suppose my "20" years statement is false according to "OpenSecrets", but if you change my statement to "15" years, it is probably correct, since General Electric did most of their lobbying closer to 1998 than 2018.
    - spenczar5 8y ago
      > I would say your statement is even more misleading than mine. Fair enough, I guess. I think of groups like Blue Cross/Blue Shield or the NAR as representing corporations, but I can see how you'd disagree on a strict reading. I didn't mean to mislead. > General Electric has fallen off a cliff since 1998. In 1998, GE spent $7.28m on lobbying. Nearly every year since then, they have spent more than that, generally staying north of $16m per year, with the exception of the last three years. Your statement is true only for the last three years. Same source: https://www.opensecrets.org/lobby/clientsum.php?id=D000000125&year=2018 https://www.opensecrets.org/lobby/clientsum.php?id=D00000012... for GE, https://www.opensecrets.org/lobby/clientsum.php?id=D000000100&year=2018 https://www.opensecrets.org/lobby/clientsum.php?id=D00000010... for Boeing.

michaelfeathers 8y ago

I'd go further. Use of software to make a plane airworthy when it isn't natively airworthy is a problem. Civilian aircraft should have a different standard than fighter aircraft: one that allows graceful degradation under failure.

clavoie 8y ago
Being a total layman in terms of aircraft design -- I've always wondered why they weren't built more akin to distributed systems. Instead of 1-3 large fuel tanks, have hundreds of small ones, so losing one is not a disaster. Similar with hydraulics and electronic wiring -- instead of a single cable running along wings, why not a redundant network that can survive everything short of a wing being sheared off?
- BillinghamJ 8y ago
  The things you've mentioned are likely to have significant downsides specifically in terms of weight and complexity
  - inetsee 8y ago
    The A-10 Close Air Support aircraft manages to survive in exceptionally stressful circumstances. The flight control systems are triply redundant: two independent hydraulic systems, and a set of cables if both hydraulic systems fail. Having a second angle of attack sensor on the Boeing 737 MAX (which is critical to aircraft safety) be an extra cost option is unconscionable.
- kenneth 8y ago
  That's pretty much the case. Fuel tanks are split into multiple smaller tanks in the wings, and most systems have one or multiple redundancy (multiple hydraulics systems, multiple flight computers, multiple black boxes, graceful failure protocols, etc.)
- Aloha 8y ago
  How do you fill hundreds of small fuel tanks?

phkahler 8y ago

One would hope MCAS would not do anything if the AoA input was known bad. This looks really bad for the company. I'd be asking for FMEAs and/or FTAs on this, what the conclusions were, and how they intended to mitigate this exact failure - because it's got to be in there.

zaphar 8y ago
If you've only got 2 sensors and you aren't even reading from one of them then "bad" input is hard to know. You can do nothing if they both disagree but you can't actually know which sensor is bad and which is good without a tie breaker.
- Gibbon1 8y ago
  Who knows but it the chatter makes it sound like the system doesn't have redundant sensors. You have two systems with one sensor each. One system is active and the other is inactive. And no voting logic.
- ddalex 8y ago
  You don't need to automatically fix the disagreement. You just need to stop taking nose-down action and inform the human in charge . Problem is, if you need to inform the human, you also need to train the human, which leads to extra costs.

cameldrv 8y ago

(I originally wrote what my Brother in law posted on Twitter) I believe that the package is for Cat III hand flown landings. I think that to do that, you need AoA on the HUD, and in the event of an AoA failure you have to go around. It is strange for sure that you need AoA consistency checks for that, but not for MCAS.

ip26 8y ago

Read somewhere a few days ago that Southwest, operator of maybe the most 737's in the world, didn't like the MCAS on the MAX and that extra "package" was essentially customizations/modifications created specifically to make them happy.

Gibbon1 8y ago

I'll also point this out "Unlike the EFS system, MCAS can make huge nose down trim changes." That's a potential reason why an unaware pilot wouldn't disengage the auto trim system. Because he wouldn't think the trim system as being capable of causing an extreme pitch down. Also this response. > Interesting. So how did Boeing get a plane without redundancy in a critical sub system approved? Answered own question, by hiding it in what is normally a non-critical subsystem.

Alex3917 8y ago

> What I find downright criminal is this Not much different than how with most car companies, the only way to get headlights that don't blind oncoming cars is to pay to upgrade to leather seats. C.f.: https://www.iihs.org/iihs/news/desktopnews/night-vision-headlights-improve-but-base-models-leave-drivers-in-the-dark https://www.iihs.org/iihs/news/desktopnews/night-vision-head...

Reason077 8y ago

> "The fact that the redundancy of a sensor on which a system capable of sudden, large control inputs relies is an optional package to be purchased separately... I simply have no words." As far as I'm aware, all Boeing 737 aircraft have 2 AoA sensors. There isn't an option to add a 3rd redundant one like the Twitter poster suggests. From what I understand, what is optional is the display of the AoA on the main screen and/or HUD, and the "AOA DISAGREE" warning which cross-checks the data from the 2 sensors. (Airbus A320s have 3 AoA sensors. A380, A350, and Bombardier C-Series, aka A220, have 4.)

gaius_baltar 8y ago
I got this in the other way: there is no actual redundancy even with the optional package, the new sensor is only used to compare inputs with the one used by the control system and warn the pilots if they disagree. Pretty sure I'm misinterpreting things (I hope!). In the other hand, if there are two sensors, what the control system does when they disagree and there is no hardware to show this to the pilots?
- Reason077 8y ago
  Right. The optional packages don't add additional sensors and don't add redundancy. But they may improve situational awareness by drawing pilot's attention to a sensor problem faster. The MCAS system, apparently, always takes its input from one of the two AoA sensors - it alternates each flight. Lots of details here: http://www.b737.org.uk/mcas.htm http://www.b737.org.uk/mcas.htm

m3kw9 8y ago

Stuff like this is usually triple redundant

gok 8y ago

Safety as an option is pretty standard in aviation, unfortunately. For example, collision avoidance systems have been in development since the 1950s, but TCAS remains only partially required to this day.

WalterBright 8y ago

> quite spot on > Nowhere in here is there a software problem. I worked on the design of the 757 stab trim system, am trained in aerospace engineering, and am a programmer. I'm not a pilot. I am not explicitly familiar with the 737 stab trim system. It is indeed at least partially a software problem. I had thought of two possible improvements: one is to limit the authority of the MCAS's commands to the trim, and the other is to not issue further commands if the pilot fights the trim commands. I read today that Boeing's proposed software fix includes both of those.

sgustard 8y ago
I have to agree. If you're saying "the software is working to spec, but the specs are bad," then you're still describing a software problem.
- speedplane 8y ago
  > If you're saying "the software is working to spec, but the specs are bad," then you're still describing a software problem. This is a pretty bold statement. The spec includes many non-software related features.
  - cashsterling 8y ago
    In aerospace, lack of proper requirements in the flight software would be a system engineering problem... a semantic difference really, but important nonetheless. System engineering and validation has a metric ton of 'tools' to catch bad system design and improper algorithm implementation... I really question what the hell happened here: - Did they just pencil whip the FMEA on this or what? (failure mode and effects analysis) - What happened with the Hardware-in-the-loop flight simulation when they tested the scenario where the AoA sensor givesg spurious data, both high and low (but especially high)? I mean... they did test this, right?
JorgeGT 8y ago
In that sense, I agree with you. I meant to say that this wasn't a software bug or programmer's fault like "oh, someone wrote max_authority = 6º instead of 0.5º in line 8745679", but something bigger: the very design and specs of the system and a certification process that didn't challenge those aspects.

neurotech1 8y ago

Airbus A320 used to have GPWS (Ground Proximity Warning System) as an optional extra. A budget airline didn't order it, and quite possibly encouraged pilots to make excessively fast approaches, and led to a CFIT (Controlled Flight Into Terrain) crash of a perfectly good jet.[0] Another crash related to the 737 MAX crash, is Scandinavian 751 [1]. On a MD-81 airliner, with "Automatic Thrust Restoration", A system the pilots were not trained on, caused the thrust to be increased, when the pilot tried to throttle back the damaged engines and clear the compressor stall. "Pilot Betrayed" was the Air Crash Investigations episode. [0] https://en.wikipedia.org/wiki/Air_Inter_Flight_148 https://en.wikipedia.org/wiki/Air_Inter_Flight_148 [1] https://en.wikipedia.org/wiki/Scandinavian_Airlines_Flight_751 https://en.wikipedia.org/wiki/Scandinavian_Airlines_Flight_7... [2] https://www.imdb.com/title/tt1862850/ https://www.imdb.com/title/tt1862850/

jacurtis 8y ago

For anyone who is old enough to remember when airbags first came onto the scene in cars, they were sold as optional packages. Most car manufacturers called these "safety packages". These optional packages were sold basically in the same way as you jokingly mentioned above. "Buy this optional package and when you hit a tree at 50mph, this will keep you from flying through the windshield and will probably save your life." I actually remember a conversation that my parents had with eachother when I was young about whether it was "worth it" to buy airbags in their new car. Yes, that was a REAL conversation that Millions of people had and a decision they had to make for nearly 25 years before airbags became standard in cars in the late 90s. Lots of people felt that the optional saftey package weren't worth the money and many people died through the '80s and '90s from car crashes that they would have survived had they splurged for the "safety package" when they purchased their car from the dealer. It turns out that when people are buying cars, they aren't planning on crashing them. So spending extra money that only helps them when they crash felt unnecessary. These packages could cost in the neighborhood of $2,500 extra. Which on a $20,000 car is over 10% extra! Would you pay 10% extra for a safety package if cars today didn't come with standard safety equipment? This is what led to the USA making airbags mandatory in 1998, forcing this safety feature on manufacturers in order to save lives. I imagine the 737 AoA sensors were sold as the same thing. Boeing sold this as a "additional redundancy package" or "safety package" and probably included many other features as part of a larger package. Just like thrifty car buyers of the 70s, 80s, and 90s who figured they didn't need airbags in their cars and opted out of the safety package. The budget airlines of Lion Air and Ethiopian Airlines had a good history with the normal 737 planes and figured this additional safety package wasn't worth the money.

heavenlyblue 8y ago
You do understand that a single light in an aircraft doesn’t cost 10% of it’s total budget? That’s like saying that car manufacturers decided to add a light saying their ABS is disengaged depending on whether you paid for it. That’s more like it.

cashsterling 8y ago

I, too, have no words to communicate my astonishment. A phrase comes to mind: "To screw up this bad you need an MBA." Which is not really intended as a dig on MBA's... but rather to indicate that this debacle has the feeling of business decisions overruling sound engineering design. This seems poor system engineering on multiple levels: - the whole airframe / engine thrust composite flight characteristic requiring an agressive AoA limiting safety... on a commercial airliner? what? - The default MCAS relying on a single sensor, which seems to be occasionally quite unreliable, without sensor truth'ing based on other measurements (i.e. dividing ground speed by rate of altitude change to estimate AoA, then also figuring for control surface position feedback. - The MCAS not having proper situational behavior: i.e. "I'm only 500 feet of the deck... maybe this not a great time to try to tip the nose down too hard... you know, in case that stupid sensor is wrong. Oh... and the pilot is also pulling back on the yoke trying to correct something I'm doing. yeah... my bad, MCAS will chill right now and assume the pilot is NOT trying to fly this thing like a stunt plane 500 feet off the deck".

chipsa 8y ago
Angle of Attack cannot be inferred from ground speed and vertical speed. It's quite possible to have a massively negative vertical speed (10000 feet per minute descent), and have an Angle of Attack greater than 45 degrees. Angle of Attack is strictly the angle of the airflow compared to the aircraft.
- cashsterling 8y ago
  We could argue about this (technically, you are not wrong) but the point is, the aircraft MCAS should be able to do this robustly with available sensor data.* Without going into a lengthy analysis... you can infer AoA from groundspeed, rate of climb/descent, pitch/yaw/roll, and position feedback of control surfaces. Recent information on these data points can be used to calculate present and near future flight dynamics. The fields of Kalman filtering, adaptive control, and optimal control as applied to aerospace engineering are decades old. There are many examples where adaptive flight controls can fly an aircraft that has inherently unstable flight dynamics... In fact most modern jet fighter require computer controlled flight controls to maintain stable flight. Applying these algorithms to an airliner for sensor truth'ing is a cake walk. * I think you actually can infer situational AoA from ground speed and altitude data: An example you could build a regression model of standard take-off scenarios and using recent groundspeed, altitude change, and the rate of change of (groundspeed/altitude). You could build this off of flight test data and simulation. In practice, there may be no point to doing this because a wealth of additional flight sensor information is available.

engire 8y ago

Can I first correct you on two major errors in your statement. TWO AOA vanes are NOT an option on any aircraft. Every aircraft have two vanes as standard. the option is for a small AOA position indicator icon on the top corner on the main screen for the captains side. also there is no AOA disagree light. This is an amber AOA disagree text that appears on the screen. I can only say that on my airline this is standard as it was on the 737 NG in our fleet.