15 ms·

How the pilots of Lion Air Flight 610 lost control

105 points by skilled 8y ago

gumby 8y ago
It doesn't feel like the NYT is trying to communicate something to their reader. Or if they are I can't figure out what the message is. Apart from the whizzbang combined with tiny paragraphs (making it hard to read or comprehend), not all the text is available to Safari's reader mode.
AceyMan 8y ago
The page is graphics loaded page with floating paragraphs over a single "sheet of wallpaper" showing interior and exterior diagrams of the aircraft. I can see it not working worth a damn in any kind of Reader View. FWIW, nothing new here, but it's a good 3rd grade overview for the layperson. And I am a Boeing guy, through and through, but they screwed the pooch on this one. RIP, Lion Air 610. /Acey (me: FAA licensed dispatcher)
- gumby 8y ago
  I can't see how you can learn anything from it not in a reader view either. oh, and if you need any assistive technology (e.g. perhaps you're blind): tough luck!
  - tapland 8y ago
    If you didn't read about it after the accident this seems like a pretty easy to understand explanation. Why couldn't anyone learn from it? It explains the issue pretty well in my opinion
    - gumby 8y ago
      Because it was little dribs of info you had to scroll between, all with a distracting backdrop.
- sargun 8y ago
  Can you point us to a better article?
  - LeoPanthera 8y ago
    The Wikipedia article is quite good. https://en.wikipedia.org/wiki/Lion_Air_Flight_610 https://en.wikipedia.org/wiki/Lion_Air_Flight_610
  - trevyn 8y ago
    Here’s the actual safety board report, including flight data recorder graphs: http://knkt.dephub.go.id/knkt/ntsc_aviation/baru/pre/2018/2018%20-%20035%20-%20PK-LQP%20Preliminary%20Report.pdf http://knkt.dephub.go.id/knkt/ntsc_aviation/baru/pre/2018/20... Looks like there was a problem with the autopilot trim, likely due to a faulty sensor, and the pilot just needed to switch off the autopilot instead of fighting it — the previous flight of that aircraft experienced a similar issue, and they just turned the autopilot trim off and carried on normally. :( The graphs on PDF page 23 are particularly morbid, and you’ll note that two of the final auto nose-down commands are of longer duration than all the rest.
    - trevyn 8y ago
      Ah, this is quite good as well: https://leehamnews.com/2018/11/14/boeings-automatic-trim-for-the-737-max-was-not-disclosed-to-the-pilots/ https://leehamnews.com/2018/11/14/boeings-automatic-trim-for... Apparently it wasn’t the standard autopilot trim, but a special safety program specific to the 737 MAX. Which still is just a switch to disable, if you know that switch is there, or that there is a checklist for dealing with runaway trim, or that you’re aware that auto-trim commands are the reason your plane keeps nosing down...
      - caf 8y ago
        I think the most important point described in that article is that the MCAS trim, unlike the other kinds of elevator trim, isn't defeated by limit switches on the yoke - and that this critical information was not in the training or flight manual.
- Havoc 8y ago
  Yeah the layout is quite annoying
WatchDog 8y ago
You would think it would make sense to automatically disable the MCAS system and sound a warning if the aircraft detects conflicting readings from it's sensors.
- ams6110 8y ago
  Yes, especially since there are only 2 AOA sensors so you can't know which one is incorrect by comparing to a third sensor. I've read that it's possible to derive AOA from other sensors but don't think these aircraft have that ability.
  - heyjudy 8y ago
    Serious lack of redundancy in a life-safety critical-system. It should have: - 5+ AOA sensors of different types - discard any sensor values that are stuck or providing incorrect readings - have a spray, heating system (some do) and camera system for automatically deicing and verifying AOA sensors in-flight Without a camera system to verify sensors, crews won't have a definitive way to know if they're getting true or false information from clogged pitot, stuck AOA or malfunction.
    - deleted 8y ago
      [deleted]
  - kwhitefoot 8y ago
    You should always have at least three and preferably an odd number of similar sensors so that you can get at least a vague idea of consensus. The British Royal Navy rule was: take one chronometer or three, never two because with two you will always be unsure which one is correct while with three you can, usually, just average the two that are closest together and regard the third as an outlier. See also https://blog.ipspace.net/2017/01/never-take-two-chronometers-to-sea.html https://blog.ipspace.net/2017/01/never-take-two-chronometers...
- akira2501 8y ago
  Due to the way the engines are mounted on the frame, which enhances their efficiency, they also cause more of a "upward thrust vector." This makes it very easy for this plane to reach dangerously high AOA in certain scenarios, particularly during turns. The MCAS has a specific and important function and just turning it off is probably not going to increase safety. The real problem was the Boeing did not disclose the existence of this device and it's functions in aircraft training, according to one source, because they did not want to inundate new pilots with too much information about the plane and it's attendant safety systems. Perhaps, had the pilots known, they would have seen the stick shaker/stall warning system activating on _one side only_ as a serious indication of an Airspeed/AOA system fault and the potential for incorrect MCAS outputs being generated. They might have known to disable the electronic trim control, bypassing the MCAS, and then to manually fly and trim the plane with the aforementioned thrust vectoring taken into consideration. They could have trained for this. That would have all given them the best safety margin for survival here.
heyjudy 8y ago
https://outline.com/pELX5F https://outline.com/pELX5F
iamhamm 8y ago
These human-machine interface failures always fascinate me. The NYT makes the quip about not being able to look down and note the trim, but speed doesn’t need to be high to have these failures. It seems to be more related to system complexity misunderstandings or compounding interpretation issues. Look at the grounding of the Royal Majesty - not exactly some high speed object; it all amounted to a misunderstood icon. See: https://ti.arc.nasa.gov/m/profile/adegani/Grounding%20of%20the%20Royal%20Majesty.pdf https://ti.arc.nasa.gov/m/profile/adegani/Grounding%20of%20t...
jumelles 8y ago
I'm amazed that Boeing isn't the focus of more - much, much more - criticism. It seems clear to me they are vastly more culpable than the pilots or airline.
- pcurve 8y ago
  I agree that Boeing's share of responsibility is big. I think media coverage has largely reflected it too. I don't recall reading any article where it put the blame on the pilots. Based on what I've read, it felt more like, 70% Boeing, 29% airline, 1% pilot.
  - privateSFacct 8y ago
    If this is the case we seriously need to consider getting rid of the pilots. Stab trim issues are a memory checklist item under existing training with clear cutoffs easily available. 1% pilot fault is ridiculous.
    - SomeHacker44 8y ago
      "Getting rid of the pilots" has already mostly happened. You know this. Autopilot on at 400-1000' and off at MDA/200'/DH to rollout. In between they are system administrators. Heck, even I can and often do fly my much smaller planes this way. And, as you also probably know, autopilot is required to be flying the plane above 28,000' (Google RVSM for interested readers). I am sometimes glad I have access to Cubs, Decathlons, and other simple planes to keep me an honest pilot. :)
      - privateSFacct 8y ago
        So true, and overseas actually a bigger issue I think because they don't always have the recreational / GA flying world going on. Some pilots in airlines trying to develop local skills are going basically straight onto an Airbus flight deck. Not saying it's bad, but they don't seem as comfortable with manual flying. And frankly manual flying is more complicated on these planes - the number of autopilot and throttle modes etc etc - a cessna - pitch, power... I do wonder if adding a basic safe flight pitch power button (ignores everything and just goes to a safe pitch power setting smack in middle of power curve) might help. Literally ignore AoA, ignore speed, ignore everything. Just trim to middle and set power. Air france? This flight? These planes are flying fine otherwise. You don't 100% need all the automation (though the MAX did introduce some weird instability near stall which is annoying in a big way). My own view - make a naturally stable plane (MAX moved away from this), and have a way to fall back to a basic setting that takes advantage of that.
Animats 8y ago
There have been a surprising number of air disasters in recent years caused primarily by air data sensors returning false data. - Birgenair Flight 301 - B757. Pitot tube clogged possibly by insect nest, false overspeed indication, autopilot commanded pitch-up, alarms, stall warning, crew confused about speed, loss of control. 189 dead. - Air France Flight 447 - Airbus A330. Well known. Pitot tube clogged by ice, confusion about airspeed, loss of control. 228 dead. - Saratov Flight 703 - An-198. Pitot tube frozen. Three airspeed indicators all disagreed. Loss of control. 71 dead. - Lion Air - Angle of attack vane failure, from parent article. That information could be checked against GPS, and at least one aircraft does this. But that has its own problems.[1] Checking against an inertial system is another possibility. Those are complicated, though. The classic airspeed, altimeter, and angle of attack vane are so simple. [1] https://www.gpsworld.com/gps-disruption-a-full-fledged-aviation-problem/ https://www.gpsworld.com/gps-disruption-a-full-fledged-aviat...
- gugagore 8y ago
  Do aircraft inertial measurement systems ever provide linear velocity information that isn't trash? I had assumed not (except for special cases where you can do e.g. a zero-velocity update) An inertial measurement system cannot measure velocity. It's insensitive to which inertial frame it is in. (to move from a non moving frame to a moving frame, you must accelerate, which it could detect. but then you have to integrate these accelerations over time)
  - SomeHacker44 8y ago
    Your assertion is simply demonstrated to be untrue. Inertial systems integrate all measured accelerations over time to compute present velocity and position. They are periodically corrected with external references. The accuracy is limited to the accuracy of the accelerometers, the sampling frequency and the accuracy of the digital math, among other things. This IMU idea could be used, but I am not aware of any current commercial IMUs in airliners - but I am not an airline pilot so that does not count for much. If I were given conflicting information I would see how the plane is acting. I have flown planes with various primary instrument failures over my decades, some in actual instrument conditions, and always had sufficient secondary sources of data to determine the situation. As an example in a piston plane: Tachometer fails? Cross check fuel flow, manifold pressure and airspeed - subject to mixture control which can be reasonably set using "lean to roughness and enrich" method. Been there, done that. I have had an altimeter fail insidiously slowly, when water got into the static lines (despite redundant ports and properly routing them upward), while in actual near minimums in a non precision approach. (I used GPS altitude cross reference and odd seeming behavior of altimeter needle to diagnose and switch to alternate static.) Importantly, if the plane is flying straight and level and sufficiently fast, over time this becomes a "feel" and you can tell if something is extraordinary. Maybe not on airliners, but definitely the piston planes I fly a lot and probably the small jets I fly on occasion. This is why some of these things are mystifying to me. If your engines are set to a certain power and seem to be operating properly, and your attitude is just so, in the absence of unusual meteorological conditions your plane should be performing in a manner consistent to your last experience. If your stall warning is going off, well, maybe the instrument broke. Air France is really scary mystifying. Look, I am a software engineer so I get it, people can get locked into a misconception about a fault and not step away long/far enough to consider other alternatives. It happens a lot in debugging. But for whatever reason my brain works much faster and more dynamically when shit happens in the left seat of an airplane having problems.
reactor 8y ago
From the article, "Outside the plane, one of the plane’s angle of attack sensors falsely indicated that the plane’s nose was pointed too high, and the aircraft could stall." I'm not in any way eligible to comment on aviation systems, but why that equipment needs to measure the angle of airplane be outside? Couldn't something like gyroscope mounted inside do the job?
- txcwpalpha 8y ago
  Angle of Attack is based on the angle between the plane and the surrounding airflow, not based on gravity/landscape/etc.
  - deleted 8y ago
    [deleted]
- deleted 8y ago
  [deleted]
- JshWright 8y ago
  It's measuring the angle of the plane relative to the air moving past it (i.e. the angle the plane is "attacking" the air). Knowing the angle of the plane in an absolute sense isn't as useful when it comes to detecting stalls, etc.
  - reactor 8y ago
    Thanks, make sense.
  - kwhitefoot 8y ago
    It's not as useful but it is far from useless. The ordinary artificial horizon indicator should have been able to show that the external sensor was far out. I don't know anything about the aircraft or Boeing but as a software designer and implementor I would have wanted to use the output of the artificial horizon as well as the angle of attack sensor in an attempt to gauge the quality of the readings. I'm guessing here but it seems likely the aircraft was equipped with a normal gyroscopic instrument in addition to the external angle of attack sensor. Of course the Boeing engineers who worked on this could easily tell us exactly how it works and why. I don't suppose that will ever happen though.
1stranger 8y ago
Would it be possible to have a single switch that disables all automatic systems and puts the plane in complete manual control as much as possible? Can these planes even operate in a complete manual mode?
- ReptileMan 8y ago
  Fly by wire. So no. Though Boing I think have law of control at - you better know what and why are doing buddy - level.
- drpixie 8y ago
  Certainly possible but not required. In the B737 family, most of the main controls are essentially power assisted direct controls. Boeing consistently has preferred more-or-less manual controls, while Airbus took the other direction (in which the pilot guides the computer, which operates the controls). All airlines can fly happily in completely "manual" mode. Boeings are doing this most of the time, Airbus's only do this when the situation is outside the computer's envelope.
Havoc 8y ago
Both inconsistent readings and a plane actively working against the pilots will. That seems like a pretty massive failing on the manufacturers part.
- privateSFacct 8y ago
  Except that runaway trim is a MEMORY checklist item. Literally - they should have the procedure here memorized, and there are LOTS of cutout options, from temp right at fingertips to cutoff switches.
  - Havoc 8y ago
    The fact that the FAA issued an emergency order with instructions after this crash suggests to me this wasn't sufficiently covered by existing procedure. In fact: > The FAA's directive orders airlines within three days to update flight manuals to include specific steps pilots should take to recover.
- StreamBright 8y ago
  Human technology has it limits, reliability is one of them. The more complex systems we produce the less reliable they become. The correct course of action is not just blame the manufacturer but have a run book for every scenario of failure that makes sure the best outcome. System operation people are aware of this as much as emergency unit workers and pilots too. In this situation the pilots failed to follow the standard protocol to disengage the auto-pilot and safely navigate the plane like the previous flight team did.
WalterBright 8y ago
What the pilots experienced was indistinguishable from runaway stab trim, and shutting off the stab trim from the switches on the console is the correct response. There's a loud distinctive sound when the trim runs, and the wheels that bracket the console turn, so it's pretty obvious. The previous flight's pilots had indeed done this. The pilots are trained for this. They has 12 minutes to shut off the repeated action of the runaway stab trim. Additionally, the airplane should have been grounded after the previous flight, as runaway stab trim is a serious problem, until the fault was found and corrected. Equally badly, the flight crew was probably not informed of what had happened on the previous flight.
- lsh123 8y ago
  Exactly. While Boeing indeed added a new system, the malfunctions and resolutions matrix didn’t change. Thus Boeing didn’t need to change training and claimed that new planes can be flown without any additional training. It’s hard to fault Boeing here.
  - dotancohen 8y ago
    Are you saying that pilots can transition from a 737 to a 737MAX with no sim time? I thought that they had different instrumentation, at the very least.
- cameldrv 8y ago
  Well sort of but not quite. When the pilots clicked the trim up switch on the yoke, it disabled the MCAS system for 5 seconds -- i.e. it made the problem go away temporarily. Then MCAS comes right back with more nose down trim. This is not a typical runaway trim situation where it's just continuously rolling in more trim. The pilots had no idea that this system existed or that a "runaway trim" failure could have these characteristics. Sure, it's easy to say from the ground with what we know now that all they had to do was flip a couple of switches, and that a previous crew managed to land safely. However, the job of an airplane is not to be safe only with quick thinking, above average pilots. If a single sensor failure can present a situation that 99% of pilots will successfully diagnose and recover from, you're looking at multiple crashes per year.
privateSFacct 8y ago
What is never mentioned is that since forever runaway stabilizer is one of the FEW memory items. Here it is. To be qualified as a pilot you have to have this memorized. I. Runaway Stabilizer CONTROL COLUMN - HOLD FIRMLY AUTOPILOT (if engaged) - DISENGAGE Do not re-engage the autopilot. If the Runaway Continues STAB TRIM CUTOUT SWITCHES (both) - CUTOUT All this drama around "fighting the controls" and "fighting the plane" is weird. This is not some procedure you need to lookup, this is one of a few memory items.
- ratsimihah 8y ago
  That would make sense. Do you think they lacked experience or received improper training? From what I remember when preparing for the helicopter license, it was emphasized that being able to use manual sensors and controls was critical in case of such failures.
  - privateSFacct 8y ago
    A couple of factors played into this. Terrible maintenance (well known issue with this airline). Seriously, a plane in US is HIGHLY unlikely to fly this many times with this type of issue. So in the US, risk of this even happening is MUCH lower. We don't hear about the situations pilots respond properly too. Despite the other person correcting me, most pilots do know this stuff in US. Go arounds, single engine landings, stab trim cutout etc all happen with some frequency. Runaway stabilizer has a cutout switch and wheel for a reason, the manual control you describe. It's actually a memory item - memory or recall items are used for time critical situations with risks of severe aircraft damage. You do the memory items and then look at checklist. There are not that many. My own view here - despite what others are saying, if pilots had cut out the electric trim, they would have been fine. 99% of time flights go per the plan and are boring, so pilots do sometimes take shortcuts. They don't brief the flight properly, they don't maintain a sterile cockpit, they don't look over past issues on a plane before takeoff (maintenance log book) and they don't follow procedures exactly. Planes are so safe a lot of pilots lean HEAVILY on the automation. So I'd be very curious about cockpit voice recorder. It seems almost impossible to go 12 minutes without doing the 20 second memory item. An example - properly using an elevator - you hit the button, when doors open, you enter. If doors closing you hit the button outside again or hit the door hold button inside and then enter or WAIT if doors don't open. The sensors in the door to avoid crushing you are meant as a backup. BUT, folks are so confident in those backup sensors they skip the procedure, and actually stick their hand into a closing door on purpose to keep it open. It works 99% of time I'm sure - but there will be that .01% where it doesn't. That's like flying. 99.999% it actually works if you take a shortcut (ie, in many cases you can cut out trim with control column input). But that IS NOT the actual procedure. So you see pilots say things like, "I've always done it this way", and yeah, it works 99.9% of the time BUT the designers intended you to follow the damn procedure - in which case you'll almost never have a problem. Another problem, you remember what you do. So if you are used to cutting out stab trim with control column as a shortcut and not switches, then yes, you could be in trouble here, especially if you never fall back to the actual procedure. There was a change here which mean more cases in which stab trim would be flight commanded and some shortcuts wouldn't work on it.
cameldrv 8y ago
What the plane did here is not technically reasonable. We had optimal estimation and sensor fusion in the sixties. With two AoA sensors, three pitot/static systems, GPS, and a full IMU, the plane had more than enough information to determine that there was no high AoA situation requiring its intervention.calling for the computer's intervention. Since this dangerous high AoA situation is so rare, even a simple rule requiring both AoA sensors to agree that AoA was dangerously high to override the pilot completely solves the problem. Even with just one AoA sensor and a little memory, the simple fact that the system had rolled in so much nose down trim, which should have lowered AoA, with no apparent effect should have clued it into the fact that it’s model of the system was wrong and caused it to stop. This is not a particularly sophisticated insight. The engineers all knew this when they designined the system. They had a reason for doing it the way they did, though, and that was to slip the new system in a bit under the radar of the FAA. Since the MCAS is required to maintain the stability of the plane at high angle of attack, it should have been certified as a Stability Augmentation System. That would have subjected it to more redundancy requirements and eliminated this failure. The problem was that Boeing wanted the MAX to not require significant training or certification beyond the 737/737NG. A big new Stability Augmentation System would have required extra certification and probably pilot training. Instead, they chose to sort of launder this system through an existing one, the Elevator Feel Shift system. The EFS adds some nose down trim at certain speeds and altitudes to make the 737NG feel like a classic 737. Since the FAA already determined that the EFS wasn't a stability augmentation system, but it could control the elevator trim, Boeing figured they could piggyback the new MCAS onto it, adding just one new input, the AoA sensor, and no new outputs. Since it only controlled the trim and not the main flight controls, Boeing could keep to its manual control philosophy, and they could slip it through certification. They couldn’t give the computer both AoA inputs, because the air data system is supposed to have two totally independent and manually selectable sets of sensors. If you give a computer both AoA inputs, you lose that redundancy concept. So why did the EFS system never cause any problems despite having the same lack of redundancy? My guess is that the EFS is essentially an open-loop controller. It applies a fixed amount of forward trim for a given speed/altitude combination. If the pitot/static system goes haywire, the worst that happens is fixed, moderate amount of nose down trim, easily and naturally compensated for by the pilot or autopilot. The MCAS appears to be closed loop in that it will just keep adding more and more nose down trim until the AoA sensor says things are OK again, the pilot pulls its circuit breaker, or the plane smashes into the ocean. People in the industry blaming the pilots one bit are making a mistake the industry collectively stopped making seventy years ago. We don’t blame the pilot anymore, we blame the system. Given enough time, humans will make any mistake that can be made. If the plane cannot be flown completely safely by significantly below average pilots, it's an unsafe plane. Demanding that the system be safe even with imperfect pilots is why commerical aviation is so amazingly safe today. It’s also ironically why the MCAS is in there in the first place. Boeing could have just put in the manual to never exceed 14 degrees of AoA, and even average pilots would never have a problem. The FAA would never certify such a plane to carry paying passengers with such a limitation though. It would be too dangerous. Eventually someone would screw up and the plane would go out of control, so there had to be a computer to prevent this. As it stands, the plane does not IMO meet certification requirements for the transport category, and the only reason it isn't grounded is because there are literally half a trillion dollars worth of these things in service or on order. I'm racking my brain to think of another single product produced anywhere that is so valuable.
NelsonMinar 8y ago
All these years of listening to Boeing bigots talk about how bad Airbus planes are because they are too automated, flown by machines, unsafe.. And then a Boeing automated system seems to a significant cause of a fatal accident. Not sure what if any conclusion to draw from that, just the context.
hotswapster 8y ago
It does my head in how a subscription is required to read these articles. Is there a way to read them without subscribing? I’m not from the US and I’m not interesting in paying for the NY Times. I am interested in what the HN community has to share however
deleted 8y ago
[deleted]
BXLE_1-1-BitIs1 8y ago
There was an erroneous stick shaker that captured the crew's attention and diverted attention from the trim stuffing the nose down. The NYT graphics show a miniscule part of the problem. Most likely the FDR data has been played through a full motion simulator - it would be a hairy ride, but the crews have to keep it confidential until the Indonesian authorities release the data.

105 points by skilled 8y ago

It doesn't feel like the NYT is trying to communicate something to their reader. Or if they are I can't figure out what the message is. Apart from the whizzbang combined with tiny paragraphs (making it hard to read or comprehend), not all the text is available to Safari's reader mode.

AceyMan 8y ago

The page is graphics loaded page with floating paragraphs over a single "sheet of wallpaper" showing interior and exterior diagrams of the aircraft. I can see it not working worth a damn in any kind of Reader View. FWIW, nothing new here, but it's a good 3rd grade overview for the layperson. And I am a Boeing guy, through and through, but they screwed the pooch on this one. RIP, Lion Air 610. /Acey (me: FAA licensed dispatcher)

gumby 8y ago
I can't see how you can learn anything from it not in a reader view either. oh, and if you need any assistive technology (e.g. perhaps you're blind): tough luck!
- tapland 8y ago
  If you didn't read about it after the accident this seems like a pretty easy to understand explanation. Why couldn't anyone learn from it? It explains the issue pretty well in my opinion
  - gumby 8y ago
    Because it was little dribs of info you had to scroll between, all with a distracting backdrop.
sargun 8y ago
Can you point us to a better article?
- LeoPanthera 8y ago
  The Wikipedia article is quite good. https://en.wikipedia.org/wiki/Lion_Air_Flight_610 https://en.wikipedia.org/wiki/Lion_Air_Flight_610
- trevyn 8y ago
  Here’s the actual safety board report, including flight data recorder graphs: http://knkt.dephub.go.id/knkt/ntsc_aviation/baru/pre/2018/2018%20-%20035%20-%20PK-LQP%20Preliminary%20Report.pdf http://knkt.dephub.go.id/knkt/ntsc_aviation/baru/pre/2018/20... Looks like there was a problem with the autopilot trim, likely due to a faulty sensor, and the pilot just needed to switch off the autopilot instead of fighting it — the previous flight of that aircraft experienced a similar issue, and they just turned the autopilot trim off and carried on normally. :( The graphs on PDF page 23 are particularly morbid, and you’ll note that two of the final auto nose-down commands are of longer duration than all the rest.
  - trevyn 8y ago
    Ah, this is quite good as well: https://leehamnews.com/2018/11/14/boeings-automatic-trim-for-the-737-max-was-not-disclosed-to-the-pilots/ https://leehamnews.com/2018/11/14/boeings-automatic-trim-for... Apparently it wasn’t the standard autopilot trim, but a special safety program specific to the 737 MAX. Which still is just a switch to disable, if you know that switch is there, or that there is a checklist for dealing with runaway trim, or that you’re aware that auto-trim commands are the reason your plane keeps nosing down...
    - caf 8y ago
      I think the most important point described in that article is that the MCAS trim, unlike the other kinds of elevator trim, isn't defeated by limit switches on the yoke - and that this critical information was not in the training or flight manual.
Havoc 8y ago
Yeah the layout is quite annoying

WatchDog 8y ago

You would think it would make sense to automatically disable the MCAS system and sound a warning if the aircraft detects conflicting readings from it's sensors.

ams6110 8y ago
Yes, especially since there are only 2 AOA sensors so you can't know which one is incorrect by comparing to a third sensor. I've read that it's possible to derive AOA from other sensors but don't think these aircraft have that ability.
- heyjudy 8y ago
  Serious lack of redundancy in a life-safety critical-system. It should have: - 5+ AOA sensors of different types - discard any sensor values that are stuck or providing incorrect readings - have a spray, heating system (some do) and camera system for automatically deicing and verifying AOA sensors in-flight Without a camera system to verify sensors, crews won't have a definitive way to know if they're getting true or false information from clogged pitot, stuck AOA or malfunction.
  - deleted 8y ago
    [deleted]
- kwhitefoot 8y ago
  You should always have at least three and preferably an odd number of similar sensors so that you can get at least a vague idea of consensus. The British Royal Navy rule was: take one chronometer or three, never two because with two you will always be unsure which one is correct while with three you can, usually, just average the two that are closest together and regard the third as an outlier. See also https://blog.ipspace.net/2017/01/never-take-two-chronometers-to-sea.html https://blog.ipspace.net/2017/01/never-take-two-chronometers...
akira2501 8y ago
Due to the way the engines are mounted on the frame, which enhances their efficiency, they also cause more of a "upward thrust vector." This makes it very easy for this plane to reach dangerously high AOA in certain scenarios, particularly during turns. The MCAS has a specific and important function and just turning it off is probably not going to increase safety. The real problem was the Boeing did not disclose the existence of this device and it's functions in aircraft training, according to one source, because they did not want to inundate new pilots with too much information about the plane and it's attendant safety systems. Perhaps, had the pilots known, they would have seen the stick shaker/stall warning system activating on _one side only_ as a serious indication of an Airspeed/AOA system fault and the potential for incorrect MCAS outputs being generated. They might have known to disable the electronic trim control, bypassing the MCAS, and then to manually fly and trim the plane with the aforementioned thrust vectoring taken into consideration. They could have trained for this. That would have all given them the best safety margin for survival here.

heyjudy 8y ago

https://outline.com/pELX5F https://outline.com/pELX5F

iamhamm 8y ago

These human-machine interface failures always fascinate me. The NYT makes the quip about not being able to look down and note the trim, but speed doesn’t need to be high to have these failures. It seems to be more related to system complexity misunderstandings or compounding interpretation issues. Look at the grounding of the Royal Majesty - not exactly some high speed object; it all amounted to a misunderstood icon. See: https://ti.arc.nasa.gov/m/profile/adegani/Grounding%20of%20the%20Royal%20Majesty.pdf https://ti.arc.nasa.gov/m/profile/adegani/Grounding%20of%20t...

jumelles 8y ago

I'm amazed that Boeing isn't the focus of more - much, much more - criticism. It seems clear to me they are vastly more culpable than the pilots or airline.

pcurve 8y ago
I agree that Boeing's share of responsibility is big. I think media coverage has largely reflected it too. I don't recall reading any article where it put the blame on the pilots. Based on what I've read, it felt more like, 70% Boeing, 29% airline, 1% pilot.
- privateSFacct 8y ago
  If this is the case we seriously need to consider getting rid of the pilots. Stab trim issues are a memory checklist item under existing training with clear cutoffs easily available. 1% pilot fault is ridiculous.
  - SomeHacker44 8y ago
    "Getting rid of the pilots" has already mostly happened. You know this. Autopilot on at 400-1000' and off at MDA/200'/DH to rollout. In between they are system administrators. Heck, even I can and often do fly my much smaller planes this way. And, as you also probably know, autopilot is required to be flying the plane above 28,000' (Google RVSM for interested readers). I am sometimes glad I have access to Cubs, Decathlons, and other simple planes to keep me an honest pilot. :)
    - privateSFacct 8y ago
      So true, and overseas actually a bigger issue I think because they don't always have the recreational / GA flying world going on. Some pilots in airlines trying to develop local skills are going basically straight onto an Airbus flight deck. Not saying it's bad, but they don't seem as comfortable with manual flying. And frankly manual flying is more complicated on these planes - the number of autopilot and throttle modes etc etc - a cessna - pitch, power... I do wonder if adding a basic safe flight pitch power button (ignores everything and just goes to a safe pitch power setting smack in middle of power curve) might help. Literally ignore AoA, ignore speed, ignore everything. Just trim to middle and set power. Air france? This flight? These planes are flying fine otherwise. You don't 100% need all the automation (though the MAX did introduce some weird instability near stall which is annoying in a big way). My own view - make a naturally stable plane (MAX moved away from this), and have a way to fall back to a basic setting that takes advantage of that.

Animats 8y ago

There have been a surprising number of air disasters in recent years caused primarily by air data sensors returning false data. - Birgenair Flight 301 - B757. Pitot tube clogged possibly by insect nest, false overspeed indication, autopilot commanded pitch-up, alarms, stall warning, crew confused about speed, loss of control. 189 dead. - Air France Flight 447 - Airbus A330. Well known. Pitot tube clogged by ice, confusion about airspeed, loss of control. 228 dead. - Saratov Flight 703 - An-198. Pitot tube frozen. Three airspeed indicators all disagreed. Loss of control. 71 dead. - Lion Air - Angle of attack vane failure, from parent article. That information could be checked against GPS, and at least one aircraft does this. But that has its own problems.[1] Checking against an inertial system is another possibility. Those are complicated, though. The classic airspeed, altimeter, and angle of attack vane are so simple. [1] https://www.gpsworld.com/gps-disruption-a-full-fledged-aviation-problem/ https://www.gpsworld.com/gps-disruption-a-full-fledged-aviat...

gugagore 8y ago
Do aircraft inertial measurement systems ever provide linear velocity information that isn't trash? I had assumed not (except for special cases where you can do e.g. a zero-velocity update) An inertial measurement system cannot measure velocity. It's insensitive to which inertial frame it is in. (to move from a non moving frame to a moving frame, you must accelerate, which it could detect. but then you have to integrate these accelerations over time)
- SomeHacker44 8y ago
  Your assertion is simply demonstrated to be untrue. Inertial systems integrate all measured accelerations over time to compute present velocity and position. They are periodically corrected with external references. The accuracy is limited to the accuracy of the accelerometers, the sampling frequency and the accuracy of the digital math, among other things. This IMU idea could be used, but I am not aware of any current commercial IMUs in airliners - but I am not an airline pilot so that does not count for much. If I were given conflicting information I would see how the plane is acting. I have flown planes with various primary instrument failures over my decades, some in actual instrument conditions, and always had sufficient secondary sources of data to determine the situation. As an example in a piston plane: Tachometer fails? Cross check fuel flow, manifold pressure and airspeed - subject to mixture control which can be reasonably set using "lean to roughness and enrich" method. Been there, done that. I have had an altimeter fail insidiously slowly, when water got into the static lines (despite redundant ports and properly routing them upward), while in actual near minimums in a non precision approach. (I used GPS altitude cross reference and odd seeming behavior of altimeter needle to diagnose and switch to alternate static.) Importantly, if the plane is flying straight and level and sufficiently fast, over time this becomes a "feel" and you can tell if something is extraordinary. Maybe not on airliners, but definitely the piston planes I fly a lot and probably the small jets I fly on occasion. This is why some of these things are mystifying to me. If your engines are set to a certain power and seem to be operating properly, and your attitude is just so, in the absence of unusual meteorological conditions your plane should be performing in a manner consistent to your last experience. If your stall warning is going off, well, maybe the instrument broke. Air France is really scary mystifying. Look, I am a software engineer so I get it, people can get locked into a misconception about a fault and not step away long/far enough to consider other alternatives. It happens a lot in debugging. But for whatever reason my brain works much faster and more dynamically when shit happens in the left seat of an airplane having problems.

reactor 8y ago

From the article, "Outside the plane, one of the plane’s angle of attack sensors falsely indicated that the plane’s nose was pointed too high, and the aircraft could stall." I'm not in any way eligible to comment on aviation systems, but why that equipment needs to measure the angle of airplane be outside? Couldn't something like gyroscope mounted inside do the job?

txcwpalpha 8y ago
Angle of Attack is based on the angle between the plane and the surrounding airflow, not based on gravity/landscape/etc.
- deleted 8y ago
  [deleted]
deleted 8y ago
[deleted]
JshWright 8y ago
It's measuring the angle of the plane relative to the air moving past it (i.e. the angle the plane is "attacking" the air). Knowing the angle of the plane in an absolute sense isn't as useful when it comes to detecting stalls, etc.
- reactor 8y ago
  Thanks, make sense.
- kwhitefoot 8y ago
  It's not as useful but it is far from useless. The ordinary artificial horizon indicator should have been able to show that the external sensor was far out. I don't know anything about the aircraft or Boeing but as a software designer and implementor I would have wanted to use the output of the artificial horizon as well as the angle of attack sensor in an attempt to gauge the quality of the readings. I'm guessing here but it seems likely the aircraft was equipped with a normal gyroscopic instrument in addition to the external angle of attack sensor. Of course the Boeing engineers who worked on this could easily tell us exactly how it works and why. I don't suppose that will ever happen though.

1stranger 8y ago

Would it be possible to have a single switch that disables all automatic systems and puts the plane in complete manual control as much as possible? Can these planes even operate in a complete manual mode?

ReptileMan 8y ago
Fly by wire. So no. Though Boing I think have law of control at - you better know what and why are doing buddy - level.
drpixie 8y ago
Certainly possible but not required. In the B737 family, most of the main controls are essentially power assisted direct controls. Boeing consistently has preferred more-or-less manual controls, while Airbus took the other direction (in which the pilot guides the computer, which operates the controls). All airlines can fly happily in completely "manual" mode. Boeings are doing this most of the time, Airbus's only do this when the situation is outside the computer's envelope.

Havoc 8y ago

Both inconsistent readings and a plane actively working against the pilots will. That seems like a pretty massive failing on the manufacturers part.

privateSFacct 8y ago
Except that runaway trim is a MEMORY checklist item. Literally - they should have the procedure here memorized, and there are LOTS of cutout options, from temp right at fingertips to cutoff switches.
- Havoc 8y ago
  The fact that the FAA issued an emergency order with instructions after this crash suggests to me this wasn't sufficiently covered by existing procedure. In fact: > The FAA's directive orders airlines within three days to update flight manuals to include specific steps pilots should take to recover.
StreamBright 8y ago
Human technology has it limits, reliability is one of them. The more complex systems we produce the less reliable they become. The correct course of action is not just blame the manufacturer but have a run book for every scenario of failure that makes sure the best outcome. System operation people are aware of this as much as emergency unit workers and pilots too. In this situation the pilots failed to follow the standard protocol to disengage the auto-pilot and safely navigate the plane like the previous flight team did.

WalterBright 8y ago

What the pilots experienced was indistinguishable from runaway stab trim, and shutting off the stab trim from the switches on the console is the correct response. There's a loud distinctive sound when the trim runs, and the wheels that bracket the console turn, so it's pretty obvious. The previous flight's pilots had indeed done this. The pilots are trained for this. They has 12 minutes to shut off the repeated action of the runaway stab trim. Additionally, the airplane should have been grounded after the previous flight, as runaway stab trim is a serious problem, until the fault was found and corrected. Equally badly, the flight crew was probably not informed of what had happened on the previous flight.

lsh123 8y ago
Exactly. While Boeing indeed added a new system, the malfunctions and resolutions matrix didn’t change. Thus Boeing didn’t need to change training and claimed that new planes can be flown without any additional training. It’s hard to fault Boeing here.
- dotancohen 8y ago
  Are you saying that pilots can transition from a 737 to a 737MAX with no sim time? I thought that they had different instrumentation, at the very least.
cameldrv 8y ago
Well sort of but not quite. When the pilots clicked the trim up switch on the yoke, it disabled the MCAS system for 5 seconds -- i.e. it made the problem go away temporarily. Then MCAS comes right back with more nose down trim. This is not a typical runaway trim situation where it's just continuously rolling in more trim. The pilots had no idea that this system existed or that a "runaway trim" failure could have these characteristics. Sure, it's easy to say from the ground with what we know now that all they had to do was flip a couple of switches, and that a previous crew managed to land safely. However, the job of an airplane is not to be safe only with quick thinking, above average pilots. If a single sensor failure can present a situation that 99% of pilots will successfully diagnose and recover from, you're looking at multiple crashes per year.

privateSFacct 8y ago

What is never mentioned is that since forever runaway stabilizer is one of the FEW memory items. Here it is. To be qualified as a pilot you have to have this memorized. I. Runaway Stabilizer CONTROL COLUMN - HOLD FIRMLY AUTOPILOT (if engaged) - DISENGAGE Do not re-engage the autopilot. If the Runaway Continues STAB TRIM CUTOUT SWITCHES (both) - CUTOUT All this drama around "fighting the controls" and "fighting the plane" is weird. This is not some procedure you need to lookup, this is one of a few memory items.

ratsimihah 8y ago
That would make sense. Do you think they lacked experience or received improper training? From what I remember when preparing for the helicopter license, it was emphasized that being able to use manual sensors and controls was critical in case of such failures.
- privateSFacct 8y ago
  A couple of factors played into this. Terrible maintenance (well known issue with this airline). Seriously, a plane in US is HIGHLY unlikely to fly this many times with this type of issue. So in the US, risk of this even happening is MUCH lower. We don't hear about the situations pilots respond properly too. Despite the other person correcting me, most pilots do know this stuff in US. Go arounds, single engine landings, stab trim cutout etc all happen with some frequency. Runaway stabilizer has a cutout switch and wheel for a reason, the manual control you describe. It's actually a memory item - memory or recall items are used for time critical situations with risks of severe aircraft damage. You do the memory items and then look at checklist. There are not that many. My own view here - despite what others are saying, if pilots had cut out the electric trim, they would have been fine. 99% of time flights go per the plan and are boring, so pilots do sometimes take shortcuts. They don't brief the flight properly, they don't maintain a sterile cockpit, they don't look over past issues on a plane before takeoff (maintenance log book) and they don't follow procedures exactly. Planes are so safe a lot of pilots lean HEAVILY on the automation. So I'd be very curious about cockpit voice recorder. It seems almost impossible to go 12 minutes without doing the 20 second memory item. An example - properly using an elevator - you hit the button, when doors open, you enter. If doors closing you hit the button outside again or hit the door hold button inside and then enter or WAIT if doors don't open. The sensors in the door to avoid crushing you are meant as a backup. BUT, folks are so confident in those backup sensors they skip the procedure, and actually stick their hand into a closing door on purpose to keep it open. It works 99% of time I'm sure - but there will be that .01% where it doesn't. That's like flying. 99.999% it actually works if you take a shortcut (ie, in many cases you can cut out trim with control column input). But that IS NOT the actual procedure. So you see pilots say things like, "I've always done it this way", and yeah, it works 99.9% of the time BUT the designers intended you to follow the damn procedure - in which case you'll almost never have a problem. Another problem, you remember what you do. So if you are used to cutting out stab trim with control column as a shortcut and not switches, then yes, you could be in trouble here, especially if you never fall back to the actual procedure. There was a change here which mean more cases in which stab trim would be flight commanded and some shortcuts wouldn't work on it.

cameldrv 8y ago

What the plane did here is not technically reasonable. We had optimal estimation and sensor fusion in the sixties. With two AoA sensors, three pitot/static systems, GPS, and a full IMU, the plane had more than enough information to determine that there was no high AoA situation requiring its intervention.calling for the computer's intervention. Since this dangerous high AoA situation is so rare, even a simple rule requiring both AoA sensors to agree that AoA was dangerously high to override the pilot completely solves the problem. Even with just one AoA sensor and a little memory, the simple fact that the system had rolled in so much nose down trim, which should have lowered AoA, with no apparent effect should have clued it into the fact that it’s model of the system was wrong and caused it to stop. This is not a particularly sophisticated insight. The engineers all knew this when they designined the system. They had a reason for doing it the way they did, though, and that was to slip the new system in a bit under the radar of the FAA. Since the MCAS is required to maintain the stability of the plane at high angle of attack, it should have been certified as a Stability Augmentation System. That would have subjected it to more redundancy requirements and eliminated this failure. The problem was that Boeing wanted the MAX to not require significant training or certification beyond the 737/737NG. A big new Stability Augmentation System would have required extra certification and probably pilot training. Instead, they chose to sort of launder this system through an existing one, the Elevator Feel Shift system. The EFS adds some nose down trim at certain speeds and altitudes to make the 737NG feel like a classic 737. Since the FAA already determined that the EFS wasn't a stability augmentation system, but it could control the elevator trim, Boeing figured they could piggyback the new MCAS onto it, adding just one new input, the AoA sensor, and no new outputs. Since it only controlled the trim and not the main flight controls, Boeing could keep to its manual control philosophy, and they could slip it through certification. They couldn’t give the computer both AoA inputs, because the air data system is supposed to have two totally independent and manually selectable sets of sensors. If you give a computer both AoA inputs, you lose that redundancy concept. So why did the EFS system never cause any problems despite having the same lack of redundancy? My guess is that the EFS is essentially an open-loop controller. It applies a fixed amount of forward trim for a given speed/altitude combination. If the pitot/static system goes haywire, the worst that happens is fixed, moderate amount of nose down trim, easily and naturally compensated for by the pilot or autopilot. The MCAS appears to be closed loop in that it will just keep adding more and more nose down trim until the AoA sensor says things are OK again, the pilot pulls its circuit breaker, or the plane smashes into the ocean. People in the industry blaming the pilots one bit are making a mistake the industry collectively stopped making seventy years ago. We don’t blame the pilot anymore, we blame the system. Given enough time, humans will make any mistake that can be made. If the plane cannot be flown completely safely by significantly below average pilots, it's an unsafe plane. Demanding that the system be safe even with imperfect pilots is why commerical aviation is so amazingly safe today. It’s also ironically why the MCAS is in there in the first place. Boeing could have just put in the manual to never exceed 14 degrees of AoA, and even average pilots would never have a problem. The FAA would never certify such a plane to carry paying passengers with such a limitation though. It would be too dangerous. Eventually someone would screw up and the plane would go out of control, so there had to be a computer to prevent this. As it stands, the plane does not IMO meet certification requirements for the transport category, and the only reason it isn't grounded is because there are literally half a trillion dollars worth of these things in service or on order. I'm racking my brain to think of another single product produced anywhere that is so valuable.

NelsonMinar 8y ago

All these years of listening to Boeing bigots talk about how bad Airbus planes are because they are too automated, flown by machines, unsafe.. And then a Boeing automated system seems to a significant cause of a fatal accident. Not sure what if any conclusion to draw from that, just the context.

hotswapster 8y ago

It does my head in how a subscription is required to read these articles. Is there a way to read them without subscribing? I’m not from the US and I’m not interesting in paying for the NY Times. I am interested in what the HN community has to share however

deleted 8y ago

[deleted]

BXLE_1-1-BitIs1 8y ago

There was an erroneous stick shaker that captured the crew's attention and diverted attention from the trim stuffing the nose down. The NYT graphics show a miniscule part of the problem. Most likely the FDR data has been played through a full motion simulator - it would be a hairy ride, but the crews have to keep it confidential until the Indonesian authorities release the data.