7 ms·

This appears to answer the question of why, given that there is a "memory item" for dealing with runaway trim that would have helped, it wasn't followed by the

This appears to answer the question of why, given that there is a "memory item" for dealing with runaway trim that would have helped, it wasn't followed by the crew: > “They didn’t seem to know the trim was moving down,” the third source said. “They thought only about airspeed and altitude. That was the only thing they talked about.” If this is true, not only did the pilots not realize the trim was the cause of the dive and try to fix it, it seems that they didn't even realize the trim wheel was moving. "Runaway trim" would be something they trained for, but has a presentation so vastly different (of a visibly spinning trim wheel causing quick descent) as to not be recognizably the same emergency to these pilots. Of course, this crew had no way of knowing that there was a system on the plane that could slowly make erroneous trim changes over a long period with autopilot off, and it makes sense that they might miss it happening while focusing on their stall alerts and speeds and trying to come up with an explanation and also trying to retain control.

RidingPegasus 8y ago
Was the trim wheel not visibly spinning out of their control? Simply because the symptoms are different should mean that stopping it via taking manual control is a recurring option.
- cjbprime 8y ago
  The spin was not visible to these pilots presumably because it was happening in small increments with a delay between them.
  - acqq 8y ago
    Exactly. It was not "runaway" in the sense of "continuously doing the same." Something was happening, from the point of view of the pilot and co-pilot "at random." (Not to mention that on the previous models exactly such pattern was effectively impossible to happen). The faulty non-redundant sensor was producing false reading and that was the input to the to the pilots completely unknown MCAS which then moved the nose down based on the difference between the input and MCAS' expected target, turning "the correction" on only from time to time. The presence of the third, off-duty pilot on the flight before additionally explains even better how in that case the crash was avoided. While the pilot and co-pilot were at the controls, the third one, looking from behind, didn't have the direct duties to do, so he had enough time to both just observe everything that was going on and the less immediate stress to allow him to come to the solution that worked. Like when one programmer spends half an hour on something "not working" and complains to the colleague who just takes a look and immediately sees what the first one hasn't for half an hour. Additionally the movement of the trim wheel is actually easier to follow from the place in the back (see https://news.ycombinator.com/item?id=19440045 https://news.ycombinator.com/item?id=19440045 ). And there's also a possibility that the false sensor reading and MCAS together produced a slightly different differential signal which resulted in more obvious manifestation of the problem. Possibly both elements contributed to the lucky outcome in that previous flight. In the old times of civil aviation there was the third person on duty in the cockpit, the "flight-engineer." As the planes got more computerized controls, they started to be certified to fly with two-pilot crews. The computerized systems, when working, do reduce the chance of the crash and overall number of accidents did decrease simply because the humans were on the average less often in control, and therefore there was less chance to do anything wrong. The problem is when the unreported computerized controls secretly depend on the single sensor that can be faulty. And when they actively confuse the pilot and ruin the flight instead of helping him.
    - kayfox 8y ago
      "Runaway" in the sense of "its moving in a undesired and unexpected direction and continues to do so despite attempts to stop it".
      - jeremysmyth 8y ago
        "Runaway" is a term of art here. Redefining it for this conversation is unhelpful.
  - ahoka 8y ago
    If anyone wonders this is how it looks like: https://youtu.be/-X01uY0bJs0?t=48 https://youtu.be/-X01uY0bJs0?t=48
    - TheSpiceIsLife 8y ago
      And sounds? The trim wheels moving also makes a sound?
      - deleted 8y ago
        [deleted]
      - ekianjo 8y ago
        From the videos I have seen in cockpit it does seem to make a sound as it rolls.
      - WalterBright 8y ago
        Yes. The ratcheting sound is deliberate, so the pilots don't need to see the wheels spin to know the trim is running. Also note that there's a white stripe on the wheel so it's easy to see it is running.
        jacobush 8y ago
        Someone else in the thread said newer models don't make a sound.
      - michaelt 8y ago
        You'll note in that video, the left-hand pilot has a headset on, presumably the right-hand pilot does too. Headsets are seen in other videos [1] of the cockpit. So even if the spinning trim wheels make noise, they may not have heard it. (My google image search shows some pilots wearing two-ear headsets, some wearing one-ear headsets, some wearing two-ear headsets with side not on their ear, and some not wearing headsets. But some of those are pilots posing for photos, and presumably pilots don't let people visit the cockpit during critical stages of the flight) [1] https://youtu.be/wz2Q3COfveg?t=409 https://youtu.be/wz2Q3COfveg?t=409
- frenchy 8y ago
  I doubt it would have been "out of control", it would have just been slow changes over time as trim increased. The pilots might not have been fighting it much either, if they thought that airspeed was the problem, and not pitch.
  - Someone1234 8y ago
    The graph on page 14 of this report[0] is pretty telling about how slow the changes were... Note Trim Manual, Trim Automatic, and Pitch Trim Position. [0] https://reports.aviation-safety.net/2018/20181029-0_B38M_PK-LQP_PRELIMINARY.pdf https://reports.aviation-safety.net/2018/20181029-0_B38M_PK-...
    - acqq 8y ago
      Thanks! Page 23 of the PDF or 14 as printed at the bottom. Sadly it's hard to compare with the previous flight (pg. 25 or 16) as for it they published the graph of the whole flight instead of only the part until the (at the time still secret) MCAS was turned off (or probably more correctly, the motor controlled by the MCAS, which actually steered the "trim").
- crooked-v 8y ago
  Part of the problem is that the MCAS system has a 20-second delay after the last time the pilot touched controls for the flaps, and it adjusts itself to a new position and then stops. There are no other indicators that it's happening, and since it's noncontinuous and has a delay, it would be easy to miss the only visual sign of the trim wheel readjusting itself once, especially since the wheel itself is by the pilot's thigh outside of forward line of sight. In a normal runaway trim situation, by comparison, the trim wheel continues obviously spinning as long as the problem is going on, so it only takes a single glance at any time to notice it.
  - oasisbob 8y ago
    I'm not sure that's accurate. Other descriptions of the MCAS system state that it will continually modify the trim on the horizontal stabilizer for up to 10 seconds at a time. http://www.b737.org.uk/mcas.htm http://www.b737.org.uk/mcas.htm
    - inferiorhuman 8y ago
      To summarise; MCAS will trim the Stabilizer down for 10 seconds (2.5 deg nose down) and pause for 5 seconds and repeat if the conditions (high angle of attack, flaps up and autopilot disengaged) continue to be met. Using electric pitch trim will only pause MCAS, to deactivate it you need to switch off the STAB TRIM SUTOUT switches.
    - usrusr 8y ago
      Well, the point is that the experience from MCAS trying to be clever and subtle is very different from what one would expect in an old-school runaway trim situation caused by, say, a stuck relay. Easy to chase down a wrong line of inquiry when the symptoms only appear sporadically and can be temporarily stopped by a whole range of actions that won't permanently fix it.
- dukoid 8y ago
  To my understanding, the trim wheels are moving frequently for small adjustments -- and it's possible to switch the wheel movement off if this becomes too annoying.
  - losvedir 8y ago
    Also not a pilot but I don't think you can switch the wheel off. You can, however, disable the electric motors that spin the wheel, which is what the checklist for runaway trim tells you to do, and what was supposed to have been done here.
    - WalterBright 8y ago
      The spinning wheels are directly connected to the stab trim via cables. If the trim motors fail, the pilots can turn those wheels by hand (there's a folding crank on them) to manually adjust the trim.
  - VBprogrammer 8y ago
    It's running for 10 seconds at a time for 2.5 units of trim out of a possible ~15. That's quite a big trim change. I find it hard to believe that it's not noticed because of the 'boiling a frog' situation. It not possible to disable wheel movement. It's mechanically linked to the jackscrew.
- verytrivial 8y ago
  The Lion Air directive has instructions on page 41 of their preliminary report that include: 5. If the runaway continues after the autopilot is disengaged: STAB TRIM CUTOUT switches (both) ... CUTOUT If the runaway continues: Stabilizer trim wheel .... *Grasp and hold* This may be an overabundance of caution, or to cover other runaway events, but it reads like "you may need to physically fight the computer even when it has been disabled". Not exactly heartening. EDIT: A comment below points out haptic feedback to disengage as a reason to grab-and-hold. https://reports.aviation-safety.net/2018/20181029-0_B38M_PK-LQP_PRELIMINARY.pdf https://reports.aviation-safety.net/2018/20181029-0_B38M_PK-...
  - mlindner 8y ago
    That's incorrect, these have haptic feedback, once you grasp it, it's not going to fight you. It looks for resistance and if it sees the resistance it doesn't continue.
    - verytrivial 8y ago
      Ah, Okay, thanks. What remaining automation would be driving the trim wheel in this case that grab-and-hold would countermand?
    - heavenlyblue 8y ago
      You are still fighting it, aren't you?
    - userbinator 8y ago
      Really? Here's a video of someone doing it, and it certainly looks like it tries to fight you (and the trim motor continues running) but there's a clutch that starts slipping: https://www.youtube.com/watch?v=cQirIH_DuAs https://www.youtube.com/watch?v=cQirIH_DuAs
  - ProAm 8y ago
    > you may need to physically fight the computer This is the future I look forward to. AI/ML/Autopilot/Companies moving fast and breaking things.... Boeing took that one literally.
l33tman 8y ago
This is not true, on the released FDR data, it shows clearly that the pilot flying corrected the MCAS system's trim down adjustments at every cycle (each 10 seconds) for most of the level flight. So at least the PF must have been aware of this. It's a bit weird if they didn't talk about it though, maybe it was obvious to both pilots and they were concentrating on if they were keeping altitude and airspeed at all (the most important factors that would keep them flying).
- mannykannot 8y ago
  This got my attention too. For at least 6 minutes, the crew were coping with the problem - the aircraft was holding altitude and speed, if erratically, and they must have been getting the trim back to approximately where it should have been, or else MCAS would have driven the stabilizer to the point where it overrode the elevator authority. In about the last minute of that recording, the pattern changes: the pilot trim inputs decreased and became much shorter, and the stick forces changed from going up and down to a sustained, high, back-force. The article says that near the end, the captain handed over control to the 1st officer so he could look for answers. I am speculating wildly here, but could that coincide with the change in the pattern of the pilot's responses? Could it be that the 1st officer did not attempt to re-trim, or did not do it completely? Did he try to trim manually, but perhaps could not because of the load on the stabilizer? [1]. IIRC, according to Dominic Gates, it would only take two cycles of MCAS intervention until the stabilizer had reached a point where full nose-up elevator could not keep the airplane level. The chart was published in one of Dominic Gates' Seattle Times articles, but I am not in a position to find it right now... [1] https://news.ycombinator.com/item?id=19400157 https://news.ycombinator.com/item?id=19400157
  - VBprogrammer 8y ago
    Anyone who's ever piloted an aircraft would try to re-trim against pitch forces. It's inconceivable that the pilot for some reason just failed to do that. I'm not sure exactly what force it takes to get full nose up elevator at flying speed but it will be in the range of dozens of kilograms. I personally wonder if there is potentially a thermal cut-off on the trim motor which could have been activated by this trimming and re-triming. Or even an outright failure of some part of the system.
dvdkhlng 8y ago
I'm just wondering: it looks like the absolute trim angle is not visible anywhere in the cockpit? The trim wheel takes multiple revolutions to adjust trim between min/max limits, so looking at the wheel itself, you can only every see changes, not values. Unless you notice the change while it happens, you won't have any indication as to the unusual trim angle? Thinking about that I wonder whether there was even a measurement signal that the computers could rely on to determine the current state of the trim angle in absolute terms. Given how the MCAS did incremental changes without stopping before reaching the mechanical limits, I do imagine that there might have been no hardware feedback to assess the current trim state, thereby making it impossible hardware-wise to limit MCAS to a sensible (and safe) range of angles. [edit] Looking at the lion air crash report [1] p.14 ff it seems like also the black box only recorded trim changes, not absolute trim state. [1] https://reports.aviation-safety.net/2018/20181029-0_B38M_PK-LQP_PRELIMINARY.pdf https://reports.aviation-safety.net/2018/20181029-0_B38M_PK-...
- VBprogrammer 8y ago
  There is a trim position indication. It's to the left and right of the throttles. It's a critical thing to check before starting the takeoff roll. You can see it on the copilot's side here by the trim wheel. http://www.b737.org.uk/images/stabtrim_200.jpg http://www.b737.org.uk/images/stabtrim_200.jpg
  - dvdkhlng 8y ago
    Yes, thinking about that, not having any pilot-visible indication would be quite some point of failure at lift-off. Still wondering whether MCAS and other automatation systems have a representation of that position indication. I can't currently come up with an idea of how to implement that in an easy, yet error-resilient way. Usually you would initially drive into a reference position, then integrate the motor's actions to arive at an absolute value, but integration would potentially add up subsequent errors, and obviously you can't re-do the referencing step while in the air.
    - VBprogrammer 8y ago

RidingPegasus 8y ago

Was the trim wheel not visibly spinning out of their control? Simply because the symptoms are different should mean that stopping it via taking manual control is a recurring option.

cjbprime 8y ago
The spin was not visible to these pilots presumably because it was happening in small increments with a delay between them.
- acqq 8y ago
  Exactly. It was not "runaway" in the sense of "continuously doing the same." Something was happening, from the point of view of the pilot and co-pilot "at random." (Not to mention that on the previous models exactly such pattern was effectively impossible to happen). The faulty non-redundant sensor was producing false reading and that was the input to the to the pilots completely unknown MCAS which then moved the nose down based on the difference between the input and MCAS' expected target, turning "the correction" on only from time to time. The presence of the third, off-duty pilot on the flight before additionally explains even better how in that case the crash was avoided. While the pilot and co-pilot were at the controls, the third one, looking from behind, didn't have the direct duties to do, so he had enough time to both just observe everything that was going on and the less immediate stress to allow him to come to the solution that worked. Like when one programmer spends half an hour on something "not working" and complains to the colleague who just takes a look and immediately sees what the first one hasn't for half an hour. Additionally the movement of the trim wheel is actually easier to follow from the place in the back (see https://news.ycombinator.com/item?id=19440045 https://news.ycombinator.com/item?id=19440045 ). And there's also a possibility that the false sensor reading and MCAS together produced a slightly different differential signal which resulted in more obvious manifestation of the problem. Possibly both elements contributed to the lucky outcome in that previous flight. In the old times of civil aviation there was the third person on duty in the cockpit, the "flight-engineer." As the planes got more computerized controls, they started to be certified to fly with two-pilot crews. The computerized systems, when working, do reduce the chance of the crash and overall number of accidents did decrease simply because the humans were on the average less often in control, and therefore there was less chance to do anything wrong. The problem is when the unreported computerized controls secretly depend on the single sensor that can be faulty. And when they actively confuse the pilot and ruin the flight instead of helping him.
  - kayfox 8y ago
    "Runaway" in the sense of "its moving in a undesired and unexpected direction and continues to do so despite attempts to stop it".
    - jeremysmyth 8y ago
      "Runaway" is a term of art here. Redefining it for this conversation is unhelpful.
- ahoka 8y ago
  If anyone wonders this is how it looks like: https://youtu.be/-X01uY0bJs0?t=48 https://youtu.be/-X01uY0bJs0?t=48
  - TheSpiceIsLife 8y ago
    And sounds? The trim wheels moving also makes a sound?
    - deleted 8y ago
      [deleted]
    - ekianjo 8y ago
      From the videos I have seen in cockpit it does seem to make a sound as it rolls.
    - WalterBright 8y ago
      Yes. The ratcheting sound is deliberate, so the pilots don't need to see the wheels spin to know the trim is running. Also note that there's a white stripe on the wheel so it's easy to see it is running.
      - jacobush 8y ago
        Someone else in the thread said newer models don't make a sound.
    - michaelt 8y ago
      You'll note in that video, the left-hand pilot has a headset on, presumably the right-hand pilot does too. Headsets are seen in other videos [1] of the cockpit. So even if the spinning trim wheels make noise, they may not have heard it. (My google image search shows some pilots wearing two-ear headsets, some wearing one-ear headsets, some wearing two-ear headsets with side not on their ear, and some not wearing headsets. But some of those are pilots posing for photos, and presumably pilots don't let people visit the cockpit during critical stages of the flight) [1] https://youtu.be/wz2Q3COfveg?t=409 https://youtu.be/wz2Q3COfveg?t=409
frenchy 8y ago
I doubt it would have been "out of control", it would have just been slow changes over time as trim increased. The pilots might not have been fighting it much either, if they thought that airspeed was the problem, and not pitch.
- Someone1234 8y ago
  The graph on page 14 of this report[0] is pretty telling about how slow the changes were... Note Trim Manual, Trim Automatic, and Pitch Trim Position. [0] https://reports.aviation-safety.net/2018/20181029-0_B38M_PK-LQP_PRELIMINARY.pdf https://reports.aviation-safety.net/2018/20181029-0_B38M_PK-...
  - acqq 8y ago
    Thanks! Page 23 of the PDF or 14 as printed at the bottom. Sadly it's hard to compare with the previous flight (pg. 25 or 16) as for it they published the graph of the whole flight instead of only the part until the (at the time still secret) MCAS was turned off (or probably more correctly, the motor controlled by the MCAS, which actually steered the "trim").
crooked-v 8y ago
Part of the problem is that the MCAS system has a 20-second delay after the last time the pilot touched controls for the flaps, and it adjusts itself to a new position and then stops. There are no other indicators that it's happening, and since it's noncontinuous and has a delay, it would be easy to miss the only visual sign of the trim wheel readjusting itself once, especially since the wheel itself is by the pilot's thigh outside of forward line of sight. In a normal runaway trim situation, by comparison, the trim wheel continues obviously spinning as long as the problem is going on, so it only takes a single glance at any time to notice it.
- oasisbob 8y ago
  I'm not sure that's accurate. Other descriptions of the MCAS system state that it will continually modify the trim on the horizontal stabilizer for up to 10 seconds at a time. http://www.b737.org.uk/mcas.htm http://www.b737.org.uk/mcas.htm
  - inferiorhuman 8y ago
    To summarise; MCAS will trim the Stabilizer down for 10 seconds (2.5 deg nose down) and pause for 5 seconds and repeat if the conditions (high angle of attack, flaps up and autopilot disengaged) continue to be met. Using electric pitch trim will only pause MCAS, to deactivate it you need to switch off the STAB TRIM SUTOUT switches.
  - usrusr 8y ago
    Well, the point is that the experience from MCAS trying to be clever and subtle is very different from what one would expect in an old-school runaway trim situation caused by, say, a stuck relay. Easy to chase down a wrong line of inquiry when the symptoms only appear sporadically and can be temporarily stopped by a whole range of actions that won't permanently fix it.
dukoid 8y ago
To my understanding, the trim wheels are moving frequently for small adjustments -- and it's possible to switch the wheel movement off if this becomes too annoying.
- losvedir 8y ago
  Also not a pilot but I don't think you can switch the wheel off. You can, however, disable the electric motors that spin the wheel, which is what the checklist for runaway trim tells you to do, and what was supposed to have been done here.
  - WalterBright 8y ago
    The spinning wheels are directly connected to the stab trim via cables. If the trim motors fail, the pilots can turn those wheels by hand (there's a folding crank on them) to manually adjust the trim.
- VBprogrammer 8y ago
  It's running for 10 seconds at a time for 2.5 units of trim out of a possible ~15. That's quite a big trim change. I find it hard to believe that it's not noticed because of the 'boiling a frog' situation. It not possible to disable wheel movement. It's mechanically linked to the jackscrew.
verytrivial 8y ago
The Lion Air directive has instructions on page 41 of their preliminary report that include: 5. If the runaway continues after the autopilot is disengaged: STAB TRIM CUTOUT switches (both) ... CUTOUT If the runaway continues: Stabilizer trim wheel .... *Grasp and hold* This may be an overabundance of caution, or to cover other runaway events, but it reads like "you may need to physically fight the computer even when it has been disabled". Not exactly heartening. EDIT: A comment below points out haptic feedback to disengage as a reason to grab-and-hold. https://reports.aviation-safety.net/2018/20181029-0_B38M_PK-LQP_PRELIMINARY.pdf https://reports.aviation-safety.net/2018/20181029-0_B38M_PK-...
- mlindner 8y ago
  That's incorrect, these have haptic feedback, once you grasp it, it's not going to fight you. It looks for resistance and if it sees the resistance it doesn't continue.
  - verytrivial 8y ago
    Ah, Okay, thanks. What remaining automation would be driving the trim wheel in this case that grab-and-hold would countermand?
  - heavenlyblue 8y ago
    You are still fighting it, aren't you?
  - userbinator 8y ago
    Really? Here's a video of someone doing it, and it certainly looks like it tries to fight you (and the trim motor continues running) but there's a clutch that starts slipping: https://www.youtube.com/watch?v=cQirIH_DuAs https://www.youtube.com/watch?v=cQirIH_DuAs
- ProAm 8y ago
  > you may need to physically fight the computer This is the future I look forward to. AI/ML/Autopilot/Companies moving fast and breaking things.... Boeing took that one literally.

l33tman 8y ago

This is not true, on the released FDR data, it shows clearly that the pilot flying corrected the MCAS system's trim down adjustments at every cycle (each 10 seconds) for most of the level flight. So at least the PF must have been aware of this. It's a bit weird if they didn't talk about it though, maybe it was obvious to both pilots and they were concentrating on if they were keeping altitude and airspeed at all (the most important factors that would keep them flying).

mannykannot 8y ago
This got my attention too. For at least 6 minutes, the crew were coping with the problem - the aircraft was holding altitude and speed, if erratically, and they must have been getting the trim back to approximately where it should have been, or else MCAS would have driven the stabilizer to the point where it overrode the elevator authority. In about the last minute of that recording, the pattern changes: the pilot trim inputs decreased and became much shorter, and the stick forces changed from going up and down to a sustained, high, back-force. The article says that near the end, the captain handed over control to the 1st officer so he could look for answers. I am speculating wildly here, but could that coincide with the change in the pattern of the pilot's responses? Could it be that the 1st officer did not attempt to re-trim, or did not do it completely? Did he try to trim manually, but perhaps could not because of the load on the stabilizer? [1]. IIRC, according to Dominic Gates, it would only take two cycles of MCAS intervention until the stabilizer had reached a point where full nose-up elevator could not keep the airplane level. The chart was published in one of Dominic Gates' Seattle Times articles, but I am not in a position to find it right now... [1] https://news.ycombinator.com/item?id=19400157 https://news.ycombinator.com/item?id=19400157
- VBprogrammer 8y ago
  Anyone who's ever piloted an aircraft would try to re-trim against pitch forces. It's inconceivable that the pilot for some reason just failed to do that. I'm not sure exactly what force it takes to get full nose up elevator at flying speed but it will be in the range of dozens of kilograms. I personally wonder if there is potentially a thermal cut-off on the trim motor which could have been activated by this trimming and re-triming. Or even an outright failure of some part of the system.

dvdkhlng 8y ago

I'm just wondering: it looks like the absolute trim angle is not visible anywhere in the cockpit? The trim wheel takes multiple revolutions to adjust trim between min/max limits, so looking at the wheel itself, you can only every see changes, not values. Unless you notice the change while it happens, you won't have any indication as to the unusual trim angle? Thinking about that I wonder whether there was even a measurement signal that the computers could rely on to determine the current state of the trim angle in absolute terms. Given how the MCAS did incremental changes without stopping before reaching the mechanical limits, I do imagine that there might have been no hardware feedback to assess the current trim state, thereby making it impossible hardware-wise to limit MCAS to a sensible (and safe) range of angles. [edit] Looking at the lion air crash report [1] p.14 ff it seems like also the black box only recorded trim changes, not absolute trim state. [1] https://reports.aviation-safety.net/2018/20181029-0_B38M_PK-LQP_PRELIMINARY.pdf https://reports.aviation-safety.net/2018/20181029-0_B38M_PK-...

VBprogrammer 8y ago
There is a trim position indication. It's to the left and right of the throttles. It's a critical thing to check before starting the takeoff roll. You can see it on the copilot's side here by the trim wheel. http://www.b737.org.uk/images/stabtrim_200.jpg http://www.b737.org.uk/images/stabtrim_200.jpg
- dvdkhlng 8y ago
  Yes, thinking about that, not having any pilot-visible indication would be quite some point of failure at lift-off. Still wondering whether MCAS and other automatation systems have a representation of that position indication. I can't currently come up with an idea of how to implement that in an easy, yet error-resilient way. Usually you would initially drive into a reference position, then integrate the motor's actions to arive at an absolute value, but integration would potentially add up subsequent errors, and obviously you can't re-do the referencing step while in the air.
  - VBprogrammer 8y ago