8 ms·

I don't entirely grok the vocabulary, but it sounds like the computers each said "This input is outside of limits, something must be wrong with me" so shut off

I don't entirely grok the vocabulary, but it sounds like the computers each said "This input is outside of limits, something must be wrong with me" so shut off for safety. But is there a mechanism that says "we can't all be wrong, it must be the sensor", to avoid situations like this?

noir_lord 5y ago
> "we can't all be wrong, it must be the sensor", to avoid situations like this? Not an expert but my understanding is that typically with these systems they take a poll and vote, if 1/3 disagree it's ignored if 2/3 disagree they scream and switch to manual/fallback simpler systems.
- zymhan 5y ago
  Indeed, it seems like the computers were not in agreement > the combination of a high COM/MON channels asynchronism In this case, since the automation failed, the plane reverted to "Direct law", where the pilot's inputs directly control the plane, instead of passing through computer checks first.
- sgtnoodle 5y ago
  For a binary signal, it's impossible for all three to mismatch. For a more analog signal, all three will generally mismatch to some extent whether it's in time or in space or both. Fault tolerance and fault detection are two separate but often coupled concepts. Systems can be designed to be inherently tolerant to a fault without detecting the fault (and good designs often are). All faults need to be detected eventually, though, so that they can be repaired before more faults occur and compound into a broken system. There's typically very tight timing requirements for fault tolerance, and significantly looser timing requirements for detection. As a result, you can often solve them differently. In a 3 string system with voting, it's often the case that the median signal is used for control without any interpretation of "goodness". That strategy works fine for short time periods of fault tolerance, as two strings would have to produce bad signals for the system to be affected. Separate from the median voting control path, you would then have a variety of consistency checking algorithms looking at the three signals and trying to intelligently determine whether any of the strings have failed. Those algorithms are often stateful and complicated, and rely on heavy filtering to avoid false positives. When a fault is detected, at minimum it needs to be communicated to an operator. In some cases, the detected fault will also trigger a "fault response", i.e. disabling the offending computer. In this case, it sounds like maybe a fault detection algorithm had a false positive that disabled the computer, and the same algorithm was running on all three computers. Despite there being 3 computers, it doesn't sound like this is a 3 string voting system. Rather, each of the three computers are independently able to control the system. The 3 strings exist for redundancy rather than for fault tolerance. Fault tolerance is provided by having two computers that cross-check everything they do, and third computer is there so that the cross-checking is fault tolerant. Two string redundancy is very common in automotive and aerospace.
- jstanley 5y ago
  How can you tell the difference between 1/3 disagreeing and 2/3 disagreeing? If 2/3 disagree, don't they just agree with each other, and therefore it still only looks like 1/3 disagreeing?
  - ddalex 5y ago
    1/3 disagrees - two sets of outputs are converging, one is diverging 2/3 disagrees - all 3 sets of output diverge from each other; there is no clear course of action
zymhan 5y ago
I'm not sure there was a sensor issue here. It appears that the Flight Computer monitoring routine for the Rudder position somehow caused all three computers to crash. This was somehow exacerbated by the pilot's rudder inputs, which I don't fully understand > the combination of a high COM/MON channels asynchronism and the pilot pedal inputs resulted in the rudder order difference between the two channels to exceed the monitoring threshold The flight computer failure resulted in the inability to use two braking mechanisms, Thrust Reversers (make air from the engine go forwardsish) and Spoilers (stop the wing from producing lift, putting more weight on the wheels and making the brakes more effective.
- salawat 5y ago
  The thing that bugs me is "did the pilots engage those systems manually?" And How was their reaction time doing so compared to someone who would be used to doing so via muscle memory on the regular? Automating things is nice and all, but there is something to be said for keeping manual skills sharp.
  - avianlyric 5y ago
    Well in this case the entire system failed safe after a pretty catastrophic failure of the automated systems. So on the whole I would say this incident demonstrates that the current safety standards, contingency plans, and pilot train all work as needed. I don’t think there’s anything here to suggest that the pilots manual skills are rusty. And when talking about the specific systems that didn’t active. They didn’t activate because they require a positive indication from the flight computers that’s it’s safe to activate. Something that probably can’t be overridden by the pilots. Which is why the planning process for flights requires pilots to assume they won’t work, and ensure the runway is long enough for the worst possible scenario.
    - scoopertrooper 5y ago
      > So on the whole I would say this incident demonstrates that the current safety standards, contingency plans, and pilot train all work as needed. I don’t think there’s anything here to suggest that the pilots manual skills are rusty According to the article they had 30 feet of runway remaining when they brought the plane to a halt. So, yeah I guess, everything worked out, but I wouldn't say that was indicative of a well oiled machine.
ak217 5y ago
It sounds like it was more complex than that. The computers shut down because of a COM-MON order mismatch. In other words, there is a watchdog in the system that monitors the three computers and their orders (control surface movements, etc. - in this case rudder inputs). There are 3 computers and one of them is designated COM (command) while another is designated MON (monitoring). If the values of the inputs that the computers want to send to the actuators diverge too much for too long, the command computer is disconnected and another computer is designated command. In this case it sounds like the command and monitoring computers had a significant delay between them when they transitioned from flight law (settings used in flight) to ground law (settings used while on the ground) while the pilot was also applying rudder (which is very common during landing, to correct for crosswind). When on the ground, the rudder is set to not deflect as much from the pilot's inputs compared to when in the air. It sounds like normally the computers switch to ground law close enough in time that the resulting mismatch is ignored, but in this case the significant delay caused a "split brain" situation and the watchdog disconnected all 3 computers. Perhaps the root cause is an issue in the weight-on-wheel sensor inputs that the computers use to transition. The computers need to use redundant sensors so they don't all rely on the same faulty sensor, but maybe the sensors are not appropriately cross-fed or the sensor input combination logic is too different between the computers. The problem with not disconnecting a computer in this situation is that bad inputs from that computer may also cause a crash. But maybe if no other computer is available to take its place the logic should be different, at least during takeoff/landing.
- avianlyric 5y ago
  Im gonna re-explain what you’ve written in language I’m more familiar with to check my understanding, and hopefully you can correct any errors I make. As I understand it the A330 has three primary flight computers, all observing the same inputs (which might come from different physical sensors monitoring the same thing?) and producing outputs, also know as “orders” for other systems in the plan, like actuators. Of these three computers, one will act as the primary command computer (COM), one as a monitoring computer (MON), the third is spare and normally ignored. Only orders from the COM machine is sent to downstream systems like actuators. There’s a separate watchdog system that monitors the outputs (orders) of both the COM and MON, and if their order values diverge by too much for too long, it shuts down the COM computer and passes control to the MON computer and the spare. As part of this process one of those two computers becomes COM the other MON. I assume the size of the divergence determines how long they can diverge for. A large divergence is only allowable for a very short period of time, and a small divergence is allowed for longer. In addition to all of this, the computers have different operating modes which changes how they respond to inputs. In this case the relevant states are “normal law” for in the air, and “ground law” for on the ground. One things that’s different between these states is how tightly coupled rudder inputs from the pilot are to rudder orders to the actuator. In the air the rudder is less tightly coupled than on the ground. E.g. commanding full left rudder in the air results in less physical movement of the rudder than the same command on the ground. When the plane lands, detected by a pressure sensor in the landing gear, the computers transition from “normal law” to “ground law”. Which for some outputs, like the rudder, might result in a step change in outputs (orders) from the computers. So in this specific scenario what happened is that the flight computers for some reason didn’t transition between “normal law” and “ground law” simultaneously (or close enough to simultaneously). So the COM computer significantly changed its rudder output as a result of changing law, but the MON computer didn’t, because it hadn’t changed law yet (the inverse of this is also possible). As a result they were producing very large differences in rudder orders, resulting in the monitoring watchdog killing the COM computer and failing over to the spare. Where the above situation happened a second time, resulting in all computers being shut down. Is all of the above correct? All of this does make me wonder, if changing law can result in computer outputs quickly changing, doesn’t that make law changes inherently dangerous? If you’re a pilot landing a plane applying significant rudder inputs, doesn’t the above me that those inputs will have a vastly different effect once the wheels touch the runway?
phkahler 5y ago
That's how I read it too. The discrepancy was deemed "I'm broken and need to shut down" instead of something more benign in a condition they didnt realize might happen. That a critical difference since all 3 will think "I'm broken" as happened here. That stuff can be hard to get right. Glad they found this one with 10 meters to spare.
- avianlyric 5y ago
  It seems that flight planning is done on the basis that the loss of all flight computers is possible, and to make sure that your runway is long enough to accommodate that situation. So thankfully the loss of all three computers isn’t inherently dangerous, and is demonstrated by this incident. Based on my reading of the report the primary take away from this (apart from a an issue with the flight computers software), is that flight plans aren’t and prep by the airport wasn’t conservative enough, so they ended up with slightly less safety margin than they expected in this scenario.
  - ncmncm 5y ago
    > loss of all three computers isn’t inherently dangerous You seem to be using a different definition of "dangerous" from the rest of us. Loss of systems that pilots are used to depending on is inherently dangerous. After that, degrees of luck, skill, and conservative planning become major factors in the outcome. The loss of spoilers and reversers is itself dangerous. In slightly different conditions, e.g. snow, the aircraft would not have stopped where it did. There was a Lockheed aircraft that ran off the end of a runway at an above-usual touchdown speed: the spoilers would not go up because for lack of weight on wheels, and brakes had no traction because spoilers were not up.
    - avianlyric 5y ago
      I don’t know what to say. Pilots are trained for this exact situation, flight plans are designed for this situation. Every reasonable measure is taken to ensure that this exact situation isn’t unnecessarily more dangerous than it absolutely needs to be. As proven by this exact incident. You can talk about hypotheticals like snow, but this is a commercial airport. They will either have snow clearing equipment or they’ll redirect planes if they can’t clear the snow. It’s also Taipei where it basically never snows around the city. Simply put if it was unsafe to land this plane in this state with snow (and snow was believed to fall that day), then the plane wouldn’t even take off. Lockheed aircraft aren’t passenger airlines, they have a completely different risk profile and risk appetite. I imagine Lockheed aircraft occasionally get taken down my enemy fire, but we don’t build commercial aircraft to handle that situation.

noir_lord 5y ago

> "we can't all be wrong, it must be the sensor", to avoid situations like this? Not an expert but my understanding is that typically with these systems they take a poll and vote, if 1/3 disagree it's ignored if 2/3 disagree they scream and switch to manual/fallback simpler systems.

zymhan 5y ago
Indeed, it seems like the computers were not in agreement > the combination of a high COM/MON channels asynchronism In this case, since the automation failed, the plane reverted to "Direct law", where the pilot's inputs directly control the plane, instead of passing through computer checks first.
sgtnoodle 5y ago
For a binary signal, it's impossible for all three to mismatch. For a more analog signal, all three will generally mismatch to some extent whether it's in time or in space or both. Fault tolerance and fault detection are two separate but often coupled concepts. Systems can be designed to be inherently tolerant to a fault without detecting the fault (and good designs often are). All faults need to be detected eventually, though, so that they can be repaired before more faults occur and compound into a broken system. There's typically very tight timing requirements for fault tolerance, and significantly looser timing requirements for detection. As a result, you can often solve them differently. In a 3 string system with voting, it's often the case that the median signal is used for control without any interpretation of "goodness". That strategy works fine for short time periods of fault tolerance, as two strings would have to produce bad signals for the system to be affected. Separate from the median voting control path, you would then have a variety of consistency checking algorithms looking at the three signals and trying to intelligently determine whether any of the strings have failed. Those algorithms are often stateful and complicated, and rely on heavy filtering to avoid false positives. When a fault is detected, at minimum it needs to be communicated to an operator. In some cases, the detected fault will also trigger a "fault response", i.e. disabling the offending computer. In this case, it sounds like maybe a fault detection algorithm had a false positive that disabled the computer, and the same algorithm was running on all three computers. Despite there being 3 computers, it doesn't sound like this is a 3 string voting system. Rather, each of the three computers are independently able to control the system. The 3 strings exist for redundancy rather than for fault tolerance. Fault tolerance is provided by having two computers that cross-check everything they do, and third computer is there so that the cross-checking is fault tolerant. Two string redundancy is very common in automotive and aerospace.
jstanley 5y ago
How can you tell the difference between 1/3 disagreeing and 2/3 disagreeing? If 2/3 disagree, don't they just agree with each other, and therefore it still only looks like 1/3 disagreeing?
- ddalex 5y ago
  1/3 disagrees - two sets of outputs are converging, one is diverging 2/3 disagrees - all 3 sets of output diverge from each other; there is no clear course of action

zymhan 5y ago

I'm not sure there was a sensor issue here. It appears that the Flight Computer monitoring routine for the Rudder position somehow caused all three computers to crash. This was somehow exacerbated by the pilot's rudder inputs, which I don't fully understand > the combination of a high COM/MON channels asynchronism and the pilot pedal inputs resulted in the rudder order difference between the two channels to exceed the monitoring threshold The flight computer failure resulted in the inability to use two braking mechanisms, Thrust Reversers (make air from the engine go forwardsish) and Spoilers (stop the wing from producing lift, putting more weight on the wheels and making the brakes more effective.

salawat 5y ago
The thing that bugs me is "did the pilots engage those systems manually?" And How was their reaction time doing so compared to someone who would be used to doing so via muscle memory on the regular? Automating things is nice and all, but there is something to be said for keeping manual skills sharp.
- avianlyric 5y ago
  Well in this case the entire system failed safe after a pretty catastrophic failure of the automated systems. So on the whole I would say this incident demonstrates that the current safety standards, contingency plans, and pilot train all work as needed. I don’t think there’s anything here to suggest that the pilots manual skills are rusty. And when talking about the specific systems that didn’t active. They didn’t activate because they require a positive indication from the flight computers that’s it’s safe to activate. Something that probably can’t be overridden by the pilots. Which is why the planning process for flights requires pilots to assume they won’t work, and ensure the runway is long enough for the worst possible scenario.
  - scoopertrooper 5y ago
    > So on the whole I would say this incident demonstrates that the current safety standards, contingency plans, and pilot train all work as needed. I don’t think there’s anything here to suggest that the pilots manual skills are rusty According to the article they had 30 feet of runway remaining when they brought the plane to a halt. So, yeah I guess, everything worked out, but I wouldn't say that was indicative of a well oiled machine.

ak217 5y ago

It sounds like it was more complex than that. The computers shut down because of a COM-MON order mismatch. In other words, there is a watchdog in the system that monitors the three computers and their orders (control surface movements, etc. - in this case rudder inputs). There are 3 computers and one of them is designated COM (command) while another is designated MON (monitoring). If the values of the inputs that the computers want to send to the actuators diverge too much for too long, the command computer is disconnected and another computer is designated command. In this case it sounds like the command and monitoring computers had a significant delay between them when they transitioned from flight law (settings used in flight) to ground law (settings used while on the ground) while the pilot was also applying rudder (which is very common during landing, to correct for crosswind). When on the ground, the rudder is set to not deflect as much from the pilot's inputs compared to when in the air. It sounds like normally the computers switch to ground law close enough in time that the resulting mismatch is ignored, but in this case the significant delay caused a "split brain" situation and the watchdog disconnected all 3 computers. Perhaps the root cause is an issue in the weight-on-wheel sensor inputs that the computers use to transition. The computers need to use redundant sensors so they don't all rely on the same faulty sensor, but maybe the sensors are not appropriately cross-fed or the sensor input combination logic is too different between the computers. The problem with not disconnecting a computer in this situation is that bad inputs from that computer may also cause a crash. But maybe if no other computer is available to take its place the logic should be different, at least during takeoff/landing.

avianlyric 5y ago
Im gonna re-explain what you’ve written in language I’m more familiar with to check my understanding, and hopefully you can correct any errors I make. As I understand it the A330 has three primary flight computers, all observing the same inputs (which might come from different physical sensors monitoring the same thing?) and producing outputs, also know as “orders” for other systems in the plan, like actuators. Of these three computers, one will act as the primary command computer (COM), one as a monitoring computer (MON), the third is spare and normally ignored. Only orders from the COM machine is sent to downstream systems like actuators. There’s a separate watchdog system that monitors the outputs (orders) of both the COM and MON, and if their order values diverge by too much for too long, it shuts down the COM computer and passes control to the MON computer and the spare. As part of this process one of those two computers becomes COM the other MON. I assume the size of the divergence determines how long they can diverge for. A large divergence is only allowable for a very short period of time, and a small divergence is allowed for longer. In addition to all of this, the computers have different operating modes which changes how they respond to inputs. In this case the relevant states are “normal law” for in the air, and “ground law” for on the ground. One things that’s different between these states is how tightly coupled rudder inputs from the pilot are to rudder orders to the actuator. In the air the rudder is less tightly coupled than on the ground. E.g. commanding full left rudder in the air results in less physical movement of the rudder than the same command on the ground. When the plane lands, detected by a pressure sensor in the landing gear, the computers transition from “normal law” to “ground law”. Which for some outputs, like the rudder, might result in a step change in outputs (orders) from the computers. So in this specific scenario what happened is that the flight computers for some reason didn’t transition between “normal law” and “ground law” simultaneously (or close enough to simultaneously). So the COM computer significantly changed its rudder output as a result of changing law, but the MON computer didn’t, because it hadn’t changed law yet (the inverse of this is also possible). As a result they were producing very large differences in rudder orders, resulting in the monitoring watchdog killing the COM computer and failing over to the spare. Where the above situation happened a second time, resulting in all computers being shut down. Is all of the above correct? All of this does make me wonder, if changing law can result in computer outputs quickly changing, doesn’t that make law changes inherently dangerous? If you’re a pilot landing a plane applying significant rudder inputs, doesn’t the above me that those inputs will have a vastly different effect once the wheels touch the runway?

phkahler 5y ago

That's how I read it too. The discrepancy was deemed "I'm broken and need to shut down" instead of something more benign in a condition they didnt realize might happen. That a critical difference since all 3 will think "I'm broken" as happened here. That stuff can be hard to get right. Glad they found this one with 10 meters to spare.

avianlyric 5y ago
It seems that flight planning is done on the basis that the loss of all flight computers is possible, and to make sure that your runway is long enough to accommodate that situation. So thankfully the loss of all three computers isn’t inherently dangerous, and is demonstrated by this incident. Based on my reading of the report the primary take away from this (apart from a an issue with the flight computers software), is that flight plans aren’t and prep by the airport wasn’t conservative enough, so they ended up with slightly less safety margin than they expected in this scenario.
- ncmncm 5y ago
  > loss of all three computers isn’t inherently dangerous You seem to be using a different definition of "dangerous" from the rest of us. Loss of systems that pilots are used to depending on is inherently dangerous. After that, degrees of luck, skill, and conservative planning become major factors in the outcome. The loss of spoilers and reversers is itself dangerous. In slightly different conditions, e.g. snow, the aircraft would not have stopped where it did. There was a Lockheed aircraft that ran off the end of a runway at an above-usual touchdown speed: the spoilers would not go up because for lack of weight on wheels, and brakes had no traction because spoilers were not up.
  - avianlyric 5y ago
    I don’t know what to say. Pilots are trained for this exact situation, flight plans are designed for this situation. Every reasonable measure is taken to ensure that this exact situation isn’t unnecessarily more dangerous than it absolutely needs to be. As proven by this exact incident. You can talk about hypotheticals like snow, but this is a commercial airport. They will either have snow clearing equipment or they’ll redirect planes if they can’t clear the snow. It’s also Taipei where it basically never snows around the city. Simply put if it was unsafe to land this plane in this state with snow (and snow was believed to fall that day), then the plane wouldn’t even take off. Lockheed aircraft aren’t passenger airlines, they have a completely different risk profile and risk appetite. I imagine Lockheed aircraft occasionally get taken down my enemy fire, but we don’t build commercial aircraft to handle that situation.