13 ms·

Incident: Airbus A330 at Taipei, primary computers failed on touchdown (2020)

300 points by akamaka 5y ago

mplanchard 5y ago
This could probably use a 2020 in the title
- akamaka 5y ago
  Final report was just issued this week.
- alkonaut 5y ago
  Maybe explains why there were only 87 pax in a 330 too.
- deleted 5y ago
  [deleted]
mysterydip 5y ago
I don't entirely grok the vocabulary, but it sounds like the computers each said "This input is outside of limits, something must be wrong with me" so shut off for safety. But is there a mechanism that says "we can't all be wrong, it must be the sensor", to avoid situations like this?
- noir_lord 5y ago
  > "we can't all be wrong, it must be the sensor", to avoid situations like this? Not an expert but my understanding is that typically with these systems they take a poll and vote, if 1/3 disagree it's ignored if 2/3 disagree they scream and switch to manual/fallback simpler systems.
  - zymhan 5y ago
    Indeed, it seems like the computers were not in agreement > the combination of a high COM/MON channels asynchronism In this case, since the automation failed, the plane reverted to "Direct law", where the pilot's inputs directly control the plane, instead of passing through computer checks first.
  - sgtnoodle 5y ago
    For a binary signal, it's impossible for all three to mismatch. For a more analog signal, all three will generally mismatch to some extent whether it's in time or in space or both. Fault tolerance and fault detection are two separate but often coupled concepts. Systems can be designed to be inherently tolerant to a fault without detecting the fault (and good designs often are). All faults need to be detected eventually, though, so that they can be repaired before more faults occur and compound into a broken system. There's typically very tight timing requirements for fault tolerance, and significantly looser timing requirements for detection. As a result, you can often solve them differently. In a 3 string system with voting, it's often the case that the median signal is used for control without any interpretation of "goodness". That strategy works fine for short time periods of fault tolerance, as two strings would have to produce bad signals for the system to be affected. Separate from the median voting control path, you would then have a variety of consistency checking algorithms looking at the three signals and trying to intelligently determine whether any of the strings have failed. Those algorithms are often stateful and complicated, and rely on heavy filtering to avoid false positives. When a fault is detected, at minimum it needs to be communicated to an operator. In some cases, the detected fault will also trigger a "fault response", i.e. disabling the offending computer. In this case, it sounds like maybe a fault detection algorithm had a false positive that disabled the computer, and the same algorithm was running on all three computers. Despite there being 3 computers, it doesn't sound like this is a 3 string voting system. Rather, each of the three computers are independently able to control the system. The 3 strings exist for redundancy rather than for fault tolerance. Fault tolerance is provided by having two computers that cross-check everything they do, and third computer is there so that the cross-checking is fault tolerant. Two string redundancy is very common in automotive and aerospace.
  - jstanley 5y ago
    How can you tell the difference between 1/3 disagreeing and 2/3 disagreeing? If 2/3 disagree, don't they just agree with each other, and therefore it still only looks like 1/3 disagreeing?
    - ddalex 5y ago
      1/3 disagrees - two sets of outputs are converging, one is diverging 2/3 disagrees - all 3 sets of output diverge from each other; there is no clear course of action
- zymhan 5y ago
  I'm not sure there was a sensor issue here. It appears that the Flight Computer monitoring routine for the Rudder position somehow caused all three computers to crash. This was somehow exacerbated by the pilot's rudder inputs, which I don't fully understand > the combination of a high COM/MON channels asynchronism and the pilot pedal inputs resulted in the rudder order difference between the two channels to exceed the monitoring threshold The flight computer failure resulted in the inability to use two braking mechanisms, Thrust Reversers (make air from the engine go forwardsish) and Spoilers (stop the wing from producing lift, putting more weight on the wheels and making the brakes more effective.
  - salawat 5y ago
    The thing that bugs me is "did the pilots engage those systems manually?" And How was their reaction time doing so compared to someone who would be used to doing so via muscle memory on the regular? Automating things is nice and all, but there is something to be said for keeping manual skills sharp.
    - avianlyric 5y ago
      Well in this case the entire system failed safe after a pretty catastrophic failure of the automated systems. So on the whole I would say this incident demonstrates that the current safety standards, contingency plans, and pilot train all work as needed. I don’t think there’s anything here to suggest that the pilots manual skills are rusty. And when talking about the specific systems that didn’t active. They didn’t activate because they require a positive indication from the flight computers that’s it’s safe to activate. Something that probably can’t be overridden by the pilots. Which is why the planning process for flights requires pilots to assume they won’t work, and ensure the runway is long enough for the worst possible scenario.
      - scoopertrooper 5y ago
        > So on the whole I would say this incident demonstrates that the current safety standards, contingency plans, and pilot train all work as needed. I don’t think there’s anything here to suggest that the pilots manual skills are rusty According to the article they had 30 feet of runway remaining when they brought the plane to a halt. So, yeah I guess, everything worked out, but I wouldn't say that was indicative of a well oiled machine.
- ak217 5y ago
  It sounds like it was more complex than that. The computers shut down because of a COM-MON order mismatch. In other words, there is a watchdog in the system that monitors the three computers and their orders (control surface movements, etc. - in this case rudder inputs). There are 3 computers and one of them is designated COM (command) while another is designated MON (monitoring). If the values of the inputs that the computers want to send to the actuators diverge too much for too long, the command computer is disconnected and another computer is designated command. In this case it sounds like the command and monitoring computers had a significant delay between them when they transitioned from flight law (settings used in flight) to ground law (settings used while on the ground) while the pilot was also applying rudder (which is very common during landing, to correct for crosswind). When on the ground, the rudder is set to not deflect as much from the pilot's inputs compared to when in the air. It sounds like normally the computers switch to ground law close enough in time that the resulting mismatch is ignored, but in this case the significant delay caused a "split brain" situation and the watchdog disconnected all 3 computers. Perhaps the root cause is an issue in the weight-on-wheel sensor inputs that the computers use to transition. The computers need to use redundant sensors so they don't all rely on the same faulty sensor, but maybe the sensors are not appropriately cross-fed or the sensor input combination logic is too different between the computers. The problem with not disconnecting a computer in this situation is that bad inputs from that computer may also cause a crash. But maybe if no other computer is available to take its place the logic should be different, at least during takeoff/landing.
  - avianlyric 5y ago
    Im gonna re-explain what you’ve written in language I’m more familiar with to check my understanding, and hopefully you can correct any errors I make. As I understand it the A330 has three primary flight computers, all observing the same inputs (which might come from different physical sensors monitoring the same thing?) and producing outputs, also know as “orders” for other systems in the plan, like actuators. Of these three computers, one will act as the primary command computer (COM), one as a monitoring computer (MON), the third is spare and normally ignored. Only orders from the COM machine is sent to downstream systems like actuators. There’s a separate watchdog system that monitors the outputs (orders) of both the COM and MON, and if their order values diverge by too much for too long, it shuts down the COM computer and passes control to the MON computer and the spare. As part of this process one of those two computers becomes COM the other MON. I assume the size of the divergence determines how long they can diverge for. A large divergence is only allowable for a very short period of time, and a small divergence is allowed for longer. In addition to all of this, the computers have different operating modes which changes how they respond to inputs. In this case the relevant states are “normal law” for in the air, and “ground law” for on the ground. One things that’s different between these states is how tightly coupled rudder inputs from the pilot are to rudder orders to the actuator. In the air the rudder is less tightly coupled than on the ground. E.g. commanding full left rudder in the air results in less physical movement of the rudder than the same command on the ground. When the plane lands, detected by a pressure sensor in the landing gear, the computers transition from “normal law” to “ground law”. Which for some outputs, like the rudder, might result in a step change in outputs (orders) from the computers. So in this specific scenario what happened is that the flight computers for some reason didn’t transition between “normal law” and “ground law” simultaneously (or close enough to simultaneously). So the COM computer significantly changed its rudder output as a result of changing law, but the MON computer didn’t, because it hadn’t changed law yet (the inverse of this is also possible). As a result they were producing very large differences in rudder orders, resulting in the monitoring watchdog killing the COM computer and failing over to the spare. Where the above situation happened a second time, resulting in all computers being shut down. Is all of the above correct? All of this does make me wonder, if changing law can result in computer outputs quickly changing, doesn’t that make law changes inherently dangerous? If you’re a pilot landing a plane applying significant rudder inputs, doesn’t the above me that those inputs will have a vastly different effect once the wheels touch the runway?
- phkahler 5y ago
  That's how I read it too. The discrepancy was deemed "I'm broken and need to shut down" instead of something more benign in a condition they didnt realize might happen. That a critical difference since all 3 will think "I'm broken" as happened here. That stuff can be hard to get right. Glad they found this one with 10 meters to spare.
  - avianlyric 5y ago
    It seems that flight planning is done on the basis that the loss of all flight computers is possible, and to make sure that your runway is long enough to accommodate that situation. So thankfully the loss of all three computers isn’t inherently dangerous, and is demonstrated by this incident. Based on my reading of the report the primary take away from this (apart from a an issue with the flight computers software), is that flight plans aren’t and prep by the airport wasn’t conservative enough, so they ended up with slightly less safety margin than they expected in this scenario.
    - ncmncm 5y ago
      > loss of all three computers isn’t inherently dangerous You seem to be using a different definition of "dangerous" from the rest of us. Loss of systems that pilots are used to depending on is inherently dangerous. After that, degrees of luck, skill, and conservative planning become major factors in the outcome. The loss of spoilers and reversers is itself dangerous. In slightly different conditions, e.g. snow, the aircraft would not have stopped where it did. There was a Lockheed aircraft that ran off the end of a runway at an above-usual touchdown speed: the spoilers would not go up because for lack of weight on wheels, and brakes had no traction because spoilers were not up.
      - avianlyric 5y ago
        I don’t know what to say. Pilots are trained for this exact situation, flight plans are designed for this situation. Every reasonable measure is taken to ensure that this exact situation isn’t unnecessarily more dangerous than it absolutely needs to be. As proven by this exact incident. You can talk about hypotheticals like snow, but this is a commercial airport. They will either have snow clearing equipment or they’ll redirect planes if they can’t clear the snow. It’s also Taipei where it basically never snows around the city. Simply put if it was unsafe to land this plane in this state with snow (and snow was believed to fall that day), then the plane wouldn’t even take off. Lockheed aircraft aren’t passenger airlines, they have a completely different risk profile and risk appetite. I imagine Lockheed aircraft occasionally get taken down my enemy fire, but we don’t build commercial aircraft to handle that situation.
the_duke 5y ago
Odd that there is no information from Airbus there. FYI: the comment section contains some interesting background.
- zymhan 5y ago
  Well there is a single bullet point from/about Airbus > 7. Following the occurrence, Airbus reviewed its in-service experience, and confirmed that no other triple PRIM fault at touchdown event had been reported on A330/A340 aircraft family since entry into service. The A330/A340 fleet fitted with electrical rudder has accumulated 8.7 millions of Flight Cycles and 44.3 millions of Flight Hours (in-service data from April 2020).
  - gswdh 5y ago
    AKA, it's only happened once so we don't care...
  - thomasjudge 5y ago
    So, does this mean that the outcome was, Airbus reviewed the situation and the data, and made no changes or corrections to the system?
    - thefurrysquid 5y ago
      There will be a software update. Read page 9 of the exec summary linked in the article.
ceejayoz 5y ago
> The crew applied maximum manual braking and managed to stop the aircraft 10 meters/33 feet ahead of the runway end (runway length 2600 meters/8530 feet). Hope the crew had their brown pants on.
- encoderer 5y ago
  From the cockpit, 33’ must have looked right under their nose.
  - futhey 5y ago
    Songshan's not a large airport, it's mostly used for domestic flights, and a few nearby international destinations. Single runway, immediate obstacles on both ends (no matter which way you land). Although, runway 10 is eastbound & it's just a fence and small parking lot, minor traffic, and then you land in the river. Sounds like it would have been stressful.
rzzzt 5y ago
Why can't thrust reversers and spoilers be engaged without computer supervision? The article just says that ground spoilers require one of the three flight computers to remain operative, autobrake needs two, reversers needs one out of two specific units to be running to unlock.
- bdonlan 5y ago
  All flight controls on the A330 are fly-by-wire - they're connected via computers. The most critical surfaces have secondary, dumb controllers that can take over if the primary flight computers fail, but less critical systems may not be connected to those backup systems - it's expected that a triple fault of the primary flight computers should be very rare, and manual braking is available as a backup after all.
  - NovemberWhiskey 5y ago
    Right. In this case we have a completely-unprecedented triple failure of the primary flight control system causing loss of auto-brake, spoilers and thrust reversers; a wet runway with less than expected braking action; a tailwind landing and still a happy outcome. Honestly defense-in-depth seems to be working OK here.
    - gsnedders 5y ago
      And note that for all aircraft the landing distance is calculated _without_ thrust reversers or spoilers, and thus is based on braking performance alone. Some of the challenge here was the seconds spent at near landing speed without any braking being performed, as that'll eat up runway distance fast, following the auto brake failure prior to manual braking commencing.
      - azalemeth 5y ago
        It's also worth stating (for the OP) that thrust reversers are potentially really dangerous -- if they deploy in flight without being commanded to, the aircraft can -- or, more likely if the engine computer does not detect it and shut itself down, will become barely controllable, with previous fatal outcomes. [1] Two or more FCPCs failing is sufficient evidence that a "faecal fan incident may be occurring" that the risk of deploying them is just not worth it, especially as (pointed out by this parent above) they are effectively optional equipment and the FAA requires you to expect them to be inop to be safe. What they do save is fuel and time -- taxi time to the terminal, permit the use of high-speed runway turnoffs, etc. [1] https://en.wikipedia.org/wiki/Lauda_Air_Flight_004 https://en.wikipedia.org/wiki/Lauda_Air_Flight_004
sinuhe69 5y ago
Remind me of the Ariane 5 maiden flight: redundant hardwares does not protect against unsanitized inputs. I don't know much about the design of the airbuses, but I think it's sensible to specify the last primary computer running in a different mode as the first two, simply because double hardware failure is very very unlikely. So it boils down to either software bug and or unsanitized inputs. Assuming that, the last computer should provide some basic but safe functions.
- strictfp 5y ago
  I've seen bad user input, one very unusual char, bring down a whole self-healing geographically redundant cluster of machines running an expensive commercial cluster software. Any connected devices can bring each other down.
  - andrewnicolalde 5y ago
    That sounds like a great war story if you’re open to telling it :)
  - salawat 5y ago
    Story, story, story, story!
- avianlyric 5y ago
  Well that’s kind of what happened. The primary flight computers all failed, to the plane fell back to “direct law”, which means pilot input is directly transmitted to control systems without a computer attempting to interpret intent or protect against pilot error. As a consequence a lot of the more advanced functions became disabled because they rely on the primary flight computer to indicate when it’s safe for them to operate. Presumably because accidental operation of those features in flight is extremely dangerous.
oakmad 5y ago
For everyone commenting about different hardware failure modes and possible solutions. Take a look at the airbus « flight control laws » such as https://apstraining.com/wp-content/uploads/FCS-Airbus-Quick-Reference.pdf https://apstraining.com/wp-content/uploads/FCS-Airbus-Quick-... it governs what happens to flight controls and what protections are available in different scenarios. In this case the system worked, it crashed and reverted to direct law providing manual braking control. Off topic but I found this very interesting « The deceleration performance of the occurrence flight between 6,600 feet and 7,300 feet from the threshold of runway 10 deteriorated. It may be due to paint marking and rubber deposit on the touchdown zone of runway 28. »
- cwizou 5y ago
  > The deceleration performance of the occurrence flight between 6,600 feet and 7,300 feet from the threshold of runway 10 deteriorated. It may be due to paint marking and rubber deposit on the touchdown zone of runway 28 Just to clarify for others, this is the same runway but taken in two different directions (west to east and east to west, multiply the number by 10 and you get [rounded] angle in degree, so 100ish vs 280ish). It's 8500 ft. long and it looks like there was a loss of performance around the "touchdown" point of going the other way on that runway, where you had accumulated rubber from those landings (which is pretty normal and monitored, as pointed by leecb below), if that makes any sense.
  - leecb 5y ago
    > accumulated rubber from those landings Rubber accumulation on runways is a normal and expected situation. Part of runway maintenance at airports serving larger aircraft includes regular removal of the rubber using high pressure water or chemicals. https://en.wikipedia.org/wiki/Airfield_rubber_removal https://en.wikipedia.org/wiki/Airfield_rubber_removal
    - salawat 5y ago
      Not going to lie, I never thought this was a thing that'd need doing, but on further reflection it seems so obvious I'm kinda mad it never occurred to me in retrospect.
teeray 5y ago
I wonder what would have happened if they tried to go around? The TO/GA is configured ahead of time in the FMS… so if the flight computers are out, I wonder if there’s some hard-coded performance targets for the engines in a go around situation.
- concerned_user 5y ago
  They would push throttle manually in max power and perform the maneuver by hand.
- deleted 5y ago
  [deleted]
- hugh-avherald 5y ago
  There was barely a second between the first indication of a primary computer fault and the call for reversers to deploy. Once you select reverse thrust, you cannot go around under any circumstances.
  - strogonoff 5y ago
    It’s an interesting scenario. Your wheels touch the ground; you engage reverse thrust and it instantly fails; you are rolling at dangerous landing speed but now can’t go around? Relatedly, I wonder why did the pilots not engage the regular brakes ASAP.
    - avianlyric 5y ago
      Not quite, reading around it seems that flight planning requires you to assume that reverse thrusters and ground spoilers don’t work. So you land, your reverse thrusters immediately fail, and you’re now required to stop using your manual breaks and consume all of the breaking distance you originally planned for, with presumably a bit of margin on top.
bryan0 5y ago
The 2020 should be removed from the title. The article is mostly new information from the final report: > On Sep 3rd 2021 Taiwan's ASC released their final report in Chinese and their English Executive Summary concluding the probable causes of the incident were: …
FatalLogic 5y ago
The flight data recorder chart on page 114 of the Chinese-language report[0] seems to record that the pilots hit the brakes very hard in the last few seconds, maybe when they realized they were close to overrunning the runway. That caused a sudden peak of -0.47g deceleration, so it looks as if they could have used the brakes to slow the plane earlier on the runway, but maybe they didn't because they were expecting they could sort out the problem with their spoilers and reverse thrust and stop it in the normal way, without giving the passengers a shock. [0]https://www.ttsb.gov.tw/media/4912/ci-202-final-report_chinese_release.pdf https://www.ttsb.gov.tw/media/4912/ci-202-final-report_chine... edit: that chart's x-axis is the aircraft's position on the 8500ft-runway, and most of the recording traces seem to begin at the point when it had touched down finally with all wheels
- ysangkok 5y ago
  > hit the brakes very hard in the last few seconds In direct law, is it possible to brake too hard and have the tires skid on the runway? Is it possible that the pilots were afraid of this? I ask because people are characterizing direct law as "less smart". So how smart is it? Smart enough to include ABS?
  - FatalLogic 5y ago
    I don't know, but the computer had failed, the runway was wet, and they were confused, so, yes, it's not unlikely they were also afraid of skidding
  - oakmad 5y ago
    Oh 100%! Anti skid is only available in alternate and normal laws. Section 6.5 braking system of this document. https://www.smartcockpit.com/docs/A330_Flight_Deck_and_Systems_Briefing_For_Pilots.pdf https://www.smartcockpit.com/docs/A330_Flight_Deck_and_Syste...
    - iso8859-1 5y ago
      But that page does not mention direct law. Wouldn't anti-skid protection depend on the switch shown on the landing gear panel shown in 6.9?
pseingatl 5y ago
Airbus: what is it doing now?
aomobile 5y ago
I wonder how the pilots would have reacted if the computers had crashed 10-20 seconds earlier. Would they have landed or would they go up again to wait for reboot?
- bonzini 5y ago
  Difficult to say, since the computer crash was linked directly to having just touched down. 10-20 seconds before there was no disagreement because neither computer has detected that the wheels were on the ground.
fnord77 5y ago
> The root cause was determined to be an undue triggering of the rudder order COM/MON monitoring concomitantly in the 3 FCPC. At the time of the aircraft lateral control flight law switching to lateral ground law at touch down, the combination of a high COM/MON channels asynchronism and the pilot pedal inputs resulted in the rudder order difference between the two channels to exceed the monitoring threshold. The FCPC1 failed first. In lay-programmers terms, this sounds like a race condition. Is that correct?
- zaarn 5y ago
  Not quite. The computers disagreed, they didn't race. Likely cause might be that the sensors responsible for detecting touchdown might have been rusty or failed. The computers aren't racing towards ground law here, they switched to ground law at different times sufficiently apart that the monitoring watchdog turned them all off for commanding the aircraft to do three different things. Imagine it more like website and it's hosted by three servers with a replicated database. A monitoring tool checks that the number of entries in each server's database is the same. If one server has a different number, the tool turns it off, it now requires a human to bootstrap the database again. If two servers disagree the website goes offline, because it's no longer possible to tell which server has correct data and not operating is cheaper than the man hours required to puzzle the data back together. In this case, a likely cause might be that suddenly the network latency between the three servers got suddenly very large (cable damage maybe?) and now all three no longer agree on the dataset. The servers are shutdown.
5faulker 5y ago
Good thing that this was on a smaller airplane, otherwise it could have been a larger disaster.
- dangs_penis2 5y ago
  Wow! Thanks for the insight!
danielschonfeld 5y ago
Feels like there might be more to the story than is being let on. Curious why the PF has to specifically ask before landing for the spoilers to be called, to tell if he's touched down. AFAIK it's actually common Airbus procedure to called Spoilers and Reversers and deceleration after landing. Also telling whether or not you touched down isn't really a problem... even for the smoothest/softest of landings. Then there's the PM calling out "center line" at 130 feet...
dangs_penis2 5y ago
HackersNews is a experiment.
jimnotgym 5y ago
I note there was a tailwind serious enough to reduce breaking performance, and also that it is possible to land in the opposite direction on the runway in question. Is the practice of landing into the wind ignored within certain parameters?
- gsnedders 5y ago
  In general, aircraft are certified to land/takeoff with tailwinds (and crosswinds, etc.) up to a certain speed. You don't want to be changing runway direction all the time, after all. Ultimately, it's up to the crew on board to determine the landing distance required for the current conditions (including the wind—whatever direction that is!), and if they don't believe the conditions are safe they should not proceed with the landing. They can, of course, request the opposite runway, though obviously if there is frequent traffic ATC is likely to refuse unless they want to change the direction for all traffic.
PostThisTooFast 5y ago
"undue triggering of the rudder order COM/MON monitoring concomitantly in the 3 FCPC. At the time of the aircraft lateral control flight law switching to lateral ground law" Inexcusable jargon and bullshit. Whoever was paid to generate this report should be fired.
ddalex 5y ago
Commercial piloting is hours of boredom with few moments of raw sheer terror.
tim333 5y ago
Good thing they failed on touchdown and not up in the air.

300 points by akamaka 5y ago

mplanchard 5y ago

This could probably use a 2020 in the title

akamaka 5y ago
Final report was just issued this week.
alkonaut 5y ago
Maybe explains why there were only 87 pax in a 330 too.
deleted 5y ago
[deleted]

mysterydip 5y ago

I don't entirely grok the vocabulary, but it sounds like the computers each said "This input is outside of limits, something must be wrong with me" so shut off for safety. But is there a mechanism that says "we can't all be wrong, it must be the sensor", to avoid situations like this?

noir_lord 5y ago
> "we can't all be wrong, it must be the sensor", to avoid situations like this? Not an expert but my understanding is that typically with these systems they take a poll and vote, if 1/3 disagree it's ignored if 2/3 disagree they scream and switch to manual/fallback simpler systems.
- zymhan 5y ago
  Indeed, it seems like the computers were not in agreement > the combination of a high COM/MON channels asynchronism In this case, since the automation failed, the plane reverted to "Direct law", where the pilot's inputs directly control the plane, instead of passing through computer checks first.
- sgtnoodle 5y ago
  For a binary signal, it's impossible for all three to mismatch. For a more analog signal, all three will generally mismatch to some extent whether it's in time or in space or both. Fault tolerance and fault detection are two separate but often coupled concepts. Systems can be designed to be inherently tolerant to a fault without detecting the fault (and good designs often are). All faults need to be detected eventually, though, so that they can be repaired before more faults occur and compound into a broken system. There's typically very tight timing requirements for fault tolerance, and significantly looser timing requirements for detection. As a result, you can often solve them differently. In a 3 string system with voting, it's often the case that the median signal is used for control without any interpretation of "goodness". That strategy works fine for short time periods of fault tolerance, as two strings would have to produce bad signals for the system to be affected. Separate from the median voting control path, you would then have a variety of consistency checking algorithms looking at the three signals and trying to intelligently determine whether any of the strings have failed. Those algorithms are often stateful and complicated, and rely on heavy filtering to avoid false positives. When a fault is detected, at minimum it needs to be communicated to an operator. In some cases, the detected fault will also trigger a "fault response", i.e. disabling the offending computer. In this case, it sounds like maybe a fault detection algorithm had a false positive that disabled the computer, and the same algorithm was running on all three computers. Despite there being 3 computers, it doesn't sound like this is a 3 string voting system. Rather, each of the three computers are independently able to control the system. The 3 strings exist for redundancy rather than for fault tolerance. Fault tolerance is provided by having two computers that cross-check everything they do, and third computer is there so that the cross-checking is fault tolerant. Two string redundancy is very common in automotive and aerospace.
- jstanley 5y ago
  How can you tell the difference between 1/3 disagreeing and 2/3 disagreeing? If 2/3 disagree, don't they just agree with each other, and therefore it still only looks like 1/3 disagreeing?
  - ddalex 5y ago
    1/3 disagrees - two sets of outputs are converging, one is diverging 2/3 disagrees - all 3 sets of output diverge from each other; there is no clear course of action
zymhan 5y ago
I'm not sure there was a sensor issue here. It appears that the Flight Computer monitoring routine for the Rudder position somehow caused all three computers to crash. This was somehow exacerbated by the pilot's rudder inputs, which I don't fully understand > the combination of a high COM/MON channels asynchronism and the pilot pedal inputs resulted in the rudder order difference between the two channels to exceed the monitoring threshold The flight computer failure resulted in the inability to use two braking mechanisms, Thrust Reversers (make air from the engine go forwardsish) and Spoilers (stop the wing from producing lift, putting more weight on the wheels and making the brakes more effective.
- salawat 5y ago
  The thing that bugs me is "did the pilots engage those systems manually?" And How was their reaction time doing so compared to someone who would be used to doing so via muscle memory on the regular? Automating things is nice and all, but there is something to be said for keeping manual skills sharp.
  - avianlyric 5y ago
    Well in this case the entire system failed safe after a pretty catastrophic failure of the automated systems. So on the whole I would say this incident demonstrates that the current safety standards, contingency plans, and pilot train all work as needed. I don’t think there’s anything here to suggest that the pilots manual skills are rusty. And when talking about the specific systems that didn’t active. They didn’t activate because they require a positive indication from the flight computers that’s it’s safe to activate. Something that probably can’t be overridden by the pilots. Which is why the planning process for flights requires pilots to assume they won’t work, and ensure the runway is long enough for the worst possible scenario.
    - scoopertrooper 5y ago
      > So on the whole I would say this incident demonstrates that the current safety standards, contingency plans, and pilot train all work as needed. I don’t think there’s anything here to suggest that the pilots manual skills are rusty According to the article they had 30 feet of runway remaining when they brought the plane to a halt. So, yeah I guess, everything worked out, but I wouldn't say that was indicative of a well oiled machine.
ak217 5y ago
It sounds like it was more complex than that. The computers shut down because of a COM-MON order mismatch. In other words, there is a watchdog in the system that monitors the three computers and their orders (control surface movements, etc. - in this case rudder inputs). There are 3 computers and one of them is designated COM (command) while another is designated MON (monitoring). If the values of the inputs that the computers want to send to the actuators diverge too much for too long, the command computer is disconnected and another computer is designated command. In this case it sounds like the command and monitoring computers had a significant delay between them when they transitioned from flight law (settings used in flight) to ground law (settings used while on the ground) while the pilot was also applying rudder (which is very common during landing, to correct for crosswind). When on the ground, the rudder is set to not deflect as much from the pilot's inputs compared to when in the air. It sounds like normally the computers switch to ground law close enough in time that the resulting mismatch is ignored, but in this case the significant delay caused a "split brain" situation and the watchdog disconnected all 3 computers. Perhaps the root cause is an issue in the weight-on-wheel sensor inputs that the computers use to transition. The computers need to use redundant sensors so they don't all rely on the same faulty sensor, but maybe the sensors are not appropriately cross-fed or the sensor input combination logic is too different between the computers. The problem with not disconnecting a computer in this situation is that bad inputs from that computer may also cause a crash. But maybe if no other computer is available to take its place the logic should be different, at least during takeoff/landing.
- avianlyric 5y ago
  Im gonna re-explain what you’ve written in language I’m more familiar with to check my understanding, and hopefully you can correct any errors I make. As I understand it the A330 has three primary flight computers, all observing the same inputs (which might come from different physical sensors monitoring the same thing?) and producing outputs, also know as “orders” for other systems in the plan, like actuators. Of these three computers, one will act as the primary command computer (COM), one as a monitoring computer (MON), the third is spare and normally ignored. Only orders from the COM machine is sent to downstream systems like actuators. There’s a separate watchdog system that monitors the outputs (orders) of both the COM and MON, and if their order values diverge by too much for too long, it shuts down the COM computer and passes control to the MON computer and the spare. As part of this process one of those two computers becomes COM the other MON. I assume the size of the divergence determines how long they can diverge for. A large divergence is only allowable for a very short period of time, and a small divergence is allowed for longer. In addition to all of this, the computers have different operating modes which changes how they respond to inputs. In this case the relevant states are “normal law” for in the air, and “ground law” for on the ground. One things that’s different between these states is how tightly coupled rudder inputs from the pilot are to rudder orders to the actuator. In the air the rudder is less tightly coupled than on the ground. E.g. commanding full left rudder in the air results in less physical movement of the rudder than the same command on the ground. When the plane lands, detected by a pressure sensor in the landing gear, the computers transition from “normal law” to “ground law”. Which for some outputs, like the rudder, might result in a step change in outputs (orders) from the computers. So in this specific scenario what happened is that the flight computers for some reason didn’t transition between “normal law” and “ground law” simultaneously (or close enough to simultaneously). So the COM computer significantly changed its rudder output as a result of changing law, but the MON computer didn’t, because it hadn’t changed law yet (the inverse of this is also possible). As a result they were producing very large differences in rudder orders, resulting in the monitoring watchdog killing the COM computer and failing over to the spare. Where the above situation happened a second time, resulting in all computers being shut down. Is all of the above correct? All of this does make me wonder, if changing law can result in computer outputs quickly changing, doesn’t that make law changes inherently dangerous? If you’re a pilot landing a plane applying significant rudder inputs, doesn’t the above me that those inputs will have a vastly different effect once the wheels touch the runway?
phkahler 5y ago
That's how I read it too. The discrepancy was deemed "I'm broken and need to shut down" instead of something more benign in a condition they didnt realize might happen. That a critical difference since all 3 will think "I'm broken" as happened here. That stuff can be hard to get right. Glad they found this one with 10 meters to spare.
- avianlyric 5y ago
  It seems that flight planning is done on the basis that the loss of all flight computers is possible, and to make sure that your runway is long enough to accommodate that situation. So thankfully the loss of all three computers isn’t inherently dangerous, and is demonstrated by this incident. Based on my reading of the report the primary take away from this (apart from a an issue with the flight computers software), is that flight plans aren’t and prep by the airport wasn’t conservative enough, so they ended up with slightly less safety margin than they expected in this scenario.
  - ncmncm 5y ago
    > loss of all three computers isn’t inherently dangerous You seem to be using a different definition of "dangerous" from the rest of us. Loss of systems that pilots are used to depending on is inherently dangerous. After that, degrees of luck, skill, and conservative planning become major factors in the outcome. The loss of spoilers and reversers is itself dangerous. In slightly different conditions, e.g. snow, the aircraft would not have stopped where it did. There was a Lockheed aircraft that ran off the end of a runway at an above-usual touchdown speed: the spoilers would not go up because for lack of weight on wheels, and brakes had no traction because spoilers were not up.
    - avianlyric 5y ago
      I don’t know what to say. Pilots are trained for this exact situation, flight plans are designed for this situation. Every reasonable measure is taken to ensure that this exact situation isn’t unnecessarily more dangerous than it absolutely needs to be. As proven by this exact incident. You can talk about hypotheticals like snow, but this is a commercial airport. They will either have snow clearing equipment or they’ll redirect planes if they can’t clear the snow. It’s also Taipei where it basically never snows around the city. Simply put if it was unsafe to land this plane in this state with snow (and snow was believed to fall that day), then the plane wouldn’t even take off. Lockheed aircraft aren’t passenger airlines, they have a completely different risk profile and risk appetite. I imagine Lockheed aircraft occasionally get taken down my enemy fire, but we don’t build commercial aircraft to handle that situation.

the_duke 5y ago

Odd that there is no information from Airbus there. FYI: the comment section contains some interesting background.

zymhan 5y ago
Well there is a single bullet point from/about Airbus > 7. Following the occurrence, Airbus reviewed its in-service experience, and confirmed that no other triple PRIM fault at touchdown event had been reported on A330/A340 aircraft family since entry into service. The A330/A340 fleet fitted with electrical rudder has accumulated 8.7 millions of Flight Cycles and 44.3 millions of Flight Hours (in-service data from April 2020).
- gswdh 5y ago
  AKA, it's only happened once so we don't care...
- thomasjudge 5y ago
  So, does this mean that the outcome was, Airbus reviewed the situation and the data, and made no changes or corrections to the system?
  - thefurrysquid 5y ago
    There will be a software update. Read page 9 of the exec summary linked in the article.

ceejayoz 5y ago

> The crew applied maximum manual braking and managed to stop the aircraft 10 meters/33 feet ahead of the runway end (runway length 2600 meters/8530 feet). Hope the crew had their brown pants on.

encoderer 5y ago
From the cockpit, 33’ must have looked right under their nose.
- futhey 5y ago
  Songshan's not a large airport, it's mostly used for domestic flights, and a few nearby international destinations. Single runway, immediate obstacles on both ends (no matter which way you land). Although, runway 10 is eastbound & it's just a fence and small parking lot, minor traffic, and then you land in the river. Sounds like it would have been stressful.

rzzzt 5y ago

Why can't thrust reversers and spoilers be engaged without computer supervision? The article just says that ground spoilers require one of the three flight computers to remain operative, autobrake needs two, reversers needs one out of two specific units to be running to unlock.

bdonlan 5y ago
All flight controls on the A330 are fly-by-wire - they're connected via computers. The most critical surfaces have secondary, dumb controllers that can take over if the primary flight computers fail, but less critical systems may not be connected to those backup systems - it's expected that a triple fault of the primary flight computers should be very rare, and manual braking is available as a backup after all.
- NovemberWhiskey 5y ago
  Right. In this case we have a completely-unprecedented triple failure of the primary flight control system causing loss of auto-brake, spoilers and thrust reversers; a wet runway with less than expected braking action; a tailwind landing and still a happy outcome. Honestly defense-in-depth seems to be working OK here.
  - gsnedders 5y ago
    And note that for all aircraft the landing distance is calculated _without_ thrust reversers or spoilers, and thus is based on braking performance alone. Some of the challenge here was the seconds spent at near landing speed without any braking being performed, as that'll eat up runway distance fast, following the auto brake failure prior to manual braking commencing.
    - azalemeth 5y ago
      It's also worth stating (for the OP) that thrust reversers are potentially really dangerous -- if they deploy in flight without being commanded to, the aircraft can -- or, more likely if the engine computer does not detect it and shut itself down, will become barely controllable, with previous fatal outcomes. [1] Two or more FCPCs failing is sufficient evidence that a "faecal fan incident may be occurring" that the risk of deploying them is just not worth it, especially as (pointed out by this parent above) they are effectively optional equipment and the FAA requires you to expect them to be inop to be safe. What they do save is fuel and time -- taxi time to the terminal, permit the use of high-speed runway turnoffs, etc. [1] https://en.wikipedia.org/wiki/Lauda_Air_Flight_004 https://en.wikipedia.org/wiki/Lauda_Air_Flight_004

sinuhe69 5y ago

Remind me of the Ariane 5 maiden flight: redundant hardwares does not protect against unsanitized inputs. I don't know much about the design of the airbuses, but I think it's sensible to specify the last primary computer running in a different mode as the first two, simply because double hardware failure is very very unlikely. So it boils down to either software bug and or unsanitized inputs. Assuming that, the last computer should provide some basic but safe functions.

strictfp 5y ago
I've seen bad user input, one very unusual char, bring down a whole self-healing geographically redundant cluster of machines running an expensive commercial cluster software. Any connected devices can bring each other down.
- andrewnicolalde 5y ago
  That sounds like a great war story if you’re open to telling it :)
- salawat 5y ago
  Story, story, story, story!
avianlyric 5y ago
Well that’s kind of what happened. The primary flight computers all failed, to the plane fell back to “direct law”, which means pilot input is directly transmitted to control systems without a computer attempting to interpret intent or protect against pilot error. As a consequence a lot of the more advanced functions became disabled because they rely on the primary flight computer to indicate when it’s safe for them to operate. Presumably because accidental operation of those features in flight is extremely dangerous.

oakmad 5y ago

For everyone commenting about different hardware failure modes and possible solutions. Take a look at the airbus « flight control laws » such as https://apstraining.com/wp-content/uploads/FCS-Airbus-Quick-Reference.pdf https://apstraining.com/wp-content/uploads/FCS-Airbus-Quick-... it governs what happens to flight controls and what protections are available in different scenarios. In this case the system worked, it crashed and reverted to direct law providing manual braking control. Off topic but I found this very interesting « The deceleration performance of the occurrence flight between 6,600 feet and 7,300 feet from the threshold of runway 10 deteriorated. It may be due to paint marking and rubber deposit on the touchdown zone of runway 28. »

cwizou 5y ago
> The deceleration performance of the occurrence flight between 6,600 feet and 7,300 feet from the threshold of runway 10 deteriorated. It may be due to paint marking and rubber deposit on the touchdown zone of runway 28 Just to clarify for others, this is the same runway but taken in two different directions (west to east and east to west, multiply the number by 10 and you get [rounded] angle in degree, so 100ish vs 280ish). It's 8500 ft. long and it looks like there was a loss of performance around the "touchdown" point of going the other way on that runway, where you had accumulated rubber from those landings (which is pretty normal and monitored, as pointed by leecb below), if that makes any sense.
- leecb 5y ago
  > accumulated rubber from those landings Rubber accumulation on runways is a normal and expected situation. Part of runway maintenance at airports serving larger aircraft includes regular removal of the rubber using high pressure water or chemicals. https://en.wikipedia.org/wiki/Airfield_rubber_removal https://en.wikipedia.org/wiki/Airfield_rubber_removal
  - salawat 5y ago
    Not going to lie, I never thought this was a thing that'd need doing, but on further reflection it seems so obvious I'm kinda mad it never occurred to me in retrospect.

teeray 5y ago

I wonder what would have happened if they tried to go around? The TO/GA is configured ahead of time in the FMS… so if the flight computers are out, I wonder if there’s some hard-coded performance targets for the engines in a go around situation.

concerned_user 5y ago
They would push throttle manually in max power and perform the maneuver by hand.
deleted 5y ago
[deleted]
hugh-avherald 5y ago
There was barely a second between the first indication of a primary computer fault and the call for reversers to deploy. Once you select reverse thrust, you cannot go around under any circumstances.
- strogonoff 5y ago
  It’s an interesting scenario. Your wheels touch the ground; you engage reverse thrust and it instantly fails; you are rolling at dangerous landing speed but now can’t go around? Relatedly, I wonder why did the pilots not engage the regular brakes ASAP.
  - avianlyric 5y ago
    Not quite, reading around it seems that flight planning requires you to assume that reverse thrusters and ground spoilers don’t work. So you land, your reverse thrusters immediately fail, and you’re now required to stop using your manual breaks and consume all of the breaking distance you originally planned for, with presumably a bit of margin on top.

bryan0 5y ago

The 2020 should be removed from the title. The article is mostly new information from the final report: > On Sep 3rd 2021 Taiwan's ASC released their final report in Chinese and their English Executive Summary concluding the probable causes of the incident were: …

FatalLogic 5y ago

The flight data recorder chart on page 114 of the Chinese-language report[0] seems to record that the pilots hit the brakes very hard in the last few seconds, maybe when they realized they were close to overrunning the runway. That caused a sudden peak of -0.47g deceleration, so it looks as if they could have used the brakes to slow the plane earlier on the runway, but maybe they didn't because they were expecting they could sort out the problem with their spoilers and reverse thrust and stop it in the normal way, without giving the passengers a shock. [0]https://www.ttsb.gov.tw/media/4912/ci-202-final-report_chinese_release.pdf https://www.ttsb.gov.tw/media/4912/ci-202-final-report_chine... edit: that chart's x-axis is the aircraft's position on the 8500ft-runway, and most of the recording traces seem to begin at the point when it had touched down finally with all wheels

ysangkok 5y ago
> hit the brakes very hard in the last few seconds In direct law, is it possible to brake too hard and have the tires skid on the runway? Is it possible that the pilots were afraid of this? I ask because people are characterizing direct law as "less smart". So how smart is it? Smart enough to include ABS?
- FatalLogic 5y ago
  I don't know, but the computer had failed, the runway was wet, and they were confused, so, yes, it's not unlikely they were also afraid of skidding
- oakmad 5y ago
  Oh 100%! Anti skid is only available in alternate and normal laws. Section 6.5 braking system of this document. https://www.smartcockpit.com/docs/A330_Flight_Deck_and_Systems_Briefing_For_Pilots.pdf https://www.smartcockpit.com/docs/A330_Flight_Deck_and_Syste...
  - iso8859-1 5y ago
    But that page does not mention direct law. Wouldn't anti-skid protection depend on the switch shown on the landing gear panel shown in 6.9?

pseingatl 5y ago

Airbus: what is it doing now?

aomobile 5y ago

I wonder how the pilots would have reacted if the computers had crashed 10-20 seconds earlier. Would they have landed or would they go up again to wait for reboot?

bonzini 5y ago
Difficult to say, since the computer crash was linked directly to having just touched down. 10-20 seconds before there was no disagreement because neither computer has detected that the wheels were on the ground.

fnord77 5y ago

> The root cause was determined to be an undue triggering of the rudder order COM/MON monitoring concomitantly in the 3 FCPC. At the time of the aircraft lateral control flight law switching to lateral ground law at touch down, the combination of a high COM/MON channels asynchronism and the pilot pedal inputs resulted in the rudder order difference between the two channels to exceed the monitoring threshold. The FCPC1 failed first. In lay-programmers terms, this sounds like a race condition. Is that correct?

zaarn 5y ago
Not quite. The computers disagreed, they didn't race. Likely cause might be that the sensors responsible for detecting touchdown might have been rusty or failed. The computers aren't racing towards ground law here, they switched to ground law at different times sufficiently apart that the monitoring watchdog turned them all off for commanding the aircraft to do three different things. Imagine it more like website and it's hosted by three servers with a replicated database. A monitoring tool checks that the number of entries in each server's database is the same. If one server has a different number, the tool turns it off, it now requires a human to bootstrap the database again. If two servers disagree the website goes offline, because it's no longer possible to tell which server has correct data and not operating is cheaper than the man hours required to puzzle the data back together. In this case, a likely cause might be that suddenly the network latency between the three servers got suddenly very large (cable damage maybe?) and now all three no longer agree on the dataset. The servers are shutdown.

5faulker 5y ago

Good thing that this was on a smaller airplane, otherwise it could have been a larger disaster.

dangs_penis2 5y ago
Wow! Thanks for the insight!

danielschonfeld 5y ago

Feels like there might be more to the story than is being let on. Curious why the PF has to specifically ask before landing for the spoilers to be called, to tell if he's touched down. AFAIK it's actually common Airbus procedure to called Spoilers and Reversers and deceleration after landing. Also telling whether or not you touched down isn't really a problem... even for the smoothest/softest of landings. Then there's the PM calling out "center line" at 130 feet...

dangs_penis2 5y ago

HackersNews is a experiment.

jimnotgym 5y ago

I note there was a tailwind serious enough to reduce breaking performance, and also that it is possible to land in the opposite direction on the runway in question. Is the practice of landing into the wind ignored within certain parameters?

gsnedders 5y ago
In general, aircraft are certified to land/takeoff with tailwinds (and crosswinds, etc.) up to a certain speed. You don't want to be changing runway direction all the time, after all. Ultimately, it's up to the crew on board to determine the landing distance required for the current conditions (including the wind—whatever direction that is!), and if they don't believe the conditions are safe they should not proceed with the landing. They can, of course, request the opposite runway, though obviously if there is frequent traffic ATC is likely to refuse unless they want to change the direction for all traffic.

PostThisTooFast 5y ago

"undue triggering of the rudder order COM/MON monitoring concomitantly in the 3 FCPC. At the time of the aircraft lateral control flight law switching to lateral ground law" Inexcusable jargon and bullshit. Whoever was paid to generate this report should be fired.

ddalex 5y ago

Commercial piloting is hours of boredom with few moments of raw sheer terror.

tim333 5y ago

Good thing they failed on touchdown and not up in the air.