5 ms·

On the Qantas 72 flight (2008), the ATSB report showed the same power spike that upset the ADIRU also left tidy 1-word corruptions in the flight data recorder.

On the Qantas 72 flight (2008), the ATSB report showed the same power spike that upset the ADIRU also left tidy 1-word corruptions in the flight data recorder. Those aligned with the clock cycle, shared the same amplitude and were confined to single ARINC words. That is pretty much exactly the signature of a failing solid state relay or contactor on the shared avionics power bus (upstream of both FDR and fly by wire). Radation-driven bit flips would be Poisson distributed in time and energy. So that is one way to find out

kranke155 11mo ago
What do you think it could be ?
- dmbche 11mo ago
  "That is pretty much exactly the signature of a failing solid state relay or contactor on the shared avionics power bus (upstream of both FDR and fly by wire)."
  - kranke155 11mo ago
    Thanks. I didn’t realise what that meant in context.
captainkrtek 11mo ago
Do you think they're using the guise of "its solar radiation" as cover to do a software update to fix a more problematic "bug", and perhaps tangentially there are some changes in said-update to improve some error correcting type code (eg: related to detecting spurious bit flips).
- fooker 11mo ago
  Yeah I don't buy it either. If it was really 'solar radiation' there would be more small details.
  - chasing0entropy 11mo ago
    I would say its pretty detailed -an unknown interference caused a single crc protected 32 bit word to be corrupted simultaneously, by timestamp, in both the flight controller hardware and the black box data recorder. My concern would be what error correction mechanism did or did not catch the corruption in memory and why did it not recover without critical impact to operations?
    - fooker 11mo ago
      > corrupted simultaneously This sounds like a software bug. Something like - {copy a to b, checksum a--b} Instead of - {copy a to t, checksum a--t, copy t to b, checksum a--b} I bet the fix is along these lines, with the caveat of real time systems/etc.
  - dboreham 11mo ago
    Reading the Airbus press release, I wonder if this is what happened: Solar radiation event led to alpha particle induced data corruption in a flight control computer memory (could be DRAM, SRAM, on-chip cache, registers...). These failures are supposed to be transient (reboot and all is well). This is an anticipated failure mode. Only one (of three?) computers should be affected by such a failure and therefore the remaining two keep on running the plane. But what happened is <something> went wrong with the failover/voting mechanism (as often happens with one-off seldom-executed failover code). The result was no flight control computer functionality until the entire system was rebooted. Hence the emergency landing. The fix is to address that software error, with perhaps a secondary fix TBD to harden the hardware (add some shielding perhaps). The fact that they talk about data corruption and not just a malfunction suggests alpha bit flip rather than latch-up. Then send the whole statement through a French to English translator to make it a bit more confusing.
  - londons_explore 11mo ago
    My guess is they haven't managed to point to the single memory bit which was flipped to cause this result. The software update is probably more along the lines of 'lets just introduce a watchdog task which resets the system if the output deviates too far from the input for too long'.
- aunty_helen 11mo ago
  No, that would be straight to jail.
  - vlovich123 11mo ago
    Remind me who from Boeing went to jail?
    - stackghost 11mo ago
      Airbus is in Europe where the Rule of Law still exists
      - vlovich123 11mo ago
        That’s what we naively thought here too.
        kakacik 11mo ago
        Look at how US government treats financial behemoths which actively harm whole mankind vs how EU treats them. There is way more to this topic obviously (who wants to harm their local company), but generally US is pro-companies while Europe is pro-people.
        vlovich123 11mo ago
        Deutsche Bank and HSBC, two major European banks, have repeatedly admitted they have engaged in money laundering activities for Russia, drug cartels and terrorists and have consistently failed to meet their AML obligations. The US is the only entity that’s going after these banks for these issues winning significant judgments and even with that backdrop you don’t see any EU enforcement.
- maximilianburke 11mo ago
  Not in aviation.
  - Bratmon 11mo ago
    Does the 737-Max not count as aviation anymore?
    - edoceo 11mo ago
      It does. It is but the Max issue was well different to this one.
  - superxpro12 11mo ago
    Counterpoint: Boeing MCAS tho
- schmuckonwheels 11mo ago
  No, because aerospace is not garden-variety Silicon Valley webshittery. There is a slightly different level of discipline and engineering ethics at play.
RealityVoid 11mo ago
After reset, it went away. If it was this kind of hw issue, it should still be present. Considering those units were designed back when they did not have EDAC mandated, I can believe it could have been a bit flip (along with some other stuff they will probably address to take into consideration this failure mode). Nowadays, most MCU's have ECC on them so the time of this excuse is mostly gone now. :)
- dylan604 11mo ago
  > Nowadays, most MCU's have ECC on them so the time of this excuse is mostly gone now. :) That's kind of a misleading statement. Assuming you mean on planes built nowadays, as we clearly see that nowadays planes still flying (6K of them at least) still have issues. We don't need hand wavy comments trying to make it sound like modern day aviation is no longer susceptible, especially when it's in a thread on an article showing how that's just not true
  - alwa 11mo ago
    I think you and gp may be speaking about different stages. Gp seems to be saying that a plane being designed and specified today would use technologies hardened against this type of error. That even though they’re in widespread operation today, the aircraft types in question were designed (and certified) many years ago, before ECC was the norm. My impression is that, once their type is certified, new airframes are built to pretty much exactly that specification even all these years later.
    - dylan604 11mo ago
      > I think you and gp may be speaking about different stages Yes, that's my point. Just because new aircraft are designed with improved hardware does not automatically mean the issue is resolved industry wide. Existing equipment will still have issues. So the statement is misleading. Is the number of aircraft with ECC "most" of the equipment in the skies?
      - 11mo ago
londons_explore 11mo ago
I'm very surprised that a plane doesn't have voltage, current and glitch monitoring on every power rail, logged to the data recorder. You would pretty much be logging, every millisecond, the minimum, maximum and mean voltage for every 1ms period (and the same for current). Then any failing solid state relay would be obvious in the collected data, far before you start to get word corruption!

kranke155 11mo ago

What do you think it could be ?

dmbche 11mo ago
"That is pretty much exactly the signature of a failing solid state relay or contactor on the shared avionics power bus (upstream of both FDR and fly by wire)."
- kranke155 11mo ago
  Thanks. I didn’t realise what that meant in context.

captainkrtek 11mo ago

Do you think they're using the guise of "its solar radiation" as cover to do a software update to fix a more problematic "bug", and perhaps tangentially there are some changes in said-update to improve some error correcting type code (eg: related to detecting spurious bit flips).

fooker 11mo ago
Yeah I don't buy it either. If it was really 'solar radiation' there would be more small details.
- chasing0entropy 11mo ago
  I would say its pretty detailed -an unknown interference caused a single crc protected 32 bit word to be corrupted simultaneously, by timestamp, in both the flight controller hardware and the black box data recorder. My concern would be what error correction mechanism did or did not catch the corruption in memory and why did it not recover without critical impact to operations?
  - fooker 11mo ago
    > corrupted simultaneously This sounds like a software bug. Something like - {copy a to b, checksum a--b} Instead of - {copy a to t, checksum a--t, copy t to b, checksum a--b} I bet the fix is along these lines, with the caveat of real time systems/etc.
- dboreham 11mo ago
  Reading the Airbus press release, I wonder if this is what happened: Solar radiation event led to alpha particle induced data corruption in a flight control computer memory (could be DRAM, SRAM, on-chip cache, registers...). These failures are supposed to be transient (reboot and all is well). This is an anticipated failure mode. Only one (of three?) computers should be affected by such a failure and therefore the remaining two keep on running the plane. But what happened is <something> went wrong with the failover/voting mechanism (as often happens with one-off seldom-executed failover code). The result was no flight control computer functionality until the entire system was rebooted. Hence the emergency landing. The fix is to address that software error, with perhaps a secondary fix TBD to harden the hardware (add some shielding perhaps). The fact that they talk about data corruption and not just a malfunction suggests alpha bit flip rather than latch-up. Then send the whole statement through a French to English translator to make it a bit more confusing.
- londons_explore 11mo ago
  My guess is they haven't managed to point to the single memory bit which was flipped to cause this result. The software update is probably more along the lines of 'lets just introduce a watchdog task which resets the system if the output deviates too far from the input for too long'.
aunty_helen 11mo ago
No, that would be straight to jail.
- vlovich123 11mo ago
  Remind me who from Boeing went to jail?
  - stackghost 11mo ago
    Airbus is in Europe where the Rule of Law still exists
    - vlovich123 11mo ago
      That’s what we naively thought here too.
      - kakacik 11mo ago
        Look at how US government treats financial behemoths which actively harm whole mankind vs how EU treats them. There is way more to this topic obviously (who wants to harm their local company), but generally US is pro-companies while Europe is pro-people.
        vlovich123 11mo ago
        Deutsche Bank and HSBC, two major European banks, have repeatedly admitted they have engaged in money laundering activities for Russia, drug cartels and terrorists and have consistently failed to meet their AML obligations. The US is the only entity that’s going after these banks for these issues winning significant judgments and even with that backdrop you don’t see any EU enforcement.
maximilianburke 11mo ago
Not in aviation.
- Bratmon 11mo ago
  Does the 737-Max not count as aviation anymore?
  - edoceo 11mo ago
    It does. It is but the Max issue was well different to this one.
- superxpro12 11mo ago
  Counterpoint: Boeing MCAS tho
schmuckonwheels 11mo ago
No, because aerospace is not garden-variety Silicon Valley webshittery. There is a slightly different level of discipline and engineering ethics at play.

RealityVoid 11mo ago

After reset, it went away. If it was this kind of hw issue, it should still be present. Considering those units were designed back when they did not have EDAC mandated, I can believe it could have been a bit flip (along with some other stuff they will probably address to take into consideration this failure mode). Nowadays, most MCU's have ECC on them so the time of this excuse is mostly gone now. :)

dylan604 11mo ago
> Nowadays, most MCU's have ECC on them so the time of this excuse is mostly gone now. :) That's kind of a misleading statement. Assuming you mean on planes built nowadays, as we clearly see that nowadays planes still flying (6K of them at least) still have issues. We don't need hand wavy comments trying to make it sound like modern day aviation is no longer susceptible, especially when it's in a thread on an article showing how that's just not true
- alwa 11mo ago
  I think you and gp may be speaking about different stages. Gp seems to be saying that a plane being designed and specified today would use technologies hardened against this type of error. That even though they’re in widespread operation today, the aircraft types in question were designed (and certified) many years ago, before ECC was the norm. My impression is that, once their type is certified, new airframes are built to pretty much exactly that specification even all these years later.
  - dylan604 11mo ago
    > I think you and gp may be speaking about different stages Yes, that's my point. Just because new aircraft are designed with improved hardware does not automatically mean the issue is resolved industry wide. Existing equipment will still have issues. So the statement is misleading. Is the number of aircraft with ECC "most" of the equipment in the skies?
    - 11mo ago

londons_explore 11mo ago

I'm very surprised that a plane doesn't have voltage, current and glitch monitoring on every power rail, logged to the data recorder. You would pretty much be logging, every millisecond, the minimum, maximum and mean voltage for every 1ms period (and the same for current). Then any failing solid state relay would be obvious in the collected data, far before you start to get word corruption!