7 ms·
I’d really, really like to know what microcontroller family this was found on. Assuming that this is a safety processor (lockstep, ECC, etc) it suggests that EC
by addaon 11mo ago
I’d really, really like to know what microcontroller family this was found on. Assuming that this is a safety processor (lockstep, ECC, etc) it suggests that ECC was insufficient for the level of bit flips they’re seeing — and if the concern is data corruption, not unintended restart, it means it’s enough flips in one word to be undetectable. The environment they’re operating in isn’t that different from everyone else, so unless they ate some margin elsewhere (bad voltage corner or something), this can definitely be relevant to others. Also would be interesting to know if it’s NVM or SRAM that’s effected.
- jayanmn 11mo agoI am worried about a software fix for what looks like hardware problem.
- deleted 11mo ago[deleted]
- afavour 11mo agoIt could be as simple as storing multiple copies of the relevant data and adding a checksum, something like that. Hardware fix is the ultimate solution but it might be possible to paper over with software.
- deleted 11mo ago[deleted]
- kachapopopow 11mo agosoftware fixes are totally fine since the chance of two redundant pairs failing within the time it takes to correct these errors is more zero's than there are atoms in the universe. (each pilot has a redundant computer and because there's two pilots there's two redundant pairs)
- themerone 11mo agoGracefully handling hardware faults is a software problem. The Air France Flight 447 crash was the result of bad software and bad hardware.
- vel0city 11mo agoI'm reminded of the Apollo moon landing where the computer was rapidly rebooting and being in an OK-ish state to continue to be useful almost immediately
- CrossVR 11mo agoIt wasn't rebooting, it ran out of memory and started aborting lower priority tasks. It was a excellent example of robust programming in the face of unexpected usage scenarios.
- f1shy 11mo agoOf topic for the thread, but on for the comment: I was working in an automotive project 3 years ago. It was all about safety, and one hypothesis was the processor could get overloaded. I was astonished no one in a grouo of 20 “senior sw architecs” had any idea about the concept of load shedding. The proposed solution was “in that case, reboot”. Mind you whatever came out of that project is rolling on the street today.
- concinds 11mo agoWe really should mandate all that stuff to be open-source, so we can be aware of how defective everything is.
- K0balt 11mo agoFail safe/fail soft I still design this into many of the things I work on, especially if I’m working close to the metal on controller systems. At some point it becomes ridiculous / impossible but I’m often thinking about how a system would handle memory corruption, bit flips, invalid sensor data, etc. These days, somebody should design a triple redundant microcontroller that runs quorum on the gpio at the hardware level. It could be a 0.30 part instead of 0.10 one, but I would specify it just about everywhere. Add $3 to BOM cost to categorically eliminate an entire class of failure would be ramrodded by legal into just about every medical device, PLC, critical automotive system, etc one would think. Seems like a good gambit for a riscV startup, but what do I know.
- willis936 11mo agoIt's a system problem. The system is being fixed.
- TehCorwiz 11mo agoAn early revision of the Raspberry Pi 2 would crash if you hit it with a bright light like a camera flash. Specifically a xenon flash. https://forums.raspberrypi.com/viewtopic.php?t=99167 https://forums.raspberrypi.com/viewtopic.php?t=99167 https://forums.raspberrypi.com/viewtopic.php?f=28&t=99042 https://forums.raspberrypi.com/viewtopic.php?f=28&t=99042 https://www.raspberrypi.com/news/xenon-death-flash-a-free-physics-lesson/ https://www.raspberrypi.com/news/xenon-death-flash-a-free-ph... https://www.youtube.com/watch?v=wyptwlzRqaI https://www.youtube.com/watch?v=wyptwlzRqaI
- mlyle 11mo agoYah, but that's a case of the package not being opaque enough.
- russdill 11mo agoCompletely unrelated and due to a design failure by the rpi folks.
- hughw 11mo agoIs it really so unrelated? Isn't it a case where a similar phenomenon -- radiation impacting a computer calculation -- happened and it's one we can all relate to more easily, and reproduce if we cared to, than high altitude avionics? Not necessarily disputing but it just seems like a relatable case that helps me understand the issue better. If it's a radically different case somehow I'm interested to learn.
- anonymousiam 11mo agoproper SEU mitigation goes far beyond ECC. Satellites fly higher than the A320, and they (at least the ones I know about) use Triple Modular Redundancy: https://en.wikipedia.org/wiki/Triple_modular_redundancy https://en.wikipedia.org/wiki/Triple_modular_redundancy https://en.wikipedia.org/wiki/Single-event_upset https://en.wikipedia.org/wiki/Single-event_upset For manned spaceflight, NASA ups N from 3 to 5. Other mitigations include completely disabling all CPU caches (with a big performance hit), and continuously refreshing the ECC RAM in background. There are also a bunch of hardware mitigations to prevent "latch up" of the digital circuits.
- rkagerer 11mo agoIn redundant systems like these, how do you avoid the voting circuit becoming a single point of failure? Eg. I could understand if each subsystem had its own actuators and they were designed so any 3 could aerodynamically override the other 2, but I don't think that's how it works in practice.
- AlphaSite 11mo agoVoting can be coordinated between the N cpus rather than an external arbiter (even making that redundant eventually required the CPUs to decide what to do if they disagree so may as well handle it internally).
- exe34 11mo agoif the issue is radiation bit flipping, you could make that part overly shielded?
- RealityVoid 11mo agoSee my other comments in the other threads. This does not have EDAC. I was as surprised as you but it doesn't seems to be an MCU but a composition of several distinct chips. That flight computer was designed in the 90's and updated in 2002 with a new hw variant that does have edac. So yes, for this kind of thing, I can buy that a bit flip happened. You can see much more data in the report: https://www.atsb.gov.au/sites/default/files/media/3532398/ao2008070.pdf#%5B%7B%22num%22%3A837%2C%22gen%22%3A0%7D%2C%7B%22name%22%3A%22FitH%22%7D%2C207%5D https://www.atsb.gov.au/sites/default/files/media/3532398/ao...
- Reason077 11mo agoThe recalled aircraft include the latest A320neo model, some of which are basically brand new. Why would they be using flight computers from before 2002? Why is an old report from 2008, relating to a completely different aircraft type (A330), relevant to the A320 issue today?
- RealityVoid 11mo agoBecause the problem isn't just this. It's that the flight controller did not properly decide what to do when the data spiked because of this issue as well.
- Havoc 11mo ago> Why would they be using flight computers from before 2002? Guessing that using previously certified stuff is an advantage
- LiamPowell 11mo ago> Why would they be using flight computers from before 2002? Why would you assume they're not? I don't know about aircraft specifically, but there's plenty of hardware that uses components older than that. Microchip still makes 8051 clones 45 years after the 8051 was released.
- K0balt 11mo agoThat’s just wild to think about. We should all strive to build solutions that plague our descendants with their persistent utility.
- asdefghyk 11mo agoRE "...The environment they’re operating in isn’t that different from everyone else...." NO this is incorrect. High flying aircraft more likely to suffer increased radiation caused by 11 year peak sunspot cycle . such aircraft should be using "radiation hardened electronics" , somewhat like spacecraft use...