8 ms·
Software bug made Bombardier planes turn the wrong way
- redis_mlc 6y agoI understand the bug, and it's one of the worst imaginable. Doing an unauthorized departure turn can impact terrain. At night, it may not even be noticed by the pilots. Source: commercially-rated pilot.
- speeder 6y agoIt wasn't because software bug, but this is how "Mamonas Assassinas" died, basically during a missed approach the pilot turned the wrong direction (but with the correct radius and all) and crashed into a very tall hill near the airport.
- pravda 6y agoAh yes. That was a dark day. Well, night actually. https://www.youtube.com/watch?v=PwXplj3ssNs https://www.youtube.com/watch?v=PwXplj3ssNs
- zomglings 6y agoCould you explain the bug here? It wasn't clear to me what the exact problem was from reading the article, just that it occurred under a very specific and uncommon set of circumstances.
- NikolaeVarius 6y ago> "This issue will occur in departures and missed approaches where the shortest turn direction is different than the required turn direction onto the next leg if the crew edits the 'Climb to' altitude field." > The FMS may change the planned database turn direction to an incorrect turn direction when the altitude climb field is edited.”
- zomglings 6y agoThanks. The summary is actually very useful. What I mean, though, was I didn't understand what the bug was - not how it manifested.
- NikolaeVarius 6y agoThe bug is that the flight computer could set an incorrect turn direction for a go around abort maneuver/take off procedure. https://portal.rockwellcollins.com/documents/796122/0/OPSB+Rev4.pdf/0a43ef78-ad73-5e84-03eb-b805f6edfc18 https://portal.rockwellcollins.com/documents/796122/0/OPSB+R...
- mjg59 6y agoThere's two components in a turn - the desired heading at the end of the turn, and the direction you should turn to get there. Setting the "Climb to" altitude appears to have cleared the turn direction information. In the absence of that, the computer will turn in whichever direction results in a shorter turn to the desired heading. This is usually what you want, but not always, so I can understand it taking a while before anyone noticed it.
- mopsi 6y agoAirports have landing procedures, essentially a list of waypoints and altitudes. A landing aircraft has to fly that route down to the runway. Each list has Missed Approach Point, at which the list branches into two: landing or abort. If you are not ready to land at that point (too fast, can't see runway, previous aircraft still on the runway, etc), then you fly the abort part. Usually it tells to climb to a certain safe altitude and turn towards a waiting area for another landing attempt. These procedures can be flown manually, or activated for autopilot to fly. A bug made the autopilot turn in opposite direction to what's in the abort section. Here's a landing procedure for Helena regional airport: https://flightaware.com/resources/airport/HLN/IAP/ILS+OR+LOC+Y+RWY+27/pdf https://flightaware.com/resources/airport/HLN/IAP/ILS+OR+LOC... The narrowing beam is instrument landing system that guides you towards the runway. If you reach 4580 feet minimum altitude during approach, but can't see the runway, you must fly the abort procedure, which is drawn with dashed lines: climb immediately to 4700 feet on current heading, keep climbing to 9000 while turning to heading 021, then proceed north at 9000 on 336 degree radial from Helena radio beacon, and upon reaching waypoint WOKEN circle until further instructions. This bug could cause the aircraft turn southwest (towards mountains) instead of northeast (valley). At Helena, the bug would not reveal itself because the right turn is 114 degrees. If the procedure required to turn more than 180 degrees right, for example, 200 or so degrees right towards SWEDD, the aircraft would make a left (shortest) turn instead. Green should be flown, red would be flown: https://i.imgur.com/ojShQa2.png https://i.imgur.com/ojShQa2.png
- lutorm 6y agoIt seems pretty far from "worst imaginable". I mean, it's not like it goes into an uncontrollable dive.
- redis_mlc 6y agoPassengers fear falling out of the sky, but that's a pretty rare thing with airplanes. A dive at altitude would give you time to react, usually minutes. Hills are invisible in the dark. Also, charted departure instructions are something you bet your life on, as well as your passengers. So if you don't trust the plates or FMS, you can't fly in IMC or at nite.
- dehrmann 6y agoIf you still have control surfaces, is a dive at sufficient altitude ever uncontrollable? Flat spins are way scarier.
- redis_mlc 6y agoThe main issues are: - dive with power causing overspeed - dive causing controls to lock due to transsonic shock waves - improper recovery from a spiral dive will over-G - MCAS-style confusion - bottoming out on a phugoid oscillation - unrecoverable spins, usually in jets But in general, if you unload the wings (reduce G), then nothing breaks.
- kohtatsu 6y agoI just flew out of this airport yesterday, only 5 passengers onboard a Bombardier Q400. Thankfully there are no nearby hills for this bug to kill anyone there. Unrelated, but how many carbon offsets do I buy?
- ashtonkem 6y agoFor a mostly empty flight? A lot.
- markvdb 6y agoYou'd have consumed about 0.45 l/km of kerosene per flight km per passenger. Calculation based upon [0]. That means you'd need to offset CO² emissions of about 1.11 kg CO² per flight km [1] for yourself. [0] https://www.flyradius.com/bombardier-q400/fuel-burn-consumption https://www.flyradius.com/bombardier-q400/fuel-burn-consumpt... [1] https://www.engineeringtoolbox.com/co2-emission-fuels-d_1085.html https://www.engineeringtoolbox.com/co2-emission-fuels-d_1085...
- kohtatsu 6y agoWow that's great, thank you for finding these and doing those calculations. I'm glad to read it's one of the more fuel efficient aircrafts. Flightaware puts the actual distance at 760km, so it'd be ~844kg of CO². I found this great guide by the David Suzuki Foundation which assesses different carbon offset vendors with a few different measures: https://davidsuzuki.org/wp-content/uploads/2019/10/purchasing-carbon-offsets-guide-for-canadians.pdf https://davidsuzuki.org/wp-content/uploads/2019/10/purchasin... Pages 42-49 go into the different criteria, Page 50 is the table scoring them all. I decided to go with https://carbonzero.ca https://carbonzero.ca, it was $19.89CAD for 0.88t. Thank you for your help :)
- deleted 6y ago[deleted]
- parkovski 6y agoThis reminds me of a meetup I attended last fall, they were talking about the Spectre/Meltdown issues. I asked the presenters if anything in chip manufacturing/verification processes had changed as a result of that and they seemed surprised. To me, when a software bug shows up in a critical system, that means you actually have a logistics bug. Airplane control software should not be allowed to have bugs. CPUs should not be allowed to have bugs. And OS's should not be allowed to crash (looking at you Microsoft). When one of these things happens, in my opinion the correct response is _not_ to just release fixes and workarounds and then say "we'll try really hard to not let it happen again." You do that, sure. But the first time you see airplane software malfunction, that means you need to change the way the software is written and released so that the whole class of issues will not ever happen again. You don't stop at a public apology, you don't fire the person that unintentionally wrote the bug. If you have to hire mathematicians to formally prove the critical paths of the software, you do that. If it costs 10x more to release bug-free software, oh well, you do that. All of these corporate people thinking they can save money by spending less on quality are extremely naive. You can do a financial analysis of this, but they're doing it wrong. Did you ever consider what the cost of a whole generation just not trusting air travel at all would be?
- nickff 6y agoAre you willing to pay 10x more for the product with that supposed extra reliability (100% vs 99.99966%)? Before you answer, you must remember that perfection cannot be proven ex-ante, it can only be assured. You should also keep in mind that real systems have fault modes aside from software bugs and hardware glitches, such as unanticipated edge cases and user error, which may dominate your actual failure statistics.
- na85 6y ago>But the first time you see airplane software malfunction, that means you need to change the way the software is written and released so that the whole class of issues will not ever happen again. This is pretty good intuition but often a systemic change is not economically feasible. For avionics software at least, a rewrite of the software would likely have to be recertified from scratch before it would be allowed to fly. We do, however, have several different quality assurance programs in Aerospace that are supposed to address this sort of thing. Once you identify the root cause, the process found to be deficient is supposed to have a Process Owner who is required to create a preventive and corrective action plan to prevent a recurrence, with more severe problems requiring more robust action plans. Done right, the process owner is supposed to be empowered to make the changes that need to be made. These systems tend to be evolutions of ISO 9000 as pioneered by Toyota (IIRC). They are highly bureaucratic and soul-sucking, but they are also the least-shitty solution that's been tried.
- thePunisher 6y agoI keep noticing that more and more aviation and space missions fail because of software problems. It seems to me that the new generation of engineers are generally less competent or companies see software as an afterthought which can be outsourced to lower-wage countries. The Boeing MAX and Starliner come to mind, but the failed Moon missions by Israel and India are also examples of this trend. Cost cutting in software development is costing companies dearly. Boeing may even go bankrupt because of this.
- aidenn0 6y agoDon't underestimate the fact that there is a higher volume of software though. Many things that used to be done by analog computers or manually done by the pilot are now done in software. In addition, the Boeing MAX was a largely system design issue; the software was operating as-designed, and had it been implemented in hardware, it would have likely failed in the same manner.
- Jtsummers 6y agoIt's a hiring problem. They let go of the good and/or experienced engineers in the 00s, then replaced them primarily with EEs (as a computer scientist I was told I couldn't write code, only test it, at my first job, I did not stay long) with minimal programming experience. These were very compliant people, happy to do 60 hours or more per week (work harder, not smarter). They lacked the historical context of the systems they were maintaining/developing, and the experience to properly model the systems under development [0]. This hiring problem is compounded by the oversight problem. The program managers are similarly inexperienced. Or they came from strictly a testing side with no concept of what software development itself entails (I've seen this a lot). So they aren't bad at managing requirements, they may actually be really good at it, but they absolutely fail to understand that software is a hard problem (especially when dozens of subcontractor are involved) that extends beyond just the technical problem, and to the communication and coordination problem. That's assuming they're experienced, USAF program managers for software (IME) are straight out of college history majors. DoD programs are scary. [0] Most avionics systems, in my experience, boil down to rather straightforward state machines. Understood this way they become much simpler to write and test. The hard part is hitting your timing constraints, but that's easier to achieve with correct-but-slow-and-maintainable code than with incorrect-but-fast-and-unmaintainable code. Inexperienced developers won't see this possibility, either by failing to spend time studying the requirements or failing to understand how to implement state machines at all.
- trhway 6y ago>Most bugs in airliners tend to be unforeseen memory overflows the 21st century, planet Earth.
- 908B64B197 6y agoTurning off the feature doesn't sound so bad considering the CRJ-200 first flew in 1991, it took 26 years to identify the bug so I assume it's not used frequently at all.
- throwanem 6y agoIt's an avionics bug, so very likely the affected equipment is aftermarket.
- MaxBarraclough 6y agoHow's that? Are most avionics issues due to aftermarket equipment?
- throwanem 6y agoI don't know about that, but it'd be a surprise to see an airframe of that age in commercial service that hadn't had its avionics upgraded, since that's a relatively simple way to gain new flight management capabilities. It'd also be a surprise to see so severe a bug go undetected for so long, if it was part of the original equipment.
- dghughes 6y agoFour months ago there was a post of another article from theregister.co.uk about another bug. The cockpit/flight deck displays went blank if a Boeing 737 landed at any runway that was oriented at 270 degrees true. https://news.ycombinator.com/item?id=21991087 https://news.ycombinator.com/item?id=21991087
- t0mas88 6y agoThat sounds more dangerous than this one, but flying the wrong missed approach or departure is far more dangerous than any display/instrument failure because the crew wouldn't be aware of the problem. If the flight display fails you look at the backup instrument and keep flying, every crew is trained for that. But if the missed approach or departure procedure is wrong the crew has a high probability of not noticing it (this all happens in a very high workload situation). If you don't notice you can't fix it. What makes it worse is that this bug happens in situations whether the procedure requires a turn "the long way around", they wouldn't design the procedure that way unless it's really necessary. So there is a big chance there is terrain or an obstacle on the other side. Source: I'm a commercial pilot
- kayfox 6y agoOddly enough, I think the software in both cases is written by the same company: Collins Aerospace.
- abductee_hg 6y agonot the 1st time bombies did this: https://www.sueddeutsche.de/wirtschaft/deutsche-bahn-ic-1.4773380 https://www.sueddeutsche.de/wirtschaft/deutsche-bahn-ic-1.47...
- rrmm 6y agoAnother turn the wrong way issue that happened was on the early versions of the 737. It would cause the rudder to go hard-over to the other direction it was commanded. It happened so hard it was difficult (if not impossible) for a pilot to counteract it using the pedals (they would basically have to stand on the pedal). It killed at least 157 people. The culprit in this case iirc was a combination of a flaw in the hydraulic cylinder design with large temperature swings. The story of the guy who finally figured it out is a fun one. https://en.wikipedia.org/wiki/Boeing_737_rudder_issues https://en.wikipedia.org/wiki/Boeing_737_rudder_issues
- baybal2 6y agoAnother reverse polarity incident: https://www.airliners.net/forum/viewtopic.php?f=3&t=1424233&sid=6b6dc652243a92d64d2d4359a3f5bad5 https://www.airliners.net/forum/viewtopic.php?f=3&t=1424233&...