12 ms·
Blackout Bug: Boeing 737 cockpit screens go blank if lands on specific runways
- fit2rule 7y agoOh, it is going to be fun to see them get re-certified. Did someone forget to run the fuzzer, maybe?
- mcv 7y agoThat is an awfully specific bug. I can see how it went undetected. But what could trigger this?
- francisofascii 7y agoFrom reading the article, it seems like it occurs when the runway is precisely running east to west and the airplane is coming in westward. And also the latitude/longitude is somehow involved. You can see one of the airports here. Notice how it is exactly horizontal, i.e parallel of latitude. https://www.google.com/maps/place/Wiley+Post-Will+Rogers+Memorial+Airport/@71.2835378,-156.7756877,14.08z/data=!4m5!3m4!1s0x50c327fff620aae7:0x936de8c8fd2d094b!8m2!3d71.2874212!4d-156.7802701 https://www.google.com/maps/place/Wiley+Post-Will+Rogers+Mem...
- Scoundreller 7y agoMy thought is some combination of that and magnetic deviation. Runway headings are always magnetic headings, so what the runway says and what GPS calculates (at the lowest levels) are two different things. So maybe the issue only occurs at airports where you’re landing on 27 AND mag deviation is some specific value? Would explain why it happens at only some US and some S. American airports: https://en.m.wikipedia.org/wiki/Magnetic_declination#/media/File%3AWorld_Magnetic_Declination_2015.pdf https://en.m.wikipedia.org/wiki/Magnetic_declination#/media/...
- theideaofcoffee 7y ago> Runway headings are always magnetic headings... This is incorrect, while most runways are aligned with the magnetic heading, that changes when you have a field with multiple parallel runways. Atlanta Hartsfield comes to mind first, with its five parallels in the East-West directions, 8L/26R, 8R/26L, 9L/27R, 9R/27L, 10/28. One would naively expect these to be not parallel by just reading the name, while in fact they are all in line.
- Scoundreller 7y agoOkkkk, good point. And they're only magnetic until they get updated because the declination changes over time. Other exceptions: near the poles where the magnetic declinations change rapidly, so they use something else.
- Someone 7y agoThe affected runways have fairly similar directions (between RW25 and RW29. Runway names follow compass directions. See https://en.wikipedia.org/wiki/Runway#Naming https://en.wikipedia.org/wiki/Runway#Naming). I don’t know what the distribution of runways in the USA is, but that feels too similar to me to be a coincidence. Even if it is, I think it wouldn’t be the whole story, as there must be zillions of such runways in the USA.
- vidanay 7y agoI got ya Bro. https://trailsofwind.figures.cc/ https://trailsofwind.figures.cc/
- pintxo 7y agoI guess the automated test suite just got larger. Testing all approaches on all runways on all airports worldwide...
- rossdavidh 7y agoI think you meant to say, "I guess the automated test suite should get larger". :) Otherwise, I am afraid your optimism might be misplaced.
- pbhjpbhj 7y agoGiven these planes can take off and land on autopilot (I'm not sure that's true, just what I hear) then surely it's easy to run simulations of all those events; so if it was in scope for simulation it could be caught? There's about 20k commercial airfields, if you ran 20mins of simulation for each one then you're done within the year. Presumably Boeing can run in parallel and above realtime. Are their watchdog systems for everything, or do they primarily rely on pilots noticing problems? Wild speculation: Perhaps it's a power issue, something about the conditions causes a power demand that exceeds specs (or makes an LED? voltage drop too low). What else links all the screens?
- Piskvorrr 7y agoActually, an autopilot landing requires CATIII ground equipment...but that could be easier to set up in a simulation. But that is the nature of testing in multivariate conditions: e.g. if the bug triggers only when the heading is exactly 270.0000, I could see why a testing suite wouldn't catch it.
- jojo2000 7y agoIn the automotive and aerospace world, of course tests in the real world cannot cover all cases (cost, time make it impossible). Even most (not all, but almost none for meaningful applications) computer program cannot be proved to come to an halt [0], so complete testing is impossible by essence. We can only use more restrictive rules for programming but cannot formally guarantee anything. As those systems are tied to the physical world, a whole lot of complexity is added by uncontrolled parameters. Yet we love testing things. So a lot of techniques exist, such as SIL [1] and HIL [2]. So you could imagine using a real dashboard hooked up to a plane simulator. Which would enable testing the device in a wide array of conditions. [0] https://en.wikipedia.org/wiki/Halting_problem https://en.wikipedia.org/wiki/Halting_problem [1] https://www.quora.com/What-is-Software-in-the-Loop-SIL https://www.quora.com/What-is-Software-in-the-Loop-SIL [2] https://en.wikipedia.org/wiki/Hardware-in-the-loop_simulation https://en.wikipedia.org/wiki/Hardware-in-the-loop_simulatio...
- mikl 7y agoI’m getting happier and happier that my usual airline (Swiss) has a mostly-Airbus fleet.
- coredog64 7y agoTo quote Han Solo, “Don’t get cocky, kid” Boeing and Airbus both source avionics from the same third parties. The 737NG uses Smiths, but everything else in the Boeing fleet (and all newer Airbus airframes that I’m aware of) come from Honeywell. While you can fault Boeing for not finding this, the vast majority of the fault lies with the OEM.
- mikl 7y agoEven if this particular problem is not Boeing’s fault, but Smiths’, the whole 737MAX saga still puts Boeing in a pretty bad light – cutting corners on security, outsourcing their software to dubious vendors, etc.
- whatshisface 7y agoThere's plenty of bad light to go around. Say company A builds a faulty part, then a stack of ten subcontractors test it and fail to find the error, and finally company B tests and accepts it. Who is responsible for the eventual failure in the field? They all are, because they all had veto power.
- elliekelly 7y agoThe Daily podcast (NYTimes) did an episode last week called “Boeing’s Broken Dreams” where they interviewed a former Boeing safety manager turned whistleblower. Definitely sounds like there was a major cultural shift over the past decade or so.
- thephyber 7y agoThat podcast ep re-airing was based on this story[1]. [1] https://www.nytimes.com/2019/04/20/business/boeing-dreamliner-production-problems.amp.html https://www.nytimes.com/2019/04/20/business/boeing-dreamline...
- nradov 7y agoThat reminds me of the C-130 avionics failure when flying at negative altitude. https://www.popularmechanics.com/military/aviation/a26598/c-130-sea-level-dead-sea/ https://www.popularmechanics.com/military/aviation/a26598/c-...
- pintxo 7y agoOr the flight computer inverting the plane on flying over the equator (changing the sign of the latitude), luckily only in the simulator [1]. [1] https://catless.ncl.ac.uk/Risks/3.44.html https://catless.ncl.ac.uk/Risks/3.44.html
- willvarfar 7y agoReminds me of a story about the Black Buck raids by Vulcan bombers during the Falklands Conflict. Because they didn't have any maps of the southern hemisphere, they just turned a map of the Atlantic upside down. Source: https://youtu.be/rpRtEABdxNc?t=359 https://youtu.be/rpRtEABdxNc?t=359 (worth listening to the whole thing!)
- alistairSH 7y agoWhy would they turn the map upside down? North is still north, even if you're below the equator.
- gpm 7y agoThe world is spherical not square, which means that the farther away from the equator you are the smaller a circle parallel to the equator is. If you tried to use a map of the northern hemisphere for the southern hemisphere without turning it upside down, you would be stretching the point that is the north pole to cover the entire circle that is the equator, and condensing the entire circle that is the equator to the point that is the north pole.
- rob74 7y ago> The lowest airfield in North America, for the record, is Furnace Creek Airport, Death Valley at minus 210 feet below sea level. So, then I guess it's 210 ft _above_ sea level? (SCNR)
- izzydata 7y agoApparently another plane went down in Ukraine according to this article. That seems like a much bigger story. Maybe I happened to miss that one yesterday? Edit: A Ukrainian airline in Iran.
- DemiGuru 7y agoIt’s a different model it’s 737-800. Different plane model while still within the 737 family.
- izzydata 7y agoWhen I say "another" I was referring to that previous incident where a passenger plane was shot down in Ukraine and not the Max plane problem. Any passenger plane crashing seems like a significant story. I hope we aren't numb to that already.
- SmellyGeekBoy 7y agoWhat makes you think it's being treated as insignificant? It's all over the news.
- izzydata 7y agoNow that I think about it I'm not sure where I got that impression. You are probably correct.
- deleted 7y ago[deleted]
- Kipters 7y ago737-800 is part of the 737 NG family
- zymhan 7y agoBoth the plane that crashed, and the model referenced in this article, are the same family of 737.
- taylodl 7y agoI'd like to learn how Boeing's software delivery practice has changed. Sounds like there may be some lessons learned we can all learn from.
- Frost1x 7y agoI would hazard a guess: their process became "agile."
- eskaytwo 7y agoYou should clarify what you mean by “agile”. It means so many different things in different contexts. A large part of this is likely the cost-driven pressure that lost the CEO his job. Agile can be abused certainly.
- nybble41 7y ago> Agile can be abused certainly. ... where "abused" appears to cover most good-faith efforts to implement Agile in practice. When it works it's Agile—despite the fact that these implementations have very little in common. When it fails the mantra is always "Agile wasn't implemented properly". Agile has every characteristic of a utopian ideal that cannot survive contact with reality, or at least the realities of safety-critical software.
- Jtsummers 7y agoBig-A Agile is often described in Utopian terms. Which is unfortunate, but little-a agile is understood to be a thing you grow towards with constant improvements and changes to both the process and the system being developed based on frequent, ideally continuous, feedback. Boeing, in my experience, didn't want to change their processes, they wanted to run the same processes faster. Which is decidedly not agile (and I hate Scrum for this). The form that I've seen them adopt on programs I've been involved with is heavily inspired by Scrum. So they have sprints, they accept product releases from the subs and will, ostensibly, test it (running the integration lab they have the more complete test infrastructure, in theory). But, in practice, the released code isn't tested for months, but development continues, monthly or bi-weekly releases continue, and no feedback is ever provided to the developers. Finally, 6-12 months later most of the builds are discarded, the most recent one is tested, and it's discovered that, true-to-Waterfall, the wrong thing was built. Patches are quickly developed and applied, tests pass, and something is released to the customers. Months or years later, the software is found to be defective, the test suite is found to be insufficient (again), and the process repeats.
- jperras 7y agoIs someone dividing by zero somewhere? "All six display units (DUs) blanked with a selected instrument approach to a runway with a 270-degree true heading, and all six DUs stayed blank until a different runway was selected." cos(270°) = 0
- gpm 7y agoBut cos(90°) = 0 too, unless they thought to check for one case but they did check for the other, or something more complex is happening.
- jperras 7y agoI mean, it's probably more complex than what I postulated. It was just the first thing that came to mind, since 270° is zero in a few trigonometric functions/identities.
- FriendlyNormie 7y agoangle = arctan(dy/dx) where dx=0. I guarantee this is the mistake they made. In some languages you can use Math.atan2(dy,dx) to easily avoid this.
- phkahler 7y ago>> I guarantee this is the mistake they made. No. East-West runways are very common. If I may speculate... the blanking of 5 displays is probably not a bug. Maybe it's a (misguided) feature meant to keep people from landing at certain restricted airports and they coded the logic incorrectly?
- whatshisface 7y agoI doubt even the military would want the risk of a humongous plane landing blind on their runway, no matter how many secrets they had nearby.
- 7y ago
- dwnvoted2hell 7y agoThis is too much crap. This is one time, when I think a president should fire all the CEO and Administration, and get some newly retired NASA Engineering Management over this craziness. I don't want to fly in a Boeing ever again.
- Starkus 7y agoIran shot down that 737 last night and killed 170 people. Edit: Downvote all you want like the MSM the last two days. We all know its true.
- thephyber 7y agoCitation needed. I think you are conflating two separate events. Iran shot volleys of missiles at two different bases in Iraq. Separately, but on the same day, aUkraine-bound Boeing 737 800 crashed hear Tehran[1]. Some articles are reporting that it had technical problems. But it may take a while to verify because Iran is withholding the black box from Boeing. [1] https://www.nytimes.com/2020/01/07/world/middleeast/iran-plane-crash-boeing-ukraine.html https://www.nytimes.com/2020/01/07/world/middleeast/iran-pla...
- Starkus 7y agoWatch the video and then look at the results - that thing was either shot down or blown up. That is not a Boeing malfunction
- bagacrap 7y agoThey shot down a plane mostly full of Iranians and Canadians?
- briandear 7y agoThere were holes in the airplane consistent with air defense guns. The photos are pretty clear.
- makomk 7y agoGenerally, passenger planes are shot down due to incompetence from trigger-happy idiots, not because anyone actually wanted to attack a passenger plane. So this is entirely plausible.
- Scoundreller 7y agoEdit: the airports noted aren’t along the same magnetic declination, so maybe not that. Given that this applies to a few US airports and some S. American ones, maybe it has to do with magnetic declination calculations gone awry. Maybe only airports along one of these lines gets impacted: https://en.m.wikipedia.org/wiki/Magnetic_declination#/media/File%3AWorld_Magnetic_Declination_2015.pdf https://en.m.wikipedia.org/wiki/Magnetic_declination#/media/... And magnetic declination changes all the time. Maybe different systems are using different sources of truth? Maybe when they did initial testing, they included some airports at the extremes, but when the declinations got updated, they were no longer good test cases?
- deleted 7y ago[deleted]
- deleted 7y ago[deleted]
- ravenstine 7y agoIsn't this something that would be discovered through flight simulation? Does Boeing run their software and hardware through automated testing?
- _bxg1 7y agoI for one will never ride on a 737 again, even if they manage to get them back off the ground. For every problem that's come to light there are probably two more than haven't been discovered (or disclosed).
- markus92 7y agoThis bug affects the Boeing 737 NG, the '97 model that everyone and their model still uses. The article is not about the grounded 737 MAX model.
- johannes1234321 7y agoFact is: there are around 10,000 737s around, each if them starts and lands multiple times a day, by far mostly without problems. Risk is similar to you slipping while getting out of bed in the morning or being overrun while crossing a street. (Excluding Max versions, which are grounded)
- ta999999171 7y agoMaybe if I have a heart attack or seizure during those activities... I have 0 past history of slipping. Also, can't avoid walking. I can avoid flying on something I know could be fucked.
- some_random 7y agoAre you saying you have literally never slipped before? I'm quite impressed if so.
- ta999999171 7y agoStraying from point, haha, but only once, in a new set of footwear - jumping over a car.
- mcsoft 7y agoWhile it is easy to blame Boeing these days, I suspect we have survivorship bias here. Dig into Airbus software and all sorts of similar scary bugs will pop up. For instance, one may remember inconsistencies between airspeed measurements that resulted in the loss of AF 447 on June 1, 2009. Overall, it's great that at least some airliner's closed software finally gets more eyes looking at it.
- umvi 7y agoI seem to recall Airbus having different engineering teams across EU using different versions of the CAD tool and then when they went to build the plane the wires were too short because one version of the CAD tool accounted for the radius of wires turning a corner while the other didn't. https://www.nytimes.com/2006/12/11/business/worldbusiness/11iht-airbus.3860198.html https://www.nytimes.com/2006/12/11/business/worldbusiness/11...
- zaphirplane 7y agoHad only the version that didn’t account for turning a corner used, would things have worked out ? I don’t follow
- Piskvorrr 7y agoThe measurements would have been the same in both cases: therefore it would have been a) designed correctly and implemented correctly, or b) designed incorrectly, but caught in the design phase, looping back into a). This way, you get c) designed correctly, but implemented incorrectly or d) designed incorrectly and the design implemented as specified, in both cases leading to an unworkable implementation.
- kayfox 7y agoBoeing had an approach to this issue: Your either running the correct version of the CAD software, or you can't check in designs. This also meant that every time it was updated there was a rush to get engineers who's update did not work back up and working. Fortunately those were a small minority and it was usually disk space issues.
- lmaophp 7y agoThe absolute state of pajeet outsourcing
- awinter-py 7y agoThe first F-22 they delivered to japan had serious trouble when it crossed the international date line Sounds like nobody was hurt & they didn't lose any planes, but there was some period of time where the onboard computers were not good https://www.defenseindustrydaily.com/f22-squadron-shot-down-by-the-international-date-line-03087/ https://www.defenseindustrydaily.com/f22-squadron-shot-down-... Yet another reason to exercise low-level math functions with a wide range of inputs -- maybe not on every run, but at least in pre-release integration tests There are only 4 billion floats https://randomascii.wordpress.com/2014/01/27/theres-only-four-billion-floatsso-test-them-all/ https://randomascii.wordpress.com/2014/01/27/theres-only-fou...
- sudosysgen 7y agoF-22 weren't exported, it was probably just at Okinawa
- vmchale 7y agodon't land on those runways!