11 ms·
Boeing 787s must be rebooted every 51 days to prevent 'misleading data' (2020)
- trasz 4y agoHere’s a properly technical explanation of the problem: https://news.ycombinator.com/item?id=33239443 https://news.ycombinator.com/item?id=33239443
- hbbwebw 4y agoI have lab equipment that needs a power cycle after 48 hours of power on time for it to continue to produce high quality data. This is because the internal calibration code only runs upon boot. Maybe these planes have something similar. So, to me, the power cycle thing is not super surprising. That fact that the FAA felt the need to order Boeing to do it seems a bit suspect though. That's probably the real story.
- ChrisArchitect 4y agoAnything new here? Some discussion from 6 years ago: https://news.ycombinator.com/item?id=13094600 https://news.ycombinator.com/item?id=13094600
- deleted 4y ago[deleted]
- dundercoder 4y agoIt’s probably more complex than a slow memory leak, but is very reminiscent of one.
- netsharc 4y agoMaybe something to do with a clock. Space shuttle missions couldn't fly during the new year clock/date changeover...
- tkanarsky 4y agoHumans have to be rebooted at most every 2 days to avoid misleading auditory data among other things.
- anonym29 4y agoYour analogy made me laugh, but also made me think about how common fatal mental illness is among humans, and contrast that rate with the fatal plane crashes attributable to computing errors. It's a pretty amazing accomplishment for humanity to have developed systems as complex as modern jumbo jets with "brains" that are essentially more reliable than the ones that created them.
- jstarfish 4y ago> It's a pretty amazing accomplishment for humanity to have developed systems as complex as modern jumbo jets with "brains" that are essentially more reliable than the ones that created them. Load the airplane's "brain" with algorithms for how to survive, cooperate, adapt and replicate under every conceivable condition and you'll end up with a comparable system more complicated and worse-performing than ourselves. Not really a fair comparison. Self-driving cars (and robotics) would be better examples, since they attempt to emulate a broader swath of our sensory capabilities.
- masklinn 4y agoThat development was built up over time and very much in blood though. Channels like Mentour are quite illuminating on that subject. And in fairness, reliability is a major point of automation.
- akiselev 4y agoYou need more scare quotes around “reliable” than brain. The airplane requires constant almost daily maintenance by dozens of highly trained professionals and yet none of its chips will survive decades save for a few lucky passive components. Whereas going decades without seeing a qualified doctor has been the default state since the dawn of homo sapiens with nothing more than a good night’s sleep.
- 4y ago
- spicyusername 4y agoThe old turn it off and on again...
- dtgriscom 4y agoI learned programming in the late '70s on a Wang 2200T. The manual had a procedure it would recommend called "Master Initialization." Yes: turning the power off and on.
- dylan604 4y agomid-flight though?
- stonemetal12 4y agoHow many flights last for 51 days?
- dylan604 4y agoIf you take off on day 50 23 hours type of thing is more of what I was getting at. It's very easy to just hit skip when presented dialogs like "3 more days to reset password" or "3 more days until scheduled reboot". As long as it doesn't end during your flight, why not push it for someone else? Otherwise, you just make it part of every flight's routine to reboot.
- bioemerl 4y ago> It's very easy to just hit skip when presented dialogs like "3 more days to reset password" or "3 more days until scheduled reboot" This simply does not and cannot happen to an airline. If it does people will be fired and processes revised.
- dylan604 4y agooh boy, you sound like someone that is big into the brochures. with the current state of Boeing, nothing would surprise me, and there's very little i would accept at face value from them.
- helipod 4y agoIt's very close to the limit of a 32 bit integer in miliseconds. Comes out to 47 days according to google.
- munch117 4y agoPerhaps you should switch to a search engine that's better at math.
- jp57 4y agoThis seems like some kind of overflow problem, but 2^32 milliseconds is 49.7 days. Is it possible that the 51 day number came from a combination of rounding and an off-by-one error in the reporting?
- gte525u 4y agoAirworthiness directive indicates 51. https://ad.easa.europa.eu/blob/2020-06-14.pdf/AD_US-2020-06-14_1 https://ad.easa.europa.eu/blob/2020-06-14.pdf/AD_US-2020-06-...
- deleted 4y ago[deleted]
- wzdd 4y ago49.7 days is equal to 2**32 milliseconds, after which Windows 95 crashes (https://web.archive.org/web/20010331174127/http://support.microsoft.com/support/kb/articles/q216/6/41.asp https://web.archive.org/web/20010331174127/http://support.mi...). That's close-ish to 51 days. Maybe they start the counter with a value other than zero.
- JTbane 4y agoI immediately though of the GetTickCount win32 function, which, as you said, wraps around to zero every fifty days or so. A lot of programs would crash if you left them open for this long.
- geoffdunbar 4y agoI worked at Microsoft on the NetShow Encoder, later known as the Windows Media Encoder, in the Windows 95/Windows NT4 era. I remember carefully writing millisecond code that would deal with differences, instead of absolutes, so that we could run longer than 49 days. Honestly I don't recall whether the limitation was GetTickCount, the recording API, the file format, or all three. In our debug build, I added a registry key that would start the concept of time at 49-ish days, so that we could test that everything was working without waiting 50 days. However, before we shipped, the intrepid test team did actually run the encoder for 50 days to prove that it worked. (On Windows NT, of course. Windows 95 with good hardware and drivers was perhaps more stable than its reputation, but still...) Though TBH we definitely wouldn't have delayed a release by 49 days if it had failed at the last minute.
- drewrv 4y agoI would hope there's a margin of error baked in and stuff doesn't break on day 52 exactly.
- dundercoder 4y agoThe 787 runs on VxWorks though. https://blogs.windriver.com/wind_river_blog/2011/09/boeing-787-the-long-certification-journey-comes-to-an-end/ https://blogs.windriver.com/wind_river_blog/2011/09/boeing-7...
- abudabi123 4y agoWho are the programmers behind this average work and who are their superiors?
- tfsh 4y agoI would not be so brazen as to call the engineers behind this "average" or their work so. Unless you can make the argument that a plane would fly for 51 days straight, being required to reboot the on-board systems is not a major deal-breaker, especially as the larger Boeing's take 60 minutes to refuel. The engineers behind Boeing planes have built an incredibly sophisticated machine which omitting a very nessescary but not so inconvient task of rebooting every n days, performs its primary directive well.
- rado 4y agoIf this is a known issue, what are the chances of a related, unknown and potentially catastrophic issue?
- deleted 4y ago[deleted]
- deleted 4y ago[deleted]
- s1artibartfast 4y agoLower by one factor than before this issue was determined.
- malfist 4y agoConsidering it's likely an issue with an integer overflow for a millisecond clock: near zero. One bug, does not imply two bugs.
- deleted 4y ago[deleted]
- 4y ago
- WalterBright 4y agoMy Windows desktop and iPhone need to be cold booted now and then. My Ubuntu box hasn't needed that.
- ThinkingGuy 4y agoChapter 1 of "The Inmates are Running the Asylum," by Alan Cooper (1999) opens with the riddle: "What do you get when you cross a computer with an airplane?" The answer, of course, is "a computer."
- GoodDreams 4y agoReminds me of: People said you can’t write a financial trading system in a garbage collected language. Turns out you can as long as you don’t garbage collect during the trading day by carefully managing allocations and manually running GC each day before trading starts. Or the old story of the junior engineer who finds a memory leak in a missile’s guidance system and the senior engineer says the memory leak is fine as long as you don’t run out of memory before the missile completes its mission.
- gumby 4y agoReminds me of a sales call to a military contractor. They were interested in an Open Source OS because of the licensing costs. The embedded OS they were using had a license that said you could run as many instances as you liked on a "single device". Their problem was that when they operated their device it was indeed a single device, but later became many more. If they paid for the extra licenses they didn't need it would be ripping off the government; if they failed to pay for the extra license at some point they would be in breach of their contract with the government as all these unlicensed devices were released. It rapidly became obvious that their application was a MIRV and that they weren't really going to change but had asked us to talk with them as part of their process of negotiating a deal with their current supplier. It's hard to imagine really worrying about your license agreements if there actually were a global thermonuclear war.
- nocoiner 4y agoThat is an amazing story.
- personjerry 4y agoThat reminds me of a short sci-fi story I read about an alien invasion, in which humans were losing horribly until we learned about their legal system and sued them into submission
- zokier 4y agoFor many processes where the lifecycle is limited, freeing memory can be waste of time; most notably freeing memory just before exiting is complete waste.
- jaywalk 4y ago51 days is nothing. The Airbus A350 had to be rebooted every 149 hours: https://www.theregister.com/2019/07/25/a350_power_cycle_software_bug_149_hours/ https://www.theregister.com/2019/07/25/a350_power_cycle_soft...
- dschuetz 4y agoAnd now we know the 787 runs on Windows.
- ourmandave 4y agoHopefully the forced Windows 11 upgrade isn't scheduled for when you're over the Atlantic.
- pc86 4y agoIt doesn't.
- jpgvm 4y agoI believe it's called a joke.
- TremendousJudge 4y agoThe only reason this seems to be a problem is that apparently operators (or the manufacturer) didn't know about it, since I don't think there's an actual usecase for the aircraft to be powered on continuously for that long.
- Someone 4y agoFTA: It is usual for commercial airliners to spend weeks or more continuously powered on as crews change at airports, or ground power is plugged in overnight while cleaners and maintainers do their thing. I don’t know how long rebooting a 787 takes, but turn Around Time of an airliner may be too short to do a reboot. http://aviationavi.com/turn-around-time-tat/ http://aviationavi.com/turn-around-time-tat/: Ryanair a private ULCC of Europe and one of the largest ULCC of the world having a TAT of 25 minutes does not have a seatback pocket. Wondering why? Back in 2004, removal of seat back pockets greatly reduced the time for which the Ryanair aircraft stayed on the ground. That’s is because it takes more time to clean an airplane with seatback pockets. So, that’s 25 minutes from touching down to being airborne again. Chances are the plane is standing still for less than 15 minutes. Long-haul flights take longer (up to two hours, according to https://simpleflying.com/aircraft-turnaround-process/ https://simpleflying.com/aircraft-turnaround-process/) Also: it wouldn’t surprise me if you could not refuel an airliner while it is rebooting, all lights may be off for a while, etc.
- kayfox 4y agoAssuming the plane spends 50% of its time flying, it kinda lines up with an A check interval. https://en.wikipedia.org/wiki/Aircraft_maintenance_checks https://en.wikipedia.org/wiki/Aircraft_maintenance_checks
- duckmysick 4y agoI would expect rebooting to be a part of a routine maintenance and a health check between each flight. Now I'm curious: how long does it take to reboot Boeing 787.
- Steltek 4y agoThis was my first thought. Surely there are other systems on the plane that require maintenance or checks that are even more disruptive. Rebooting is a switch, button, or command. Inspecting engines, pipes, or compartments requires somebody to get out of a chair.
- ankaAr 4y agoI'm just thinking about a Liam Neeson movie in a 787 with on flight tank refill and the reboot after 2^32 milliseconds because the plane could crash and it is the president pilot in the middle of something bad like a WW3.. Edit: patent pending
- leecarraher 4y agocommercial air travel is all about critical maintenance. Add it to the checklist and move on. Feels like the article pandering to that y2k nostalgia.
- chx 4y agoAbsolutely. It's like the regular social media mini uproar when someone discovers duct tape on the plane they are flying. It's speed tape.
- kleiba 4y agoI think that's fine. I mean, what could possibly go wrong?
- laluneodyssee 4y agoReminds me of the fact that a lot of Boeing planes still operate on floppy disks for data https://www.popularmechanics.com/science/a33612762/boeing-747-floppy-disk-updates/ https://www.popularmechanics.com/science/a33612762/boeing-74...
- KptMarchewa 4y agoI wonder what's being used to update B-52s.
- guywithahat 4y agoWell a lot of planes are decades old, the fact some use floppy disks doesn't surprise me at all
- kayodelycaon 4y agoNitpick: the floppy disks are used for software updates, not flight operations.
- Melatonic 4y agoCan you imagine being a pilot halfway through a flight and while you are at 40,000 feet your airlines tech support asks you "Have you tried turning it off and on again?" :-D
- netsharc 4y agoOne time I was flying inside China, and as the plane reached cruising altitude it felt eerily quiet, as if the engines were off and we were just gliding. I honestly wondered "Is this it? Am I going to die? Surrounded by people I won't even be able to communicate with?". Of course since I'm still here, it wasn't it. After a while I felt the vibration of the engines again and was relieved. Maybe the pilots had to reboot the plane.