24 ms·
Boeing 787s must be reset every 51 days or 'misleading data' is shown (2020)
- deleted 2y ago[deleted]
- tomudding 2y ago(2020)
- shadowgovt 2y agoThis is remarkably business-as-usual for airplane electronics. As a more mundane example: the wifi on planes does temporary [edit: DHCP, not NAT] leases. But the system on many has expiration windows on the order of hours, possibly more than a day... Couple that with the number of passengers planes serve and busy routes can easily exhaust the lease pool. The solution: there's a button the flight attendants can push to reboot the router, dumping the lease table.
- JosephRedfern 2y agoNitpicking here, but you mean DHCP rather than NAT, right?
- shadowgovt 2y agoYes; thank you.
- Matheus28 2y agoEven with super long leases, couldn’t they just have a larger subnet? A /8 oughta do it. But I guess we’re talking about the same people who made the mistake in the first place…
- jmholla 2y agoTo steelman the choice, the reserved IP /8 subnet is 10.x.x.x and is often used for corporate networks and other larger subnets experience similar usage. People on the plane using WiFi are likely to access their corporate networks via VPN, potentially causing routing issues. Users VPNing into the reused address space for their own home VPN are probably knowledgeable enough to figure out what is going on and a small enough user base to not care about.
- Filligree 2y agoCouldn't we spare a single extra /8 for airplanes to use? Though I suppose it's not worth it when you can hit 'reboot'.
- AStonesThrow 2y agoHow about "we" use IPv6 instead, and nobody runs out of address space ever again?
- yard2010 2y agoWouldn't this mean we have to power cycle the whole internet at the same time? Instead of supporting both which means no IPv6 in the near future
- mrguyorama 2y agoI would vote for a once a year internet holiday. It would bring minor mental wellbeing improvements, coerce important industries and systems to exercise redundancy pathways, provide opportunities to have such a cutover like switching to IPv6, and remind a million petty tyrant product managers that no, our goddamned fart app does not need 6 9s of reliability. 100 bucks says IPv6 would still not get implemented. We need legislation at this point. There's enough stubborn assholes in the networking infrastructure industry just refusing to do their job for it to happen by itself. They will insist they need to save a few thousand bucks and hold the whole damn world back.
- jcelerier 2y ago51 days * 86400 seconds * 1000 => 4406400000 2^32 => 4294967296 the coincidence seems unlikely, it's basically ~~5 hours and a half~~ 30 hours of difference if one has a 1-ms counter increment
- Dylan16807 2y agoIt's a day and a half difference, and since 2^32 is the smaller number that would be pretty catastrophic. Pretty likely it's coincidence.
- sitkack 2y agoWatch Windows 95 crash live as it exceeds 49.7 days uptime https://news.ycombinator.com/item?id=28340101 https://news.ycombinator.com/item?id=28340101 Must be a northwest washington thing.
- throwbadubadu 2y agoNot getting it.. yeah the famous 32 bit ms overflow after 49 something days. But why then 51 here? Shouldn't they be required to reboot after 49 days please please? :D
- tines 2y agoPossibly cumulative error in the timing source?
- avelis 2y agoIn the software world I call this an end user discovered issue. But when the issue involves a plane that is carrying actual souls. That can feel very scary. I am sure this has been resolved by now since its from 2020.
- recursive 2y agoI don't think airplane software ships updates the way npm packages do. I would be more surprised if this is fixed.
- thecosmicfrog 2y ago> I don't think airplane software ships updates the way npm packages do. I'd ideally like to sleep tonight, thanks.
- advisedwang 2y agoI think from the point of view of Boeing, the FAA and the airlines, "put it in our maintenance checklist to reboot every 51 days" is a fix.
- woah 2y agoWith that framing, this sounds like one of the easiest maintenance tasks imaginable. No wrenches or grease involved.
- trollied 2y agoThey do get software updates. Watch "Stig Aviation" "Stig Shift" series on youtube. He's shown how to do updates in a few of his videos.
- Dylan16807 2y agoThat depends on how much code was having trouble, and what you mean by "resolved". The safe option might be to avoid the situation, and I could imagine that even if there is a code update it might just make the plane balk at getting ready to take off after a certain amount of uptime.
- Dylan16807 2y agoPrevious: https://news.ycombinator.com/item?id=22761395 https://news.ycombinator.com/item?id=22761395 https://news.ycombinator.com/item?id=33233827 https://news.ycombinator.com/item?id=33233827 More interesting, a root cause analysis: https://news.ycombinator.com/item?id=33239443 https://news.ycombinator.com/item?id=33239443 https://ioactive.com/reverse-engineers-perspective-on-the-boeing-787-51-days-airworthiness-directive/ https://ioactive.com/reverse-engineers-perspective-on-the-bo... The 47 bit timestamp at 32MHz would explain the duration (Though not why it isn't 33MHz?).
- dang 2y agoThanks! Macroexpanded: Reverse Engineer’s Perspective on the Boeing 787 ‘51 days’ Directive - https://news.ycombinator.com/item?id=33239443 https://news.ycombinator.com/item?id=33239443 - Oct 2022 (55 comments) Boeing 787s must be rebooted every 51 days to prevent 'misleading data' (2020) - https://news.ycombinator.com/item?id=33233827 https://news.ycombinator.com/item?id=33233827 - Oct 2022 (140 comments) Boeing 787s must be turned off and on every 51 days (2020) - https://news.ycombinator.com/item?id=27117320 https://news.ycombinator.com/item?id=27117320 - May 2021 (42 comments) Boeing 787s must be turned off and on every 51 days - https://news.ycombinator.com/item?id=27111650 https://news.ycombinator.com/item?id=27111650 - May 2021 (4 comments) Boeing 787s must be turned off and on every 51 days to prevent 'misleading data' - https://news.ycombinator.com/item?id=22761395 https://news.ycombinator.com/item?id=22761395 - April 2020 (152 comments)
- gfv 2y agoI have a way simpler explanation. IEEE 754 double can only represent integers up to 2^53 without precision loss, so if you naively average two numbers greater than 2^52, you get an erroneous result. It just so happens that 2^52 nanoseconds is a little bit over 52 days. I've seen the same thing with AMD CPUs where they hang after ~1042 days which is 2^53 10-nanosecond intervals.
- withinboredom 2y agoThis is incorrect. Very incorrect and disastrously so. Drop 0.3 in here: https://www.h-schmidt.net/FloatConverter/IEEE754.html https://www.h-schmidt.net/FloatConverter/IEEE754.html You can also drop 524535643, an integer clearly less than 2^53 and is off by 5. This is even seen here: #include <stdio.h> int main() { float b = 524535643.0f; printf("%f", b); return 0; } output: 524535648.000000
- rich_sasha 2y agoScary as it is, is there any reason for a passenger jet to have uptime if more than, say, 24hrs? Wouldn't you just switch it off and on again between every flight, regardless? If this issue was in a car, we would never know as no one keeps their car running for 50 days straight.
- sheepybloke 2y agoAirlines will run the aircraft as long as possible. As another commenter mentioned, if an aircraft isn't in flight, it's in maintenance. All of these times it's on.
- fnordpiglet 2y agoI’ll bet you the typical EV stays powered on 24/7 with reboots around OTA updates.
- garyfirestorm 2y agounsure what you mean here. most of the systems go to a sleep state in modern vehicles ev or not. the 12v battery keeps only certain ECU's up - think ECUs that control alarm, lock and unlock state and any communication with the mobile app via LTE... but the rest of the systems are OFF, you don't want an EV battery to hit 0% and 12V to also hit 0% - that would basically make it a brick from what I understand- because EV's have contactors which need to shut for the battery to be 'engaged' the 12V battery controls these contactors.
- fnordpiglet 2y agoA car with an enormous rack of high capacity batteries able to accelerate an 8000 pound object to 60mph and sustain that for hundreds of miles generally doesn’t depend on the backup battery for literally anything. It has so much excess energy storage in the form of electricity in the primary batteries it generally doesn’t power down the onboard computers at all. Indeed when you get close to exhausting the main battery rack it starts selectively shutting down everything. I’ve never personally let mine get to 0% ever - but for instance a Tesla is continuously on, and if you use sentry mode it’s not just on but the GPU is constantly doing classification of the environment to determine if someone is prowling your vehicle.
- akira2501 2y ago> This alarming-sounding situation That's not what's alarming to me. What's alarming is that the plane could possibly be in a position to be continuously powered on for 51 days in the first place.
- stavros 2y agoWhen a minute of downtime costs thousands, why wouldn't you expect planes to be in constant utilization?
- fallingknife 2y agoThe number of flights varies a lot by time of day, so there is nothing close to constant utilization.
- Filligree 2y agoThere's not much reason to turn them off outside of maintenance. When they're parked, they're connected to grid power.
- n_ary 2y agoA parked Aircraft is not kept powered when there are no maintenance or other routine(cleaning/checks/certification/preparation/restocking etc.) It is very surprising that how a lot of comments here claim the contrary. Even when parked for next flight, until resupply and cargo routines are declared, it is also not powered.
- thecosmicfrog 2y agoAirliners are regularly and routinely shut down. "Cold and dark" is a common startup procedure for the first flight of the day.
- CactusOnFire 2y agoI've flown with airlines before where there was a cascading delay due to a "plane deficit" at the terminal (not the technical term, that's my own). Not to say it's always uptime, but I imagine there are instances of constant uptime.
- smcleod 2y agoI was speaking with a 787 pilot last Sunday, I told him that the week before when I was at an airport there were two pilots sitting next to me talking about how "This is the third bloody 787 rescue we've had this month... I can't believe we had full engine and <I think he said auxiliary?> failure at the same time" - I asked him if this is common and he said "I hear of it, but I haven't had that many major failures, but lots of little things - last time I flew in from <city> a few moments after we touched down we lost auxiliary power from the rear engine, all the cabin lighting went black along with a number of other things, thankfully we'd already significantly reduced speed and were straight and already lost most of the speed we were carrying, so we were fine and taxied to the disembark location, they had it up and flying again within the day - but it certainly was disconcerting to say the least". I will be slightly paraphrasing from memory there, but certainly was quite surprised how calm he was about the whole thing, there's no way I'd board one of those things.
- Filligree 2y agoAPU failure maybe? That would be troublesome indeed; with no engines and no APU you'd lose most instrumentation and a lot of the hydraulics.
- smcleod 2y agoI thought the guy I was speaking mentioned something about instrumentation but I wasn't 100% sure and that sounded more serious so didn't mention it - but if the aux engine failing would do that - I guess that lines up!
- n_ary 2y agoThere is also a RAT at the back that can be deployed to generate some power(~5-10 minutes max) in case of severe emergency in Air. It is what you hear sometimes, when the aircraft is making a very shrill noise flying over your head. However, if it is not a test flight, a RAT deployment should make you very uncomfortable and worried…
- deleted 2y ago[deleted]
- fnordpiglet 2y agoI’d note that commercial airplanes generally operate with 6-7 9’s of availability. For anyone that’s ever built a system with 5 9’s, this is impressive. In fact it’s impressive enough you probably don’t think twice about sleeping on a flight.
- woah 2y agoIf something goes wrong, does it matter whether you are asleep or awake?
- greenchair 2y agowoosh!
- vkou 2y agoOnly when a flight attendant is asking on the intercom: "We don't mean to alarm anyone, but is anyone on board a pilot?" and you happen to be one.
- incognito124 2y agoIt's entirely a different kind of flying All together
- hooverd 2y agoHopefully you didn't have the fish.
- 2y ago
- joejohnson 2y agoThis was news in 2020. Has it been fixed?
- tedunangst 2y agoAnd 4.5 years later, what's new?
- xyst 2y agoThis company just can’t stay out of the news. Their planes are trash. Software is straight garbage. Many people have died because of this company and suffered undue stress/anxiety because of the massive dip in quality. Boeing engineers/builders caught on audio stating they wouldn’t be caught dead in their own planes unless feeling suicidal.
- zamadatix 2y agoThe company definitely can't stay out of the news and it's gone downhill over the recent years but you've picked an interesting post to lament about those on. The news they can't stay out of is over 4 years old in this case. The model of plane it's about (787) has never had a single fatality despite >15 years of operations and >1,000 units operating today. In all, deaths are probably the worst possible metric to berate Boeing on - including every death (e.g. hijackings, not just engineering failures) their popular 747 line has had comes to <6,000 fatalities despite carrying billions of passengers over a period of >50 years. Despite their ever increasing incompetence on delivery speed, test compliance, and innovation... commercial air travel with Boeing (and other major air manufacturers) has always been one of, if not the, safest mechanisms of travel we've ever executed on. Particularly the last 5 years have been the safest period in terms of air travel deaths or injuries. None of that means we shouldn't criticize Boeing by any means, just that doing it over perceived death and accident counts because of what news headlines imply is complete nonsense in terms of actual numbers no matter how you slice it. It's important those kinds of things are reported but it's equally important to not get swept up in paranoia over it.
- gs17 2y agoAgreed, my 737 fears were relieved by researching how many of them are in the air at any moment, how many millions of trips they fly each year, how old airframes can get before they get retired, etc. Even the "worse" models are feats of engineering.
- boohoo123 2y agothis is what happens when you hire based on checked checkboxes and not qualifications.
- hggigg 2y agoHad a similar problem to this many years ago. Happened every 24 days approximately and lost one user setting. Had a logic analyser connected to it for days trying to reproduce the issue in some way. Went to go for a piss and get a coffee one afternoon and came back and there it was triggered! What happened? Well it turns out there was a timer that no one used that overflowed and caused an interrupt which wasn’t handled any more, the interrupt handler fell through, caused a halt and the WDT fired fire rebooting it and some idiot hadn’t stored that one setting in the NVRAM. So then we had more problems. 5000 things with EPROMs in that were rebooting every 24 days which were spread all over the planet. Many questions to ask over how the hell it ended up like that. I hope people are asking these sorts of questions at Boeing. Edit: also the source code we had did not match what was on the devices. Turned out the engineer who provided the hex file hadn’t copied that code to the file server and had left a year before hand. We didn’t find that until the WDT fired and piqued our interest and could reproduce it on the dev board because the software was different (should have checked that past the label on the ROM which was wrong!)
- dgoldstein0 2y agoThis should carry a label: 2020. This article is 4.5 years old
- dang 2y agoAdded. Thanks!
- qxfys 2y agoIt sounds like my random Raspberry Pi sitting somewhere in my server room that has to be restarted every <x> weeks.
- louwhopley 2y agoSame same but different
- olabyne 2y agoReally ? Mine has an uptime of a year or so, it resets only if a big storm stopped the main power for a few seconds. Maybe it is the new hardware ? I have the original one (arvm6, 512 Meg of RAM)
- tonyedgecombe 2y agoMy TV needs regular reboots, about every six weeks.
- abadpoli 2y agoAirbus A350s had the same issue: https://www.theregister.com/2019/07/25/a350_power_cycle_software_bug_149_hours/ https://www.theregister.com/2019/07/25/a350_power_cycle_soft... We’re just going to see more and more issues like this as more and more software is used in applications like this. I would be willing to bet that a Tesla would also spontaneously crash if left on for hundreds of hours, but they just rarely if ever are left on that long.
- flutas 2y agoFord F150 Lightning had a similar issue on a cross country road race some YT'ers put on. It died at 13% battery, Ford said it was due to not letting the truck rest.
- bandyaboot 2y agoIf problems persist after rebooting, you may need to use a giant paperclip to perform a reset.
- jmrm 2y ago51 days? That looks like the old Windows 94/98 bug, where it used a 32-bit variable to store uptime in milliseconds
- pulse7 2y ago"Reboot tut immer gut!" (Reboot is always good!)
- justmarc 2y agoThere are just too many worrying signs from Boeing in the last years. I have no idea about these things at all but some of the issues seem almost unforgivable to me. They should work very hard for the industry, and the ultimate end users to regain confidence in them again. I'm not sure they are doing this.
- tessierashpool9 2y agoeasy: while(true) { if( (date.today() - date(this.system.uptime) >= 51) && !this.sys.isFlying ) { this.sys.resetNow(); } time.sleep(1000); }
- boohoo123 2y agowell now your system doesn't do anything because its stuck in a forever loop checking the time. it's most likely programmed in C so you can remove the OOP as well.
- sgarland 2y agoThere was a similar problem with a specific generation of 688-class submarines, where a calculated temperature would slowly drift. The metric wasn’t used for any protective actions, so it wasn’t a “shut down immediately and return home surfaced on the diesel” situation, but still disconcerting. I assume that after this the software was soak-tested for weeks / months to eliminate that class of bug. Naval Reactors is many things, but repeating the same mistake twice isn’t one of them.
- DiggyJohnson 2y agoI'm honestly impressed that the Register included a prominent blurb explaining to the reader that while this sounds like a catastrophic issue, the most likely outcome if this is experienced in flight is a safe and controlled landing. > Sidenote > > Pitch and power is a simple concept. If you have the throttles, say, three-quarters open and the nose of the aeroplane is pointing a few degrees above the horizon, chances are you're probably flying straight and level at a safe speed. Training manuals normally contain a number of precise pitch and power settings (they vary between aeroplane types) so if display systems start failing, pilots can fall back to these with confidence.