12 ms·
Boeing 787s must be turned off and on every 51 days to prevent 'misleading data'
- snickerbockers 7y agoThe article doesn't mention this, but 51 days is approximately 2^32 milliseconds...
- wolfgke 7y ago2^32 ms is about 49.71 days ( (2^32)/(1000 * 3600 * 24) ), so less than the reboot cycle of 51 days.
- rightbyte 7y agoOn the 51th day you need to power cycle, counting from day one.
- SAI_Peregrinus 7y agoI mentioned this in another thread, but 2^32 * 1024us is 50.9 days. So it's probably a systick at 1.024ms overflowing a uint32_t. If you've got a 1us timer it's a lot cleaner for the CPU to make the tick happen at 1024us than at 1000.
- choeger 7y agoI have heard this anecdote before. In case you might not: 51days is awfully close to 2^32ms ...
- dan_quixote 7y agoJust for good measure: (2^32)/(1000 * 3600 * 24) = 49.71 days
- sedatk 7y agoso the correlation is meaningless here, as the reboot time is after the overflow.
- CyanBird 7y agoIt could very easily be that some manager decided to add an extra day to the schedule "just in case" as in close enough
- SAI_Peregrinus 7y ago2^32 * 1024us is 50.9 days.
- creato 7y agoThis must be a coincidence because surely there is a large safety factor in this 51 day schedule. If the reboot is required to be every 51 days, surely the issue this is trying to avoid only appears after at least 3-4x as long.
- close04 7y ago> This AD requires repetitive cycling of the airplane electrical power. This AD was prompted by a report that the stale-data monitoring function of the common core system (CCS) may be lost when continuously powered on for 51 days The issue appears after 51 days. I can't see in that AD a specific recommendation to reboot at predetermined intervals, just that it requires reboots before 51 days of continuous operation.
- dpau 7y agothere was also a conjecture that the dreamliner required rebooting every 248 days due to a similar issue https://www.engadget.com/2015-05-01-boeing-787-dreamliner-software-bug.html https://www.engadget.com/2015-05-01-boeing-787-dreamliner-so...
- JadeNB 7y agotpmx (https://news.ycombinator.com/item?id=22761698 https://news.ycombinator.com/item?id=22761698) points out a related anecdote https://sites.google.com/site/edmarkovich2/whywindows95andwindows98wouldcrashafter49.7daysofuptime https://sites.google.com/site/edmarkovich2/whywindows95andwi... .
- ndespres 7y agoThe 51 days in the headline popped out to me because I have a Sonicwall brand firewall I help manage which requires a reboot every 51 days or it starts dropping packets. Yesterday was day 51 and like clockwork, it started failing.
- SAI_Peregrinus 7y ago2^32 * 1024us is 50.9 days. So it's probably a systick at 1.024ms overflowing a uint32_t.
- hnarn 7y agoI am going out on a limb here but I seem to remember reading somewhere that airliners do have maintenance schedules that are very strictly kept, for obvious reasons. If the maintenance schedule is N days, then any news article pointing out how amusing it is that an airliner needs to be rebooted every >N days is at best sensationalism, at worst pure fearmongering. I don't know for a fact this is the case here for the 787, but I think there are far better things to worry about when it comes to technical security in airliners than how often they need to be rebooted. For example, whether the on-board WiFi is sufficiently separated from the in-flight systems, and (as discussed recently here on HN) whether the advent of touchscreens for critical flight systems is sufficiently durable, tested and redundant.
- Terr_ 7y agoIndeed, working with a Boeing subcontractor, I've seen a few cases where something is "fixed by process control", where rules for people are designed to circumvent the deficiency in the software. Basically, adding more code to make software "smarter" for those edge-cases was judged (rightly or wrongly) as having an even higher risk of introducing new bugs and creating new test procedures or invalidating previous testing.
- anentropic 7y agoIt does make you wonder what other bugs the software has though? Presumably it's not intentionally designed so that it needs rebooting periodically...
- jpollock 7y agoLots of things are designed with periodic restarts in mind. From memory, one of the JVM garbage collectors is designed with daily restarts in mind. This is done to avoid having to deal with the expense of memory fragmentation.
- ardy42 7y ago> It does make you wonder what other bugs the software has though? Presumably it's not intentionally designed so that it needs rebooting periodically... Not necessarily. If the computer will never actually need to run for 51 days continuously, it may be a reasonable trade-off to require the reboot instead of writing (potentially buggy) code to handle a scenario that can be easily prevented from happening. It reminds me of this story: https://devblogs.microsoft.com/oldnewthing/20180228-00/?p=98125 https://devblogs.microsoft.com/oldnewthing/20180228-00/?p=98...: > I was once working with a customer who was producing on-board software for a missile. In my analysis of the code, I pointed out that they had a number of problems with storage leaks. Imagine my surprise when the customers chief software engineer said "Of course it leaks". He went on to point out that they had calculated the amount of memory the application would leak in the total possible flight time for the missile and then doubled that number. They added this much additional memory to the hardware to "support" the leaks. Since the missile will explode when it hits its target or at the end of its flight, the ultimate in garbage collection is performed without programmer intervention.
- niffydroid 7y agoI think airbus 350 needs a reboot as well
- briandear 7y agoEvery 149 hours..[1] However, since Airbus != Boeing, nobody around here cares. Only stories that are pointing out problems with Boeing are allowed (or upvoted) apparently. [1] https://www.theregister.co.uk/2019/07/25/a350_power_cycle_software_bug_149_hours/ https://www.theregister.co.uk/2019/07/25/a350_power_cycle_so...
- tolien 7y agoApart from when that exact article was posted and got 65 points and 70 comments? [0] You even commented in it. 0: https://news.ycombinator.com/item?id=20524391 https://news.ycombinator.com/item?id=20524391
- cm2187 7y agoIs it because it runs on Windows 95?
- tabtab 7y agoGood thing we improved with Windows 10 and have to reboot only every 52 days. Progress! Seriously, Windows 10 does tend to get dodgy if you don't reboot for a few weeks. I'm not the only one who's noticed. Granted, it's less likely to outright crash, but it acts increasingly drunk.
- marcosdumay 7y agoIt gets slower over time. I still can not understand how an OS can make things slower over time. I made an excercize of rebooting my work computer every Monday, but it really wasn't enough. Now I restart it every time I can.
- tpmx 7y agoIt was 49.7 days for Windows 95: https://sites.google.com/site/edmarkovich2/whywindows95andwindows98wouldcrashafter49.7daysofuptime https://sites.google.com/site/edmarkovich2/whywindows95andwi... Still, it's remarkable that two separate Seattle-based companies have produced a similarly short time bomb on very expensive and highly visible product development projects.
- JJMcJ 7y agoThis wasn't noticed for a few years after Win95 was released. The joke was that nobody had ever had a Win95 system stay up for 49 days. Mwah-hah-hah.
- nomel 7y agoGetTickCount is a monotonic timestamps still supported in Windows 10, available for any application to use. These days, you should use GetTickCount64, but any application that doesn't handle the rollover of GetTickCount is buggy.
- deleted 7y ago
- renewiltord 7y agoClassic Related Stories: * Patriot Missile launchers must be rebooted periodically to account for accumulated errors from truncation: http://www-users.math.umn.edu/~arnold//disasters/patriot.html http://www-users.math.umn.edu/~arnold//disasters/patriot.htm... * Cruise missiles with memory leaks that do not matter because they'll reach the target before they allocate all memory: https://groups.google.com/forum/message/raw?msg=comp.lang.ada/E9bNCvDQ12k/1tezW24ZxdAJ https://groups.google.com/forum/message/raw?msg=comp.lang.ad...
- chrischattin 7y agoCommercial pilot here. Can confirm that turn it off and back on again works for troubleshooting avionics issues. But seriously, this is clickbait and nothing to see here. Many things on the aircraft are checked, cycled, etc before every flight, let alone on a 51 day mx schedule.
- crististm 7y agoI'm sorry to hear that people have become so accustomed to fixing failures of the mind (which these defects are) with reboots. It takes a certain type of person to fly a plane and resilience in face of unknowns and following checklists are some of the qualities they have. In the same vein, these kind of failures are known to programmers for tens of years, just like metallurgists are aware of metal fatigue _and plan for it_. Failure of software professionals to plan and mitigate for this kind of foreseeable problems is inexcusable and I liken them to some incompetent metallurgists in an alternate Univers who brushed off De Havilland Comet's like there was nothing to learn from.
- StillBored 7y agoI don't know why this is down voted. Yes software bugs happen, but they are fixing this with documentation rather than root causing the problem. That should worry people, because until its been root caused the actual implications are also unknown. For all we know its a symptom of a bug that will cause a more severe problem somewhere else.
- detaro 7y agoDo you have a source for the claim they haven't identified the root cause before determining that a regular reboot is a suitable mitigation?
- StillBored 7y agoOccam's razor. This is a bug with ""several potentially catastrophic failure scenarios". Yet, its not been fixed in the ~10 years since it first flew. Nor is it the first, there have been a number of fairly critical bugs on this airplane that took a long time to diagnose before changes were submitted for certification. So, In ~10 years and many major revisions of the aircraft, multiple re-certifications, etc none of them have bothered to fix it. One might argue they are afraid of changing the software because it might cause other catastrophic failures, but that leads down a thought process just as severe.
- 101404 7y ago> 2 Apr 2020 Not published April 1.
- RegBarclay 7y agoI scrolled back up to the top of the page to check before I finished reading it.
- mortehu 7y agoSafe use of Microsoft Windows also requires rebooting on a slightly shorter schedule, because GetTickCount will overflow. In particular if you're running a real-time simulation which is likely to use delta time as a critical parameter, and you can't audit the code or know for a fact that it uses GetTickCount.
- nomel 7y agoI've often wondered how people came to the conclusion that "Windows is unstable, you can't leave it on for more than a couple months!" due to this. At one company I worked for, all of our National Instruments test equipment would start to fail with communication problems after about two months on our Windows XP computers. Being familiar with GetTickCount, I rebooted the computers, recorded the date, verified the next failure was 49 days later, then emailed National Instruments with a link to the GetTickCount documentation. They pushed out an update with a fix 3 days later. Oops.
- commandlinefan 7y ago... but the developers made all their "sprint goal commitments".
- yingw787 7y agoDo safety-critical systems have memory burned as ROM, instead of having dynamic memory allocation? From my point of view, the plane doesn't really change, so the avionics suite shouldn't require changing either. You build a physics model of the plane, translate it into memory, then bake it in, and dynamic allocation is only needed for when you need inputs. Or is this dangerous because the physics model does change significantly for different loadouts? I'm not an aerospace or embedded engineer.
- sanguy 7y agoAnd to think of the billions given to Boeing to bail it out while the management team who got it into this state got golden parachutes? If the government deems that Boeing much be saved, it should also deem that the that prior management was negligent and cause for this situation and seize their personal assets and hold them criminally accountable.
- lovecg 7y agoIs it common to leave the electronics running for that long anyway? My naive understanding is that it would be rebooted after every flight anyway.
- detaro 7y agoIdeally a plane is spending as little time as possible not doing anything. It's on the ground for as short as possible and there's ideally always something happening that needs monitoring or communication. Restarting a bunch of low-level systems just because doesn't fit into that, so apparently a 51 day span without powering it off wouldn't be unheard of.
- _trampeltier 7y agoAt least better than 2016. Then it was just 22 days before the plane stopped working.
- S_A_P 7y agoOne thing Im curious about is what is classified as a "reboot" of the plane? It it is parked over night are all the systems shut down and restarted the next day it is put back in service? Does it sit "running" in some sleep mode? Last time I checked a plane cannot(practically) stay airborne for 51 days. Is the reboot a pain in the ass 5 day procedure? there are too many unknowns to sound alarm bells.
- cgb223 7y agoI wish some upstart would try to create a plane making startup to crush Boeing at their inefficient bloated game We need an Elon Musk of flight, Boeing has gotten away with too much mediocrity
- perl4ever 7y agoIs there as much opportunity? It could be that aircraft are much more optimized than rockets were. We already reuse airplanes.
- VBprogrammer 7y agoAre we talking about the same Elon Musk who's car company created an autopilot which has been known to crash into to side of trucks? Have I missed the obvious sarcasm?
- jdblair 7y agoOverflow is not the only kind of bug triggered by uptime. In February 1991 a Patriot missile failed to intercept and incoming Scud due to an accumulation of time-based errors. The missile system had been online for 100 hours and this resulted in enough error that the intercept calculation was incorrect. People died. http://www-users.math.umn.edu/~arnold//disasters/patriot.html http://www-users.math.umn.edu/~arnold//disasters/patriot.htm...
- benibela 7y agoI wonder if there were ever uptime issues caused by heap fragmentation.
- jdblair 7y agoThis happens on set top boxes, especially when the graphics memory heap is allocated separately from the system memory heap. The graphics memory heap can be fragmented and surfaces stop being rendered because there are no contiguous memory blocks large enough. Having two heaps on a low memory device leads to unfortunate compromises.
- superjan 7y agoI honestly think the safest solution is that an aircraft should refuse to take off after two weeks until you reboot it. In stead, Boeing and Airbus leave it to customers to test if the plane still flies after six months.
- raister 7y agoGetting late to the discussion, but people tackle this for a long time in software engineering, it's called Software Rejuvenation, with models of repairing systems, Markovian assumptions, applications in JVM, etc. Interesting topic. It was used to analyse Patriot missiles that needed the same approach to replenish its internal variables each time.
- vincnetas 7y agoI'm a bit ashamed of that but i guess im not the only one like this. At work we have a system which started crashing, and we could not figure out why. It runs normally, but restarts after some time and then again continues to function properly. So what did we do? Ran multiple instances behind a proxy and let instances crash. But cluster as a whole functions perfectly even when parts of it are restarting because of unknown error that we have no capacity to identify and fix.
- shakit 7y agoI am guessing they need bigger buffer to would clear it out and some good GPS and timestamps database integrations to clear the alt and speed data. A sensor should only look at data from the past week and not calculate from 49 days in the past. What use would pilot have other than for service and maintainence.
- shakit 7y agoI am guessing manufacturer doest have a budget to fix this. They are too busy sorting the 737 pitch controls. I am guessing they need bigger buffer to would clear it out and some good GPS and timestamps database and add a clr button on the console to clear the historic alt and speed data. The historic data can go to the black box and the new one stored to the buffer. A sensor should only look at data from the past week and not calculate stuff from 49 days in the past. What use would pilots have other than for service and maintainence. What was the OS written in objective C?