8 ms·
> is at best sensationalism, at worst pure fearmongering I don't know about the case here, but any time I've hit an issue in my work where "Thing X needs to be
by munk-a 7y ago
> is at best sensationalism, at worst pure fearmongering
I don't know about the case here, but any time I've hit an issue in my work where "Thing X needs to be done every Y or bugs start happening" it's a pretty clear sign of some deeper issues and likely a lot of underlying bad dev processes.
This issue might be as "simple" as a memory leak that will suddenly require reboots every N minutes when a seemingly unrelated patch exacerbates an issue.
- goblin89 7y agoBug-free software of any complexity is at the very least exceedingly improbable, so there is always a tradeoff to be made and a lesser evil to be chosen. Aircraft firmware requiring mandatory reboots in alignment with maintenance schedule, but working reliably otherwise, inspires more confidence than firmware advertised to run bug-free forever.
- paulryanrogers 7y agoAren't Ada and similar languages designed for safety critical cases like this? When lives are on the line software should be tested for reliability beyond 51 days. Having to restart is a symptom of reckless disregard for safety IMO.
- tjr 7y agoWhen lives are on the line software should be tested for reliability beyond 51 days. Avionics software is written a world of verifiable requirements. For how many days should the software be required to operate? Is it acceptable to add that many [more] days to the software verification schedule in order to verifiably demonstrate that it works according to requirements? Why is 51 days not long enough?
- paulryanrogers 7y agoTaking a plane from design to commercial delivery takes years. I'm sure they can spare 2-3 months to do some long running tests. Especially if those can run in parallel with other fit-and-finish work unrelated to software.
- ethbro 7y agoThere are an infinite number of tests that could be performed. That this test could have been performed does not mean that all possible tests could have been performed. Which is really what we're talking about here. Is "able to run 51 days without reboot" a requirement, or not? If not, and it's not a use case, then it shouldn't have been tested for. Instead, the limited time and resources available should have been spent on more important things.
- munk-a 7y agoThis just gets back to the issue though - since all this software is locked behind privacy laws we have no visibility into how thoroughly this bug was identified. We don't know if there is a test out there confirming that this data corruption occurs after the expected amount of time after each patch.
- ethbro 7y agoYou mean on the FAA side? I don't have as much a problem with that. There's always been tension between regulator access and proprietary corporate information. In order to get the former, you have to design a scheme to protect the latter.
- deleted 7y ago[deleted]
- henvic 7y ago1. You say like aircrafts are being manufactured by laymen. Despite all recent problems with Boeing, it's not the case. 2. Running a battery of formal proof tests is expensive and way more complicated than running a unit test suite for software. 3. Probably more complexity is required to solve this issue, and where is more complexity there, there might be more risk. I'm not saying that this is even acceptable or a great trade-off, but the way you worded your comment is presumptuous.
- munk-a 7y ago
- threeseed 7y ago> Having to restart is a symptom of reckless disregard for safety IMO No it's a symptom of having bugs in your code. And they can be there for a host of reasons ranging from "this is a once off accident" to "systematic failure in the software engineering process".
- paulryanrogers 7y agoBugs may be inevitable but reasons and outcomes matter. If the entertainment system goes down then no big deal. If the pilot is misinformed and the plane crashes then how is it possible for the company to get such slop certified?
- goblin89 7y agoThe pilots apparently start getting misinformed 51 days in software runtime. What if the certification process only tests for 42 days?
- goblin89 7y agoI’d be more alarmed about the fact that FAA had to issue a directive to deal with this situation. Either Boeing did not include the reboot in operation or maintenance procedures, or operators did not follow those procedures. The requirement of a reboot on its own, though, would not strike me as a blatant disregard for safety, as long as the period between reboots is long enough to exceed the maximum possible length of flight (taking any emergencies into account) with leeway to spare.
- munk-a 7y agoI wonder how many days before you hit 51 is the cutoff where planes need to be grounded - obviously a continuous 51 day flight can't happen without a lot of fancy stuff going on that doesn't apply to the 787, but let's say you're at 49.2 days and considering a .6 day flight, is that allowed or is .2 days within the expected probable variance of your flight time? What if it's closer?
- goblin89 7y agoI believe a reboot in such circumstances belongs to the A check[0], i.e. each 25 days of flight time or more frequently than that. (Franky, I am surprised airplane software runs for days—somehow I assumed they get shut down completely every time they refuel.) [0] https://en.wikipedia.org/wiki/Aircraft_maintenance_checks#A_check https://en.wikipedia.org/wiki/Aircraft_maintenance_checks#A_...
- nitrogen 7y agoI have audio devices which, when installed, will run flawlessly until power or hardware failure. The firmware isn't bug free, but the operation never encounters bugs.
- im_down_w_otp 7y agoDevices in systems like this are full of monotonically increasing sequence numbers used for all manner of coordination and diagnostic functions. In this case it appears to be a way to ensure some recency constraint on critical data. This is an extremely common method of attempting to assess/identify staleness of critical data (i.e. "Is the sequence number I'm looking at before or after the last one I saw, and by how much?") in critical real-time systems. Probably this is a counter that rolls over if it's not reset, the predictability of needing to reset it before time T is an indicator it's a sequence number that's driven by a hard real-time trigger with extremely predictable cadence.
- panda88888 7y agoI think you are right. 50 days is 4.32e9 ms, which is just a bit under max value of unsigned 32-bit int.
- munk-a 7y agoIt's actually a bit over the max value[1] - I agree though that I'd strongly suspect this issue is related to overflowing a millisecond counter stored in a 32-bit int. The numbers are way too close. Hey, maybe <51 was just a off-by-one error... or maybe the actual advisory is to be <50 and some PM decided that number was too round or violated an SLA. 1. 4,294,967,296 or 4.29e9
- 3pt14159 7y agoBut with engineering we almost always have safety factors. I'd say it's probably a 64-bit int, but that would be way too much of a safety factor.
- pdpi 7y agoSafety factors are a thing, sure. Safety factors of 4.29e9x (which is what you get when you go from 32- to 64-bit ints) are possibly a bit excessive, and not at all worthy of an FAA airworthiness directive.
- shadowgovt 7y agoIf it goes from 58 days to N minutes, it'll be caught real quick by QA testing for the patch, won't it?
- munk-a 7y agoI have no idea, hopefully? Or maybe this bug just manifests as a minor fuzziness on calculations that'd fall within acceptable error for seemingly unrelated tests. I also have no idea what Boeing's QA is like and I feel like assuming the best is clearly incorrect.
- hnarn 7y agoHave you heard the anecdote about buffer overflows in missiles? The short story is: they don't matter if they happen after you know the missile has exploded. It doesn't by definition make it an "underlying bad dev process".
- hinkley 7y agoContinuous Deployment environments are susceptible to versions of this. You have a slow leak, and nobody notices until Thanksgiving, when the processes that used to run for 3 days now run for 10. And just about the time you think you got that sorted out, Christmas comes along and busts the rest of your teeth out.
- liquidgecka 7y agoI worked at Google way way back when. We had an emergency code red situation where dozens of engineers from all over the company had to sit in a room and figure out what was making out network overload. After a bit of debugging it became clear that Gmail services where talking to Calendar services with an exceeding amount of traffic that nobody would have expected.A little debugging later and it became clear that restarting the gmail server fixed the issue. One global rolling restart later and all was well. But then the debugging started. Turns out the service discovery component would health check backend destinations once a second. This was fine as it made sure we would never try to call against a server that was long gone. The bug was that it never stopped health checking a backend. Even if the service discovery had removed a host from the pool long ago. Gmail had stopped deploying while it got ready for Christmas, and Calendar was doing a ton of small stability improvement deploys. We created the perfect storm for this specific bug. The most alarming part? This bug existed in the shared code that did RPC calls/health checking for all services across Google and had existed for quite a long time. In the end though, Gmail almost took Google offline by not deploying. =)
- hinkley 7y agoStatistics being what they are, eventually you will have, in the same build, an unfixed bug that requires a restart, and an unfixed bug that only works until you restart. That is never a fun day.
- jb3689 7y agoWhat about running repairs in something like Cassandra? In some cases it is by design. Here I'm a little surprised an airliner would even go that long without a reboot