14 ms·
Half of European flights delayed due to system failure
- dang 9y agoUrl changed from https://www.theverge.com/2018/4/3/17193814/eu-eurocontrol-brussels-system-failure-flight-plans-airlines-delays-grounded https://www.theverge.com/2018/4/3/17193814/eu-eurocontrol-br..., which points to this.
- kzrdude 9y agoJust like there are no bugs without security impact, is it real to say that this has no impact on flight security? Any error can be a contributing factor.
- gsich 9y agoSimilar to every incident at a nuclear power plant. "no danger for the population"
- webreac 9y agoSafety is ensured by ATC controlers. ETFMS is there to ensure that traffic does not increase beyond controlers capacity. I have read that without ETFMS, the traffic is reduced by 10%. I have been involved in ATC simulations where controlers had to land about 40 flights per hour using new procedures and tools. We have tried with 38 flights per hour, it was too easy for the controlers: their work was perfect even without tools. With 42 flights, controlers were getting angry because the traffic could not be managed. At 40, we could see the benefits of out new procedures and tools (more regular separation of flights). IMHO 10% less traffic gives far enough capacity margin to ensure safety.
- thaumasiotes 9y ago> Just like there are no bugs without security impact I'm compelled to once again bring up the report that went "when I zoom in, the text becomes blurry".
- jumelles 9y agoI fear this sort of thing is going to become more and more common at airports.
- isostatic 9y agoWhy? Sure, the skies are more and more crowded, meaning more and more people will be affected by a once-in-20 year failure, but why would it happen more and more common?
- SteveNuts 9y agoWell the archaic systems are getting older and older. From what I've heard it's almost impossible to get replacement parts for some of the computer systems that run airports/airlines.
- tetha 9y agoAnd on the other hand, there are massive requirements at all levels for this kind of software. I'm not sure what the right answer is at that point. From dev-experience, I wouldn't want half-assed software directing flights. From ops-experience, I don't want software directing flights requiring specific hardware so you can just replace the hardware with easily available hardware.
- drinchev 9y agoFrom dev-experience I would be surprised if SpaceX rockets run with decades old assembler code-base. I still believe we can create bug-free solution with modern toolset. It's a matter of zeros behind the project funding.
- tetha 9y agoThat's a good point. I'm just a bit disillusioned with the quality of some of the legacy stuff at my current place. Especially because robustness and probability of failures is mostly determined by requirements.
- isostatic 9y ago> Eurocontrol announced the system restart later in the day, after what it called extensive testing. So they turned it off and on again.
- Numberwang 9y agoBastards.
- jwbensley 9y agoIts not brain surgery!
- jwbensley 9y agoWhats with the downvotes? That guys username is and my comment are both references to a TV show.
- isostatic 9y agoDecade old UK tv series on BBC two? I imagine most have never heard of it. I was thinking more of an even older show: https://www.youtube.com/watch?v=nn2FB1P_Mn8 https://www.youtube.com/watch?v=nn2FB1P_Mn8
- jwbensley 9y agoIt's very common that someone hasn't heard of everything. It's why down votes without explanation are bad, like on StackOverflow. It was in reference to That Mitchell and Webb Look: https://youtu.be/AIxz6BDmTNU https://youtu.be/AIxz6BDmTNU https://youtu.be/THNPmhBl-8I https://youtu.be/THNPmhBl-8I
- isostatic 9y agoIndeed, and as usual XKCD has an explanation for the "someone hasn't heard of everything" phenomenon. https://www.xkcd.com/1053/ https://www.xkcd.com/1053/ However HN does seem to curate specific on topic informative posts, especially early on in the conversation (when everyone is reading, rather than just people looking for replies to their comments). Overall I think I agree with the policy.
- icebraining 9y ago"departures are now limited to 10/hour at #brusselsairport" Impressive that they maintain this rhythm even during a once-in-a-decade unexpected system malfunction.
- ttul 9y agoI guess that’s the “paper and pencil” speed.
- ehudla 9y agoIs that the Ada code base?
- sleavey 9y agoI was on a plane just about to push back at Heathrow, and the pilot informed us we'd be delayed 15 minutes due to this failure. In the end it was 10 minutes, and we landed only 5 minutes late at my destination. Doesn't appear to have been a big deal, at least for me.
- dghughes 9y agoI wonder if that is due to Heathrow using Time-Based Separation software. https://www.airport-technology.com/news/optimised-runway-tool-improve-efficiency-heathrow/ https://www.airport-technology.com/news/optimised-runway-too...
- hightowk 9y agoTechnically, Time Based Separation helps only arrivals since it tries to negate the effects of headwinds on final approach, but it does add a lot of resilience to the airport to absorb delays that would normally ripple to departures.
- dghughes 9y agoI'm pretty sure it doesn't I was reading about it and it mentioned even when the pilot can start the engines is calculated. It saves millions of liters of fuel just waiting a few minutes. There is a digital display facing the pilot showing him when he can depart. Even which size aircraft are allowed to depart. It's extremely well organized.
- dx034 9y ago10% reduction probably meant that they tried to keep most flights on time and "strategically" delayed some flights. Makes it worse for some passengers but keeps knock-on effects for transfer passengers under control.
- emersonrsantos 9y agoFor those curious, Eurocontrol MUAC (Maastricht Upper Area Control Center) migrated to 50 virtual SUSE Linux Enterprise servers running under IBM z/VM hypervisor on a IBM z196 mainframe system in 2013.
- justadudeama 9y agoDoes any of this effect the nature of the failure? Have they said what went wrong?
- reaperducer 9y agoLikely not. Just geek curiosity. /Manufactured in Fishkill, New York. Color me surprised.
- dx034 9y agoNo but the fact that this is the first outage in 20 years could indicate that mainframes still have their place if you need 100% reliability. Which makes plans of banks to move core systems from mainframes to the cloud even more worrying.
- amaccuish 9y agoanywhere we can read up on this? :)
- christoph 9y agohttps://www.suse.com/success/stories/eurocontrol/ https://www.suse.com/success/stories/eurocontrol/
- matte_black 9y agoIn situations like this I’m glad I book my flights with Chase Sapphire Reserve. It comes with a trip delay reimbursement for up to $500 per ticket if a flight is delayed more than 6 hours. No sweat!
- ytwySXpMbS 9y agoI'm glad I live in the EU in such circumstances: EU regulation 261 [1] covers so much, with €250 to €600 compensation depending on flight distance for delays over 4 hours, with a percentage of full compensation for shorter delays. No specific credit card required. [1] https://en.wikipedia.org/wiki/Flight_Compensation_Regulation_261/2004 https://en.wikipedia.org/wiki/Flight_Compensation_Regulation...
- jimktrains2 9y agoDoesn't the end of this article state that that may not be in effect for extrodinary circumstances out of the airlines' control?
- lucb1e 9y agoI kind of doubt that matte_black's gamble (because that's what "insurance" is for costs which you could easily carry yourself if need be, unlike e.g. medical costs) would include that either. Then again, I'm surprised how many "insurances" cover things under your control (like dropping a device) so I might be wrong.
- matte_black 9y agoI would be 100% covered up to $500. I’d rather deal with a credit card company than a government anyway.
- kylegordon 9y agoYour 'Merkin is showing. Wrong again. You deal with the airline. Not the 'government' If the airline does you wrong, at that point the CAA takes it up on your behalf. CAA ~= FAA
- njitbew 9y agoYes, this sucked. I just had a 1 hour delay on a 1 hour flight (AMS-ZHR). Unfortunately, no compensation until its a 2 hour delay (but thank god it was only one hour!). Passengers who had a layover were noticeably less happy.
- CaptainZapp 9y agoYou'd probably won't have been eligable for compensation since this delay was definitely beyond the airline's control. While a lot of airlines try to weasel out of their obligations (mechanical failue, for example, which however is the airline's responsibility) I would think such a case is pretty clear cut.
- DavidAdams 9y agoCan confirm. My flight from Zurich was delayed by over an hour today, causing my family and me to need to run, OJ Simpson-style, through the Philadelphia airport to make our connection.
- tnolet 9y ago5hr delay on a 1,5hr flight from Berlin to Paris. If they'd use Docker with React this would have never happened!
- reificator 9y ago> If they'd use Docker with React this would have never happened! It's probably because they wrote it in Go. Did you know Go doesn't even have generics?
- jfktrey 9y agoWhat do you mean? Go has generics. Kind of. https://www.reddit.com/r/rust/comments/5penft/parallelizing_enjarify_in_go_and_rust/dcsgk7n/ https://www.reddit.com/r/rust/comments/5penft/parallelizing_...
- deleted 9y ago[deleted]
- Pxtl 9y agoWait, is that string using inuktitut characters as fake angle-brackets? That is monstrously evil.
- nkurz 9y agoYup, that's even evil enough to quote in full: [–] pftbest 114 points 1 year ago can you please explain this go syntax to me? type ImmutableTreeListᐸElementTᐳ struct { I thought go doesn't have generics. [–]Uncaffeinated[S] 239 points 1 year ago It doesn't. That's just a "template" file, which I use search and replace in order to generate the three monomorphized go files. If you look closely, those aren't angle brackets, they're characters from the Canadian Aboriginal Syllabics block, which are allowed in Go identifiers. From Go's perspective, that's just one long identifier. https://www.reddit.com/r/rust/comments/5penft/parallelizing_enjarify_in_go_and_rust/dcsgk7n/ https://www.reddit.com/r/rust/comments/5penft/parallelizing_...
- bluedino 9y agoTwenty years from now...uber car service system failure affecting 90% of North America
- Zenst 9y agoMaybe, though I can imagine a news release along the lines of an old Steve Coogan comedy sketch: https://youtu.be/zUoT5AxFpRs?t=12s https://youtu.be/zUoT5AxFpRs?t=12s
- jlgaddis 9y agoThe only reason this will never happen is because there's a only snowball's chance in hell that Uber will still be around in 20 years.
- tvanantwerp 9y agoI was at a conference in Northern Virginia some months ago and saw a presentation from the folks at Upside, a startup specializing in booking business travel. They described the legacy system which handles pretty much all booking in the US, a system called SABRE. They described it as an ancient 6-bit computer system in Texas with no modern API. Everything they do tech-wise is a modern wrapper around that system. So I'm not at all surprised by any air travel computer failures if tech like that is central to the system.
- dhimes 9y agoWow. I remember sabre from genie- General Electric Network Information Exchange. In the 1980s. I’m not sure compuserve was invented yet. My father and I could use “electronic mail” to stay in touch. For you youngsters: I hsd never yet had a “remote control” for a tv yet. Yep. Had to get up to sdjust the volume or change the channel (don’t get me started on adjusting the antenna). Wish I didn’t know this.
- userbinator 9y agoI've heard of enough misguided "modernisations" (and failures thereof) that I think the "legacy system" was the part that stayed working throughout, and it's the newer stuff added around it that failed. The old stuff may be old but there's a reason it's old... it's outlasted any attempts at replacing it.
- tyingq 9y ago
- tim333 9y agoApparently the Enhanced Tactical Flow Management System (ETFMS) packed up. >“ETFMS facilitates improvements in flight management from the pre-planning stage to the arrival of the flight. It maximises the updating of flight-related data and thus improves the real picture of a given flight, thereby contributing to the Gate-to-Gate Concept,” Eurocontrol explains on its website. >The agency initially reported that contingency procedures were immediately put in place which reduced the capacity of the European network by around 10 per cent. http://www.airtrafficmanagement.net/2018/04/eurocontrol-gives-allclear-for-normal-efms-operations-to-resume/ http://www.airtrafficmanagement.net/2018/04/eurocontrol-give... They don't seem to say what went wrong but do say >In over 20 years of operation, the ETFMS has only had one other outage which occurred in 2001. The system currently manages up to 36,000 flights a day. Tech details from wikipedia fr: Written in ADA , and running on HP-UX , the system is based on an exchange of messages between the airlines (who will file / change / update flight plans), the air traffic control bodies and the CFMU , the messages are written in ADEXP format. ETFMS uses at least 5 fundamental notions: flight plan : describes the 4D trajectory of an airplane. regulation: aircraft rate applied to a "volume traffic". Example: 50 aircraft / hour "Traffic volume": association of a geographical reference (air sector, waypoint, airport, etc.) and a set of aircraft flows. list of takeoff slots or called slot . Example: if the rate of regulation is 30 planes / h, there will be a slot every 2 minutes: 10 hours, 10 h 2, etc. the delay: difference in time between the take-off time desired by the company and the schedule calculated by ETFMS.
- makmanalp 9y agoIt would be fascinating to read about what bug took down a system with one outage in 20 years. I remember reading in the chubby paper (from google) that user error was the cause of more than half the downtime incidents. Wonder if that's the case here too.
- tim333 9y agoYeah, I recall last year "BA’s £150,000,000 outage was caused by someone turning computers on and off too quickly" I think companies often try to stay quite about the dumb stuff.
- maartn 9y agoSo Trump and Putin finally hooked up. No need to get all SuSe over it... nerds
- candiodari 9y agoNo worries. Everyone at that organisation is paid boatloads of money. [1] They don't pay taxes on it. (~10%, "for solidarity", which means they get to enjoy healthcare paid by ~55% taxed nationals) And 90% of the organisation (especially the management) has absolutely nothing, nothing whatsoever, to do with guiding planes anywhere. In fact, those departments are severely understaffed. The department doing "regulatory support" is about 2/3rds of the organisation (tldr: making sure half the local government officials don't have to get their own coffee - and before you say it, no, Eurocontrol employees don't get them coffee, they're merely in charge of making sure someone's there to get them coffee, and steak, cake, and ... The coffees, I might add, are baffling. Done from a steam boiler machine in front of you, with fair trade beans, sweetened not with sugar, but with expensive imported bars of chocolate meled in milk that's frothed in front of you (they put in the chocolate somehow while they're frothing the milk with steam, melting it in while not getting the steam on the chocolate somehow), and you get the rest of the case of that (expensive) chocolate to take home for the kids. No, not when you ask, they'll ask you if you want that. Btw, it's not really the rest of the case they give you, you get a fresh case. Oh and of course, of the bar they opened they prepare just one coffee (about 1/4th of the bar). The rest gets thrown into the trashcan, they don't use the same bar for the next coffee. As for the steaks ... oh my God) And in case you're wondering: the odds of 2 planes colliding with zero guidance outside of the ATC zones around airports (which aren't covered by Eurocontrol) over even a region as big as Europe are more than 10 billion to 1, against, per year. So if Eurocontrol didn't exist at all, and we just allowed every plane to fly wherever ... nothing would go wrong at all. So ... what is the problem here ? Disruption of millions of travelers for no reason whatsoever ? Let's please not pretend anyone at Eurocontrol cares (well, they care about not being interfered with, and that will make them care NOW, but if one thing's guaranteed it's that the Software/ATC departments will remain the same size, and only the bribery departments will grow) [1] http://www.eurocontrol.int/sites/default/files/content/documents/jobs-section/scale-basic-monthly-salaries-officials-annex3-staff-regulations.pdf http://www.eurocontrol.int/sites/default/files/content/docum...
- markrb 9y agoNEED A HACKED ATM CARD WITH PIN? We Sell Physical Loaded ATM Cards . It Is A Crooned Card That Can Be Used To Withdraw Cash At Any ATM Machine. This Cards Comes In Visa/MasterCard. Therefore It Works At Any ATM Machine That Accept Visa/MasterCard Worldwide. CAN I USE THIS READY MADE ATM CARD TO BUY STUFF IN STORES? OR ONLINE SHOPPING? PAY BILLS? Yes, With This Physical ATM Card, You Can Use It To Pay Stuff At Stores Through POS. With This ATM Card Information, You Can Use It Online To Pay Bills Or Do Online Shopping. When You Order For This Card, Full Information About The Card Will Be Given To You. We Also Reload Your Card When Funds Exhausted. DO I NEED TO ORDER NEW CARD EVERY TIME THE FUNDS IS FINISH? No, If You Have Already Ordered Our Card, There Is No Need To Keep Ordering New Cards, Just Contact Us For A Reload. We Shall Easily Reload The ATM Card Already In Your Possession HOW LONG DOES IT TAKE TO RECEIVE ATM CARD IN MY COUNTRY? If You Are In ASIA, You Will Receive Your Card In 2-3 DAYS With Guaranteed. If You Are Outside Asia Your Card Will Arrive To You Between 3 – 5 Business Days Guaranteed. HOW SAFE IS THIS CARD? It Is 100% Safe To Use This Card. Because It Will Be Shipped To You As A Gift Card. DO YOU ALSO RELOAD ANY OTHER CARD NOT FROM THIS CLONED CARDS? Yes, We Can Reload Any Active And Valid Cards, Any Type Of Card Just Contact Us For A Reload (Prepaid Cards, Credit/Debit Cards). HOW DO I ORDER FOR THE ATM CARD? EMAIL: legithackers@outlook.com HOW DO WE MAKE THIS CARD? We Use A Machine MSR To Crone This Cards . You Can Also Buy This Machine From Us Also. You Can Order For The ATM Card Either The Designed Card Or The Blank Card But Still Same Information On Them. EMAIL: (legithackers@outlook.com)