10 ms·
Thousands of Delta passengers delayed by computer outage
- timthelion 10y agoPart of me, thinks that their systems were attacked, but being into security, I've though a lot about how to attack a large computer system like this. Since server failiures are so common, the system should be able to handle N failiures before anything bad happens. If this was an attack, I guess the attacker must have found a flaw that allowed them to instantly compromise any server. But it actually seems more likely that this was caused, merely, by incompetence. If the attack was possible, than it was just as likely, if not more, that Delta managed to merely shoot itself in the foot.
- LaSombra 10y agoI tend to think it might have been a software/hardware update failure of some kind.
- abhianet 10y ago"Never attribute to attacks that which is adequately explained by bugs."
- nradov 10y agoIt's like how some of my relatives immediately suspect a "virus" whenever they suffer any sort of computer failure. Nope, no malware, just typical buggy software.
- spyrosg 10y agoI'm one of the lucky people who have to wait. The airport personnel tells us it's the system that prints out weather information that's out. Not sure why one would want to attack that.
- phil21 10y agoConsidering check-in and reservation status in general appears to be down I'm guessing this outage is relatively widespread. Assuming what you've heard is correct this seems to span multiple Delta systems.
- ethbro 10y agoIf it is power issues and some bit of infrastructure deep in the back end took a nosedive, wouldn't be surprised if it effects multiple front end systems.
- snuxoll 10y agoConsidering it's an airline, it's likely either a IBM i or a IBM Z System where they track everything relating to their flights (schedules, reservations, checked luggage, etc). A lot of companies (mine included) seem to thing these devices are high-availability giants and never test what happens should they go down. I know if ours ever took a dump we would be in some big trouble (though we could function for at least 24 hours).
- rasz_pl 10y agoI am having a strong Deja Vu right now. I remember EXACTLY the same scenario happening a ~year ago. Hmmm http://qz.com/417617/every-united-airlines-flight-was-grounded-this-morning-in-the-us/ http://qz.com/417617/every-united-airlines-flight-was-ground... and again http://www.pcworld.com/article/2945552/united-airlines-flights-grounded-by-network-connectivity-issues.html http://www.pcworld.com/article/2945552/united-airlines-fligh... and again http://newyork.cbslocal.com/2016/05/29/computer-system-shuts-down-at-jfk-terminal-7-passengers-say-airline-staff-handwriting-boarding-passes/ http://newyork.cbslocal.com/2016/05/29/computer-system-shuts...
- vetler 10y ago> Not sure why one would want to attack that. Well, that would make it less obvious. 1. Take a short-position on Delta on the stock exchange 2. Sabotage a piece of infrastructure that's important to operations 3. ??? 4. Profit! :)
- peter303 10y agoAttack the power systems which security overlooks. Sounds like this case.
- lordnacho 10y agoI'm sure a lot of people are asking how this is possible. - Legacy systems: the thing is probably old and has had a lot of patchwork done to it by various people, many of whom are retired by now. - The thing is also large and distributed, which makes noticing potential failures quite hard. There are loads of postmortems on the web about how some minor issue like a line going down caused some cascade of unforeseen problems. There's a lot of lessons being learned about how to solve this problem though, so I suppose there's hope. - Always on: it's hard to change something once the business is relying on it. Loads of staff need to be shown how it works, which makes it hard to incrementally improve the thing because "how it works" would be something that people would have to be updated on. - Never finished: large organisations will always have loads of feature requests and bug fixes. If you start responding to some of them, you might find yourself swamped. Either you work on the next item, or you spend valuable time making sure fewer items are fixed. It's a balancing act.
- planetjones 10y agoI would add Resourcing to the list. Have Delta airlines joined the mob of traditional big businesses who decide technology is expensive and send it to the cheapest possible? Alternatively it could just be an catastrophic infrastructure failure.
- brightball 10y agoNot likely. All of Delta's scheduling software is in straight assembly. Deltas generally regarded as the well oiled machine of the airline industry. You're paying for always on time and consistent quality. It's one of the reasons a story like this is surprising.
- spyrosg 10y agoInteresting! Are there any source listings anywhere? Do they have particular programming tools or methods that help them deal with the complexity? If it's all assembly, I'm surprised this is only happening now.
- mrweasel 10y agoThe most annoying part of bugs like this, is that we rarely get a good post-mortem. To frequently the media just reports it as "a computer glitch". You would think that with computers being so important to modern day life, we should get better answers. I'm not suggestion that the bug should be explained in details in the evening news, but at least give some indication of the nature of the problem. They could just refer to a website, if people want the technical details. When a car has a problem, it's not reported as a "glitch". Imagine the stupidity of a media report claiming "A glitch in the latests Ford models cause several drivers to crash". You don't do that, you explain that it's a fault in the steering, breaks, electric systems, or where ever the issue lies. I think we should reasonably expect the same when errors in computer systems hit the news.
- xixixao 10y agoI'm afraid that's because the target audience has (on average) that level of technical knowledge (in both cases)...
- planetjones 10y agoOne of the main reasons that you don't get a good post mortem is that it's often directly related to human error or a severe defect brought about by a decision to cut costs. There have been several huge outages in UK banking over the past few years and management have a convenient excuse in blaming "legacy systems". While I respect private enterprise, for systems of systemic importance I would like to see some legislation to implement a mandatory and detailed post mortem like you propose.
- mrweasel 10y ago>it's often directly related to human error or a severe defect brought about by a decision to cut costs I wonder if companies do cost-benefit analysis on those kinds of situation. "We saved X amount of £ on cutting costs. Outages as a result of cost cutting Y amount of £". You would think that at least stock holders would insist.
- 10y ago
- cstross 10y agoIt's not software or hardware this time: there's a crippling power outage at ATL, which is Delta's main hub (and presumably where their mainframe herd is based): http://fox61.com/2016/08/08/delta-airline-reports-outage-everywhere-systems-down-and-flights-grounded/ http://fox61.com/2016/08/08/delta-airline-reports-outage-eve... The outage began at 0230 and is presumably ongoing and beyond the capacity of their backup generators to cope with.
- gtirloni 10y agoSame here: http://news.delta.com/730-am-et-update-outage-affects-departing-flights-today http://news.delta.com/730-am-et-update-outage-affects-depart... A power outage in Atlanta, which began at approximately 2:30 a.m. ET, has impacted Delta computer systems and operations worldwide, resulting in flight delays 8:40 a.m. ET UPDATE: A Delta ground stop has been lifted and limited departures are resuming following a power outage in Atlanta that impacted Delta computer systems and operations worldwide. Cancellations and delays continue.
- bgroins 10y agoI'm quite surprised that their whole operation is region dependent. I work with hospitals on high availability/continuous availability of critical systems and would think that a company as big as Delta would consider a regional power outage as a high risk and plausible scenario.
- chiph 10y agoHave you flown through Hartsfield-Jackson? There's six concourses with 30+ gates each, nearly all of which is Delta. Their operations are highly-concentrated there. That being said, I expect them to upgrade their generators to handle outages like this, and distribute more of their IT infrastructure. Unlike the typical summer storms in the Atlanta area that delay flights, this affected them worldwide.
- hueving 10y agoAs a passenger on the west coast flying to another west coast city, I give no shits about the number of concourses in ATL. LAX still has electricity but delta's poor infrastructure planning has left a single point of failure in the deep south affecting flights everywhere in the world.
- yardie 10y agoI just don't understand how a large carrier such as Delta can get so much wrong. What's the DR plan? Are they using ASNs? A company is product, people, and computers. This is not the 20th century, flight plans aren't filed by telephone calls for these guys. I've tested our DR plans and we were operational after 4.5hours, optimally. And we were 3 people. This outage reeks of PHBs doing lots of kicking the can and not a whole of implementing.
- xjlin0 10y agoSounds like your DR plan or your network don't require power. Care to share?
- yardie 10y agoYes, 2 datacentres, Microsoft's DFS, cloud backups, and staff trained to know what to do. For the datacentre I have a friend that we worked out a mutually beneficial arrangement. I keep a few spare servers in his racks and he has a few in ours. Just enough to run the essentials. Testing was literally walking up to the rack and switching off the power. Then documenting what happens next. If you've configured the ASN just right the domain should roll right over to the backup servers. Offsite DR servers needed a little kick to bring up to speed (I can't remember why this wasn't fully automated). Most of the time was waiting for backup files to restore and verify. Since we were using someone else's network we were shaped down to a small percentage of their bandwidth. After that we had a debriefing to see what worked, what didn't work. The usual IT stuff.
- coldcode 10y agoPower failure, clearly their back power systems are inadequate. I believe some of Delta's systems are also backed by SABRE which while ancient is generally rock solid (it lives in a nuclear hardened bunker in Tulsa). This problem is in Delta's own systems not SABRE.
- useful 10y agothe main host of delta is not sabre, it is deltamatic, a homebrew system that is run by delta. Most other airline systems like sabre communicate via teletype to one another
- smn1234 10y agoshould be on AWS... all the cool kids are doing it
- peter303 10y agoMerged airlines with multiple legacy systems from 1980s. The legacy systems are well battle tested over the decades, but brittle to add new capabilities like mobile. Only about a third of of from-scratch large corporate and government software systems succeed. Having to hire the cheapest contractor doesnt always work.
- smn1234 10y agofinance industry as well... They're sunsetting legacy systems from many years of M&A, refactoring apps, adding API tiers, using novel tech stacks, building in true-HA and cross-region active-active.
- hiimnate 10y agoI was expecting a "computer bug" story on the front page of HN to actually have some interesting content.
- DickingAround 10y agoTough day for them. Moving to 'cloud' hosting... it doesn't just make it someone else's problem. It encourages good architecture.
- martin1b 10y agoWhy is it reported as a bug rather than a power outage?? There is a huge difference.
- tedmiston 10y agoIt seems a bug in the infrastructure when they switched generators is what caused the power outage.
- baus 10y agoInterestingly, 10 flights were canceled on Allegiant today. Including one I'm booked on. But apparently it is a normal mode of operations for Allegiant. Absolutely horrid airline. https://www.allegiantair.com/travel-alerts https://www.allegiantair.com/travel-alerts
- tedmiston 10y agoAllegiant a budget airline that's okay when it works and a pile of junk when it doesn't. They fly routes a few days per week out of my city. In a recent cancellation (by choice), they told customers that they could rebook them on the same day next week. Somehow they managed to do this without offering refunds or alternative flights back that day, the next, etc. I've only flown with them once, but it was a popular route to Vegas. It was fine for the price, especially since they don't surge as much as normal domestic airlines when you book with less than 2 weeks notice.
- intrasight 10y agoTime for Delta to move to AWS
- rbc 10y agoAWS has outages too. There might be some perceived attraction to transferring the operational risk to AWS, but Amazon's agreements limit their liability.
- rbc 10y agoIt sounds like some kind of single point of failure in the Delta architecture was uncovered.
- tedmiston 10y agoOfficial post from Delta Operations - http://news.delta.com/more-flights-resume-delays-cancels-continue-after-power-outage http://news.delta.com/more-flights-resume-delays-cancels-con...
- tedmiston 10y agoA post in the ongoing outage thread on FlyerTalk mentions a fire in the datacenter. > According to the flight captain of JFK-SLC this morning, a routine scheduled switch to the backup generator this morning at 2:30am caused a fire that destroyed both the backup and the primary. Firefighters took a while to extinguish the fire. Power is now back up and 400 out of the 500 servers rebooted, still waiting for the last 100 to have the whole system fully functional. http://www.flyertalk.com/forum/27032000-post135.html http://www.flyertalk.com/forum/27032000-post135.html
- fflluuxx 10y agoSouthwest had an outage two weeks ago. Its starting to feel fishy.
- _acme 10y agoWould you mind elaborating on what is starting to feel fishy to you and how whatever that is is starting to feel fishy?
- nodesocket 10y agoStrangely, their stock price was not affected by the outage and this news at all. Honestly, I don't get it. https://www.google.com/finance?q=NYSE%3ADAL&ei=ncSpV_m8MsG9iwKpnrSQCQ https://www.google.com/finance?q=NYSE%3ADAL&ei=ncSpV_m8MsG9i...