13 ms·
United Airlines System-Wide Computer Problem
- danso 11y agoVery little news about this on Google News, but heard over the local Chicago ABC affiliate that the FAA attributed this to an "automation error" Edit: And its Twitter account has been relatively inactive, with more than 30 minutes since the last reply-to or general tweet...presumably a lot complaining tweets have come in in the last 30 minutes https://twitter.com/united/with_replies https://twitter.com/united/with_replies
- lectrick 11y agoJust a minute ago they tweeted about it: https://twitter.com/united/status/618769524544942081 https://twitter.com/united/status/618769524544942081
- ceejayoz 11y agoTook them until just now to go "oh, we should probably post something that isn't a @reply". https://twitter.com/united/status/618777538865799168 https://twitter.com/united/status/618777538865799168
- fnordfnordfnord 11y agoI bet they could have saved themselves untold numbers of telephone calls in the support queue, and thousands of in-person queries to ticket agents and other airport staff with one or two tweets and a facebook post or two.
- caractacus 11y agoNot a word about it on United's web site. Flight status page doesn't load correctly. "Today's Operations" gives an error message. United's Twitter is silent. Meanwhile news articles and twitter complaints abound. http://mashable.com/2015/07/08/united-computer-problems-flights-grounded http://mashable.com/2015/07/08/united-computer-problems-flig...
- DangerousPie 11y agoOfficial status page: http://www.fly.faa.gov/ois/jsp/summary_sys.jsp http://www.fly.faa.gov/ois/jsp/summary_sys.jsp Currently says: ATCSCC ADVZY 027 DCC 07/08/15 UAL GROUND STOP REVISION DESTINATION AIRPORT: ALL AIRPORTS FACILITIES INCLUDED: ALL FACILITIES GROUND STOP PERIOD: 08/1200Z - 08/1315Z REASON: USER REQUEST AUTOMATION ISSUES
- kieranelby 11y agoGood to see the Sun Microsystems favicon on the FAA status page - not seen one of those for a while!
- fnordfnordfnord 11y agoLast one I noticed was the FCC's comment system.
- gesman 11y agoInteresting bits: "Departing DEN; taxied and then returned to gate. Pilot says nationwide failure of "three or four" computer systems. Only information from airport staff is that since the computers are down UAL can't book pax onto any other airline ..." "Systemwide Ground Stop posted at FAA: Due to USER REQUEST DUE TO AUTOMATION ISSUES. UAL AND SUBS ONLY., departure traffic destined to ALL airport will not be allowed to depart until at or after 13:15 UTC."
- scrumper 11y agoThat's twice in a short time now - 'coincidence' on Ian Fleming's scale. A third time is enemy action. Airlines do seem like a pretty juicy target for cyber war operations - you can cause a gigantic amount of disruption with a successful attack on a single system.
- eli 11y agoThat seems like quite a leap given how often complex computer systems fail without any malicious act.
- mnw21cam 11y agoIndeed. Never attribute to malice that which is adequately explained by stupidity. https://en.wikipedia.org/wiki/Hanlon's_razor https://en.wikipedia.org/wiki/Hanlon's_razor
- dlgeek 11y agoOr, you know, the load is increasing over time, and the system is failing to scale to it, and so will fail more and more frequently as the load continues to increase?
- joezydeco 11y agoThe amount of daily UAL passenger seats doesn't change or grow radically over weeks or months. What is the scale-up here? Everyone using their phones to check rez/status/boarding passes?
- dlgeek 11y agoSome quick googling suggests they take delivery of several-many dozen planes a year (based on various order/delivery/etc. coverage), which would absolutely cause a step function in the number of passenger seats. I expect there's growth on all of these: - Total planes - Total flights - Number of routes - Number of passengers - Passenger utilization of electronic boarding - Passenger utilization of reservation modification (seat changes, upgrades, etc.) - Passenger utilization of in-flight electronic amenities (wifi, in flight entertainment, etc.) that are billed through the system (United has a proprietary WiFi that's tied to your frequent flier account) - Online booking - Travel agent/reseller pricing queries - Travel agent/reseller booking I'm not saying any of these are dramatically increasing, but I'd bet they're all going up slowly, which will add more and more load to the system.
- johnrydell 11y agoLet's just be thankful that we haven't had a major air-based catastrophe due to an outage or hacking!
- userbinator 11y agoPlanes with pilots in them are mostly autonomous and can avoid each other as well as the ground. https://en.wikipedia.org/wiki/Ground_proximity_warning_system https://en.wikipedia.org/wiki/Ground_proximity_warning_syste... https://en.wikipedia.org/wiki/Traffic_Collision_Avoidance_System https://en.wikipedia.org/wiki/Traffic_Collision_Avoidance_Sy... https://en.wikipedia.org/wiki/%C3%9Cberlingen_mid-air_collision#TCAS_and_conflicting_orders https://en.wikipedia.org/wiki/%C3%9Cberlingen_mid-air_collis...
- chinathrow 11y agoYes, mostly. https://en.wikipedia.org/wiki/Vitaly_Kaloyev https://en.wikipedia.org/wiki/Vitaly_Kaloyev
- drzaiusapelord 11y agoI wouldn't get on a Russia autonomous flight for a million dollars. Between the corruption, hackey engineering, and complete disregard for safety standards, Russian accidents surprise no one. The Tu-154 accident list is long and scary. Hell, look at all the fires that broke out. https://en.wikipedia.org/wiki/Tupolev_Tu-154#Incidents_and_accidents https://en.wikipedia.org/wiki/Tupolev_Tu-154#Incidents_and_a...
- JHof 11y agoAutomated, yes. Autonomous, no. The plane voices an alert, but the pilots still have to react to it.
- userbinator 11y agoI meant autonomous with respect to the ground.
- lectrick 11y agoI have a nagging (intuition? feeling?) that software safety/reliability/security needs are going to explode soon (because unreliabilities multiply in non-resilient systems interacting with each other) and that these are simply foreshocks. (yeah, I know security is already a huge deal, but as we come to trust software systems more and more, the safety/reliability factor will come more into play) EDIT: This is also part of the reason I've been learning Elixir (http://elixir-lang.org/ http://elixir-lang.org/) since it's based on the highly-resilient Erlang and is designed to embrace failure. This was also informed by me reading Nassim Taleb's book "Antifragile" as well as "Thinking in Systems: A Primer" by the (late) Donella Meadows.
- yAnonymous 11y agoI doubt it. Nobody wants to pay for that.
- DougWebb 11y agoYou might be right, unfortunately. If it's cheaper to buy insurance that will cover the (expected) losses caused by outages, most organizations will choose to do that instead of making the software more failure-resistant. The problem is that insurance only works well for isolated incidents, but a software failure can cause a cascading failure with a huge impact. Insurance companies generally aren't prepared for that and don't have the resources to pay out to everyone.
- sopooneo 11y agoBut aren't the insurance companies smart enough to figure this out and start correcting their rates to be much higher?And if they actually have their acts together, wouldn't those same insurance companies start insisting on basic audits of their client's systems? I actually don't know about this stuff, so any correction of my thoughts is appreciated.
- TeMPOraL 11y agoA cynic in me feels that someone will figure out the problems with cascade case and insure from failure of insurance companies to pay out insure money. Just like during 2008 financial crisis.
- chinathrow 11y agoNice reminder about a "glitch" happening in one of the datacenters use at the airline I used to work. Went like this: Guy who shows around the new datacenter/ops guy demonstrated how the emergency power off works by lifting the protection plate. Protection plate unhinges suddenly and droppes onto emergency power off button. Hilarity ensues.
- ianhawes 11y agoI remember one of the LiveJournal outages (IIRC) was caused by a dude in the datacenter thinking that the emergency power off button opened one of the doors.
- misterbwong 11y agoHad a similar experience at an old company. One of the light switches in server room was placed next to a switch connected to the UPS of certain servers (I have no idea how/why this even came about). Confused the heck out of us when we were trying to figure out why some of our servers went on UPS power randomly at night. Turns out we'd get the notifications on nights the cleaning crew decided to flick the lights off then on again.
- js2 11y agoAt former managed-hosting startup, we would deploy a pair of EMC symmetrixes into our DC cage as storage for our DB tier, with volumes mirrored over the pair. While equipment was being moved into the cage one day, someone accidentally banged the power-switch protection plate on one of the EMCs. Fortunately the power switch had to be held depressed for several seconds to turn off the equipment. Unfortunately, the protection plate itself got jammed in against power switch keeping it depressed. Hoisted by its own petard, as it were. Good thing we deployed those EMCs in pairs. But then curiosity gets the better of a DC ops guy wondering how the plate got jammed. So he punches the plate on the 2nd EMC causing it to similarly jam and power-off. Doh. When we told EMC about this flaw in the design, they deployed a fix to the production line - a rubber stopper under the plate next to the switch to protect the switch from the protection plate.
- TWAndrews 11y agoI was able to get checked in via the United android App around 9am ET.
- raus22 11y agoOff-topic: Please use ISO 8601 format(YYYY-MM-DD) for dates in titles. The US date format hurts my poor logical soul. https://en.wikipedia.org/wiki/ISO_8601 https://en.wikipedia.org/wiki/ISO_8601
- interdrift 11y agoIt's still month 7 so it's hard to be confused. PS. From Europe.
- notNow 11y agoBut a month from now, some people will be confused for sure. So, it's more future-proof to settle for ISO or alphanumeric format to avoid confusion and miscommunication.
- mnw21cam 11y agoOr just use words (8th July 2015). But yes, YYYY-MM-DD actually sorts correctly.
- r00fus 11y agoISO all the way. Words have issues for different month languages (e.g. Auot vs. August), but otherwise is more elegant (no delimiter needed, e.g. 08JUL15)
- pavel_lishin 11y agohttp://www.telegraph.co.uk/news/ww1-archive/11721622/Daily-Telegraph-July-8-1915.html http://www.telegraph.co.uk/news/ww1-archive/11721622/Daily-T...
- anc84 11y agoIt's called "8. Juli 2015" where I live. Use numbers, please.
- sk5t 11y agoSee also: https://xkcd.com/1179/ https://xkcd.com/1179/
- peterjmag 11y agoOuch. And only a month after another major systems outage: http://www.wired.com/2015/06/united-flights-grounded-mysterious-problem/ http://www.wired.com/2015/06/united-flights-grounded-mysteri...
- jonawesomegreen 11y agoI bet this is an issue with an old mainframe used somewhere in the booking system, something that has worked well but is difficult to fix when things go wrong. I think there is / will be a lot of money to be made trying to solve the problem of software security and reliability. This is obviously an extremely difficult problem, however the number of ancient systems that we currently have interconnected I think more large scale outages like this are inevitable.
- caminante 11y agoI submit there's more money to be [saved] from not replacing old systems. Case in point, take this HackerNews fan favorite[1] about a school district using an "ancient system" and avoiding $M's in incremental costs. [1] https://news.ycombinator.com/item?id=9705830 https://news.ycombinator.com/item?id=9705830 edit: As mentioned in [1], I assume that at least someone is aware of the cost-benefit of potential projects; in turn, I assume that someone would've pulled the trigger if the $'s make sense.
- igrekel 11y agoTruth is that most reservation systems are built on TPF and it isn't really easy to replace.
- imgabe 11y agoUnited is a mess. I had the misfortune of flying with them a couple months ago. I ended up in a city a 2 hour drive from my actual destination and had to rent a car on my own to get to where I was going. That was the worst of it, but almost every flight I saw on the way (both ones I was on and other flights at nearby gates) was delayed or overbooked or otherwise messed up in some way.
- smackfu 11y agoWas there a storm or something? Usually there is some motivating reason why flights are screwed up.
- imgabe 11y agoNot that I know of. My particular flight was delayed because they had to replace a piece of the navigation equipment on the plane. This delay caused me to miss a connecting flight and that's how I ended up in an entirely different city.
- drzaiusapelord 11y agoThis happens with every airline. No airline has magical planes that never break and they all have similar maintenance schedules on the exact same planes running the exact same workload. Blame Boeing for making a crappy plane if they keep breaking down.
- imgabe 11y agoI wasn't so annoyed that the flight got delayed and I got stranded in the wrong city, it was the way they handled it. United's response was: 1. Do nothing. You're stuck here. We'll get you on the next flight. Oh, the next available flight isn't for over a week. Sorry. No, we can't pay for your rental car. 2. Upon renting my own car, and writing to customer service to complain I got a $125 voucher for United. Great. Not enough to buy an entire ticket, so it just ensures I'll have to continue giving money to this airline that failed to deliver what I gave them money for in the first place. 3. After several weeks of emails back-and-forth with customer service finally a manager agrees to issue a reimbursement for the rental car as a one-time exception to their policy of never doing that. (but not the $7 worth of gas. I guess they just have to draw a line somewhere? Oh, and they also rescinded the voucher. No big deal since I'm not too keen to fly with them again anyway) So, yeah. Sometimes shit goes wrong when you travel. Everybody knows that. How airlines act to fix it makes all the difference. If you hamstring all your customer service reps so they can't actually solve someone's problem, it makes something that's already annoying way more frustrating.
- vaadu 11y agoHow do you know it's a glitch(whatever that means) and not a big problem such as a data center outage or hacker caused?
- oaktowner 11y agoYeah, "glitch" seems to imply something minor, while this seems anything but.
- aaronkrolik 11y agoIs anyone familiar with the UA tech stack? I'd be curious to see what they're running.
- imroot 11y agoThey're running SHARES internally; most everything else is specific applications with hooks into shares for data processing and return.
- jaybna 11y agoI knew someone at United that once offered to give me a tour of one of their data centers: "It is like a computer museum - we have one of everything." Hard to imagine that they would have problems as a result. United is a really, really bad airline.
- Sukotto 11y agoWhat does "all airports" mean? Global or just US domestic?
- fnordfnordfnord 11y agoUS domestic would be my guess. ie: All airports under FAA authority.
- kendallpark 11y agoWas reading this on HN and heard it on NPR simultaneously. I have a sneaking suspicion that booking systems for most airlines run atop legacyware. It just seems like the type of thing that would've been put in place long ago and then be very expensive to migrate/updgrade.
- joezydeco 11y agoOh, it's not a suspicion. http://www.pacbiztimes.com/2012/04/06/united-takes-a-step-back-in-time-by-adopting-outdated-system/ http://www.pacbiztimes.com/2012/04/06/united-takes-a-step-ba...
- kendallpark 11y ago> The biggest problem, one that would drive any tech-savvy user crazy, is that United junked an award-winning, state-of-the-art reservation system and adopted the Continental Airlines model based on older technology known as System One. Well, that answers that.
- DannyBee 11y agoConsidering just yesterday their flight system didn't believe they flew from SFO for 2+ hours in the morning (I have screenshots), i'm not all that shocked.
- carnesen 11y agoI worked on United's computer systems for a year (never that one though), and so I get nervous when I see a headline like that. True story: one of their systems still runs on a mainframe that has 9 bits in a byte!
- themartorana 11y agoI did not know that was ever a thing. I knew that bit counts had been lower - 5 or 6 - but assumed once we hit 8, the whole power-of-2 thing was too comfortable in a binary system to ever be anything besides a power of 2 again - and sure enough, we get 16 bit, 32 bit, and 64 bit systems, and double-byte and triple-byte char sets, etc. Need more space? Take another byte. Was the 9th bit special? Or just a standard bit in the byte?
- Lx1oG-AWb6h_ZG0 11y agoSomeone probably read too much Iain M. Banks and decided to adopt Marain's base-9 system :) If I'm not mistaken, some old consoles used a 9-bit RGB encoding, so this could theoretically help there. Minecraft also uses a 9-bit system for Redstone, afaik.
- chiph 11y agoThe PowerPC based AS/400 systems were technically 65-bit machines, as the extra bit was a privilege flag to separate system code from user code. Hardware enforced security - one reason why it is such a stable machine.
- Someone 11y ago36 bit was popular for scientific computing because 35 gives you a sign bit and 10 decimal digits (yes, that's weird, but that's the argument I read everywhere, including at https://en.m.wikipedia.org/wiki/36-bit https://en.m.wikipedia.org/wiki/36-bit. 35 likely was skipped because it its only divisors are 5 and 7, limiting instructions to 7 bits even then was felt to be too restricting. For some architectures, the DoD had a say in this, too. See https://en.m.wikipedia.org/wiki/Unisys_2200_Series_system_architecture https://en.m.wikipedia.org/wiki/Unisys_2200_Series_system_ar...) 36 bits got us the 6-bit character (10 digits, 26 letters, and punctuation) with six characters in a word. Because of that, some OSes had six-character file names. If you want to get upper- and lowercase, you need more than 6 bits. 9 is the smallest divisor larger than 6 of 36, so nine-bit characters made sense. On such systems, file names could still use 6-bit characters, while applications used 9-bit ones. Also, some instructions could work on words, half words, quarter words, or sixth words.
- bst287 11y agoCurrently in the air on a United flight, EWR -> SFO. Took off at 7:30ish. No mention of this in the airport or on the plane. Yikes
- pavel_lishin 11y ago"Automation error" rings a funny bell in my head, since I'm currently re-reading "A Fire Upon The Deep".
- rwestergren 11y agoPerhaps someone wasn't following the bug bounty rules?