10 ms·
All American Airlines Flights Grounded, Experiencing Nationwide Computer Outage
- akcreek 14y agoI'm at DFW right now heading to LAX on AA. Computers just came up and they are going to begin boarding shortly. Lots of unhappy travelers here as you can imagine I'm sure. Edit-- To follow up. It's been about an hour since the computers came back up in the terminal and I'm finally in my seat on the plane. About 3 hours after it was supposed to depart. There were several canceled flights in my terminal and no other flights going out so people had to stay overnight. Most were very mad as American was not supplying food or hotel vouchers - all those people are coming out of pocket I guess. I heard quite a few saying things like, "No wonder they are going out of business" and "I will never fly American again no matter what".
- deleted 14y ago[deleted]
- w1ntermute 14y ago> I heard quite a few saying things like, "No wonder they are going out of business" and "I will never fly American again no matter what". Yeah, they'll never fly American again no matter what...until it has the lowest price ticket for the time/destination they want.
- impendia 14y ago> American was not supplying food or hotel vouchers - all those people are coming out of pocket I guess. Why are they not subject to a class action lawsuit? The problems rather obviously seem to be American's fault. Seems like some enterprising lawyer should buy a ticket, drive to the airport, go to the AA gates, and start handing out business cards to everyone in sight.
- bdunbar 14y ago> go to the AA gates You have to be a ticketed passenger to get _to_ the gates.
- w1ntermute 14y agoThen buy a ticket. It'd be a worthwhile expenditure.
- joelrunyon 14y agoGreat answer (but if the computers are down, can he buy a ticket?).
- saryant 14y agoSchedule a meeting in the AA Admiral's Club. The agents will give you a pass to get through security without a ticket.
- dredmorbius 14y agoIt'd be worth the cost of a cheap ticket. Problem is: the reservation system's down.
- dragonwriter 14y ago> Why are they not subject to a class action lawsuit? Uh, because it takes time to file those. Also, I suspect, because they have binding arbitration and anti-class-action clauses in the fine print of ticket purchase agreements.
- akcreek 14y agoAnother update. I'm still here. We are parked at the gate waiting for additional fuel before we can leave. Not sure why they boarded us just to have us sit here for another hour (or more - no clue how long this will end up taking). The good news is the plane has AC power in my row so I was able to build the navigation for a site I'm working on.
- hkmurakami 14y ago>Not sure why they boarded us just to have us sit here for another hour (or more - no clue how long this will end up taking). Every airline seems to have been doing this for decades now, and it boggles my mind why they do this. Do they think that getting people "on board" is demonstrating "progress! breakthroughs! we're moving forward folks!" or something and it's supposed to make us suddenly happy? I'd so much rather spend the extra 2 hours at the gate than crammed into a 737 for the same amount of time, especially when they decide that they can't turn on the AC (when I was little, we were stuck in a fully packed 747 for 4 hours with no AC. They opened the front and back doors b/c it was so unbearable in there...)
- shalmanese 14y agoIt's because that's one of the metrics which is tracked and, therefore, optimized for, even at the detriment to the core business.
- sebcat 14y agoHaving spent time in the military, it makes sense to me. Hurry up and wait :) I actually like that way of doing things. Boarded a flight recently and fell asleep at once after sitting down. Woke up three hours later, plane was descending. It was supposed to be a 1.5h flight. Oh well.
- ivix 14y agoDeparture slots. When one comes up you have to be ready to go immediately.
- 14y ago
- bargl 14y agoBecause it's a mechanical failure and not an "act of God" they are responsible for any lodging/food for their passengers due to this delay... I don't know if that's a legal thing, but I do know that's how every airline I've ever flown on works.
- pc86 14y ago"Mechanical failure" typically refers to things that are... mechanical in nature, e.g. not software. They are most likely interpreting this under the unforeseen events clause of the ticketing agreement (not sure of the exact wording) to see if they can get away with not paying anybody. If enough of the passengers raise a fuss they will likely offer everyone some sort of voucher, but 90% of people unfortunately will forget about the whole thing 30 minutes after they land.
- bargl 14y agoYeah that's a pity, it sucks when only the loud obnoxious people get their way...
- reagan83 14y agoI fly the DFW to LAX route once a week and have done it on many different carriers (United, AA, and Virgin America). Even though VA offers fewer flights on this route, that has become my sole carrier (even outside of today's events). They handily beat AA on experience, and sell at the exact same ticket price (main cabin, not FC).
- alex_doom 14y agoVirgin has won me over completely.
- akcreek 14y agoOkay, last update. I made it to LAX, picked up my rental car then received an email from American notifying me that my flight was on time - the same flight I just got off of that landed over 4 hours after it should have. I had to laugh at that one.
- edandersen 14y agoNice to see their legacy systems were kept up to date and maintained.
- umsm 14y agoI guess this is part of the "maintenance"... :)
- billiam 14y agoIf you board, you will be on the tarmac for a while: http://www.dallasnews.com/business/airline-industry/20130416-all-american-airlines-flights-now-grounded-nationwide-till-at-least-7-p.m..ece http://www.dallasnews.com/business/airline-industry/20130416...
- uvdiv 14y ago"A while" -- no kidding! UPDATE: 3:02 P.M.: The Federal Aviation Administration has advised that American Airlines has extended its ground stop to at least 7 p.m.
- deleted 14y ago[deleted]
- SODaniel 14y agoFor a minute I had a '24' flashback and started packing canned goods and ammunition into a 'bug out bag'..
- TallboyOne 14y agoI like how it says "apologize for ANY inconvenience" as if theres the possibility that all planes being grounded making hundreds of people late for meetings offers any room for there not to be an inconvenience.
- akcreek 14y agoI don't have anything scheduled until the morning luckily so no big deal for me. I was more inconvenienced by the fact that they don't have enough power outlets to charge devices here (DFW).
- jmathai 14y agoOr worse, parents with young children stuck at airports losing their mind.
- ryanmarsh 14y agoTo me the tone of that tweet sounded very insincere.
- pchristensen 14y agohttp://37signals.com/svn/posts/1528-the-bullshit-of-outage-language http://37signals.com/svn/posts/1528-the-bullshit-of-outage-l...
- _red 14y agoThats a great post, but the thing is the phrase "we apologize for any inconvenience this may've caused" has a very useful psychological purpose: Diffusion. Lets assume that I know you are mad, possibly pissed. Possibly so pissed that you are going to move your business away to someone else. By saying "sorry for the inconvenience", I diffuse your feelings....make it sound like "I'm sure you preferred to get there earlier....but no biggie, right?" By giving you a sincere, heartfelt apology I validate all those reasons of why you are pissed. Gives you justification for your further actions. Regardless, I don't see why it even matters to AA in this case. Thanks to FAA/Local Airport they have de facto monopolies for certain routes, in reality they don't need to apologize at all. (god, I wish competition existed in air travel).
- w1ntermute 14y agoWhy haven't carriers like AA been displaced by more competent (and passenger friendly) alternatives yet?
- soupboy 14y ago"How do you become a millionaire? Make a billion dollars and then buy an airline." -- Warren Buffett [http://www.fool.com/investing/general/2012/05/23/if-you-have-to-invest-in-airlines-dont.aspx http://www.fool.com/investing/general/2012/05/23/if-you-have...]
- callmeed 14y agoTo add to that, Virgin America is a favorite airline yet struggles to make a profit http://www.economist.com/blogs/gulliver/2012/10/virgin-america http://www.economist.com/blogs/gulliver/2012/10/virgin-ameri...
- drpancake 14y agoThought that was a Branson quote: "If you want to be a millionaire, start with a billion dollars and launch a new airline." -- Richard Branson
- Encosia 14y agoThat sounds like something Richard Branson would say (I think I remember reading it in Business Stripped Bare), but I believe Buffett's quote predates Virgin Airlines entirely.
- nickpinkston 14y agoAirlines are a pretty shitty business: commodity product, many competitors, low margins, high capital costs, high regulatory risk, high exposure to ever-rising oil prices, etc. There's some areas in the higher end stuff, but right now JetBlue / Southwest are about as progressive as you get.
- mindcrime 14y ago
- drcoopster 14y agoAs a frequent US Airways flier, this makes me feel really warm and fuzzy about the upcoming merger with American Airlines.
- maxdemarzi 14y agoI got stuck for 2 hours waiting for "paperwork" flying from Chicago to Philly. At least I managed to finish my book.
- nwh 14y agoI'm picturing a disk failure, or a RAID5 with two dead drives. Someone at AA franticly running disk recovery software.
- toomuchtodo 14y ago"Oh god please work ddrescue...."
- monsterix 14y agoAndroid anyone?
- tpowell 14y agoI can't wrap my head around how extraordinarily complex airline reservations/logistics systems are. I'm kind of surprised you don't see this more often.
- mimiflynn 14y agoLooks like they use Sabre: UPDATE, 1:43 P.M.: The Federal Aviation Administration has confirmed the ground stop. “GROUNDSTOP IMPLEMENTED BY ATCSCC ON AMERICAN AIRLINES NATIONWIDE AND AMR AMERICAN EAGLE INTO DFW, LGA, AND ORD AT THE REQUEST OF AMERICAN OPS. STOP REQUESTED AS RESULT OF AAL SABRE SYSTEM FAILURE.” from http://www.dallasnews.com/business/airline-industry/20130416-all-american-airlines-flights-now-grounded-nationwide-till-at-least-7-p.m..ece http://www.dallasnews.com/business/airline-industry/20130416...
- mjn 14y agoLooking at Sabre's own timeline, I cringe at what I suspect the codebase must look like. From what I read between the lines here, it looks like neither a carefully maintained legacy system, nor a proper rewrite at any point along the way, but more like 50 years of things bolted onto each other: http://www.sabre.com/home/about/sabre_history http://www.sabre.com/home/about/sabre_history
- nessus42 14y agoI've been told that various companies have tried to rewrite Sabre on a number of occasions and always failed.
- clueless123 14y agoThere are other systems that work fine. Sabres real competive advantage is the existing customer base.
- rbanffy 14y agoI have the utmost respect for systems that have been running for decades without major rewrites, but having said that, I imagine there is a rock-solid core that has, literally, withstood the test of time, surrounded by semi-compatible systems that accreted around it since the mid-80's, with most current systems only touching this peripheral mass rather than the core services. I've had contact with many systems that have their cores running on IBM mainframes with peripheral systems running on .NET. I fear those.
- eddmc 14y agoThe hosting company that American Airlines uses is Sabre This is the same company used by Virgin America, US Airways, JetBlue and Alaska Airlines
- sjm-lbm 14y agoAlso, a bit of trivia: AA invented Sabre. Sabre can actually trace itself back to AA's first attempts at using computerized business process automation in the 1950s, before it was spun off into a separate company in 2000. Should anyone be bored enough for a read, I actually think SABRE's history is sort of interesting: http://en.wikipedia.org/wiki/Sabre_(computer_system) http://en.wikipedia.org/wiki/Sabre_(computer_system)
- eddmc 14y agoA really interesting book I read about this is Hard Landing by Thomas Petzinger Jr. This contains more than just a history of Sabre. I read it more because I work in the travel industry ;-) http://www.amazon.com/Hard-Landing-Contest-Profits-Airlines/dp/0812928350 http://www.amazon.com/Hard-Landing-Contest-Profits-Airlines/...
- SonicSoul 14y agoon the other hand Sabre's acquisition of Dunder Mifflin was a move in a questionable direction. http://dundermifflin.sabre-corp.com/ http://dundermifflin.sabre-corp.com/
- futhey 14y agoSABRE is more of a platform than a hosting company. It even had it's own programming language at one point.
- eddmc 14y agoYes, you're right. Hosting has a different meaning in the software industry - the correct term is 'Airline reservations system' but the colloquial term used in the travel industry is 'airline hosting'.
- tquai 14y agoJust a general comment on airlines: With US-based carriers, expect low prices, inflating awards points/miles, and bottom-level service. With non-US based carriers, expect higher prices, no awards programs, and excellent service. This is my experience flying in N. America, Europe, and Asia.
- mjn 14y agoYou must not be flying the low-cost airlines in Europe. I have never seen any US-based airline that rivals the combination of low-price and bottom-level service provided by Ryanair. Unfortunately, the low-end airlines are so popular within Europe that it's increasingly hard to avoid them, even if you want to pay more, because they have many of the direct flights (especially to vacation destinations).
- onemorepassword 14y agoUnless your destination is as low-budget as the airlines, it's pretty easy to avoid them. They're mostly the equivalent of a cheap holiday bus. Also, although I will avoid the EasyJets and Ryanairs as much as possible, for a direct flight the duration of a bus ride it doesn't really make that much difference.
- k-mcgrady 14y agoI don't think this is true at all. Europe's low cost airline business is major. Easyjet, Ryanair, Flybe and Jet2 just to name a few. Obviously the service on these carriers is awful but flying between European cities for $50 a flight isn't uncommon. Personally my most consistent good service experiences have been with Air Canada. BA was good too as was Aer Lingus. I've only had two flights with them but I wasn't fond of AA.
- toomuchtodo 14y agoSingapore Air is amazing, has a great rewards program, and is affordable.
- pyre 14y ago
- capkutay 14y agoI'd like to see a paper on American Airlines software architecture. Although I doubt it'll pop up on "High Scalability" any time soon. Do we know anything about Sabre's platform? I imagine its highly redundant..how could they have a nationwide outage?
- dunham 14y agoOn twitter, Sabre says they're not having any issues, and American Airlines retracted their statement pointing the finger at Sabre: Clarification: The issue is w/ our ability to access our res system & not w/ @SabreNews. We apologize to Sabre & customers for confusion. https://twitter.com/AmericanAir/status/324227267066736640 https://twitter.com/AmericanAir/status/324227267066736640
- dia80 14y agoI feel for the poor sods thrashing away trying to make this right. Do developers in mission critical areas earn a premium? This could knock a couple of years off your life in stress.
- Irishsteve 14y agoMost likely no. But the good news is everyone will assume they caused the problem.
- ryanmarsh 14y agoLOL, yah right. No they don't earn more. AA does not pay well. Also most of the legacy systems are maintained by contractors.
- runawaybottle 14y agoWhat do they earn?
- ryanmarsh 14y agoI'm told, "well below market". I've talked to probably 40 programmers there. Only one of them knew what any of the SOLID principles were as I named them off.
- georgemcbay 14y agoI have every reason to believe you when you say AA has bad programmers/contractors, but SOLID is just some random software process marketing buzzspeak and not knowing those particular "principles" isn't a useful indicator of anything other than not being exposed to that particular brand of process marketing buzzspeak. FWIW, I'm not attack any of the ideas of SOLID here, just the idea that knowing of or not knowing of any specific brand of process marketing buzzspeak is a useful indication of anyone's skill.
- ams6110 14y agoWhy would knowing the SOLID principles have any real relevance when the systems in question are probably written in COBOL and date back at least 4 decades?
- ari_elle 14y ago"American said the issue was caused by an inability to get access to its reservations system, called Sabre. The electronic system, often described as the brains of an airline, is responsible for bookings and reservations but also manages a wide variety of functions related to flights, including printing boarding passes, online check-ins, ticketing, and tracking checked bags. [...] Sabre, meanwhile, said American’s system outage did not come from its own computer systems. Other airlines, including Southwest Airlines and JetBlue, use the reservation system and have not experienced any outages, said Nancy St. Pierre, a spokeswoman for Sabre." Source: http://www.nytimes.com/2013/04/17/business/american-airlines-cancels-flights-after-outage.html http://www.nytimes.com/2013/04/17/business/american-airlines...
- cyanoacry 14y agoA similarly described SABRE outage in 1989 seems to have caused minimum trouble, because they had the real capability to work off-line with paper: http://www.nytimes.com/1989/05/13/us/computer-chaos-for-air-travelers.html http://www.nytimes.com/1989/05/13/us/computer-chaos-for-air-... > Apparently, no information about reservations and other travel plans was lost, and American said there were no serious disruptions of its 2,300 daily flights.
- pjbrunet 14y agoFrom the NYT article, sounds like it was a connection problem. Which makes you wonder, was it a DoS attack (either end of the connection?) preventing AA from connecting to Sabre?
- ryanmarsh 14y agoSABRE was fine. Some of the systems at AA couldn't talk to it. Having seen their systems none of this surprises me.
- jchrisa 14y agoSomebody probably let an API key expire...
- mikeash 14y ago
- nlh 14y agoPutting my tinfoil hat on for a moment... I wonder if there was a threat against the airline? Perhaps something enough to be credible but not so specific as to be pinpointable, and rather than cause widespread panic by saying "ground every AA flight until we sort it out" (which would freak people out beyond belief), someone figured out that pulling the plug on their mainframe for a few hours would have the same result without the panic. Just some pissed off passengers. Or maybe I've just been watching too much Homeland. Sigh.
- ryanmarsh 14y agoThat's just your paranoia. I wish I could speak more freely than I already have (see my other comments) but the fact is, crappy software, written by crappy contractors all tied to a 53 year old system of record.
- clueless123 14y agoNever ascribe to malice that which is adequately explained by incompetence. Hanlon's razor.
- jsight 14y agoAgreed... FWIW, I remember something similar happening to both US Airways and United Airlines within a few weeks of each other about 1.5 years ago.
- simcop2387 14y agoI can't find the article, but this isn't the first time that an airline has had a problem that grounded all of their flights. There was one in recent memory (last 20 years?) that had a bug where if there were more than 32k changes to the flights in a day it would cause a horrific crash in their system. They never hit it for so long, and then one day, boom all flights just disappeared.
- sterlingross 14y agoI am surprised no one has mentioned Rule 240. I have, more than once, gone up to the agent when a flight was delayed and asked about this rule. They always hesitate at first, but if you insist, they quickly make provisions to put you on another flight. This only works when it is the airlines fault and not an issue with weather. Rule 240: http://en.wikipedia.org/wiki/Rule_240 http://en.wikipedia.org/wiki/Rule_240
- meemo 14y agoPrescient article: http://www.technologyreview.com/view/513391/why-a-botched-it-project-will-destroy-a-major-corporation-in-the-near-future/ http://www.technologyreview.com/view/513391/why-a-botched-it...