10 ms·

United Airlines Domestic Flights Grounded for 2 Hours by Computer Outage

163 points by perseusprime11 10y ago

ngoldbaum 10y ago
Currently sitting on a plane at O'Hare, can't get to gate because they won't let any planes leave to clear a spot for my plane. What a nightmare.
erubin 10y ago
I have no reason to believe that there's any malice involved here. I also think that incidents like this might become more common as the state of cyber warfare progresses. As engineers, we should take care to build secure software, especially when that software underlies important systems. We should impress upon non-technical management the importance of doing so, even if it may take a little more time or money in the short term.
stevenhubertron 10y ago
Waiting in SLC terminal. Crew has to be off the clock in an hour. Not likely getting home tonight.
- teej 10y ago
  I got stuck in SLC two Sundays ago. Hotel tonight should have decent options. If you end up in town, the place will be dead. Bourbon House is decent and serves food late.
  - deleted 10y ago
    [deleted]
  - seanmcdirmid 10y ago
    Isn't Sundance going on now? That might overflow into SLC?
- freehunter 10y ago
  I love flying and getting to experience some of the weird issues that crop up. I was on a flight in December when there was only one ground crew working at my mid-sized airport. I was really hoping to beat the winter storm coming in, as I live pretty far from the airport. If there was a delay, I'd be stuck driving home in a blizzard. They were prioritizing getting planes to gates based on if the plane was picking up passengers and heading out again. Mine was the last flight of the day for my plane, so we were at the back of the line. After 30 minutes, we finally had an opening and were moving... until another flight crew reminded the tower that they were reaching their FAA allowed hours and needed to be parked immediately. They took our spot, and then more planes came in that had to leave. Of course I know all of this because our pilot was pissed and would relay all of the info to us with a very sarcastic voice. He was supposed to be home by now too. We ended up getting into our gate almost two hours after we landed. I normally live 30 minutes from the airport, and it took me another hour and a half to get home with the weather. The only reason we got in when we did is because the weather forced a shutdown of the airport, so we were literally the last flight to get parked at a gate, and only because there were no new flights coming in. It's fun (after the fact) to think about these edge cases, how to keep the entire airspace running smoothly when one airport only has a single ground crew working, how to maximize efficiency to make sure the fewest number of people are delayed. Sucks to live through it though.
  - guard-of-terra 10y ago
    The solution here to have more crews & gates. Will create some jobs along the way, and surely will cost less than the delays you described.
    - dingo_bat 10y ago
      How will hiring more crews and gates be cheaper than one flight getting abnormally delayed? I mean maybe it will be cheaper, but there is no surety about it.
      - guard-of-terra 10y ago
        You said there was always a line of planes. All those planes were delayed, weren't they? This is even before we factor in all the time lost by passengers of these planes, after which it would be a no-brainer.
- tyingq 10y ago
  >Crew has to be off the clock in an hour That's a good observation. Often, the original issue, if it's not fixed in an hour or two...starts creating cascading problems. Say just the boarding pass / checkin function is down. If you don't fix it quickly... - Crews (pilots and/or fa's) become illegal to fly due to various rules around crew work, hours, rest periods, etc. - They can't be replaced by other crews that need to be flown in, because the gates are occupied by the planes that can't leave - Downstream flight connections no longer match up, so a massive process to change the current tail numbers <--> flight numbers plan has to happen, followed by matching crews and passengers to the new plan. Especially fun if you have a mixed fleet where only certain crews can fly certain models, and the models have different passenger capacities. Basically, once you go past an hour or so, a giant shitshow starts...and gets worse as time goes on.
rm999 10y ago
The US airline industry has a serious problem with these outages. By my count, 4 major airlines have been hit with system-wide computer-related outages in the last year (southest, jetblue, delta, united). This article goes into some detail of why it happens: http://money.cnn.com/2016/08/08/technology/delta-airline-computer-failure http://money.cnn.com/2016/08/08/technology/delta-airline-com.... It seems like human error and fragile computer systems are the biggest issues.
- JoshuaJB 10y ago
  If the datacenters are really too complex for the people running them to understand, I would expect failures like this to drive the airlines towards AWS, Azure, or some similar service.
  - nodesocket 10y ago
    I'm not advocating that it is ok, but I am sure these airline systems are super old and wouldn't be surprised if they use IBM DB2 or similar ancient database technology. Moving to the cloud is not a trivial task for these sorts of mission critical antiquated systems.
    - jimktrains2 10y ago
      > Moving to the cloud is not a trivial task for these sorts of mission critical antiquated systems The question is if moving to _anything_ more modern is less trivial/costly than keeping the current systems which appear to have many single points of failure/
      - shakna 10y ago
        Often I've seen businesses reason that the failures are cheaper than the upgrades. That metric is often miscalculated as the initial cost is more of a deal than future revenue. So they stay with antiquated mainframes and DOS interfaces. Providing jobs for people like myself who can code in PL1, though I'd much rather take a brick to the face than do it again.
mactyler 10y ago
Stuck on ground in plane at ORD, just got an update that the "mainframes" are starting to come back online. EDIT: We're now about to take off.
- cydonian_monk 10y ago
  While in this case they probably are actual mainframes, at my firm we use the term to refer to the VMs running in tiny boxes that replaced what once were also mainframes. Sometimes the language outlives the hardware.
- hindsightbias 10y ago
  Uh, they aren't using mainframes. 10:1, Linux fail.
  - pc86 10y ago
    They most certainly are. The primary systems running all the major US airlines are all mainframe at least in part.
falcolas 10y ago
May this never happen to the Tesla fleet. As hesitant as I am about self driving cars, this would be an almost irreparable setback. And it's happened twice in my recent memory to our air fleets...
- ec109685 10y ago
  Why would it be irreparable?
  - falcolas 10y ago
    Can you picture the news, and loss of user confidence, if one day everyone with a Tesla were suddenly unable to drive their car? Would you trust a car which breaks down when AWS (for example) has an outage?
- viraptor 10y ago
  Why would it happen? Teslas are not centrally managed / scheduled / booked. The planes themselves could fly - it's the other things that failed.
  - rohit89 10y ago
    Could be an issue for their self driving taxi fleet.
  - falcolas 10y ago
    They are centrally managed, though. They are authorized to run over the Tesla network, and they get patches from a centralized source. Theoretically the cars would be mechanically sound to drive, but couldn't due to problems with the software.
- pc86 10y ago
  Are we playing word association now or something? Airline mainframe management systems are not in any way related to Tesla.
  - 10y ago
fapjacks 10y ago
This doesn't surprise me. US airlines seem like breeding grounds for management silos and the technology is going to reflect that, no matter how much you spend.
- shados 10y ago
  When's the last time you heard a great software/IT engineer go "You know what I really want to do? Work for an Airlines company. It will be sweet!" Yeah, that's part of the problem.
  - acchow 10y ago
    No, the problem is the airlines don't compensate or treat their engineers well enough to make anyone think "I really want to work for an airline".

163 points by perseusprime11 10y ago

Currently sitting on a plane at O'Hare, can't get to gate because they won't let any planes leave to clear a spot for my plane. What a nightmare.

erubin 10y ago

I have no reason to believe that there's any malice involved here. I also think that incidents like this might become more common as the state of cyber warfare progresses. As engineers, we should take care to build secure software, especially when that software underlies important systems. We should impress upon non-technical management the importance of doing so, even if it may take a little more time or money in the short term.

stevenhubertron 10y ago

Waiting in SLC terminal. Crew has to be off the clock in an hour. Not likely getting home tonight.

teej 10y ago
I got stuck in SLC two Sundays ago. Hotel tonight should have decent options. If you end up in town, the place will be dead. Bourbon House is decent and serves food late.
- deleted 10y ago
  [deleted]
- seanmcdirmid 10y ago
  Isn't Sundance going on now? That might overflow into SLC?
freehunter 10y ago
I love flying and getting to experience some of the weird issues that crop up. I was on a flight in December when there was only one ground crew working at my mid-sized airport. I was really hoping to beat the winter storm coming in, as I live pretty far from the airport. If there was a delay, I'd be stuck driving home in a blizzard. They were prioritizing getting planes to gates based on if the plane was picking up passengers and heading out again. Mine was the last flight of the day for my plane, so we were at the back of the line. After 30 minutes, we finally had an opening and were moving... until another flight crew reminded the tower that they were reaching their FAA allowed hours and needed to be parked immediately. They took our spot, and then more planes came in that had to leave. Of course I know all of this because our pilot was pissed and would relay all of the info to us with a very sarcastic voice. He was supposed to be home by now too. We ended up getting into our gate almost two hours after we landed. I normally live 30 minutes from the airport, and it took me another hour and a half to get home with the weather. The only reason we got in when we did is because the weather forced a shutdown of the airport, so we were literally the last flight to get parked at a gate, and only because there were no new flights coming in. It's fun (after the fact) to think about these edge cases, how to keep the entire airspace running smoothly when one airport only has a single ground crew working, how to maximize efficiency to make sure the fewest number of people are delayed. Sucks to live through it though.
- guard-of-terra 10y ago
  The solution here to have more crews & gates. Will create some jobs along the way, and surely will cost less than the delays you described.
  - dingo_bat 10y ago
    How will hiring more crews and gates be cheaper than one flight getting abnormally delayed? I mean maybe it will be cheaper, but there is no surety about it.
    - guard-of-terra 10y ago
      You said there was always a line of planes. All those planes were delayed, weren't they? This is even before we factor in all the time lost by passengers of these planes, after which it would be a no-brainer.
tyingq 10y ago
>Crew has to be off the clock in an hour That's a good observation. Often, the original issue, if it's not fixed in an hour or two...starts creating cascading problems. Say just the boarding pass / checkin function is down. If you don't fix it quickly... - Crews (pilots and/or fa's) become illegal to fly due to various rules around crew work, hours, rest periods, etc. - They can't be replaced by other crews that need to be flown in, because the gates are occupied by the planes that can't leave - Downstream flight connections no longer match up, so a massive process to change the current tail numbers <--> flight numbers plan has to happen, followed by matching crews and passengers to the new plan. Especially fun if you have a mixed fleet where only certain crews can fly certain models, and the models have different passenger capacities. Basically, once you go past an hour or so, a giant shitshow starts...and gets worse as time goes on.

rm999 10y ago

The US airline industry has a serious problem with these outages. By my count, 4 major airlines have been hit with system-wide computer-related outages in the last year (southest, jetblue, delta, united). This article goes into some detail of why it happens: http://money.cnn.com/2016/08/08/technology/delta-airline-computer-failure http://money.cnn.com/2016/08/08/technology/delta-airline-com.... It seems like human error and fragile computer systems are the biggest issues.

JoshuaJB 10y ago
If the datacenters are really too complex for the people running them to understand, I would expect failures like this to drive the airlines towards AWS, Azure, or some similar service.
- nodesocket 10y ago
  I'm not advocating that it is ok, but I am sure these airline systems are super old and wouldn't be surprised if they use IBM DB2 or similar ancient database technology. Moving to the cloud is not a trivial task for these sorts of mission critical antiquated systems.
  - jimktrains2 10y ago
    > Moving to the cloud is not a trivial task for these sorts of mission critical antiquated systems The question is if moving to _anything_ more modern is less trivial/costly than keeping the current systems which appear to have many single points of failure/
    - shakna 10y ago
      Often I've seen businesses reason that the failures are cheaper than the upgrades. That metric is often miscalculated as the initial cost is more of a deal than future revenue. So they stay with antiquated mainframes and DOS interfaces. Providing jobs for people like myself who can code in PL1, though I'd much rather take a brick to the face than do it again.

mactyler 10y ago

Stuck on ground in plane at ORD, just got an update that the "mainframes" are starting to come back online. EDIT: We're now about to take off.

cydonian_monk 10y ago
While in this case they probably are actual mainframes, at my firm we use the term to refer to the VMs running in tiny boxes that replaced what once were also mainframes. Sometimes the language outlives the hardware.
hindsightbias 10y ago
Uh, they aren't using mainframes. 10:1, Linux fail.
- pc86 10y ago
  They most certainly are. The primary systems running all the major US airlines are all mainframe at least in part.

falcolas 10y ago

May this never happen to the Tesla fleet. As hesitant as I am about self driving cars, this would be an almost irreparable setback. And it's happened twice in my recent memory to our air fleets...

ec109685 10y ago
Why would it be irreparable?
- falcolas 10y ago
  Can you picture the news, and loss of user confidence, if one day everyone with a Tesla were suddenly unable to drive their car? Would you trust a car which breaks down when AWS (for example) has an outage?
viraptor 10y ago
Why would it happen? Teslas are not centrally managed / scheduled / booked. The planes themselves could fly - it's the other things that failed.
- rohit89 10y ago
  Could be an issue for their self driving taxi fleet.
- falcolas 10y ago
  They are centrally managed, though. They are authorized to run over the Tesla network, and they get patches from a centralized source. Theoretically the cars would be mechanically sound to drive, but couldn't due to problems with the software.
pc86 10y ago
Are we playing word association now or something? Airline mainframe management systems are not in any way related to Tesla.
- 10y ago

fapjacks 10y ago

This doesn't surprise me. US airlines seem like breeding grounds for management silos and the technology is going to reflect that, no matter how much you spend.

shados 10y ago
When's the last time you heard a great software/IT engineer go "You know what I really want to do? Work for an Airlines company. It will be sweet!" Yeah, that's part of the problem.
- acchow 10y ago
  No, the problem is the airlines don't compensate or treat their engineers well enough to make anyone think "I really want to work for an airline".