6 ms·

It's not software or hardware this time: there's a crippling power outage at ATL, which is Delta's main hub (and presumably where their mainframe herd is based)

by cstross 10y ago

It's not software or hardware this time: there's a crippling power outage at ATL, which is Delta's main hub (and presumably where their mainframe herd is based): http://fox61.com/2016/08/08/delta-airline-reports-outage-everywhere-systems-down-and-flights-grounded/ http://fox61.com/2016/08/08/delta-airline-reports-outage-eve... The outage began at 0230 and is presumably ongoing and beyond the capacity of their backup generators to cope with.

gtirloni 10y ago
Same here: http://news.delta.com/730-am-et-update-outage-affects-departing-flights-today http://news.delta.com/730-am-et-update-outage-affects-depart... A power outage in Atlanta, which began at approximately 2:30 a.m. ET, has impacted Delta computer systems and operations worldwide, resulting in flight delays 8:40 a.m. ET UPDATE: A Delta ground stop has been lifted and limited departures are resuming following a power outage in Atlanta that impacted Delta computer systems and operations worldwide. Cancellations and delays continue.
bgroins 10y ago
I'm quite surprised that their whole operation is region dependent. I work with hospitals on high availability/continuous availability of critical systems and would think that a company as big as Delta would consider a regional power outage as a high risk and plausible scenario.
- chiph 10y ago
  Have you flown through Hartsfield-Jackson? There's six concourses with 30+ gates each, nearly all of which is Delta. Their operations are highly-concentrated there. That being said, I expect them to upgrade their generators to handle outages like this, and distribute more of their IT infrastructure. Unlike the typical summer storms in the Atlanta area that delay flights, this affected them worldwide.
  - hueving 10y ago
    As a passenger on the west coast flying to another west coast city, I give no shits about the number of concourses in ATL. LAX still has electricity but delta's poor infrastructure planning has left a single point of failure in the deep south affecting flights everywhere in the world.
    - chiph 10y ago
      So - too big to fail? ;)
    - ubernostrum 10y ago
      LAX still has electricity but delta's poor infrastructure planning has left a single point of failure in the deep south affecting flights everywhere in the world. The "all over the world" bit is why there's a single point of failure. A while back there was a story that made the rounds of aviation geeks, about Delta flying an empty 747 to Korea. That was to replace a 747 which had been badly damaged by hail, to the point that it would be unable to operate its scheduled return flight to the US. Do you want to guess how many people, parts and places were involved in the "simple" task of dealing with this problem? At first, the replacement 747 is in storage in a "boneyard" facility in Arizona, due to having been recently retired. So first it has to be pulled from the storage facility, put through basic airworthiness checks and fueled up, and then a flight crew has to be present to fly it to a Delta hub where it can be readied for a trans-Pacific flight. The hub in question turned out to be Minneapolis. There, the plane has to undergo more work to get it ready for a long flight, and now multiple flight crews have to be present, since they need to rotate in and out over the duration of the flight (that's how you do long flights). Oh, and Minneapolis isn't normally a 747 base; it only gets them during peak travel seasons and on the occasional charter. So crews probably have to be brought in, stores and maintenance setups need to be brought online, etc. Then the plane can -- finally -- fly out to replace its damaged counterpart, pick up any stranded passengers and bring them to the US. Which will mean flying into yet another hub, since the flight doesn't go back to Minneapolis. Meanwhile the damaged plane is still sitting there in Korea, and needs to be repaired on-site to get it into minimum airworthy condition to fly home (empty of passengers). It's going to need parts, maintenance crew, flight crews, etc. just like the replacement plane did. And the deeper you dig the more stuff you'll find like this. Running an airline with global, or even national-across-the-US, service is not something you can decentralize to avoid problems at one operations center. The amount of coordination just of people, parts and planes across widely disparate locations requires centralized operational control instead of devolved regional centers with high autonomy.
bradleyjg 10y ago
I find that pretty odd. A decade and a half ago now I rented part of a rack in a not particularly high end co-lo facility. They had battery backup for an hour and then generators that could be fired up in twenty minutes with 3 days worth of diesel. Delta airlines, with $40B of revenue last year, has less than 8 hours of backup power for their mission critical computers?
- johnward 10y ago
  or even a second site to fail over to?
  - bluejekyll 10y ago
    Exactly my question, why was there no failover? This outage is going to cost them substantially. Even on classic DBs, setting up a failover with a lag of 15 minutes isn't a herculean effort. It's expensive, yes, but probably worth it...
- Someone1234 10y ago
  We once had a long power outage (3.5 hrs ish), the data center remained on the entire time, but after about 60 minutes the data center lost internet access(!) because their ISP only had a battery backup with a limited capacity. So it does happen. Plus sometimes backup power isn't reliable (e.g. generator problem) or they have different tiers of coverage (and Delta didn't pay for it). We'll just have to wait and see.
  - philipov 10y ago
    That's why anyone who actually cares about keeping their business running keeps a disaster recovery site in a different data center, and regularly tests their failover procedure.
    - ht85 10y ago
      It depends how much it would cost you to maintain vs the cost of a possible downtime. There are things that are fairly easy to make redundant. I don't know anything about their infrastructure, but I doubt building a failover would be trivial. Even if you had it in place, the risk of something occuring as you switch back and forth might not be worth it if you expect your regular setup to come back up within a reasonable time frame. Most people around here know how close to impossible it is to replicate real world usage on a test system, and how much overhead it adds to any change you make to your production system if you want the others to follow.
clueless123 10y ago
I call BS.. Keeping the power on is a well managed process. I doubt a Data center like that will go down for power reasons. My bet is on ancient mainframes giving up. Been there, done that.
- bluejekyll 10y ago
  What are you basing this on? I've seen many datacenters fail to kickover to backup power for various reasons. This is a process that has to work in under like 30 seconds. For places with spinning mass, that's the amount of time that the spinning mass will power the systems while the generators are kicked on. [1] The generators have been know to fail hard, and then the entire DC is down. On top of that, generators often are only able to run for a limited amount of time hours to days [2]. [1] https://en.wikipedia.org/wiki/Flywheel_energy_storage https://en.wikipedia.org/wiki/Flywheel_energy_storage [2] http://www.graniteblockglobaldatacenter.com/powersystems.html http://www.graniteblockglobaldatacenter.com/powersystems.htm...
ryandrake 10y ago
I wonder if Amazon would consider it an acceptable risk that a power outage in Seattle turns their stores off worldwide. Highly doubt it.

by cstross 10y ago

gtirloni 10y ago

Same here: http://news.delta.com/730-am-et-update-outage-affects-departing-flights-today http://news.delta.com/730-am-et-update-outage-affects-depart... A power outage in Atlanta, which began at approximately 2:30 a.m. ET, has impacted Delta computer systems and operations worldwide, resulting in flight delays 8:40 a.m. ET UPDATE: A Delta ground stop has been lifted and limited departures are resuming following a power outage in Atlanta that impacted Delta computer systems and operations worldwide. Cancellations and delays continue.

bgroins 10y ago

I'm quite surprised that their whole operation is region dependent. I work with hospitals on high availability/continuous availability of critical systems and would think that a company as big as Delta would consider a regional power outage as a high risk and plausible scenario.

chiph 10y ago
Have you flown through Hartsfield-Jackson? There's six concourses with 30+ gates each, nearly all of which is Delta. Their operations are highly-concentrated there. That being said, I expect them to upgrade their generators to handle outages like this, and distribute more of their IT infrastructure. Unlike the typical summer storms in the Atlanta area that delay flights, this affected them worldwide.
- hueving 10y ago
  As a passenger on the west coast flying to another west coast city, I give no shits about the number of concourses in ATL. LAX still has electricity but delta's poor infrastructure planning has left a single point of failure in the deep south affecting flights everywhere in the world.
  - chiph 10y ago
    So - too big to fail? ;)
  - ubernostrum 10y ago
    LAX still has electricity but delta's poor infrastructure planning has left a single point of failure in the deep south affecting flights everywhere in the world. The "all over the world" bit is why there's a single point of failure. A while back there was a story that made the rounds of aviation geeks, about Delta flying an empty 747 to Korea. That was to replace a 747 which had been badly damaged by hail, to the point that it would be unable to operate its scheduled return flight to the US. Do you want to guess how many people, parts and places were involved in the "simple" task of dealing with this problem? At first, the replacement 747 is in storage in a "boneyard" facility in Arizona, due to having been recently retired. So first it has to be pulled from the storage facility, put through basic airworthiness checks and fueled up, and then a flight crew has to be present to fly it to a Delta hub where it can be readied for a trans-Pacific flight. The hub in question turned out to be Minneapolis. There, the plane has to undergo more work to get it ready for a long flight, and now multiple flight crews have to be present, since they need to rotate in and out over the duration of the flight (that's how you do long flights). Oh, and Minneapolis isn't normally a 747 base; it only gets them during peak travel seasons and on the occasional charter. So crews probably have to be brought in, stores and maintenance setups need to be brought online, etc. Then the plane can -- finally -- fly out to replace its damaged counterpart, pick up any stranded passengers and bring them to the US. Which will mean flying into yet another hub, since the flight doesn't go back to Minneapolis. Meanwhile the damaged plane is still sitting there in Korea, and needs to be repaired on-site to get it into minimum airworthy condition to fly home (empty of passengers). It's going to need parts, maintenance crew, flight crews, etc. just like the replacement plane did. And the deeper you dig the more stuff you'll find like this. Running an airline with global, or even national-across-the-US, service is not something you can decentralize to avoid problems at one operations center. The amount of coordination just of people, parts and planes across widely disparate locations requires centralized operational control instead of devolved regional centers with high autonomy.

bradleyjg 10y ago

I find that pretty odd. A decade and a half ago now I rented part of a rack in a not particularly high end co-lo facility. They had battery backup for an hour and then generators that could be fired up in twenty minutes with 3 days worth of diesel. Delta airlines, with $40B of revenue last year, has less than 8 hours of backup power for their mission critical computers?

johnward 10y ago
or even a second site to fail over to?
- bluejekyll 10y ago
  Exactly my question, why was there no failover? This outage is going to cost them substantially. Even on classic DBs, setting up a failover with a lag of 15 minutes isn't a herculean effort. It's expensive, yes, but probably worth it...
Someone1234 10y ago
We once had a long power outage (3.5 hrs ish), the data center remained on the entire time, but after about 60 minutes the data center lost internet access(!) because their ISP only had a battery backup with a limited capacity. So it does happen. Plus sometimes backup power isn't reliable (e.g. generator problem) or they have different tiers of coverage (and Delta didn't pay for it). We'll just have to wait and see.
- philipov 10y ago
  That's why anyone who actually cares about keeping their business running keeps a disaster recovery site in a different data center, and regularly tests their failover procedure.
  - ht85 10y ago
    It depends how much it would cost you to maintain vs the cost of a possible downtime. There are things that are fairly easy to make redundant. I don't know anything about their infrastructure, but I doubt building a failover would be trivial. Even if you had it in place, the risk of something occuring as you switch back and forth might not be worth it if you expect your regular setup to come back up within a reasonable time frame. Most people around here know how close to impossible it is to replicate real world usage on a test system, and how much overhead it adds to any change you make to your production system if you want the others to follow.

clueless123 10y ago

I call BS.. Keeping the power on is a well managed process. I doubt a Data center like that will go down for power reasons. My bet is on ancient mainframes giving up. Been there, done that.

bluejekyll 10y ago
What are you basing this on? I've seen many datacenters fail to kickover to backup power for various reasons. This is a process that has to work in under like 30 seconds. For places with spinning mass, that's the amount of time that the spinning mass will power the systems while the generators are kicked on. [1] The generators have been know to fail hard, and then the entire DC is down. On top of that, generators often are only able to run for a limited amount of time hours to days [2]. [1] https://en.wikipedia.org/wiki/Flywheel_energy_storage https://en.wikipedia.org/wiki/Flywheel_energy_storage [2] http://www.graniteblockglobaldatacenter.com/powersystems.html http://www.graniteblockglobaldatacenter.com/powersystems.htm...

ryandrake 10y ago

I wonder if Amazon would consider it an acceptable risk that a power outage in Seattle turns their stores off worldwide. Highly doubt it.