6 ms·

I find that pretty odd. A decade and a half ago now I rented part of a rack in a not particularly high end co-lo facility. They had battery backup for an hour a

I find that pretty odd. A decade and a half ago now I rented part of a rack in a not particularly high end co-lo facility. They had battery backup for an hour and then generators that could be fired up in twenty minutes with 3 days worth of diesel. Delta airlines, with $40B of revenue last year, has less than 8 hours of backup power for their mission critical computers?

johnward 10y ago
or even a second site to fail over to?
- bluejekyll 10y ago
  Exactly my question, why was there no failover? This outage is going to cost them substantially. Even on classic DBs, setting up a failover with a lag of 15 minutes isn't a herculean effort. It's expensive, yes, but probably worth it...
Someone1234 10y ago
We once had a long power outage (3.5 hrs ish), the data center remained on the entire time, but after about 60 minutes the data center lost internet access(!) because their ISP only had a battery backup with a limited capacity. So it does happen. Plus sometimes backup power isn't reliable (e.g. generator problem) or they have different tiers of coverage (and Delta didn't pay for it). We'll just have to wait and see.
- philipov 10y ago
  That's why anyone who actually cares about keeping their business running keeps a disaster recovery site in a different data center, and regularly tests their failover procedure.
  - ht85 10y ago
    It depends how much it would cost you to maintain vs the cost of a possible downtime. There are things that are fairly easy to make redundant. I don't know anything about their infrastructure, but I doubt building a failover would be trivial. Even if you had it in place, the risk of something occuring as you switch back and forth might not be worth it if you expect your regular setup to come back up within a reasonable time frame. Most people around here know how close to impossible it is to replicate real world usage on a test system, and how much overhead it adds to any change you make to your production system if you want the others to follow.
  - toomuchtodo 10y ago
    Both Equinix and AWS have had cases where the genset tests pass, but entire DC blackouts occurred due to a fault only a few months later.
  - kpil 10y ago
    Yup. I've worked at several places where there has been battery backup, diesel generators, and a a disaster recovery site with generally nice and thorough strategies for replication including really expensive "sysplex-mode" for mainframes, topped with regular contigency tests where one half of the site is shut down. And somehow, at every place, eventually something bad happened in a way that took down the applications. It could be a power outage in the exactly the wrong circuit, fluctuating power messed up exactly the wrong storage system, someone dug up exactly the wrong fiber, the fire extinguisher set off by mistake and sprayed just the wrong cabinet with water fog through a door(!) that was supposed to automatically close, the diesel generator caught fire and the backup battery had to be taken offline, etc etc. I'm not saying that you shouldn't try, but it's damn hard.
  - deleted 10y ago
    [deleted]
- DoubleGlazing 10y ago
  I worked for a company that thought it had a really top level power backup system. The server room had 15 minutes of UPS, all PCs had at least 5 mins. Out the back was a big generator with 24 hours worth of diesel. It was all controlled by a fancy control system that fed light displays in every room indicating if we were on mains power, UPS power or generator power. One day during a power cut the generator kicked in as it was supposed to, but a minute later it spluttered to a halt. After much fiddling around by maintenance staff they gave up and we were all told to go home. The postmortem revealed that the diesel had become contaminated with all sorts of gunk - water, grit/sand and lumps of unidentifiable "stuff". The best laid plans and all that. Edit: And cigarette butts as well.
mseebach 10y ago
> They had battery backup for an hour and then generators that could be fired up in twenty minutes with 3 days worth of diesel. How often did you test that? > has less than 8 hours of backup power for their mission critical computers? It's plausible that they thought they had that, but some part of it failed.
- bradleyjg 10y ago
  I don't remember, but when we were taking the tour the guide did mention that the tested the batteries and generators on some schedule. I bet there's some industry standard that has a minimum testing frequency.
  - dbcurtis 10y ago
    In my experience (radio repeater and cell sites on mountain tops) an auto-start timer runs the generator once per week for 30 minutes or so. Diesels dont like to sit too long without being run. And these are 30-50 Kw generators, relatively small in comparison to a data center. I expect a weekly schedule is common practice.
- usea 10y ago
  > How often did you test that? Not parent, but here is a single data point. When I was a student, I worked for some sysadmins who managed a small, internal data center. ~200 IBM servers, a SAN, an old VAX, etc. It had a backup gas generator that was tested loudly, every single week. It was also tested whenever the power went out for too long. Obviously lots can go wrong, even with the best plans, but at least at the tiny little place I worked, the generator was tested every week.
- dpcx 10y ago
  The data center where my employer used to house stuff tested their generators for short bursts weekly, and would run them for 8 hours every quarter.
- stygiansonic 10y ago
  Very good points. Backups or backup systems should not be considered as true "backups" unless you've successfully failed over to them.
jsolson 10y ago
It can quite difficult to get a large diesel generator permit for a data center in the Atlanta area due in part to automotive emissions but also (ironically) the high greenhouse gas emissions caused by Hartsfield-Jackson Atlanta International Airport. Mind you, I'm not saying they don't have one, just that it can be difficult to get -- there can also be limits on the number of hours you can run them.
- kpil 10y ago
  What would stop you from just parking a large container outside the house? On top of a truck in the worst case... There are several providers of mobile power.
tedmiston 10y ago
It looks like switching to the backup generator may have been what caused the fire that caused the outage in the first place. > According to the flight captain of JFK-SLC this morning, a routine scheduled switch to the backup generator this morning at 2:30am caused a fire that destroyed both the backup and the primary. Firefighters took a while to extinguish the fire. Power is now back up and 400 out of the 500 servers rebooted, still waiting for the last 100 to have the whole system fully functional. http://www.flyertalk.com/forum/27032000-post135.html http://www.flyertalk.com/forum/27032000-post135.html Update: Confirmed. > But Georgia Power said the issue had to do with Delta's own equipment, not a larger power outage. "We believe Delta Air Lines experienced an equipment outage; other Georgia Power customers were not affected," said John Kraft, spokesperson for the utility. Georgia power has staff on site trying to assist Delta, he said. https://www.washingtonpost.com/news/the-switch/wp/2016/08/08/what-you-need-to-know-about-the-massive-delta-computer-outage/ https://www.washingtonpost.com/news/the-switch/wp/2016/08/08...
rhizome 10y ago
And of course, one of the benefits of diesel generators is that they aren't limited to that three days of fuel because they can be refilled while running. :)

johnward 10y ago

or even a second site to fail over to?

bluejekyll 10y ago
Exactly my question, why was there no failover? This outage is going to cost them substantially. Even on classic DBs, setting up a failover with a lag of 15 minutes isn't a herculean effort. It's expensive, yes, but probably worth it...

Someone1234 10y ago

We once had a long power outage (3.5 hrs ish), the data center remained on the entire time, but after about 60 minutes the data center lost internet access(!) because their ISP only had a battery backup with a limited capacity. So it does happen. Plus sometimes backup power isn't reliable (e.g. generator problem) or they have different tiers of coverage (and Delta didn't pay for it). We'll just have to wait and see.

philipov 10y ago
That's why anyone who actually cares about keeping their business running keeps a disaster recovery site in a different data center, and regularly tests their failover procedure.
- ht85 10y ago
  It depends how much it would cost you to maintain vs the cost of a possible downtime. There are things that are fairly easy to make redundant. I don't know anything about their infrastructure, but I doubt building a failover would be trivial. Even if you had it in place, the risk of something occuring as you switch back and forth might not be worth it if you expect your regular setup to come back up within a reasonable time frame. Most people around here know how close to impossible it is to replicate real world usage on a test system, and how much overhead it adds to any change you make to your production system if you want the others to follow.
- toomuchtodo 10y ago
  Both Equinix and AWS have had cases where the genset tests pass, but entire DC blackouts occurred due to a fault only a few months later.
- kpil 10y ago
  Yup. I've worked at several places where there has been battery backup, diesel generators, and a a disaster recovery site with generally nice and thorough strategies for replication including really expensive "sysplex-mode" for mainframes, topped with regular contigency tests where one half of the site is shut down. And somehow, at every place, eventually something bad happened in a way that took down the applications. It could be a power outage in the exactly the wrong circuit, fluctuating power messed up exactly the wrong storage system, someone dug up exactly the wrong fiber, the fire extinguisher set off by mistake and sprayed just the wrong cabinet with water fog through a door(!) that was supposed to automatically close, the diesel generator caught fire and the backup battery had to be taken offline, etc etc. I'm not saying that you shouldn't try, but it's damn hard.
- deleted 10y ago
  [deleted]
DoubleGlazing 10y ago
I worked for a company that thought it had a really top level power backup system. The server room had 15 minutes of UPS, all PCs had at least 5 mins. Out the back was a big generator with 24 hours worth of diesel. It was all controlled by a fancy control system that fed light displays in every room indicating if we were on mains power, UPS power or generator power. One day during a power cut the generator kicked in as it was supposed to, but a minute later it spluttered to a halt. After much fiddling around by maintenance staff they gave up and we were all told to go home. The postmortem revealed that the diesel had become contaminated with all sorts of gunk - water, grit/sand and lumps of unidentifiable "stuff". The best laid plans and all that. Edit: And cigarette butts as well.

mseebach 10y ago

> They had battery backup for an hour and then generators that could be fired up in twenty minutes with 3 days worth of diesel. How often did you test that? > has less than 8 hours of backup power for their mission critical computers? It's plausible that they thought they had that, but some part of it failed.

bradleyjg 10y ago
I don't remember, but when we were taking the tour the guide did mention that the tested the batteries and generators on some schedule. I bet there's some industry standard that has a minimum testing frequency.
- dbcurtis 10y ago
  In my experience (radio repeater and cell sites on mountain tops) an auto-start timer runs the generator once per week for 30 minutes or so. Diesels dont like to sit too long without being run. And these are 30-50 Kw generators, relatively small in comparison to a data center. I expect a weekly schedule is common practice.
usea 10y ago
> How often did you test that? Not parent, but here is a single data point. When I was a student, I worked for some sysadmins who managed a small, internal data center. ~200 IBM servers, a SAN, an old VAX, etc. It had a backup gas generator that was tested loudly, every single week. It was also tested whenever the power went out for too long. Obviously lots can go wrong, even with the best plans, but at least at the tiny little place I worked, the generator was tested every week.
dpcx 10y ago
The data center where my employer used to house stuff tested their generators for short bursts weekly, and would run them for 8 hours every quarter.
stygiansonic 10y ago
Very good points. Backups or backup systems should not be considered as true "backups" unless you've successfully failed over to them.

jsolson 10y ago

It can quite difficult to get a large diesel generator permit for a data center in the Atlanta area due in part to automotive emissions but also (ironically) the high greenhouse gas emissions caused by Hartsfield-Jackson Atlanta International Airport. Mind you, I'm not saying they don't have one, just that it can be difficult to get -- there can also be limits on the number of hours you can run them.

kpil 10y ago
What would stop you from just parking a large container outside the house? On top of a truck in the worst case... There are several providers of mobile power.

tedmiston 10y ago

It looks like switching to the backup generator may have been what caused the fire that caused the outage in the first place. > According to the flight captain of JFK-SLC this morning, a routine scheduled switch to the backup generator this morning at 2:30am caused a fire that destroyed both the backup and the primary. Firefighters took a while to extinguish the fire. Power is now back up and 400 out of the 500 servers rebooted, still waiting for the last 100 to have the whole system fully functional. http://www.flyertalk.com/forum/27032000-post135.html http://www.flyertalk.com/forum/27032000-post135.html Update: Confirmed. > But Georgia Power said the issue had to do with Delta's own equipment, not a larger power outage. "We believe Delta Air Lines experienced an equipment outage; other Georgia Power customers were not affected," said John Kraft, spokesperson for the utility. Georgia power has staff on site trying to assist Delta, he said. https://www.washingtonpost.com/news/the-switch/wp/2016/08/08/what-you-need-to-know-about-the-massive-delta-computer-outage/ https://www.washingtonpost.com/news/the-switch/wp/2016/08/08...

rhizome 10y ago

And of course, one of the benefits of diesel generators is that they aren't limited to that three days of fuel because they can be refilled while running. :)