3 ms·

We had a similar problem at our hosting facility last winter during that "Ice Vortex" storm that was all over the news. The facilities guys had been very proud

We had a similar problem at our hosting facility last winter during that "Ice Vortex" storm that was all over the news. The facilities guys had been very proud of never having a power outage, I've had servers with them for 15 years now. The morning of the worst of the storm, we completely lose access to all services at that facility. Super unusual, everything we have there is redundant. So I do some minor investigation, and get on the horn to them. They were being super cagey. "Hey, we lost all access to our systems." "Ok, I'll open a ticket and we will investigate." "Uhhh. It feels like it's a big problem with the data center, are you guys having problems or is it just us?" "I can't say anything more until we've completed an investigation." "I'm trying to decide if we need to start failing over to our DR site, or if I need to put chains on the truck to drive 50 miles to the data center in this storm. Is anyone else having problems? Are fire alarms going off?" "We have received multiple reports of problems." Power was back on in less than half an hour, but they still weren't saying anything for a few hours. Spent that time trying to figure out if we should shut everything down and ride it out, or if they were back in business. We had one system that suffered disk corruption, despite having a (according to the weekly testing) correctly operating BBU on the RAID. So what happened? It shouldn't have been possible, our cabinet was being fed by two lines from independent PDUs. Each PDU is fed by 2 independent UPSes (one shared between the two PDUs), each UPS fed by a dedicated generator. Should have required 3 failures to bring us down. They eventually reveal that they had had one of their UPSes down for weeks, waiting for replacement parts. The other two UPSes had independent failures (one was a controller board, one was battery related). They said they still did quarterly full load tests of the power systems, but reading between the lines I think they weren't testing these two UPSes because the other one was not there to back them up. Still, one power event in 15 years isn't too shabby.

jgalentine007 7y ago
I had a similar situation in the early 2000's at NTT/Verio in Sterling, VA. They lost power because someone dug through a utility line and introduced a ground a fault into their system. They switched to generators and then against protocol (basically human error) tried to force override a transfer switch onto their other utility which ended up killing the generators. Eventually the UPSs were all drained, but we had to shutdown servers long ahead of that because with no AC, they were overheating. People were taking servers out by pickup truck to stand them up at their other offices and data centers. 48 hours of pain.
- macintux 7y ago
  Power seems like the big wildcard in data center management. So tough to properly test your failover preparations, and so many different ways things can go wrong. I know of a large company that had the data center emergency cutoff button next to the automatic doors on the way out. Sure enough, a contractor hit it one day thinking it was the way to open the doors.
linsomniac 7y ago
Oh, I forgot to mention that their solution was to run off generators for the impacted UPSes until repairs could be completed. I think one of them ended up running for 2 weeks. It seems strange that replacement parts for this UPS were so hard to source. That's why it was down in the first place. Someone making them by hand?

jgalentine007 7y ago

I had a similar situation in the early 2000's at NTT/Verio in Sterling, VA. They lost power because someone dug through a utility line and introduced a ground a fault into their system. They switched to generators and then against protocol (basically human error) tried to force override a transfer switch onto their other utility which ended up killing the generators. Eventually the UPSs were all drained, but we had to shutdown servers long ahead of that because with no AC, they were overheating. People were taking servers out by pickup truck to stand them up at their other offices and data centers. 48 hours of pain.

macintux 7y ago
Power seems like the big wildcard in data center management. So tough to properly test your failover preparations, and so many different ways things can go wrong. I know of a large company that had the data center emergency cutoff button next to the automatic doors on the way out. Sure enough, a contractor hit it one day thinking it was the way to open the doors.

linsomniac 7y ago

Oh, I forgot to mention that their solution was to run off generators for the impacted UPSes until repairs could be completed. I think one of them ended up running for 2 weeks. It seems strange that replacement parts for this UPS were so hard to source. That's why it was down in the first place. Someone making them by hand?