3 ms·

I worked in a datacenter (serving many companies you've heard of) during a catastrophic power failure that lasted almost 24 hours. It's kind of like a plane cra

I worked in a datacenter (serving many companies you've heard of) during a catastrophic power failure that lasted almost 24 hours. It's kind of like a plane crash - it's never one thing failing that causes the problem (that's what redundancy is for) - it's that perfect chain of events, multiple 'once in a lifetime' failures that causes it. For example, power outage occurs at the same time the UPS batteries are being changed. Bypass fails and Diesel generators fail to kick in. Circuit breakers blow everywhere making it extremely difficult to get the generators back on line. This all happens in the middle of the night in a winter storm (or 'lightning storm) which causes a further delay in response time. Been there, done that. Edit: Also, bureaucracy, lack of documentation, and a manager CF added several hours to the outage. Sometimes you just have to STFU and let the geeks fix the problem.

jmcnevin 16y ago
What you've described here would make an AWESOME movie. Granted, I'm not sure how one would work "Datacenter" into the title and still sell many tickets.
- dholowiski 16y ago
  What makes it even more exciting is that some of the companies were a bank and a hospital. Lesson learned - a bank gets far more anxious than a hospital when their servers go down. I'm pretty sure there were Zombies roaming the aisles of servers where the power was out, that might help the box office sales.
- rryyan 16y ago
  Cory Doctorow (of Boing Boing fame) wrote a sci-fi novelette about a datacenter during a crisis: http://craphound.com/overclocked/Cory_Doctorow_-_Overclocked_-_When_Sysadmins_Ruled_the_Earth.html http://craphound.com/overclocked/Cory_Doctorow_-_Overclocked... I personally didn't find this piece particularly great, but there's an entry in the genre for you.
d2viant 16y ago
Except, this isn't a once in a lifetime occurrence. As of today it's twice in a lifetime. That's indicates to me they didn't just hit that perfect chain of events, it tells me they have problems in their architecture or implementation that need to be addressed and tested before the next crisis occurs. I worked in a datacenter as well. The whole point is that you test those scenarios when it's not an emergency so you can be confident everything will operate smoothly.
- dholowiski 16y ago
  Don't discount the chance that this was caused by two totally different 'once in a lifetime' chain of events... i've seen it happen. However, as you point out it is far more likely that there are architecture or implementation issues that caused the repeated outages. There are so many reasons that I don't want to have servers in California... high costs, dense population vulnerability to earthquakes and a hot terrorism target. Massive bandwidth but massive utilization, somewhat shaky power grid, heck even the state has financial issues. I picked Atlanta for my Linode server which is better (it's still a little close to the coast for me) and I host my WordPress sites at a company in Dallas, TX. Did you know texas has their own power grid - separate from the rest of the USA?
tbrownaw 16y ago
> multiple 'once in a lifetime' failures that causes it. That reminds me of hearing mention of "multiple six-sigma events" I think it was, when things were crashing in 2008: the only sane conclusion is that the models are broken.

jmcnevin 16y ago

What you've described here would make an AWESOME movie. Granted, I'm not sure how one would work "Datacenter" into the title and still sell many tickets.

dholowiski 16y ago
What makes it even more exciting is that some of the companies were a bank and a hospital. Lesson learned - a bank gets far more anxious than a hospital when their servers go down. I'm pretty sure there were Zombies roaming the aisles of servers where the power was out, that might help the box office sales.
rryyan 16y ago
Cory Doctorow (of Boing Boing fame) wrote a sci-fi novelette about a datacenter during a crisis: http://craphound.com/overclocked/Cory_Doctorow_-_Overclocked_-_When_Sysadmins_Ruled_the_Earth.html http://craphound.com/overclocked/Cory_Doctorow_-_Overclocked... I personally didn't find this piece particularly great, but there's an entry in the genre for you.

d2viant 16y ago

Except, this isn't a once in a lifetime occurrence. As of today it's twice in a lifetime. That's indicates to me they didn't just hit that perfect chain of events, it tells me they have problems in their architecture or implementation that need to be addressed and tested before the next crisis occurs. I worked in a datacenter as well. The whole point is that you test those scenarios when it's not an emergency so you can be confident everything will operate smoothly.

dholowiski 16y ago
Don't discount the chance that this was caused by two totally different 'once in a lifetime' chain of events... i've seen it happen. However, as you point out it is far more likely that there are architecture or implementation issues that caused the repeated outages. There are so many reasons that I don't want to have servers in California... high costs, dense population vulnerability to earthquakes and a hot terrorism target. Massive bandwidth but massive utilization, somewhat shaky power grid, heck even the state has financial issues. I picked Atlanta for my Linode server which is better (it's still a little close to the coast for me) and I host my WordPress sites at a company in Dallas, TX. Did you know texas has their own power grid - separate from the rest of the USA?

tbrownaw 16y ago

> multiple 'once in a lifetime' failures that causes it. That reminds me of hearing mention of "multiple six-sigma events" I think it was, when things were crashing in 2008: the only sane conclusion is that the models are broken.