5 ms·

Typically the operators will have an agreement so that they can get the generators refueled should the outage last even longer than what fuel they have on premi

by jorts 10y ago

Typically the operators will have an agreement so that they can get the generators refueled should the outage last even longer than what fuel they have on premise, so they theoretically could run indefinitely on generator power.

barkingcat 10y ago
An outage requiring generator power for longer than 1 week might not allow transportation of fuel. Just want to bring that up in case people really did think you can run "indefinitely" on generator power if the roads, bridges, and gas supply network is out, or if there's an emergency state of war or terrorism. At the 1wk+ territory, it will start to look something like Katrina, where the roads were all blocked by flooding waters and fuel resupply missions required the protection of the National Guard. I mean you can charter helicopters to bring in fuel, but I think there are more urgent things to power than a datacentre in those cases, for example, hospitals and shelters for emergency humanitarian aid.
- chinathrow 10y ago
  Never underestimate what people would do to keep the stock market running...
  - stephengillie 10y ago
    These days, I'd hope they can failover to standby hosts in "IL#" or "VA#" or another datacenter in a nearby region. The DR/BC space has become pretty robust, and with the right vendor, a datacenter loss can be recovered from simply by powering on fully-replicated VMs in another datacenter.
  - minimax 10y ago
    Or do! They closed the markets in the US for two days during Sandy even though NY4 (and the other US equity exchange data centers) were fine. http://www.wsj.com/articles/SB10001424052970204789304578087131092892180 http://www.wsj.com/articles/SB100014240529702047893045780871...
  - capote 10y ago
    Sometimes it feels like the opposite. Any day that has the slightest hint of holiday to it is "markets closed! remember no work tomorrow! hooray for national silly lala day!"
- jtakkala 10y ago
  Where I used to work we had refuelling agreements with multiple vendors. Yes, hospitals and critical infrastructure are prioritized first, but we ran on generators for an extended spell during the Northeast Blackout of 2003[1] (the outage affecting our facility only lasted a couple of days, but due to the utilities' appeal to reduce load on the grid until it was fully functional we decided to run on generators for over a week). Edit: One of my fond memories of that time was actually greeting the refuelling truck and directing it to the point where he could plug a hose into a pipe on our building and start pumping. [1] https://en.wikipedia.org/wiki/Northeast_blackout_of_2003 https://en.wikipedia.org/wiki/Northeast_blackout_of_2003
- MichaelGG 10y ago
  If terrorism, what stops someone from renting cabinets, bringing in "servers" loaded with explosives, and blowing the place up? Or even a molotov thrown at some of those big fibre entrances. Sorting that out, I'd imagine, would take a while.
- cfallin 10y ago
  > At the 1wk+ territory, it will start to look something like Katrina ... This reminds me of a blog that was live-updated during Katrina by a data center operator: [1]. They managed to keep everything running through the whole storm, IIRC, but it was pretty dicey -- moving barrels of diesel by hand, staying on-site and surviving with water rationing, losing all but one upstream fiber, etc. Fascinating reading in hindsight. [1] https://en.wikipedia.org/wiki/Interdictor_(blog) https://en.wikipedia.org/wiki/Interdictor_(blog)
- seany 10y ago
  This was an issue after katrina

by jorts 10y ago

barkingcat 10y ago

An outage requiring generator power for longer than 1 week might not allow transportation of fuel. Just want to bring that up in case people really did think you can run "indefinitely" on generator power if the roads, bridges, and gas supply network is out, or if there's an emergency state of war or terrorism. At the 1wk+ territory, it will start to look something like Katrina, where the roads were all blocked by flooding waters and fuel resupply missions required the protection of the National Guard. I mean you can charter helicopters to bring in fuel, but I think there are more urgent things to power than a datacentre in those cases, for example, hospitals and shelters for emergency humanitarian aid.

chinathrow 10y ago
Never underestimate what people would do to keep the stock market running...
- stephengillie 10y ago
  These days, I'd hope they can failover to standby hosts in "IL#" or "VA#" or another datacenter in a nearby region. The DR/BC space has become pretty robust, and with the right vendor, a datacenter loss can be recovered from simply by powering on fully-replicated VMs in another datacenter.
- minimax 10y ago
  Or do! They closed the markets in the US for two days during Sandy even though NY4 (and the other US equity exchange data centers) were fine. http://www.wsj.com/articles/SB10001424052970204789304578087131092892180 http://www.wsj.com/articles/SB100014240529702047893045780871...
- capote 10y ago
  Sometimes it feels like the opposite. Any day that has the slightest hint of holiday to it is "markets closed! remember no work tomorrow! hooray for national silly lala day!"
jtakkala 10y ago
Where I used to work we had refuelling agreements with multiple vendors. Yes, hospitals and critical infrastructure are prioritized first, but we ran on generators for an extended spell during the Northeast Blackout of 2003[1] (the outage affecting our facility only lasted a couple of days, but due to the utilities' appeal to reduce load on the grid until it was fully functional we decided to run on generators for over a week). Edit: One of my fond memories of that time was actually greeting the refuelling truck and directing it to the point where he could plug a hose into a pipe on our building and start pumping. [1] https://en.wikipedia.org/wiki/Northeast_blackout_of_2003 https://en.wikipedia.org/wiki/Northeast_blackout_of_2003
MichaelGG 10y ago
If terrorism, what stops someone from renting cabinets, bringing in "servers" loaded with explosives, and blowing the place up? Or even a molotov thrown at some of those big fibre entrances. Sorting that out, I'd imagine, would take a while.
cfallin 10y ago
> At the 1wk+ territory, it will start to look something like Katrina ... This reminds me of a blog that was live-updated during Katrina by a data center operator: [1]. They managed to keep everything running through the whole storm, IIRC, but it was pretty dicey -- moving barrels of diesel by hand, staying on-site and surviving with water rationing, losing all but one upstream fiber, etc. Fascinating reading in hindsight. [1] https://en.wikipedia.org/wiki/Interdictor_(blog) https://en.wikipedia.org/wiki/Interdictor_(blog)
seany 10y ago
This was an issue after katrina