4 ms·

Maybe folks will start to realize that an EDG (Emergency Diesel Generator) without critical support systems is useless. I was just reading about the Fukushima

Maybe folks will start to realize that an EDG (Emergency Diesel Generator) without critical support systems is useless. I was just reading about the Fukushima accident, and how most of the EDG's failed because they were water-cooled, and while the EDG's were located out of harms way, the pumps for providing cooling water were located on low ground and were damaged by the tsunami. http://fukushima.ans.org/report/accident-analysis http://fukushima.ans.org/report/accident-analysis Yesterday I saw a story about the Datagram data center in NYC having to shut down. From their reports about the damage (http://www.datagram.com http://www.datagram.com): "As of 5pm on October 29, 2012, Datagram had thoroughly tested its emergency systems at 33 Whitehall, NYC fully staffed and awaiting the storm to hit Manhattan's shores. Once ConEd lost power to Lower Manhattan, Datagram's emergency systems kicked on maintaining power to Datagram's datacenter. Unfortunately, within a couple hours of the storm hitting Manhattan's shores, the building's entire basement, which houses the building's fuel tank pumps and sump pumps, was completely filled with water and a few feet into the lobby. Due to electrical systems being underwater the building was forced to shut down to avoid fire and permanent damage." It's pretty obvious that, despite all the disaster planning done in the past, Datagram (and TEPCO, and others) have really neglected to appreciate the potential modes of failure for their backup power systems. In both cases, they misunderstood the threat to critical backup power infrastructure. If your EDG is on the roof but the fuel pumps and electrical switchgear is in the basement, what will happen during a flood? If your EDG's are on high ground but the pumps to cool them are not, what happens during a tsunami?

TallGuyShort 14y ago
Do you have better suggestions? (Not being a troll - you make a fair point and I'm genuinely interested to hear what you have to say if you would have done it differently)
- mjwalshe 14y ago
  Submesible pumps or rig a block and tackle out of a window and haul jerry cans up that way safer thna carying open buckets by hand.
- X-Istence 14y ago
  To reply to mjwalshe ... who from the looks of it is hellbanned: > Submesible pumps or rig a block and tackle out of a window and haul jerry cans up that way safer thna carying open buckets by hand. Diesel fuel can only be ignited by a high power flame (such as from a propane torch), and will not simply ignite due to sparks. Diesel fuel is amazingly safe to handle, much like fryer grease is safe to handle.
  - deleted 14y ago
    [deleted]
  - walshemj 14y ago
    True but a slipping on spilled diesel and falling down a flight of stairs is one danger still think rigging block an tackle would be much more efficient.
- Pyrodogg 14y ago
  Shouldn't it be possible to make the fuel pump and it's power supply a closed, water proof system? Such that the fuel tank could be flooded and the pump would still run?
  - TallGuyShort 14y ago
    Probably a good idea, but I can't help but think what other weakness that system might turn out to have in the next disaster. I don't know if it's just me, but I was definitely very surprised by the degree of flooding - I would have expected debris, structural damage and power outages, but even as a former New Yorker, I was surprised it all happened.
- deleted 14y ago
  [deleted]
- JagMicker 14y ago
  No, I don't. But I think this shows a lack of understanding on the part of the data centers. Datagram said they'd tested their generators prior to the storm, but apparently they were unaware that the electrical switchgear was located in an area that would readily flood. During a flood, what good is a generator if the electrical equipment can't be used? Would a possible solution be to have the generator output run directly to the datacenter, and to have the ability to isolate it from the building ("mains") power?
  - TallGuyShort 14y ago
    Also, see my sibling response to Pyrodogg - it was kind of a joint response to both :)
tghw 14y ago
Unfortunately, physics dictates this setup. To get to a top floor generator (which is generally necessary for ventilation), you have to pump from the bottom. For most buildings, that means from the basement, where the fuel tanks are placed for other obvious reasons. (Consider if one started leaking, would you want it dripping down to all of the floors below it?) These data centers are well designed, but it's impossible to cover every disaster scenario. It's easy for you to sit here after the fact and snipe at them. If you really think you can do better, go start designing them yourself. If you succeed, you'll do quite well for yourself.
- danudey 14y ago
  Regardless of whether they can be designed better, the data centres were assuring everyone that everything will be fine because they have backup generators, ignoring the probability that flooding would take out those generators. If I had a website or service hosted at a data centre likely to go offline, I'd be annoyed if I were reassured that everything was taken care of right before the system went offline.
- DannyBee 14y ago
  Your one complaint about diesel fuel dripping is not a big deal in the short term. Diesel fuel is quite safe and hard to ignite. THere are plenty of places/situations where significantly more toxic substances are stored in tanks that are not in basements. Your logic would apply more there (could you imagine if acetone was dripping from the ceiling?). As a practical matter, anything leaking through floors is really bad, but the problem is solvable and not a good reason to keep fuel tanks in a basement. So what other reasons are fuel tanks placed in basements that outweigh the risk of putting them in the basement (everything except tornadoes would seem to be a con for basements)? Additionally, "Flooding" would seem to be a fairly typical disaster scenario. Again, what does putting them in the basement prevent against that makes it a good alternative. Plenty of people are designing better datacenters and using them, there is no need for the parent to go out and do anything. If customers hold internap/et al accountable, the problem should take care of itself. (of course, they won't, but that's another matter :P)
  - 14y ago
imgabe 14y ago
As someone who designs these sorts of things for a living, there are a host of problems with having diesel fuel on the room. For one thing, fire codes limit the amount of diesel fuel you can store on the roof (for obvious reasons). For another thing, the trucks that carry in the diesel fuel to refill the tanks are located on the street. One way or another you're going to have to pump fuel from the street up to the roof, which is going to leave the pumps that do so susceptible to flooding.
- jbert 14y ago
  Would a closed, pressurised air system work as a one-off "push all diesel from this tank to the roof" system? So you could store: - tank full of diesel - some compressed air cylinders at a low level, and use them in the "everything is submerged" event.
  - imgabe 14y ago
    My expertise is on the electrical side, rather than the plumbing side, but I suspect the amount of compressed air you would need to propel 500 gallons or so of diesel up 17 stories would be impractical.
  - jordanb 14y ago
    One can buy pneumatic submersible pumps. I could see having a compressor on the upper levels of the building drive the pump in the basement. http://www.atlascopco.us/usus/products/navigationbyproduct/productgroup.aspx?id=1602137 http://www.atlascopco.us/usus/products/navigationbyproduct/p... Of course, if the diesel has water in it it's no good anyway. And the submersible pumps are mainly sludge or slurry pumps and (I suspect) not rated to handle fuel.
  - danshapiro 14y ago
    500 gal diesel = 1622 kg 17 stories = 46 meters Energy required = 1621 * 46 * g = 732 kJ Integrating PV=nRT, W = -nRT*ln(V₂/V₁) assume you're going to compress the air to 150 psi, or 10 atm, so V₂/V₁ = 10. Assume T=10 degrees C. Solving for n, you need 880 moles of air. Putting this back through PV=nRT, we get a modest 190 gallons of air (compressed). So you need a 200 gallon tank with 150 psi storage. Look, here's one on craigslist for $400. http://eugene.craigslist.org/gms/3306858286.html http://eugene.craigslist.org/gms/3306858286.html
lesterbuck 14y ago
I would have thought Tropical Storm Allison in 2001 was the teaching moment for the data center business, when all the underground infrastructure of downtown Houston was completely flooded. Apparently planners had not realized that critical backup power systems (or research animals, or symphony archives, or ...) shouldn't be below the level of potential flooding. Houston is 25 miles from the Gulf, so this wasn't a storm surge, just a huge steady rain event. I guess the lesson will be repeated until building management learns. http://en.wikipedia.org/wiki/Tropical_Storm_Allison http://en.wikipedia.org/wiki/Tropical_Storm_Allison
jordanb 14y ago
It's hard to call the diesel generators "useless" when this company is successfully using them to power its facilities. The bucket brigade is clearly not an optimal solution, but it is a solution that is working so far.
hollerith 14y ago
Refraining from locating a data center in a floodplain seems like the easiest way to get greater reliability.
- mjwalshe 14y ago
  True back in the day I worked for Telecom Gold(early online service) based next to the thames river and eventulay we moved all our machines to a DC well away from the river becuse of the flood risk Fun listning to our network guy who when we complained about the slow 10Mbps link (in 1984) he commeneted dont knock it I had oxford street dug up for this.

TallGuyShort 14y ago

Do you have better suggestions? (Not being a troll - you make a fair point and I'm genuinely interested to hear what you have to say if you would have done it differently)

mjwalshe 14y ago
Submesible pumps or rig a block and tackle out of a window and haul jerry cans up that way safer thna carying open buckets by hand.
X-Istence 14y ago
To reply to mjwalshe ... who from the looks of it is hellbanned: > Submesible pumps or rig a block and tackle out of a window and haul jerry cans up that way safer thna carying open buckets by hand. Diesel fuel can only be ignited by a high power flame (such as from a propane torch), and will not simply ignite due to sparks. Diesel fuel is amazingly safe to handle, much like fryer grease is safe to handle.
- deleted 14y ago
  [deleted]
- walshemj 14y ago
  True but a slipping on spilled diesel and falling down a flight of stairs is one danger still think rigging block an tackle would be much more efficient.
Pyrodogg 14y ago
Shouldn't it be possible to make the fuel pump and it's power supply a closed, water proof system? Such that the fuel tank could be flooded and the pump would still run?
- TallGuyShort 14y ago
  Probably a good idea, but I can't help but think what other weakness that system might turn out to have in the next disaster. I don't know if it's just me, but I was definitely very surprised by the degree of flooding - I would have expected debris, structural damage and power outages, but even as a former New Yorker, I was surprised it all happened.
deleted 14y ago
[deleted]
JagMicker 14y ago
No, I don't. But I think this shows a lack of understanding on the part of the data centers. Datagram said they'd tested their generators prior to the storm, but apparently they were unaware that the electrical switchgear was located in an area that would readily flood. During a flood, what good is a generator if the electrical equipment can't be used? Would a possible solution be to have the generator output run directly to the datacenter, and to have the ability to isolate it from the building ("mains") power?
- TallGuyShort 14y ago
  Also, see my sibling response to Pyrodogg - it was kind of a joint response to both :)

tghw 14y ago

Unfortunately, physics dictates this setup. To get to a top floor generator (which is generally necessary for ventilation), you have to pump from the bottom. For most buildings, that means from the basement, where the fuel tanks are placed for other obvious reasons. (Consider if one started leaking, would you want it dripping down to all of the floors below it?) These data centers are well designed, but it's impossible to cover every disaster scenario. It's easy for you to sit here after the fact and snipe at them. If you really think you can do better, go start designing them yourself. If you succeed, you'll do quite well for yourself.

danudey 14y ago
Regardless of whether they can be designed better, the data centres were assuring everyone that everything will be fine because they have backup generators, ignoring the probability that flooding would take out those generators. If I had a website or service hosted at a data centre likely to go offline, I'd be annoyed if I were reassured that everything was taken care of right before the system went offline.
DannyBee 14y ago
Your one complaint about diesel fuel dripping is not a big deal in the short term. Diesel fuel is quite safe and hard to ignite. THere are plenty of places/situations where significantly more toxic substances are stored in tanks that are not in basements. Your logic would apply more there (could you imagine if acetone was dripping from the ceiling?). As a practical matter, anything leaking through floors is really bad, but the problem is solvable and not a good reason to keep fuel tanks in a basement. So what other reasons are fuel tanks placed in basements that outweigh the risk of putting them in the basement (everything except tornadoes would seem to be a con for basements)? Additionally, "Flooding" would seem to be a fairly typical disaster scenario. Again, what does putting them in the basement prevent against that makes it a good alternative. Plenty of people are designing better datacenters and using them, there is no need for the parent to go out and do anything. If customers hold internap/et al accountable, the problem should take care of itself. (of course, they won't, but that's another matter :P)
- 14y ago

imgabe 14y ago

As someone who designs these sorts of things for a living, there are a host of problems with having diesel fuel on the room. For one thing, fire codes limit the amount of diesel fuel you can store on the roof (for obvious reasons). For another thing, the trucks that carry in the diesel fuel to refill the tanks are located on the street. One way or another you're going to have to pump fuel from the street up to the roof, which is going to leave the pumps that do so susceptible to flooding.

jbert 14y ago
Would a closed, pressurised air system work as a one-off "push all diesel from this tank to the roof" system? So you could store: - tank full of diesel - some compressed air cylinders at a low level, and use them in the "everything is submerged" event.
- imgabe 14y ago
  My expertise is on the electrical side, rather than the plumbing side, but I suspect the amount of compressed air you would need to propel 500 gallons or so of diesel up 17 stories would be impractical.
- jordanb 14y ago
  One can buy pneumatic submersible pumps. I could see having a compressor on the upper levels of the building drive the pump in the basement. http://www.atlascopco.us/usus/products/navigationbyproduct/productgroup.aspx?id=1602137 http://www.atlascopco.us/usus/products/navigationbyproduct/p... Of course, if the diesel has water in it it's no good anyway. And the submersible pumps are mainly sludge or slurry pumps and (I suspect) not rated to handle fuel.
- danshapiro 14y ago
  500 gal diesel = 1622 kg 17 stories = 46 meters Energy required = 1621 * 46 * g = 732 kJ Integrating PV=nRT, W = -nRT*ln(V₂/V₁) assume you're going to compress the air to 150 psi, or 10 atm, so V₂/V₁ = 10. Assume T=10 degrees C. Solving for n, you need 880 moles of air. Putting this back through PV=nRT, we get a modest 190 gallons of air (compressed). So you need a 200 gallon tank with 150 psi storage. Look, here's one on craigslist for $400. http://eugene.craigslist.org/gms/3306858286.html http://eugene.craigslist.org/gms/3306858286.html

lesterbuck 14y ago

I would have thought Tropical Storm Allison in 2001 was the teaching moment for the data center business, when all the underground infrastructure of downtown Houston was completely flooded. Apparently planners had not realized that critical backup power systems (or research animals, or symphony archives, or ...) shouldn't be below the level of potential flooding. Houston is 25 miles from the Gulf, so this wasn't a storm surge, just a huge steady rain event. I guess the lesson will be repeated until building management learns. http://en.wikipedia.org/wiki/Tropical_Storm_Allison http://en.wikipedia.org/wiki/Tropical_Storm_Allison

jordanb 14y ago

It's hard to call the diesel generators "useless" when this company is successfully using them to power its facilities. The bucket brigade is clearly not an optimal solution, but it is a solution that is working so far.

hollerith 14y ago

Refraining from locating a data center in a floodplain seems like the easiest way to get greater reliability.

mjwalshe 14y ago
True back in the day I worked for Telecom Gold(early online service) based next to the thames river and eventulay we moved all our machines to a DC well away from the river becuse of the flood risk Fun listning to our network guy who when we complained about the slow 10Mbps link (in 1984) he commeneted dont knock it I had oxford street dug up for this.