10 ms·
Bucket Brigade Hauls Diesel Fuel up 17 Stories to Keep NYC Data Center Online
- jaipilot747 14y agoWhat dedication!
- deleted 14y ago[deleted]
- dfj225 14y agoSquarespace is one of the companies in that data center. See here for their story (and some photos): http://blog.squarespace.com/ http://blog.squarespace.com/ http://status.squarespace.com/ http://status.squarespace.com/
- adanto6840 14y agoThanks for the link! It had exactly what I was hoping for from the actual article / OP.
- 7beersonthewall 14y agoAmazing!!!
- activepeanut 14y agoIs that dangerous?
- pyre 14y agoI imagine that spilling diesel down the stairs would be hazardous. At the very least, it's extremely viscous (slippery). In addition, that's probably one of the fire escape routes.
- dredmorbius 14y agoAppropriate username is appropriate.
- dsl 14y agoAt the minimum it's against fire code. But I think the chances of an inspection are pretty low at this point.
- tlrobinson 14y agoNot nearly as dangerous as gasoline would be. Diesel's flash point is like 60°C.
- ryan_s 14y agoWas it worth it? Really?
- ollybee 14y agoCompared to the amount of work involved after unscheduled power loss in a data centre yes. Been there done that, and I would defiantly haul buckets of diesel for 12 hours rather than spent the next months sorting out systems that had not restarted cleanly.
- Moto7451 14y agoI don't think so. At work we set up some servers on the west coast to take over in case our main provider on the east coast went down. I'd say the chance of one of their people getting hurt isn't really worth anyone's uptime. Also all the single site prep in the world doesn't help if that one site is taken out completely. Keeping multiple servers in multiple areas is a must if 100% uptime, even during events like this, is key.
- thomblake 14y agoIf you know of some way to swap out servers with 100% uptime, I'd like to hear it. Even Stack Exchange, which had that sort of plan in place, had to go 'static' for about a half hour.
- mindslight 14y agoAny bets as to whether they will still be singing that common 'implementation efficiency doesn't matter, you can always scale horizontally' tune afterwards?
- derfniw 14y agoMakes me think about http://xkcd.com/705/ http://xkcd.com/705/ .
- JagMicker 14y agoMaybe folks will start to realize that an EDG (Emergency Diesel Generator) without critical support systems is useless. I was just reading about the Fukushima accident, and how most of the EDG's failed because they were water-cooled, and while the EDG's were located out of harms way, the pumps for providing cooling water were located on low ground and were damaged by the tsunami. http://fukushima.ans.org/report/accident-analysis http://fukushima.ans.org/report/accident-analysis Yesterday I saw a story about the Datagram data center in NYC having to shut down. From their reports about the damage (http://www.datagram.com http://www.datagram.com): "As of 5pm on October 29, 2012, Datagram had thoroughly tested its emergency systems at 33 Whitehall, NYC fully staffed and awaiting the storm to hit Manhattan's shores. Once ConEd lost power to Lower Manhattan, Datagram's emergency systems kicked on maintaining power to Datagram's datacenter. Unfortunately, within a couple hours of the storm hitting Manhattan's shores, the building's entire basement, which houses the building's fuel tank pumps and sump pumps, was completely filled with water and a few feet into the lobby. Due to electrical systems being underwater the building was forced to shut down to avoid fire and permanent damage." It's pretty obvious that, despite all the disaster planning done in the past, Datagram (and TEPCO, and others) have really neglected to appreciate the potential modes of failure for their backup power systems. In both cases, they misunderstood the threat to critical backup power infrastructure. If your EDG is on the roof but the fuel pumps and electrical switchgear is in the basement, what will happen during a flood? If your EDG's are on high ground but the pumps to cool them are not, what happens during a tsunami?
- TallGuyShort 14y agoDo you have better suggestions? (Not being a troll - you make a fair point and I'm genuinely interested to hear what you have to say if you would have done it differently)
- tlb 14y agoSome numbers: a 1 megawatt generator burns 70 gallons / hour. If someone can carry 10 gallons (60 lbs), they need to make 7 trips / hour up 17 stories. I think one soldier could manage it.
- FireBeyond 14y agoHighly optimistic. Some numbers: Columbia Tower in Seattle, Firefighter Stairclimb event (I think you could agree a firefighter is probably on par with a soldier for fitness) - 63 stories carrying 50ish pounds of gear, average finish time, 48 minutes. 7 trips an hour up and down 17 stories = 119 stories. Oh, and the firefighters are exhausted, drenched in sweat, require cooling down and up to an hour in rehab for each climb, with legs near collapse, burning like fire... If I had to do this, I'd be going with the bucket brigade, every time (spoken as someone who has completed that stairclimb event).
- deleted 14y ago[deleted]
- mkr-hn 14y agoCan you use the elevators? If the generator doesn't have the extra power to run them, offer some customers credit and a mention in the post-mortem if they'll let you shut them down temporarily to power the elevators. Then you can bring fuel up in drums instead of buckets. edit: Nope. http://news.ycombinator.com/item?id=4723814 http://news.ycombinator.com/item?id=4723814 :(
- eropple 14y agoThe elevators are down for safety reasons, IIRC. And I doubt anyone who's in that building needs "credit" or "a mention in the post-mortem" more than they need the uptime they've paid for.
- deleted 14y ago[deleted]
- anigbrowl 14y agoYeah, seriously. Hauling drums up 17 flights of stairs is both foolishly risky and inefficient. I suspect there's a little marketing going on here.
- nicholassmith 14y agoThis got mentioned on an irc channel a few hours ago and my response was: "Why don't they build a pulley? They're nerds, they have the skills". Obviously shifting diesel about has some risks involved but a basic pulley system would help save a lot of time.
- eropple 14y agoOnly if they have a place to rig it and the tools to do it. Neither are guaranteed (and if they don't, they won't be getting them quickly).
- nicholassmith 14y agoDepends how advanced and fancy they wanted to get it, there's probably enough bits of random hardware in a data centre to improvise something. It'd be more a 'thereifixedit.com' solution than something you'd use on a daily basis, but it could (possibly) work.
- tedunangst 14y agoWho keeps 17 stories of rope in a data center? Do you know how much 17 stories of rope weighs?
- nicholassmith 14y agoA data centre with a really comprehensive disaster recovery plan. Don't necessarily need 17 stories depending on internal layout.
- deleted 14y ago[deleted]
- wukkuan 14y agoBuilding a pulley system would take some significant time and requires finding the necessary tools and resources. The bucket brigade is much simpler and immediately available. Not nearly as much fun, though.
- minikites 14y agoI use Fastmail.fm for email and they're hosted at NYI, which seems to be fine. I wonder what their facilities are like? http://www.nyistatus.com/ http://www.nyistatus.com/
- mkr-hn 14y agohttp://www.nyi.net/facilities http://www.nyi.net/facilities
- hga 14y agoParticularly important is that they're just in Zone C, i.e. apocalyptic flooding required. They also kept hefty onsite fuel reserves, i.e. at one point a reported 30 hours before they needed their first delivery, then 5 days.... This site, the/a main one for the Huffington Post (Datagram), they're all in Zone A, when Zone B flooding was considered to be likely :-( http://project.wnyc.org/news-maps/hurricane-zones/hurricane-zones.html http://project.wnyc.org/news-maps/hurricane-zones/hurricane-... ). I'm sure Manhattan Island datacenter space in Zones A and B cost less, but.... (I too use them for email and their siting has always been one of my biggest concerns.)
- asher_ 14y agoAwesome dedication, but it makes me wonder why geo-redundancy isn't in place for companies the size of these.
- smackfu 14y agoIs there an advantage of 17th floor vs. say 3rd?
- grayrest 14y agoThat's where the generator is.
- smackfu 14y agoOk. Is there some advantage to putting the generator on the 17th floor vs. the 3rd?
- fr0sty 14y agoThe building is 35 stories tall. the 17th and 18th floors are right in the middle. Besides that, I don't think the powers that be want people belching diesel exhaust just a few feet above street level.
- Pyrodogg 14y agoWith all of this effort put into keeping the data center running, I've been wondering about a few things. Was it actually connected to the outside world throughout the storm? I have a hard time imaging that with the power out in large sections of the city some key router on the line wouldn't have also lost power. If that's the case, the effort was put in just to keep the computers warm to prevent unplanned shutdown, not to actually provide uninterrupted service to the customer? I'm not familiar with data center operation. If you're already cut off from the larger network at what point does it make sense to keep the machines running vs. shutting them down? Or perhaps i'm just mistaken and they were actually connected throughout. In which case I find it amazing that the water knocked out pumps and necessitated other shutdowns but their network wasn't damaged in some way.
- Retric 14y agoFiber can be underwater with no problems so it's really just the end points you have to worry about. Honestly, I would be surprised if most networks had much issue with this storm as they generally last until the first time you need to refill the generators. That said available bandwidth is probably significant issue, I had a lot of network issues though the storm, but slow is often a lot better than down.
- madkangas 14y agoYes, it's still connected to the outside world, and has been continuously thus far. Example of a site still being served from machines at Peer1: http://blog.squarespace.com/ http://blog.squarespace.com/ (I am a Squarespace employee)
- bjornsteffanson 14y agoThis shows great dedication on behalf of the team to provide a temporary solution to a more permanent problem. Well done. More importantly, though - and not to discredit any of the hard work that's been done - hopefully the companies take a look at why the problem was created in the first place. For instance: why were the generators on the 17th floor? Why were the pumps below ground? Why was the datacenter built in a floodzone in the first place? This is not unlike a lot of problems we face in software - developers bearing the consequences of poor planning.
- jerrya 14y agoWhy are the backup generators on the 17th floor and not the 3rd floor? Assuming there is a very good reason for that, Why wasn't there an additional pumping room on the 3rd floor, pre-built, with a legal amount of diesel in reserve, and a additional pumps to take over from the basement pumps when those fail, thus saving your bucket brigade 14 floors of climbing? Why are you carrying diesel in the open in 5 gallon buckets and not in fuel containers that were purchased years ago? All in all seems somewhat half-assed.
- jmillikin 14y agoGenerators are placed on the top floor to simplify the exhaust path, which must terminate at the roof. Pumps are placed next to the fuel because pumping liquid over any significant vertical distance requires the pump to "push" rather than "pull". The fuel is placed in the basement because nobody wants to sit next to a tank full of diesel.
- jerrya 14y ago"Generators are placed on the top floor to simplify the exhaust path, which must terminate at the roof." Thank you. "Pumps are placed next to the fuel because pumping liquid over any significant vertical distance requires the pump to "push" rather than "pull". The fuel is placed in the basement because nobody wants to sit next to a tank full of diesel." The proper design would seem to have two pumping stages. One from basement to 3rd floor, the other from 3rd floor to 17th floor. If fire codes are such that one can't safely store 24 - 72 hours of fuel above flood level, don't advertise that your data center has reliable emergency power backup.
- mkr-hn 14y agoI'm surprised to see so many building experts on HN.
- fr0sty 14y agoHere is some information on the setup @ 75 Broad St: "The 17th and 18th floors of 75 Broad have been reserved for generator farms that can accommodate as many as 40 machines. Big doors will be installed in the facade of both floors so the generators can be rigged into the building. A 41,000-gallon fuel tank is being installed in the basement, with a separate generator and three redundant pumps to supply the generators on the 17th and 18th floors. Each tenant will own its own generator -- E-Spire already has one installed outside on the setback on the 17th floor -- but the building will sell them fuel." http://www.nytimes.com/1999/10/10/realestate/commercial-property-75-broad-street-turning-buildings-into-telecommunications.html?pagewanted=all&src=pm http://www.nytimes.com/1999/10/10/realestate/commercial-prop...
- qq66 14y agoThe long-term correct solution to this problem is cloud infrastructure with multi-provider failover. If you have a server in California hosted by Amazon and a server in Texas hosted by Rackspace it's unlikely that you'll find yourself hauling diesel fuel up a staircase.
- gallerytungsten 14y agoThis story reminds me of the efforts one guy in New Orleans went through to keep his data center running after Katrina. There were some tales of diesel hauling in that blog as well. http://interdictor.livejournal.com/57475.html http://interdictor.livejournal.com/57475.html http://interdictor.livejournal.com/40720.html http://interdictor.livejournal.com/40720.html http://en.wikipedia.org/wiki/Interdictor_%28blog%29 http://en.wikipedia.org/wiki/Interdictor_%28blog%29