6 ms·
Internap evacuates LGA datacenter
- ChuckMcM 14y agoI really don't know what the proper procedure should be in a data center when the big klaxon goes off and you here those fateful words over the intercom, "Close all doors, DIVE, DIVE, DIVE!"
- shimms 14y agoIf it is a halon equipped datacenter you listen. And then do as it says. Quickly.
- ChuckMcM 14y agoSo true, my first experience with halon was a briefing at IBM about their machine room policies (I was a student intern) and they stressed that when the fire alarm went off, even if you knew there wasn't a fire, you evacuated the room because when the halon dumped if you didn't have your own source of oxygen it was game over.
- patrickgzill 14y agoAre they unable to bypass the destroyed pumps and take fuel to the generators via 55-gallon drums? If so, why not?
- mikescar 14y agoThe building is being evacuated.
- eurleif 14y agoI wonder what the chances are that the water will go down, and they will be able to supply fuel to the pumps, before power is actually lost.
- siculars 14y agoZero. Their location got flooded with a 12 foot storm surge. I doubt the police are letting anyone near that area. I'm up on 96th and Park. We've had flooding and power outages along the river inland to 2nd ave up here as well. Also, there is practically no way in or out of the island of Manhattan now. All bridges and tunnels are closed except for _possibly_ the Lincoln tunnel.
- patrickgzill 14y agoSo why didn't they do it earlier? That was my point. Of course you have to keep your people safe, but this shows poor planning.
- ghshephard 14y agoStorm surge was higher than expected.
- greyboy 14y agoHow long would some <X> 55-gallon drums full of fuel have helped, anyway? Most cars can easily run through 55 gallons of fuel in a matter of hours (<10), much less a datacenter! There well may have been poor planning at some point, but these things happen so infrequently there must be an allowance for this-is-a-disaster that cannot easily be worked around. Another question is for the customers: are they running all their services from a single datacenter? Sounds like it shows poor planning.
- patrickgzill 14y agoIt is an easy question to answer, e.g. http://www.pmsi-inc.com/pdf/GeneracPowerSystem.pdf http://www.pmsi-inc.com/pdf/GeneracPowerSystem.pdf shows 1MW generator uses about 63 gallons per hour, just a little more than 1 drum. I don't know the size of the facility, so I can't tell how many drums it would take, per hour, to keep the place running.
- greyboy 14y agoSo, now they're expected to move around an unknown quantity of 400-500 lb 55-gallon drums of fuel? And where would they have sourced these so easily and quickly (another unknown, I would think)? We don't know how much energy is being used nor how much fuel is required to run the generator(s) per hour, among other things. That seems like a lot of important information that's missing to call this situation 'poor planning.' But, I'll respond to your single datapoint with mine: according to [1] that's 2 full drums for every one hour of generator running. My point isn't to be simple argumentative but to look at things from a more appropriate perspective. Generators in these circumstances (and my professional experience) are not meant for very long periods or indefinite usage. I'm filing this proposed drum-filling plan into the 'unrealistic' category. 1) http://datacentersmadesimple.com/tech_highlights.html http://datacentersmadesimple.com/tech_highlights.html
- potch 14y ago"The building itself is being evacuated and no remote hands support will be available to assist in any equipment shutdown. Life safety is our number one priority and we are making plans to completely exit the facility." Uptime that requires people to risk their lives isn't worth asking for.
- shimms 14y agoAt the point that a customer notification includes the words "Life safety is our number one priority" things like "can't they bypass the pumps" become far less significant. People's lives are at stake. If the service is important enough that you'd expect them to stay behind and keep it running for you, it is probably important enough that it exists in multiple, geographically separated data centres.
- mmwanga 14y agoOnly rackspace have fanatical support. They would lay down their lives for 1 more minute of uptime.
- sgill 14y agoF*$cked Ironic they tweeted this re AWS 12h ago: Internap @Internap Could 'Frankenstorm' Lead To Another AWS #Outage? http://onforb.es/Vtr45J http://onforb.es/Vtr45J
- acdha 14y agoDid you by any chance save that? They deleted it…
- jart 14y agoSome websites affected by catastrophe: OccupyWallSt.org, Alternet.org
- Zaheer 14y agoLifehacker/Gizmodo and its affiliate sites are down too
- yuhong 14y agohttp://qz.com/21606/how-hurricane-sandy-broke-the-internet/ http://qz.com/21606/how-hurricane-sandy-broke-the-internet/ http://buzzfeed.tumblr.com/post/34607165930/major-media-isp-goes-down http://buzzfeed.tumblr.com/post/34607165930/major-media-isp-...
- bdickason 14y agoWe (Shapeways) are also down.
- frozenport 14y agoClouds are bad for cloud computing.
- silentOpen 14y agoVirtual machines can migrate at lightspeed?
- KyleBrandt 14y agoStackoverflow and Stack Exchange are usually hosted in that building at peer 1, but we failed over to our secondary dc out in OR ealier in the evening.
- tlrobinson 14y agoSee also https://twitter.com/DEVOPS_PORN https://twitter.com/DEVOPS_PORN
- lucb1e 14y agoIt's interesting to see how a storm can singlehandedly cut out some major websites and cause millions to go without power. I think it shows how fragile this tech world really is, and how little we should rely on it in case of a major disaster.
- veemjeem 14y agoAt least with a storm, it's somewhat predictable so one can take action before it hits the datacenter. Nobody was able to predict the outage at Amazon's datacenter which took out large sites like Airbnb & Reddit almost instantly.
- achille 14y agoDon't underestimate "a storm". A hurricane releases about 1.3 x 10^17 joules/day [1] of raw kinetic energy (wind), and that's not including rain and flooding. That's equivalent to 31 megatons per day. The most powerful US nuclear weapons today are about 25 megatons. [1] - http://www.aoml.noaa.gov/hrd/tcfaq/D7.html http://www.aoml.noaa.gov/hrd/tcfaq/D7.html
- deleted 14y ago[deleted]
- rossjudson 14y agoHere's the part I don't understand. This was a record 13 foot storm surge. Is it true that, had the pumps been located above the storm surge, service would have continued uninterrupted? It sounds like everything else in their data center is just fine. This makes it seem like there was a serious design flaw. These pumps were a critical part of the system, but were located in a vulnerable area.
- aes256 14y agoI'm guessing a complete evacuation of the building would have been required regardless, which makes things a little trickier. Would they be able — or would it be advisable — to leave the fuel pumping, generators running and all operations unaffected without any staff on site?
- kalleboo 14y agoSo they have more sense than DirectNIC had during Katrina http://www.wired.com/science/discoveries/news/2005/09/68725 http://www.wired.com/science/discoveries/news/2005/09/68725
- brokentone 14y agoUpdate at 10:17ET http://pastebin.com/NUQNHHJi http://pastebin.com/NUQNHHJi Interesting they don't have a public status page of any sort.
- KevBurnsJr 14y agoInternap's cloud service taken down by Sandy's cloud service.
- brokentone 14y agoInternap LGA11 lost power at 11:48AM ET http://pastebin.com/6AxvbzF1 http://pastebin.com/6AxvbzF1