8 ms·

Server room with seismic isolation floor in Japan earthquake disaster [video]

234 points by DamnInteresting 9y ago

rjbwork 9y ago
Poor sysadmins! They have to wear full 3 pieces suits on the DC Floor! Besides that, the video/tech is really cool.
- hvidgaard 9y ago
  At least it's a nice cool temperature.
  - robinson-wall 9y ago
    Depends which aisle you're in :)
- wst_ 9y ago
  That day, I wouldn't call them poor. Probably the safest place out there.
johnflan 9y ago
That earthquake goes on for way longer than I would have expected.
- russdill 9y ago
  The duration and intensity of an Earthquake are strongly correlated. You can even use it to estimate how strong a quake is and how far away you might be. The Sendai earthquake in 2011 was 6 minutes long. There's a story which I now cannot find about a seismic conference being held in Japan some distance from the epicenter. They felt only moderate shaking, but were immediately aware of the severity of the quake due to the duration.
  - wlesieutre 9y ago
    Relevant portion starts right at the top: http://www.newyorker.com/magazine/2015/07/20/the-really-big-one http://www.newyorker.com/magazine/2015/07/20/the-really-big-...
sjbase 9y ago
This is a great example of one of those investments that might feel like a waste, until it suddenly REALLY isn't. I wish I had this video during my infosec & HA consulting days.
- Symbiote 9y ago
  I live in seismicly boring Europe, so I have no idea what damage would be caused without this floor. Many broken hard drives? Or worse?
  - mschuster91 9y ago
    All drives broken at the very least, a 10k rpm drive doesn't like ordinary vibrations already, much less 40cm+ displacement in a fraction of a second. Heavy servers and especially UPS battery rows would likely take the whole rack down/toppling over with them. Plus: most cabling is secured to the walls/floors with not very much slack, which means a LOT of damaged/torn cables and FUBAR'd ports. All in all, if an earthquake of the scale in the video would hit a typical European DC, it's bye-bye and rebuild.
    - dmix 9y ago
      Forgive my uninformed question but I remember seeing a video many years ago where a (Samsung?) hard-drive had built-in protection from being dropped. The disk will stop spinning within a millisecond when it detects movement. They could survive a sudden drop onto a floor. I assumed this was a common feature of modern disk based hard-drives. Could built-in protections like this in harddrives be a sufficient protection in an earthquake? What other moving parts are that at risk here? Assuming the servers themselves are locked in cages, I wouldn't expect them to crash to the floor. Although the big difference here is it's not a sudden drop but a prolonged forceful shaking.
      - AlephGarden 9y ago
        That keeps your data safe, but (and I'm guessing here) quite plausibly means filesystem writes may likely start failing, causing services on the affected server to start failing. If you need uptime during an earthquake, then that's not sufficient.
laurentl 9y ago
I visited a couple of DCs in Tokyo, and the tours systematically feature a look at the anti-seismic system. It's actually pretty unimpressive : basically the seismic-protected part of the building is mounted on big rubber dampers, with some huge pistons thrown in for active attenuation. In taller buildings (I visited a DC with around 20 stories IIRC) the stories are not rigidly connected together, so that instead of swaying (and possibly toppling) during an earthquake, the building just kind of wobbles. Very effective though, as the video shows. My visits were post-2011, and each DC had a record of the building's movement during the big earthquake; max amplitude on the seismic-protected part was a couple of centimeters, vs 50 cms or more for the rest of the building.
- cryptonector 9y ago
  Unimpressive?! Well, I am quite impressed.
  - laurentl 9y ago
    Just to clarify: I meant that the equipment itself is unimpressive (mundane would be a better term I guess) but the results are definitely impressive :D
- finid 9y ago
  Very impressive, actually. More impressive are the guys walking around during all that swaying as if they were on coffee brake.
- walrus01 9y ago
  whole building isolation/damper system vs no dampening comparison: https://www.youtube.com/watch?v=kzVvd4Dk6sw https://www.youtube.com/watch?v=kzVvd4Dk6sw
  - factsaresacred 9y ago
    This is another neat demo: https://www.youtube.com/watch?v=xp2pGxFzrzI https://www.youtube.com/watch?v=xp2pGxFzrzI
cryptonector 9y ago
I've been in a datacenter in Tokyo where the floors were not seismically isolated, but the rack cabinets were made to sway. It's hard to tell many floors up that an earthquake is in progress, but you can see it when you see the racks swaying!
- redm 9y ago
  I visited the Dreamworks DC in SOCAL back in 2005 and they had per rack isolation plates as you remember. They also had a 25' long tape drive on one huge isolation plate. Those plates basically are on a central piston and rotate in a circular pattern. Seems a lot cheaper than proofing the entire floor.
pavement 9y ago
I guess a similar level of movement isolation would probably be required for all those repurposed oil rigs and maritime-oriented machine rooms that anticipate taking advatage of sea water as server coolant.
factsaresacred 9y ago
"Techno Mind Corporation". Think I saw them play in Berlin once.
guscost 9y ago
Can a multimedia person make the video fixed on the servers instead?
saalweachter 9y ago
I wonder if it is disorienting standing or walking on the movement isolating floor. There's a (thought?) experiment you can do where you put someone inside a fake room built like an overturned box, and then jerk the walls in a random direction while the floor stay still. You get disoriented and fall over. Or the more common experience where you're on a stopped train and the train next to you starts to move, and it takes you a minute to figure out which of you is actually moving. I'd have to imagine standing on the isolated floor while the rest of the building moves would be similarly confusing.
- lojack 9y ago
  or like the spinning tunnel effect, where the walls are spinning around you and you have difficulty walking straight because your brain thinks you're falling over https://en.wikipedia.org/wiki/Spinning_tunnel https://en.wikipedia.org/wiki/Spinning_tunnel
- hinkley 9y ago
  The guys in that video look to me like they are experiencing some vertigo from the room shifting. They keep grabbing things to steady themselves.
- mannykannot 9y ago
  The apparent motion of the floor makes running along the edge of the gap, as one guy does towards the end, look particularly risky.
ChuckMcM 9y ago
Category "Auto and Vehicles" ? It is interesting to note that the people on the seismically isolated floor are not swaying with the video (it is explained that the camera is mounted to the non-isolated part of the building so it appears the server room floor is moving when instead it is the building that is moving). Most of the data centers I've looked at in the Bay Area just bolt the racks to the floor and are done with it assuming, I presume, that it is the shifting on the floor that damages the hardware.
- hinkley 9y ago
  Probably more an issue of shifting racks damaging the employees. Bolt those puppies down and nobody gets crushed to death.
eternalban 9y ago
Related: https://www.curbed.com/2011/3/15/10477706/japan-benefits-from-wrights-lessons-on-earthquake-preparedness https://www.curbed.com/2011/3/15/10477706/japan-benefits-fro...
i336_ 9y ago
Any video editors got a few minutes? There's an open video stabilization request here: https://www.reddit.com/r/ImageStabilization/comments/6e1mgj/request_stabilize_on_the_computer_servers_that/ https://www.reddit.com/r/ImageStabilization/comments/6e1mgj/... So far there's https://streamable.com/lrkhg https://streamable.com/lrkhg which sorta gets partway there.
vzaliva 9y ago
What is amazing is how cool are the people in the video. Instead of running out screaming they calmly do their job.
- hinkley 9y ago
  There are an astoundingly high number of quakes in Japan that are strong enough that you can feel. When I was in Tokyo years ago, I felt one every four weeks on average. You quickly come to a different relationship with the world when it refuses to stay still that often.
- matt4077 9y ago
  I sure hope that being among the racks is also the safest option for them, and that they are not risking anything just to keep these servers running.
sengork 9y ago
Commercial racks and datacenter products have optional features that can be utilised in earthquake prone environments. What you see in the video is not only those rack features but also structure of the building itself (both are important and neither one is sufficient for certain areas). For example see: https://www.ibm.com/support/knowledgecenter/STXN8P/com.ibm.storage.ssic.help.doc/f2c_prepearthqkitraised_30yyzi.html https://www.ibm.com/support/knowledgecenter/STXN8P/com.ibm.s... Without employing these techniques, you'd get something similar to this: https://i.imgur.com/Sb2M5Qo.jpg https://i.imgur.com/Sb2M5Qo.jpg

rjbwork 9y ago

Poor sysadmins! They have to wear full 3 pieces suits on the DC Floor! Besides that, the video/tech is really cool.

hvidgaard 9y ago
At least it's a nice cool temperature.
- robinson-wall 9y ago
  Depends which aisle you're in :)
wst_ 9y ago
That day, I wouldn't call them poor. Probably the safest place out there.

johnflan 9y ago

That earthquake goes on for way longer than I would have expected.

russdill 9y ago
The duration and intensity of an Earthquake are strongly correlated. You can even use it to estimate how strong a quake is and how far away you might be. The Sendai earthquake in 2011 was 6 minutes long. There's a story which I now cannot find about a seismic conference being held in Japan some distance from the epicenter. They felt only moderate shaking, but were immediately aware of the severity of the quake due to the duration.
- wlesieutre 9y ago
  Relevant portion starts right at the top: http://www.newyorker.com/magazine/2015/07/20/the-really-big-one http://www.newyorker.com/magazine/2015/07/20/the-really-big-...

sjbase 9y ago

This is a great example of one of those investments that might feel like a waste, until it suddenly REALLY isn't. I wish I had this video during my infosec & HA consulting days.

Symbiote 9y ago
I live in seismicly boring Europe, so I have no idea what damage would be caused without this floor. Many broken hard drives? Or worse?
- mschuster91 9y ago
  All drives broken at the very least, a 10k rpm drive doesn't like ordinary vibrations already, much less 40cm+ displacement in a fraction of a second. Heavy servers and especially UPS battery rows would likely take the whole rack down/toppling over with them. Plus: most cabling is secured to the walls/floors with not very much slack, which means a LOT of damaged/torn cables and FUBAR'd ports. All in all, if an earthquake of the scale in the video would hit a typical European DC, it's bye-bye and rebuild.
  - dmix 9y ago
    Forgive my uninformed question but I remember seeing a video many years ago where a (Samsung?) hard-drive had built-in protection from being dropped. The disk will stop spinning within a millisecond when it detects movement. They could survive a sudden drop onto a floor. I assumed this was a common feature of modern disk based hard-drives. Could built-in protections like this in harddrives be a sufficient protection in an earthquake? What other moving parts are that at risk here? Assuming the servers themselves are locked in cages, I wouldn't expect them to crash to the floor. Although the big difference here is it's not a sudden drop but a prolonged forceful shaking.
    - AlephGarden 9y ago
      That keeps your data safe, but (and I'm guessing here) quite plausibly means filesystem writes may likely start failing, causing services on the affected server to start failing. If you need uptime during an earthquake, then that's not sufficient.

laurentl 9y ago

I visited a couple of DCs in Tokyo, and the tours systematically feature a look at the anti-seismic system. It's actually pretty unimpressive : basically the seismic-protected part of the building is mounted on big rubber dampers, with some huge pistons thrown in for active attenuation. In taller buildings (I visited a DC with around 20 stories IIRC) the stories are not rigidly connected together, so that instead of swaying (and possibly toppling) during an earthquake, the building just kind of wobbles. Very effective though, as the video shows. My visits were post-2011, and each DC had a record of the building's movement during the big earthquake; max amplitude on the seismic-protected part was a couple of centimeters, vs 50 cms or more for the rest of the building.

cryptonector 9y ago
Unimpressive?! Well, I am quite impressed.
- laurentl 9y ago
  Just to clarify: I meant that the equipment itself is unimpressive (mundane would be a better term I guess) but the results are definitely impressive :D
finid 9y ago
Very impressive, actually. More impressive are the guys walking around during all that swaying as if they were on coffee brake.
walrus01 9y ago
whole building isolation/damper system vs no dampening comparison: https://www.youtube.com/watch?v=kzVvd4Dk6sw https://www.youtube.com/watch?v=kzVvd4Dk6sw
- factsaresacred 9y ago
  This is another neat demo: https://www.youtube.com/watch?v=xp2pGxFzrzI https://www.youtube.com/watch?v=xp2pGxFzrzI

cryptonector 9y ago

I've been in a datacenter in Tokyo where the floors were not seismically isolated, but the rack cabinets were made to sway. It's hard to tell many floors up that an earthquake is in progress, but you can see it when you see the racks swaying!

redm 9y ago
I visited the Dreamworks DC in SOCAL back in 2005 and they had per rack isolation plates as you remember. They also had a 25' long tape drive on one huge isolation plate. Those plates basically are on a central piston and rotate in a circular pattern. Seems a lot cheaper than proofing the entire floor.

pavement 9y ago

I guess a similar level of movement isolation would probably be required for all those repurposed oil rigs and maritime-oriented machine rooms that anticipate taking advatage of sea water as server coolant.

factsaresacred 9y ago

"Techno Mind Corporation". Think I saw them play in Berlin once.

guscost 9y ago

Can a multimedia person make the video fixed on the servers instead?

saalweachter 9y ago

I wonder if it is disorienting standing or walking on the movement isolating floor. There's a (thought?) experiment you can do where you put someone inside a fake room built like an overturned box, and then jerk the walls in a random direction while the floor stay still. You get disoriented and fall over. Or the more common experience where you're on a stopped train and the train next to you starts to move, and it takes you a minute to figure out which of you is actually moving. I'd have to imagine standing on the isolated floor while the rest of the building moves would be similarly confusing.

lojack 9y ago
or like the spinning tunnel effect, where the walls are spinning around you and you have difficulty walking straight because your brain thinks you're falling over https://en.wikipedia.org/wiki/Spinning_tunnel https://en.wikipedia.org/wiki/Spinning_tunnel
hinkley 9y ago
The guys in that video look to me like they are experiencing some vertigo from the room shifting. They keep grabbing things to steady themselves.
mannykannot 9y ago
The apparent motion of the floor makes running along the edge of the gap, as one guy does towards the end, look particularly risky.

ChuckMcM 9y ago

Category "Auto and Vehicles" ? It is interesting to note that the people on the seismically isolated floor are not swaying with the video (it is explained that the camera is mounted to the non-isolated part of the building so it appears the server room floor is moving when instead it is the building that is moving). Most of the data centers I've looked at in the Bay Area just bolt the racks to the floor and are done with it assuming, I presume, that it is the shifting on the floor that damages the hardware.

hinkley 9y ago
Probably more an issue of shifting racks damaging the employees. Bolt those puppies down and nobody gets crushed to death.

eternalban 9y ago

Related: https://www.curbed.com/2011/3/15/10477706/japan-benefits-from-wrights-lessons-on-earthquake-preparedness https://www.curbed.com/2011/3/15/10477706/japan-benefits-fro...

i336_ 9y ago

Any video editors got a few minutes? There's an open video stabilization request here: https://www.reddit.com/r/ImageStabilization/comments/6e1mgj/request_stabilize_on_the_computer_servers_that/ https://www.reddit.com/r/ImageStabilization/comments/6e1mgj/... So far there's https://streamable.com/lrkhg https://streamable.com/lrkhg which sorta gets partway there.

vzaliva 9y ago

What is amazing is how cool are the people in the video. Instead of running out screaming they calmly do their job.

hinkley 9y ago
There are an astoundingly high number of quakes in Japan that are strong enough that you can feel. When I was in Tokyo years ago, I felt one every four weeks on average. You quickly come to a different relationship with the world when it refuses to stay still that often.
matt4077 9y ago
I sure hope that being among the racks is also the safest option for them, and that they are not risking anything just to keep these servers running.

sengork 9y ago

Commercial racks and datacenter products have optional features that can be utilised in earthquake prone environments. What you see in the video is not only those rack features but also structure of the building itself (both are important and neither one is sufficient for certain areas). For example see: https://www.ibm.com/support/knowledgecenter/STXN8P/com.ibm.storage.ssic.help.doc/f2c_prepearthqkitraised_30yyzi.html https://www.ibm.com/support/knowledgecenter/STXN8P/com.ibm.s... Without employing these techniques, you'd get something similar to this: https://i.imgur.com/Sb2M5Qo.jpg https://i.imgur.com/Sb2M5Qo.jpg