4 ms·

Google runs a company-wide disaster recovery testing event every year. Earthquakes are among the numerous scenarios simulated. https://queue.acm.org/detail.cfm

by alxv 11y ago

Google runs a company-wide disaster recovery testing event every year. Earthquakes are among the numerous scenarios simulated. https://queue.acm.org/detail.cfm?id=2371516 https://queue.acm.org/detail.cfm?id=2371516

sneak 11y ago
Great link. Reminds me of this: http://craphound.com/overclocked/Cory_Doctorow_-_Overclocked_-_When_Sysadmins_Ruled_the_Earth.html http://craphound.com/overclocked/Cory_Doctorow_-_Overclocked...
idlewords 11y ago
The problem is that you can't really simulate the systemic effects of a really major quake (like a scenario where Mountain View is out of power for weeks).
- falcolas 11y ago
  Sure you can. Just look for the keys to the main breakers to those buildings. Of course, paying for that amount of vacation for 15 days is probably not entirely feasible for any company.
  - idlewords 11y ago
    By 'systemic' I mean not confined to one company.
- nostrademons 11y ago
  The DiRT drills include the simultaneous loss of whole offices, including the complete obliteration of the Bay Area. Participants are instructed to pretend that they've been eaten by zombies and not communicate with anyone in a remote office, and that's backed up (as described in the article) by cutting the network links to & from the Mountain View corporate office. Now, there may certainly be larger systemic effects over long time periods caused by eg. the death of the company CEO, the loss of new product development, or the complete destruction of the company's market (in, for example, a global thermonuclear apocalypse that sets us back to the stone age). But the tests absolutely do simulate the simultaneous death of 15,000+ employees, destruction of the Mountain View headquarters, and loss of all datacenters, data, and power in the Bay Area, and verify that remote teams can continue the day-to-day operations in such an event.
  - Sven7 11y ago
    What happens if Google goes down for a while? Is it really that bad? Yes, Gmail and Drive may cause all kinds of chaos but I think I can survive quite a while without the search engine and YouTube.
    - pc86 11y ago
      In the grand scheme of things, not a lot of businesses use Apps for their email, but I'd hazard a guess that you're still looking at hundreds of millions of dollars in losses if that were to suddenly disappear.
      - Mtinie 11y ago
        It's the cascading service failures and related economic costs that would be the most damaging (and hardest to really quantify until a scenario really happens).
      - sneak 11y ago
        As with most people who reason about Google, you are off by at least two orders of magnitude in your estimate.
        gaadd33 11y ago
        Wow, I had no idea that Google Apps were about 15% of Google's revenue. That is a way bigger than I would have guessed. (Hundreds of millions x 2 orders of magnitude is >= 10B, total revenue in 2014, 66B)
        pc86 11y ago
        I'm not referring to Google revenue, I'm referring to productivity/revenue for the businesses using Apps day to day.
    - rryan 11y ago
      The data could be lost. Part of DiRT is ensuring that the result of losing N data centers is not "oops we lost 20% of GMail users data permanently". On the scale of devastation this article is describing, none of Google's servicing being obliterated really matters but I'm personally pretty glad Google is prepared for multiple, massive scale disasters to happen at once.
- yokohama11 11y ago
  Most large companies do/have done exactly that. Arguably 9/11 was the big catalyst for DR planning. There were Wall St firms that literally had to get the military to escort them into Lower Manhattan in the days after to rescue servers, tapes, etc and cobble something together in a satellite office. No one wants a repeat of that, and you'll notice subsequent disasters like Sandy have barely been a blip for most as a result of policy changes. At this point basically any major company will/should have procedures in place for "perfect" DR, the ability for everyone to work from home indefinitely (and alternate offices/infrastructure waiting for the few people who need something special, like the special needs of a trading desk), as well as obviously having back up data centers with full replication.
kraig 11y ago
tsunami monkey?

by alxv 11y ago

sneak 11y ago

Great link. Reminds me of this: http://craphound.com/overclocked/Cory_Doctorow_-_Overclocked_-_When_Sysadmins_Ruled_the_Earth.html http://craphound.com/overclocked/Cory_Doctorow_-_Overclocked...

idlewords 11y ago

The problem is that you can't really simulate the systemic effects of a really major quake (like a scenario where Mountain View is out of power for weeks).

falcolas 11y ago
Sure you can. Just look for the keys to the main breakers to those buildings. Of course, paying for that amount of vacation for 15 days is probably not entirely feasible for any company.
- idlewords 11y ago
  By 'systemic' I mean not confined to one company.
nostrademons 11y ago
The DiRT drills include the simultaneous loss of whole offices, including the complete obliteration of the Bay Area. Participants are instructed to pretend that they've been eaten by zombies and not communicate with anyone in a remote office, and that's backed up (as described in the article) by cutting the network links to & from the Mountain View corporate office. Now, there may certainly be larger systemic effects over long time periods caused by eg. the death of the company CEO, the loss of new product development, or the complete destruction of the company's market (in, for example, a global thermonuclear apocalypse that sets us back to the stone age). But the tests absolutely do simulate the simultaneous death of 15,000+ employees, destruction of the Mountain View headquarters, and loss of all datacenters, data, and power in the Bay Area, and verify that remote teams can continue the day-to-day operations in such an event.
- Sven7 11y ago
  What happens if Google goes down for a while? Is it really that bad? Yes, Gmail and Drive may cause all kinds of chaos but I think I can survive quite a while without the search engine and YouTube.
  - pc86 11y ago
    In the grand scheme of things, not a lot of businesses use Apps for their email, but I'd hazard a guess that you're still looking at hundreds of millions of dollars in losses if that were to suddenly disappear.
    - Mtinie 11y ago
      It's the cascading service failures and related economic costs that would be the most damaging (and hardest to really quantify until a scenario really happens).
    - sneak 11y ago
      As with most people who reason about Google, you are off by at least two orders of magnitude in your estimate.
      - gaadd33 11y ago
        Wow, I had no idea that Google Apps were about 15% of Google's revenue. That is a way bigger than I would have guessed. (Hundreds of millions x 2 orders of magnitude is >= 10B, total revenue in 2014, 66B)
      - pc86 11y ago
        I'm not referring to Google revenue, I'm referring to productivity/revenue for the businesses using Apps day to day.
  - rryan 11y ago
    The data could be lost. Part of DiRT is ensuring that the result of losing N data centers is not "oops we lost 20% of GMail users data permanently". On the scale of devastation this article is describing, none of Google's servicing being obliterated really matters but I'm personally pretty glad Google is prepared for multiple, massive scale disasters to happen at once.
yokohama11 11y ago
Most large companies do/have done exactly that. Arguably 9/11 was the big catalyst for DR planning. There were Wall St firms that literally had to get the military to escort them into Lower Manhattan in the days after to rescue servers, tapes, etc and cobble something together in a satellite office. No one wants a repeat of that, and you'll notice subsequent disasters like Sandy have barely been a blip for most as a result of policy changes. At this point basically any major company will/should have procedures in place for "perfect" DR, the ability for everyone to work from home indefinitely (and alternate offices/infrastructure waiting for the few people who need something special, like the special needs of a trading desk), as well as obviously having back up data centers with full replication.

kraig 11y ago

tsunami monkey?