10 ms·

Hmm, in the spirit of thinking of worst-case scenarios, most of the engineering talent for Amazon and Microsoft is in Seattle and Redmond, respectively. There's

Hmm, in the spirit of thinking of worst-case scenarios, most of the engineering talent for Amazon and Microsoft is in Seattle and Redmond, respectively. There's even a sizable percentage of Google cloud platform in Fremont and Kirkland. Even if most of the employees live, it's likely basic infrastructure will be a disaster for quite a while. Maybe Google would be able to offload ops work to Mountain View, but it seems likely Amazon.com, AWS and Azure would go down, possibly for months. I wonder what the effect would be on the global economy.

serve_yay 11y ago
That does seem likely, though as far as I know, AWS's biggest zone is located in Northern Virginia. And, I'm sure the AWS people have planned as much as they can for this eventuality. (Um, I hope.)
- stephengillie 11y ago
  AWS West in Oregon are somewhat further inland from I-5. It would probably be impacted, but not to the same degree as the datacenters in other cities like Tukwila.
  - trhway 11y ago
    i think the datacenters will get soon to the status of national strategic infrastructure objects similar to bridges, power plants, etc... Given the distributed nature of the services i'd suppose the main threat though isn't from physical effect like earthquake, instead i think the main threat is introduction of something like StuxNet into say AWS (or highjacking the backdoor access that services like NSA already have there)
  - sneak 11y ago
    The loss of datacenters is irrelevant; the loss of people to keep the services running is the issue.
- Locke1689 11y ago
  I doubt the datacenter's locations matter. The services at the bottom of the stack need constant ops to stay alive. With most of the engineers out, I expect barely a few days to go by before one of the essential services needs servicing.
  - torinmr 11y ago
    For companies that size, the really important services probably have a distributed ops team, with a team in Europe/Asia alternating 12 hour oncall shifts with a team in North America. In an emergency, the non-North American team could take 24 hour shifts to keep the infrastructure alive. Though of course if headquarters was really wiped off the map, there'd be huge problems with communication, tribal knowledge that only existed in the heads of senior engineers in Seattle, and so forth.
  - jsolson 11y ago
    When supporting a service on the scale of say, EC2 and the rest of AWS, one would typically expect follow-the-sun support.
jsprogrammer 11y ago
I believe you are giving reasons why our computational infrastructure needs to be further distributed. We should not be dependent on a small number of companies who wish to locate their resources in continental subduction zones.
alxv 11y ago
Google runs a company-wide disaster recovery testing event every year. Earthquakes are among the numerous scenarios simulated. https://queue.acm.org/detail.cfm?id=2371516 https://queue.acm.org/detail.cfm?id=2371516
- sneak 11y ago
  Great link. Reminds me of this: http://craphound.com/overclocked/Cory_Doctorow_-_Overclocked_-_When_Sysadmins_Ruled_the_Earth.html http://craphound.com/overclocked/Cory_Doctorow_-_Overclocked...
- idlewords 11y ago
  The problem is that you can't really simulate the systemic effects of a really major quake (like a scenario where Mountain View is out of power for weeks).
  - falcolas 11y ago
    Sure you can. Just look for the keys to the main breakers to those buildings. Of course, paying for that amount of vacation for 15 days is probably not entirely feasible for any company.
    - idlewords 11y ago
      By 'systemic' I mean not confined to one company.
  - nostrademons 11y ago
    The DiRT drills include the simultaneous loss of whole offices, including the complete obliteration of the Bay Area. Participants are instructed to pretend that they've been eaten by zombies and not communicate with anyone in a remote office, and that's backed up (as described in the article) by cutting the network links to & from the Mountain View corporate office. Now, there may certainly be larger systemic effects over long time periods caused by eg. the death of the company CEO, the loss of new product development, or the complete destruction of the company's market (in, for example, a global thermonuclear apocalypse that sets us back to the stone age). But the tests absolutely do simulate the simultaneous death of 15,000+ employees, destruction of the Mountain View headquarters, and loss of all datacenters, data, and power in the Bay Area, and verify that remote teams can continue the day-to-day operations in such an event.
waynecochran 11y ago
Not only would there be significant downtime of major internet services, but lots of people would die as well!
- qq66 11y ago
  Downtime of major internet services could also cause deaths elsewhere, as more industries become reliant on cloud services. I could imagine hospitals losing access to health records, emergency responders losing access to mapping or route planning software, people losing access to Uber and not being able to get to the hospital because the taxi companies have gone out of business. Of course all this stuff is supposed to have clean failover, but not all of it has had its mettle tested.
  - deleted 11y ago
    [deleted]
  - zecho 11y ago
    Man. In times like these—the northwest falling into the ocean or whatever—it's unimaginable to think about the sheer amount of data loss that could occur. I'd probably be really worried about that loss of data or, worse, the loss of services if millions of people suddenly died.
    - myth_buster 11y ago
      Isn't it the case that there is replication across data centers which are geologically spread apart?
      - deegles 11y ago
        Only if you had the foresight to set that up.
  - hdunlop 11y ago
    like Uber, but for once-in-an-epoch natural disasters
hinkley 11y ago
People talk and talk about how The Cloud is about decentralizing things but really at the end of the day it's about centralizing everything with about a dozen vendors. And of course when price sensitivity causes attrition that will turn into 4 or 5 vendors. I'm not sure I see how that really counts as 'more robust'.
PotatoEngineer 11y ago
The Microsoft campus in Redmond would most likely not be knocked over by the tsunami: a rather hilly Seattle is in the way, then there's the large Lake Washington, and then Redmond is another ~40 feet above the lake on the far side, plus another ~2 miles inland. A fair amount of the energy of the tsunami is going to be expended by the time the Microsoft campus gets salted. (Kirkland is west of Redmond, adjacent to Lake Washington, so it's a similar-but-somewhat-less-optimistic story there. Fremont is adjacent to Seattle and slightly inland, so it will be worse there.) Of course, the infrastructure of the greater Seattle area is going to be trashed, but chances are good that the Redmond and Kirkland campuses of Microsoft and Google respectively would be intact. Link to the inundation zone predicted for a magnitude 7.3 earthquake on Seattle: http://wa-dnr.s3.amazonaws.com/Publications/ger_ofr2003-14_tsunami_hazard_elliottbay.pdf http://wa-dnr.s3.amazonaws.com/Publications/ger_ofr2003-14_t... This isn't even "the big one" as the OP describes, much less "the really big one", but it's a rough description of where the flooding will happen. Downtown Seattle (up and left of the word SEATTLE on the map, with streets in a dense diagonal grid) won't be strongly affected, because downtown Seattle is a hill. With an 8.5 or 9.0 earthquake, of course, Seattle will be much worse off, but there's still some hope that Redmond and Kirkland won't get too damaged.
chrismcb 11y ago
There is an entire mountain range between Seattle and the ocean. I am curious how the Sound would affect the tsunami.
kelnos 11y ago
Sure, the AWS us-west datacenters would go down (Oregon and NorCal), but us-east (VA) is still the largest of all AWS regions. While Amazon is headquartered on the west coast, I sincerely doubt a failure of even both us-west regions would take out the others. Ultimately it's up to the user: if they're going to put all their eggs in a us-west basket, they're out of luck if the west coast ends up under water.

serve_yay 11y ago

That does seem likely, though as far as I know, AWS's biggest zone is located in Northern Virginia. And, I'm sure the AWS people have planned as much as they can for this eventuality. (Um, I hope.)

stephengillie 11y ago
AWS West in Oregon are somewhat further inland from I-5. It would probably be impacted, but not to the same degree as the datacenters in other cities like Tukwila.
- trhway 11y ago
  i think the datacenters will get soon to the status of national strategic infrastructure objects similar to bridges, power plants, etc... Given the distributed nature of the services i'd suppose the main threat though isn't from physical effect like earthquake, instead i think the main threat is introduction of something like StuxNet into say AWS (or highjacking the backdoor access that services like NSA already have there)
- sneak 11y ago
  The loss of datacenters is irrelevant; the loss of people to keep the services running is the issue.
Locke1689 11y ago
I doubt the datacenter's locations matter. The services at the bottom of the stack need constant ops to stay alive. With most of the engineers out, I expect barely a few days to go by before one of the essential services needs servicing.
- torinmr 11y ago
  For companies that size, the really important services probably have a distributed ops team, with a team in Europe/Asia alternating 12 hour oncall shifts with a team in North America. In an emergency, the non-North American team could take 24 hour shifts to keep the infrastructure alive. Though of course if headquarters was really wiped off the map, there'd be huge problems with communication, tribal knowledge that only existed in the heads of senior engineers in Seattle, and so forth.
- jsolson 11y ago
  When supporting a service on the scale of say, EC2 and the rest of AWS, one would typically expect follow-the-sun support.

jsprogrammer 11y ago

I believe you are giving reasons why our computational infrastructure needs to be further distributed. We should not be dependent on a small number of companies who wish to locate their resources in continental subduction zones.

alxv 11y ago

Google runs a company-wide disaster recovery testing event every year. Earthquakes are among the numerous scenarios simulated. https://queue.acm.org/detail.cfm?id=2371516 https://queue.acm.org/detail.cfm?id=2371516

sneak 11y ago
Great link. Reminds me of this: http://craphound.com/overclocked/Cory_Doctorow_-_Overclocked_-_When_Sysadmins_Ruled_the_Earth.html http://craphound.com/overclocked/Cory_Doctorow_-_Overclocked...
idlewords 11y ago
The problem is that you can't really simulate the systemic effects of a really major quake (like a scenario where Mountain View is out of power for weeks).
- falcolas 11y ago
  Sure you can. Just look for the keys to the main breakers to those buildings. Of course, paying for that amount of vacation for 15 days is probably not entirely feasible for any company.
  - idlewords 11y ago
    By 'systemic' I mean not confined to one company.
- nostrademons 11y ago
  The DiRT drills include the simultaneous loss of whole offices, including the complete obliteration of the Bay Area. Participants are instructed to pretend that they've been eaten by zombies and not communicate with anyone in a remote office, and that's backed up (as described in the article) by cutting the network links to & from the Mountain View corporate office. Now, there may certainly be larger systemic effects over long time periods caused by eg. the death of the company CEO, the loss of new product development, or the complete destruction of the company's market (in, for example, a global thermonuclear apocalypse that sets us back to the stone age). But the tests absolutely do simulate the simultaneous death of 15,000+ employees, destruction of the Mountain View headquarters, and loss of all datacenters, data, and power in the Bay Area, and verify that remote teams can continue the day-to-day operations in such an event.

waynecochran 11y ago

Not only would there be significant downtime of major internet services, but lots of people would die as well!

qq66 11y ago
Downtime of major internet services could also cause deaths elsewhere, as more industries become reliant on cloud services. I could imagine hospitals losing access to health records, emergency responders losing access to mapping or route planning software, people losing access to Uber and not being able to get to the hospital because the taxi companies have gone out of business. Of course all this stuff is supposed to have clean failover, but not all of it has had its mettle tested.
- deleted 11y ago
  [deleted]
- zecho 11y ago
  Man. In times like these—the northwest falling into the ocean or whatever—it's unimaginable to think about the sheer amount of data loss that could occur. I'd probably be really worried about that loss of data or, worse, the loss of services if millions of people suddenly died.
  - myth_buster 11y ago
    Isn't it the case that there is replication across data centers which are geologically spread apart?
    - deegles 11y ago
      Only if you had the foresight to set that up.
- hdunlop 11y ago
  like Uber, but for once-in-an-epoch natural disasters

hinkley 11y ago

People talk and talk about how The Cloud is about decentralizing things but really at the end of the day it's about centralizing everything with about a dozen vendors. And of course when price sensitivity causes attrition that will turn into 4 or 5 vendors. I'm not sure I see how that really counts as 'more robust'.

PotatoEngineer 11y ago

The Microsoft campus in Redmond would most likely not be knocked over by the tsunami: a rather hilly Seattle is in the way, then there's the large Lake Washington, and then Redmond is another ~40 feet above the lake on the far side, plus another ~2 miles inland. A fair amount of the energy of the tsunami is going to be expended by the time the Microsoft campus gets salted. (Kirkland is west of Redmond, adjacent to Lake Washington, so it's a similar-but-somewhat-less-optimistic story there. Fremont is adjacent to Seattle and slightly inland, so it will be worse there.) Of course, the infrastructure of the greater Seattle area is going to be trashed, but chances are good that the Redmond and Kirkland campuses of Microsoft and Google respectively would be intact. Link to the inundation zone predicted for a magnitude 7.3 earthquake on Seattle: http://wa-dnr.s3.amazonaws.com/Publications/ger_ofr2003-14_tsunami_hazard_elliottbay.pdf http://wa-dnr.s3.amazonaws.com/Publications/ger_ofr2003-14_t... This isn't even "the big one" as the OP describes, much less "the really big one", but it's a rough description of where the flooding will happen. Downtown Seattle (up and left of the word SEATTLE on the map, with streets in a dense diagonal grid) won't be strongly affected, because downtown Seattle is a hill. With an 8.5 or 9.0 earthquake, of course, Seattle will be much worse off, but there's still some hope that Redmond and Kirkland won't get too damaged.

chrismcb 11y ago

There is an entire mountain range between Seattle and the ocean. I am curious how the Sound would affect the tsunami.

kelnos 11y ago

Sure, the AWS us-west datacenters would go down (Oregon and NorCal), but us-east (VA) is still the largest of all AWS regions. While Amazon is headquartered on the west coast, I sincerely doubt a failure of even both us-west regions would take out the others. Ultimately it's up to the user: if they're going to put all their eggs in a us-west basket, they're out of luck if the west coast ends up under water.