8 ms·

We unplugged a data center to test our disaster readiness

259 points by ianrahman 4y ago

aaaaaaaaata 4y ago
zoover2020 4y ago
Take that, ChaosMonkey! This is King Kong
- qw3rty01 4y ago
  Chaos Gorilla has already existed for over a decade
aftbit 4y ago
How big is metaserver these days? I might run three deployments at each data center: the primary, and secondaries for two other regions. Replicate between them at the block device level, bypassing the mysql replication situation entirely (except for on-disk consistency requirements of course). Of course this comes with a 3x increase in service infrastructure costs because of the two backups in each data center that are idle waiting for load.
- Johnny555 4y ago
  And much higher write latency, assuming write consistency is important to you.
- nickm12 4y ago
  Ah metaserver! I wonder how much of my code is still there (and if it is still Py2). I read the Atlas blog post, which implied that metaserver was rearchitected, but still 3M SLOC of Python. https://dropbox.tech/infrastructure/atlas--our-journey-from-a-python-monolith-to-a-managed-platform https://dropbox.tech/infrastructure/atlas--our-journey-from-...
aftbit 4y ago
>Given San Jose's proximity to the the San Andreas Fault, it was critical we ensured an earthquake wouldn't take Dropbox offline. >Given this context, we structured our RTO—and more broadly, our disaster readiness plans—around imminent failures where our primary region is still up, but may not be for long. IMO if the big one hits San Andreas, the SJC facilities will likely go down with ~0 warning. Certainly not enough time to drain user traffic. It's interesting to note that Dropbox realistically can probably tolerate a loss of a few second to minutes of user data in a major earthquake, but cannot tolerate the same losses to perform realistic tests (just yank the cable, no warning). If the earthquake hits at 3am in SF, it'll likely take both the metro and a significant number of the DR team out of the picture for at least a period of time. Surviving that kind of blow in the short term with 0 downtime is a very hard goal.
- paxys 4y ago
  A random (large) earthquake along the San Andreas fault is not the same as all of western USA ripping apart. A much more likely scenario is that the power grid goes down and the data center stays reasonably intact for a while on emergency power.
  - deleted 4y ago
    [deleted]
rkagerer 4y ago
This read to me as much like a history of technical debt as an article about current efforts.
bob1029 4y ago
> This complex ownership model made it hard to move just a subset of user data to another region. Welcome to basically any large-scale enterprise. I have grown to learn that the active-passive strategy is the best option if the business can tolerate the necessary human delays. You get to side-step so much complicated bullshit with this path. Trying to force active-active instant failover magic is sometimes not possible, profitable or even desirable. I can come up with a few scenarios where I would absolutely insist that a human go and check a few control points before letting another exact copy of the same system start up on its own, even if it would be possible in theory to have automatic failover work reliably 99.999% of the time.
- dboreham 4y ago
  My fear with any sort of passive standby approach is that when the disaster comes, that standby won't work, or the mechanism used to fail over to it won't work. I prefer schemes where the "failover" is happening all the time hence I can be confident it works.
  - orev 4y ago
    Having a passive strategy doesn’t mean you don’t test it, and you can even perform actual failovers once on a while to validate everything. Active-active is also valid, but the point is that it comes with a huge amount of increased complexity. At some point you need to make a value calculation to decide if you want to focus on that, or on building the product.
  - mike_d 4y ago
    A solid active/standby design should regularly flop. Every 2 weeks seems to be a sweet spot. This also balances wear across consumable hardware like disks. If your failover is happening "all the time" you basically just have a single system with failures.
    - yencabulator 4y ago
      Except for this: > However, because of that choice, replication between regions is asynchronous—meaning the remote replicas are always some number of transactions behind the primary region. Switching the active is a data loss event. Or, you can choose to drain the load and wait for the replication to catch up, but that's downtime and not a test of the real failure mechanism.
outside1234 4y ago
DiRT
dmitrygr 4y ago
Google does this yearly, with different scenarios - it is called DiRT and you can read a little here: https://cloud.google.com/blog/products/management-tools/shrinking-the-time-to-mitigate-production-incidents https://cloud.google.com/blog/products/management-tools/shri...
ckwalsh 4y ago
Similar to Facebook's Storm initiative: https://www.forbes.com/sites/roberthof/2016/09/11/interview-how-facebooks-project-storm-heads-off-data-center-disasters/ https://www.forbes.com/sites/roberthof/2016/09/11/interview-... These exercises happen several times a year.
- Shish2k 4y ago
  I used to be heavily involved in those, it was a process where we’d take a week to prepare for it, do weeks of post-mortems, and print a run of t-shirts for everyone involved to celebrate pulling one off successfully. These days the team running them announces that it’s happening in an opt-in announcement group at 8am, pulls the plug at 9am, and barely anyone even notices because the automation handles it so gracefully. Mostly I just miss the t-shirts, as the <datacenter>-storm events got the coolest graphical designs...
  - wrigby 4y ago
    Oh hey there! Can confirm that Storms got very very smooth over the course of a few years - it was really incredible to see it go from something that we planned on our roadmap to getting into work and realizing an entire datacenter was disconnected from the backbone while everything ran mostly smooth.
beckman466 4y ago
gotta prepare for the climate crisis, as Douglas Rushkoff discovered, rich capitalists "are plotting to leave us behind" https://onezero.medium.com/survival-of-the-richest-9ef6cddd0cc1 https://onezero.medium.com/survival-of-the-richest-9ef6cddd0... unpaywalled: https://archive.ph/AABsP https://archive.ph/AABsP
deathanatos 4y ago
> Not Found > The requested URL /infrastructure/disaster-readiness-test-failover-blackhole-sjc was not found on this server. > Additionally, a 404 Not Found error was encountered while trying to use an ErrorDocument to handle the request. I see which data center was hosting the article, then. Internet Archive to the rescue: https://web.archive.org/web/20220426191128/https://dropbox.tech/infrastructure/disaster-readiness-test-failover-blackhole-sjc https://web.archive.org/web/20220426191128/https://dropbox.t...
eptcyka 4y ago
Given that the blog doesn't load for me, I guess the datacenter remains unplugged?
- deleted 4y ago
  [deleted]
lloydatkinson 4y ago
I admire the forward thinking and application of common sense here. There's literally no better way of testing the system than this. It seems that a lot of big tech companies would never have the balls to do this themselves.
- mike_d 4y ago
  Netflix has Chaos Gorilla, Facebook has Storms(?), Google has DiRT. Everyone does this type of testing.
  - lloydatkinson 4y ago
    That is different to literally turning a data center off
    - mike_d 4y ago
      I am curious why you think that isn't the case? I've done it multiple times - simulating mains power failures, total power failure, split brains, and fiber cuts to the building.
exdsq 4y ago
I organised one of these as a junior dev at my first job! Glad to see I was ahead of the curve. cough.
leros 4y ago
I forget the name, but I interviewed at a medical technology company a while back. They said they failed over a data center every single day and left it offline for a while. They needed to make sure their redundancy was super reliable so they failed over daily to ensure they engineered everything super robustly.
- exdsq 4y ago
  I know this can be a good idea because it helps you catch stuff, but man how could you convince patients that you're a reliable company if you did lose something by deliberately turning stuff off and on again. I wonder if you could argue it's negligence. Every day seems like you're just inviting a mistake at some point!
  - SteveNuts 4y ago
    You could say the same thing about never testing your DR/backup capabilities. That's way more negligent.
  - sverhagen 4y ago
    Doesn't routine make you better at things? (And since everybody is now going to share war stories... When I worked at Con-way, now XPO Logistics, they had the pop-tart incident, where a kitchen mishap triggered fire regulations that mandated the data center be taken down, and since they had absolutely *zero* routine in restarting their data center, folks got to debug the backup plans in real-time, until two weeks later all service had been restored. Kind of. Me suggesting we should test this more often was only perceived as a joke in bad taste. I thought it was a swell idea, since our Sundays were scheduled downtime anyway. Yep, the whole day. Every one or two weeks. Suffice to say, it wasn't a very long stint.)
    - edmundsauto 4y ago
      People improve with routine when the environment is static, or within certain ranges. Routine degrades performance when previous assumptions shift (or are changed by dynamic agents).
323 4y ago
They seem to say that there is exactly one optical fiber going to exactly one device which is all of the connection of each of their datacenter. Is that correct? That seems very single-point-of-failureish to me.
- Bedon292 4y ago
  I think its just a simplification. For the purposes of the article, it doesn't matter if its 1 or 100. They completely unplugged the network as a whole. Later in the article they do say 'began to reconnect the network fiber', which implies multiple connections to me. They also took reference photos and ordered backup hardware. So I definitely don't think it was a single connection.
- selfhoster69 4y ago
  I wonder where that cable comes from and who maintains it.
_jal 4y ago
We do this yearly. The first time we did it was painful. It took all day, probably around 40 people involved total. We found all sorts of problems, and we actually only failed over the production site, nothing else. We'll be doing it for I think the fifth time in a few months. Mostly automated now, last year was pretty smooth, aside from a couple bad assumptions that crept back in to some code. It is like anything else, repetition makes perfect.
Knight69 4y ago
[dead]

259 points by ianrahman 4y ago

aaaaaaaaata 4y ago

zoover2020 4y ago

Take that, ChaosMonkey! This is King Kong

qw3rty01 4y ago
Chaos Gorilla has already existed for over a decade

aftbit 4y ago

How big is metaserver these days? I might run three deployments at each data center: the primary, and secondaries for two other regions. Replicate between them at the block device level, bypassing the mysql replication situation entirely (except for on-disk consistency requirements of course). Of course this comes with a 3x increase in service infrastructure costs because of the two backups in each data center that are idle waiting for load.

Johnny555 4y ago
And much higher write latency, assuming write consistency is important to you.
nickm12 4y ago
Ah metaserver! I wonder how much of my code is still there (and if it is still Py2). I read the Atlas blog post, which implied that metaserver was rearchitected, but still 3M SLOC of Python. https://dropbox.tech/infrastructure/atlas--our-journey-from-a-python-monolith-to-a-managed-platform https://dropbox.tech/infrastructure/atlas--our-journey-from-...

aftbit 4y ago

>Given San Jose's proximity to the the San Andreas Fault, it was critical we ensured an earthquake wouldn't take Dropbox offline. >Given this context, we structured our RTO—and more broadly, our disaster readiness plans—around imminent failures where our primary region is still up, but may not be for long. IMO if the big one hits San Andreas, the SJC facilities will likely go down with ~0 warning. Certainly not enough time to drain user traffic. It's interesting to note that Dropbox realistically can probably tolerate a loss of a few second to minutes of user data in a major earthquake, but cannot tolerate the same losses to perform realistic tests (just yank the cable, no warning). If the earthquake hits at 3am in SF, it'll likely take both the metro and a significant number of the DR team out of the picture for at least a period of time. Surviving that kind of blow in the short term with 0 downtime is a very hard goal.

paxys 4y ago
A random (large) earthquake along the San Andreas fault is not the same as all of western USA ripping apart. A much more likely scenario is that the power grid goes down and the data center stays reasonably intact for a while on emergency power.
- deleted 4y ago
  [deleted]

rkagerer 4y ago

This read to me as much like a history of technical debt as an article about current efforts.

bob1029 4y ago

> This complex ownership model made it hard to move just a subset of user data to another region. Welcome to basically any large-scale enterprise. I have grown to learn that the active-passive strategy is the best option if the business can tolerate the necessary human delays. You get to side-step so much complicated bullshit with this path. Trying to force active-active instant failover magic is sometimes not possible, profitable or even desirable. I can come up with a few scenarios where I would absolutely insist that a human go and check a few control points before letting another exact copy of the same system start up on its own, even if it would be possible in theory to have automatic failover work reliably 99.999% of the time.

dboreham 4y ago
My fear with any sort of passive standby approach is that when the disaster comes, that standby won't work, or the mechanism used to fail over to it won't work. I prefer schemes where the "failover" is happening all the time hence I can be confident it works.
- orev 4y ago
  Having a passive strategy doesn’t mean you don’t test it, and you can even perform actual failovers once on a while to validate everything. Active-active is also valid, but the point is that it comes with a huge amount of increased complexity. At some point you need to make a value calculation to decide if you want to focus on that, or on building the product.
- mike_d 4y ago
  A solid active/standby design should regularly flop. Every 2 weeks seems to be a sweet spot. This also balances wear across consumable hardware like disks. If your failover is happening "all the time" you basically just have a single system with failures.
  - yencabulator 4y ago
    Except for this: > However, because of that choice, replication between regions is asynchronous—meaning the remote replicas are always some number of transactions behind the primary region. Switching the active is a data loss event. Or, you can choose to drain the load and wait for the replication to catch up, but that's downtime and not a test of the real failure mechanism.

outside1234 4y ago

DiRT

dmitrygr 4y ago

Google does this yearly, with different scenarios - it is called DiRT and you can read a little here: https://cloud.google.com/blog/products/management-tools/shrinking-the-time-to-mitigate-production-incidents https://cloud.google.com/blog/products/management-tools/shri...

ckwalsh 4y ago

Similar to Facebook's Storm initiative: https://www.forbes.com/sites/roberthof/2016/09/11/interview-how-facebooks-project-storm-heads-off-data-center-disasters/ https://www.forbes.com/sites/roberthof/2016/09/11/interview-... These exercises happen several times a year.

Shish2k 4y ago
I used to be heavily involved in those, it was a process where we’d take a week to prepare for it, do weeks of post-mortems, and print a run of t-shirts for everyone involved to celebrate pulling one off successfully. These days the team running them announces that it’s happening in an opt-in announcement group at 8am, pulls the plug at 9am, and barely anyone even notices because the automation handles it so gracefully. Mostly I just miss the t-shirts, as the <datacenter>-storm events got the coolest graphical designs...
- wrigby 4y ago
  Oh hey there! Can confirm that Storms got very very smooth over the course of a few years - it was really incredible to see it go from something that we planned on our roadmap to getting into work and realizing an entire datacenter was disconnected from the backbone while everything ran mostly smooth.

beckman466 4y ago

gotta prepare for the climate crisis, as Douglas Rushkoff discovered, rich capitalists "are plotting to leave us behind" https://onezero.medium.com/survival-of-the-richest-9ef6cddd0cc1 https://onezero.medium.com/survival-of-the-richest-9ef6cddd0... unpaywalled: https://archive.ph/AABsP https://archive.ph/AABsP

deathanatos 4y ago

> Not Found > The requested URL /infrastructure/disaster-readiness-test-failover-blackhole-sjc was not found on this server. > Additionally, a 404 Not Found error was encountered while trying to use an ErrorDocument to handle the request. I see which data center was hosting the article, then. Internet Archive to the rescue: https://web.archive.org/web/20220426191128/https://dropbox.tech/infrastructure/disaster-readiness-test-failover-blackhole-sjc https://web.archive.org/web/20220426191128/https://dropbox.t...

eptcyka 4y ago

Given that the blog doesn't load for me, I guess the datacenter remains unplugged?

deleted 4y ago
[deleted]

lloydatkinson 4y ago

I admire the forward thinking and application of common sense here. There's literally no better way of testing the system than this. It seems that a lot of big tech companies would never have the balls to do this themselves.

mike_d 4y ago
Netflix has Chaos Gorilla, Facebook has Storms(?), Google has DiRT. Everyone does this type of testing.
- lloydatkinson 4y ago
  That is different to literally turning a data center off
  - mike_d 4y ago
    I am curious why you think that isn't the case? I've done it multiple times - simulating mains power failures, total power failure, split brains, and fiber cuts to the building.

exdsq 4y ago

I organised one of these as a junior dev at my first job! Glad to see I was ahead of the curve. cough.

leros 4y ago

I forget the name, but I interviewed at a medical technology company a while back. They said they failed over a data center every single day and left it offline for a while. They needed to make sure their redundancy was super reliable so they failed over daily to ensure they engineered everything super robustly.

exdsq 4y ago
I know this can be a good idea because it helps you catch stuff, but man how could you convince patients that you're a reliable company if you did lose something by deliberately turning stuff off and on again. I wonder if you could argue it's negligence. Every day seems like you're just inviting a mistake at some point!
- SteveNuts 4y ago
  You could say the same thing about never testing your DR/backup capabilities. That's way more negligent.
- sverhagen 4y ago
  Doesn't routine make you better at things? (And since everybody is now going to share war stories... When I worked at Con-way, now XPO Logistics, they had the pop-tart incident, where a kitchen mishap triggered fire regulations that mandated the data center be taken down, and since they had absolutely *zero* routine in restarting their data center, folks got to debug the backup plans in real-time, until two weeks later all service had been restored. Kind of. Me suggesting we should test this more often was only perceived as a joke in bad taste. I thought it was a swell idea, since our Sundays were scheduled downtime anyway. Yep, the whole day. Every one or two weeks. Suffice to say, it wasn't a very long stint.)
  - edmundsauto 4y ago
    People improve with routine when the environment is static, or within certain ranges. Routine degrades performance when previous assumptions shift (or are changed by dynamic agents).

323 4y ago

They seem to say that there is exactly one optical fiber going to exactly one device which is all of the connection of each of their datacenter. Is that correct? That seems very single-point-of-failureish to me.

Bedon292 4y ago
I think its just a simplification. For the purposes of the article, it doesn't matter if its 1 or 100. They completely unplugged the network as a whole. Later in the article they do say 'began to reconnect the network fiber', which implies multiple connections to me. They also took reference photos and ordered backup hardware. So I definitely don't think it was a single connection.
selfhoster69 4y ago
I wonder where that cable comes from and who maintains it.

_jal 4y ago

We do this yearly. The first time we did it was painful. It took all day, probably around 40 people involved total. We found all sorts of problems, and we actually only failed over the production site, nothing else. We'll be doing it for I think the fifth time in a few months. Mostly automated now, last year was pretty smooth, aside from a couple bad assumptions that crept back in to some code. It is like anything else, repetition makes perfect.

Knight69 4y ago

[dead]