4 ms·

Thanks for the great explanation. I agree that a Chaos Monkey test something a lot more easier to recover from than a whole datacenter getting down. I remember

Thanks for the great explanation. I agree that a Chaos Monkey test something a lot more easier to recover from than a whole datacenter getting down. I remember having read somewhere about some company (Facebook/Google/Amazon/Twitter/Dropbox or something at the same scale) that regularly simulates a whole datacenter failure, which made believe it is possible to automatically recover from this. Are you saying that even the companies I mentioned have the same issues as OVH when they recover from a complete power failure?

cesarb 9y ago
> I remember having read somewhere about some company (Facebook/Google/Amazon/Twitter/Dropbox or something at the same scale) that regularly simulates a whole datacenter failure It was Google: http://queue.acm.org/detail.cfm?id=2371516 http://queue.acm.org/detail.cfm?id=2371516 ("Weathering the Unexpected" [2012])
- ngrilly 9y ago
  Thanks! This is the article I had in mind.
mioelnir 9y ago
Simulated DC failure is more often then not just traffic flow engineering. It is more about testing the DC that takes over the traffic than it is about testing service restart in the inactive DC. There is little to test about the introduction of a hard fault, but the service resumption in the other DC is full of data to analyze. Also, in such a setup, getting the fault location running again is not on a hard clock, since it is about restoring redundancy instead of the service.