5 ms·
No, FB datacenters are geographically diverse. They do run quarterly 'storms' where a datacenter is shut down to test failover and resiliency. I have no idea i
by erobbins 8y ago
No, FB datacenters are geographically diverse.
They do run quarterly 'storms' where a datacenter is shut down to test failover and resiliency. I have no idea if today is one of those days, since I left last year.
- johannes1234321 8y agoTheoretically a real shutdown might go in a different way than previous tests or simulations. For instance in a test you might cut the connection completely, while in the real case only some power circuits go down or whatever. For instance GitHub's relatively recent shutdown was due to a fail-over heartbeat not going as expected.
- chipperyman573 8y agoInteresting. Out of curiosity, how hard is it to turn the datacenter "back on" in case they discover there's a problem with the failover?
- londons_explore 8y agoTypically that could be done with one undrain command and take about 30 seconds.
- AceyMan 8y agoHoly Batcave, I'm a baby SRE compared to that <bows solemnly>. Mind. Blown¹. /Acey ¹—I don't use FB [true] but I hear it's huge :-P
- londons_explore 8y agoit's only so quick because stuff isn't actually turned off with disks wiped. The machines are still running, with applications loaded, just with no traffic directed towards them.
- bertil 8y agoIt is an internal software problem.
- linsomniac 8y agoTest failures are all well and good, but don't always match reality. In this case, the design of the power infrastructure was solid, and their plans include running monthly generator testing and quarterly "disconnect from the grid" testing. But apparently something about this failure of both of the incoming power lines caused failures in multiple UPSes. Still waiting on the after-action review.