> It's actually pretty rare these days because all major clouds use zone-redundancy and hence their core services are robust to the loss of any single building.

> It's actually pretty rare these days because all major clouds use zone-redundancy and hence their core services are robust to the loss of any single building. Even during the recent Iberian power outages the local cloud sites mostly (entirely?) stayed up. Here's one from 2023 https://www.datacenterdynamics.com/en/news/water-leak-at-paris-global-switch-data-center-causes-fire-leads-to-outages-at-google/ https://www.datacenterdynamics.com/en/news/water-leak-at-par... I've been working with GCP hosted (cross region) services for a few years now, and the outages I remember are that one and their recent global partial outage. I've seen some things that seem to indicate major fiber cuts (or other routing woes) centered around certain locations too, but I don't remember the details. > The only way to guarantee that failover actually works is to pre-reserve 100% of the required VM capacity. This requires about 2x the spend at all times, whether that capacity is used or not. Incidentally the lesson from the global partial outage is if I wanted to survive those, I should always run all regions at 2x indicated traffic, because in the event of a similar outage in the future, competing services are likely to fail and we won't be able to scale up; instances were available, but the VM images were not, so scaling wasn't actually possible. If you can't get instances when your main region is down, it is what it is... but I suspect there's enough capacity unless everyone else has picked to be in the same two regions as you for hot and standby. > This is decidedly non-trivial unless you have bidirectional replication set up or a globally-available database like CosmosDB. > Inevitably the original site will come up and start accepting writes while the DR site is still up, and now you've got writes or transactions going to two places. Depends what's going on at the original site. If the servers are flooded, chances are they're not coming up again. Assessing the likelyhood of automatic return to service is part of the manual process; it's also part of what makes automatic failover hard. Bidirectional replication might help, or might just fail when the connection comes back. I'm a big fan of having traditional database servers start in read only and needing manual intervention to accept writes, so the human in the loop can be the Mutex... but yeah, it's tricky. And using cloud solutions for global consistency are expensive. If you honestly assess the costs and benefits and say woah, that's too expensive, that's fine with me. But you should probably have a look every once in a while. And if your deployment is big enough anyway, the costs start getting relatively lower, because maybe you want some servers here and there to reduce latency and then you need to figure out how to have the data in multiple places anyway, etc....