Cloud customers have no control on which or how many 'datacenters' are used. That's not something that's even advertised or easily available to customers. The

Cloud customers have no control on which or how many 'datacenters' are used. That's not something that's even advertised or easily available to customers. The logical units are regions and availability zones or the equivalent nomenclature in each cloud. One availability zone is expected to be one or more datacenters. We have thousands of instances in AWS. I do not know - or care - where they are physically located(other than the region name, say, Oregon). I expect at most one availability zone to get impacted if a datacenter goes up in flames (and sometimes, just a portion of one). I mention in another comment that AWS has had issues before and production systems barely got impacted. And recovered with zero intervention - instances with failed health checks get replaced by brand new ones in whatever AZs are still operational. > Data would be eventually consistent (once they find and plug the hard drives in) At the level of abstractions cloud operates, no-one is plugging drives in – someone is, but you can never see it. Most cloud workloads use network attached storage - when you can even see the logical drives (SaaS offerings may not even have that abstraction). We don't know (or care) how many physical hard drives exist, or where they are. Latency requirements probably dictate that they are close to the actual instances, but there's usually data replication going on even across DCs. In addition to that, at least in AWS, if you have saved any volume snapshots at all, they will be in S3. This data will be replicated and underlying systems can even use it to restore lost or corrupted data without you even noticing and sometimes even without a recent snapshot, as storage keeps track of what blocks have been rewritten since the last snapshot. In a particularly bad case you might have to do a restore. In almost a decade and number of volumes in the 6 digits (no clue how many drives that is!) we never had a single volume fail on AWS. Some got into a 'degraded' state and then recovered. We haven't had any failures on GCP either. In the case of GCP, even faulty hypervisors are transparently worked around - we never notice other than some audit logs saying the VM was moved. They even preserve the network connections. AWS requires a stop/start to do the same, but your VM will be up and running in a different hypervisor (sometimes a different datacenter) in a couple of minutes, with all the storage. Mind you, AWS promises eleven nines(!) of durability for S3. When you do have locally attached storage, it's treated as ephemeral and it's gone if the instance restarts. > I have to assume it’s a fault that not even distributed services can paper over. If a single datacenter fails, since it _should_ be at most one AZ(this case seems to be different) that will depend on how the application is architected. Requests in flight will obviously fail, how big of a deal depends on the problem domain. For most web apps, this will cause a retry and that's the end of the story, others will be specifically engineered to deal with receiving multiple messages or dropping messages. For example, if you need at most once delivery guarantees, you need to take extra measures Not all applications can survive an entire region going down. Some can, but that usually raises costs if you are continuously replicating data across regions. If you do that, then you should be able to steer traffic to the surviving regions. You can do that old-school by changing DNS records, or you could have fanciers solutions such as global anycast loadbalancers and have a single IP worldwide that still goes to the closest healthy region.