4 ms·
Region outages happen, which is why the guidance is to build in multiple regions, but even Amazon sometimes doesn't take their own advice. Sometimes the reason
by andrewguenther 5y ago
Region outages happen, which is why the guidance is to build in multiple regions, but even Amazon sometimes doesn't take their own advice. Sometimes the reason is good, sometimes it isn't.
- sneak 5y agoI thought the guidance was to be multi-AZ, as an AZ is the failure boundary?
- sitharus 5y agoAZs are physical boundaries, but the networking and software is interconnected. Regions are (mostly) isolated, though global services like IAM and CloudFront often have their main control plane in us-east-1
- ABeeSea 5y agoMulti-AZ deployment and multi-region failover is considered best practice.
- dharmab 5y agoAZs are physically located near each other, usually within a small enough radius that they could be all impacted by the same natural disaster. In some cloud providers and regions, AZs are simply different parts of the same building (IIRC one of the Japan regions of Azure was essentially this, but don't quote me). And evidently, the share some infrastructure. At a previous job where we needed to always be up, our disaster recovery plan assumed that the us-east-1 site had been hit by a meteor (not literally, but that's how we explained it to each other to put ourselves in the mindset.)
- treis 5y agoSoftware bugs recognize no boundary
- picardo 5y agoEveryone wants to be multi-AZ, multi-cell, but it's a multi-year project, especially for services that have been around for a while. My last team had been working on it for a couple of years when I left.
- brianpan 5y agoYou can always do better because failures can always be bigger/wider. multi-instance < multi-AZ < multi-region < multiple vendors
- chrisseaton 5y agoAZs are in one physical location. If that physical location has a problem, all AZs within it will go down.
- notatoad 5y ago>An Availability Zone (AZ) is one or more discrete data centers with redundant power, networking, and connectivity in an AWS Region... AZs are physically separated by a meaningful distance, many kilometers, from any other AZ, although all are within 100 km (60 miles) of each other. https://aws.amazon.com/about-aws/global-infrastructure/regions_az/ https://aws.amazon.com/about-aws/global-infrastructure/regio...
- notatoad 5y agoI think it'd be correct to call the AZ a failure boundary, not the failure boundary. the is hardly the first time a failure has exceeded an AZ.
- hparadiz 5y agoOur EC2 instances in us-east-1 didn't actually go down. But IT engineering was completely disrupted because our SSH login mechanism relies on the API to show you the list of instances for you to select which one to start an IAM handshake with. Our support phone line was also down. Even SQS kept chugging along just fine. I'm actually glad we're not on Lambda because of this.
- motoboi 5y agoI love GCP so much because of their ssh login infrastructure. It’s just beautiful. The cli is gorgeous, the web gui is terrible.