4 ms·
I thought the whole point of AWS that it was a distributed system with no one point of failure? In that case, how did it have an outage?
by LAC-Tech 5y ago
I thought the whole point of AWS that it was a distributed system with no one point of failure? In that case, how did it have an outage?
- andrewguenther 5y agoRegion outages happen, which is why the guidance is to build in multiple regions, but even Amazon sometimes doesn't take their own advice. Sometimes the reason is good, sometimes it isn't.
- sneak 5y agoI thought the guidance was to be multi-AZ, as an AZ is the failure boundary?
- sitharus 5y agoAZs are physical boundaries, but the networking and software is interconnected. Regions are (mostly) isolated, though global services like IAM and CloudFront often have their main control plane in us-east-1
- ABeeSea 5y agoMulti-AZ deployment and multi-region failover is considered best practice.
- dharmab 5y agoAZs are physically located near each other, usually within a small enough radius that they could be all impacted by the same natural disaster. In some cloud providers and regions, AZs are simply different parts of the same building (IIRC one of the Japan regions of Azure was essentially this, but don't quote me). And evidently, the share some infrastructure. At a previous job where we needed to always be up, our disaster recovery plan assumed that the us-east-1 site had been hit by a meteor (not literally, but that's how we explained it to each other to put ourselves in the mindset.)
- treis 5y agoSoftware bugs recognize no boundary
- picardo 5y agoEveryone wants to be multi-AZ, multi-cell, but it's a multi-year project, especially for services that have been around for a while. My last team had been working on it for a couple of years when I left.
- brianpan 5y agoYou can always do better because failures can always be bigger/wider. multi-instance < multi-AZ < multi-region < multiple vendors
- chrisseaton 5y agoAZs are in one physical location. If that physical location has a problem, all AZs within it will go down.
- notatoad 5y ago>An Availability Zone (AZ) is one or more discrete data centers with redundant power, networking, and connectivity in an AWS Region... AZs are physically separated by a meaningful distance, many kilometers, from any other AZ, although all are within 100 km (60 miles) of each other. https://aws.amazon.com/about-aws/global-infrastructure/regions_az/ https://aws.amazon.com/about-aws/global-infrastructure/regio...
- notatoad 5y agoI think it'd be correct to call the AZ a failure boundary, not the failure boundary. the is hardly the first time a failure has exceeded an AZ.
- hparadiz 5y agoOur EC2 instances in us-east-1 didn't actually go down. But IT engineering was completely disrupted because our SSH login mechanism relies on the API to show you the list of instances for you to select which one to start an IAM handshake with. Our support phone line was also down. Even SQS kept chugging along just fine. I'm actually glad we're not on Lambda because of this.
- motoboi 5y agoI love GCP so much because of their ssh login infrastructure. It’s just beautiful. The cli is gorgeous, the web gui is terrible.
- rustnote 5y agoI guess on a load balancer or DNS level? A request has to hit a single domain name / IP before it's load balanced to the distributed system right?
- dharmab 5y agoMultiple devices in very different parts of the world can respond to the same IP address. A great example are DNS servers like 1.1.1.1 and 8.8.8.8- you get routed to the nearest site that can handle your request. A common example of how this is implemented is via anycast: https://www.cloudflare.com/learning/cdn/glossary/anycast-network/ https://www.cloudflare.com/learning/cdn/glossary/anycast-net...
- riffic 5y agodo they not teach people what a failure domain is anymore? https://en.wikipedia.org/wiki/Failure_domain https://en.wikipedia.org/wiki/Failure_domain Clouds aren't magic. They require a certain amount of operational confidence in order to understand that, yes, an entire region can fall out from under you at any time and it's your responsibility to detect and deploy into an unaffected region if possible. edit: Generally, one entire region will not fail. However, core services like STS rely on us-east-1 so it's particularly susceptible to disruption.
- RyJones 5y agoIn my experience, once an item is done done done, it’s out of your hands.
- sithadmin 5y agoHonestly, no. Not frequently enough. There are also way too many successful, public cloud-native businesses running without any semblance of a Business Continuity or DR Plan among any of their teams. I wish I could name and shame some of the more egregious cases I have seen.
- deathanatos 5y ago> do they not teach people what a failure domain is anymore? Empirically? No. But seriously, instead of making every dev team consuming an AWS product hire the extra engineers required to build a system that spans multiple failure domains — which lets be honest, companies won't — why hasn't Amazon just hired the engineers required to do it for me? > Generally, one entire region will not fail. Laughs in global failures.
- Spooky23 5y agoPeople get slagged on for not having inter-region redundancy. But unless your business model can accommodate that, you’re introducing another failure domain and alot of money for duplicative infrastructure, network fees, etc. For many use cases, it’s acceptable to shrug and blame AWS for a failure. It’s harder when your high availability solution fails independently, which they almost always do more than US-East-1
- sp332 5y agohttps://status.aws.amazon.com/ https://status.aws.amazon.com/ says that multiple network devices failed.
- foobiekr 5y agoThey very carefully did not actually say that. The weasel wording used could be a routing configuration error, among other things.
- deleted 5y ago[deleted]
- staticassertion 5y agoA lot of AWS is very old and it's very hard to go back and make everything nice and reliable without breaking things when you have a shitload of customers 24/7 using your services. So yes, in theory much of AWS's services are probably very reliable and distributed across AZs and regions, but in practice there's likely a whole bunch of debt where one thing gets fucked up and it cascades.
- 6510 5y agoA profitable "accident".
- stormbeard 5y agoAs an AWS employee, you’re under too much artificial deadline pressure to address technical debt like this… let alone engineer a service properly in the first place.
- datavirtue 5y agoAs a corporate employee, you’re under too much artificial deadline pressure to address technical debt like this… let alone engineer a service properly in the first place.