4 ms·
AWS’s US-East 1 continues to be the Achilles heel of the Internet. And while yes building across multiple regions and AZs is a thing, AWS has had a string of i
by cmiles8 5mo ago
AWS’s US-East 1 continues to be the Achilles heel of the Internet.
And while yes building across multiple regions and AZs is a thing, AWS has had a string of issues where US-East 1 has broader impacts, which makes things far less redundant and resilient than AWS implies.
- keeganpoppen 5mo agoanecdotally (well, more "second-hand-ly i heard that..." it sounds like there were some carry-on effects on us-east-2 as a result of people migrating over from us-east-1, so, yeah... kinda hilarious how the multiple region / AZ thing is just so plainly a façade, but yet we all seem to just collectively believe in it as an article of faith in the Cloud Religion... or whatever...
- qaq 5mo agoIt's no magic given the size of us-east-1 there is no spare capacity to absorb all the workloads
- 8organicbits 5mo agoOne of the SRE tricks is to reserve your capacity so when the cloud runs out of capacity you're still covered. It's expensive, but you don't want to get stuck without a server when the on-demand dries up.
- cherioo 5mo agoIs it really failing more, or we just don’t hear about failure happening elsewhere? Last i heard azure outage it wasn’t even on HN frontpage
- stingraycharles 5mo agoIt really is failing more, and it’s well known amongst industry experts. It’s the oldest, largest, and most utilized region of AWS. I’ve heard people say that the underlying physical infrastructure is older, but I think that’s a bit of speculation, although reasonable. The current outage is attributed to a “thermal event”, which does indeed suggest underlying physical hardware. It’s also the most complex region for AWS themselves, as it’s the “control pad” for many of their global services.
- adriand 5mo agoWhat kind of reputation does ca-central-1 have? I’ve been using it and it seems quietly excellent. Knock on wood.
- jedberg 5mo agoMost of the other regions are fairly stable. Ohio (us-east-2) is a great choice if you're just starting out. Not sure about ca-central-1, but I've never heard anything bad about it.
- dlenski 5mo agoIt wasn't heavily utilized when I worked at AWS, until 2024. If your customers are clusterrd in Toronto and Montreal, it probably makes a lot of sense to use ca-central-1. If you've got a lot of customers in Western Canada, us-west-2 is gonna have better network latency. Other than a couple regions that had problems with their local network infrastructure (sa-east-1 was like that), there's little or nothing to differentiate the regions in terms of physical infrastructure and architecture.
- 999900000999 5mo agoToo many people are using it. In fantasy magic dream land loads are distributed evenly across different cloud providers. A single point of failure doesn't exist. It worked out with my first girlfriend. The twins are fluent in English and Korean. They know when deploying a large scale service to not only depends on AWS. Healthcare in the US is affordable. All types of magical stuff exist here. But no. It's another day. AWS US-East 1 can take town most of the internet.
- afro88 5mo agoCore AWS services use it too. Even if you are hosted in another region, you can still be affected by a US-East 1 outage
- leetrout 5mo agoBingo. This is the one most people don't know about.
- 999900000999 5mo agoThe idea would be to actually load distribute between different cloud providers. But even then , the load balancer needs to run somewhere. Which becomes a new single point of failure. I’m sure someone smarter than me has figured this out.
- jethro_tell 5mo agoyes, they have. It just costs a shit ton of money and is extremely difficult to get the suits to sign off on TWO full 'cloud services' bills. It generally doubles your cost and workload and increases your uptime by a couple hours/year, assuming you don't have bugs that affect one or the other cloud in your deployment stack. It's basically a wash for almost all organizations for twice the cost and effort.
- 999900000999 5mo agoOk... But where does the load balancer actually run. Does load balancer main run on AWS, and load balancer backup on Oracle?
- Eridrus 5mo agoPeople say this, but this this was just a single AZ, and in the last 3 years of running my startup mostly out of use-1, and we've only had one regional outage, and even that was partial, with most instances uneffected. And honestly, everybody else's stuff is in use-1, so at least your failures are correlated with your customers lol.
- linsomniac 5mo ago>And honestly, everybody else's stuff is in use-1 Yeah, but why put your eggs in that basket? I moved all our services from east to west/oregon a decade ago and haven't looked back.
- christina97 5mo agoBut it’s okay to be down when the whole internet is down.
- electroly 5mo agoNot OP, but I do single-region us-east-1 for a few reasons: 1. The severity and frequency of us-east-1 outages are vastly overstated. It's fine. These us-east-1 outages almost never affect us. This one didn't; not even our instances in the affected AZ. Only that recent IAM outage affected us a little bit, and it affected every other region, too, since IAM's control plane is centrally hosted in us-east-1. Everybody's uptime depends on us-east-1. 2. We're physically close to us-east-1 and have Direct Connect. We're 1 millisecond away from us-east-1. It would be silly to connect to us-east-1 and then take a latency hit and pay cross-region data transfer cost on all traffic to hop over to another region. That would only make sense if we were in both regions, and that is not worth the cost given #1. If we only have a single region, it has to be us-east-1. 3. us-east-1 gets new features first. New AWS features are relevant to us with shocking regularity, and we get it as soon as it's announced. 4. OP is right about the safety in numbers. Our service isn't life-or-death; nobody will die if we're down, so it's just a matter of whether they're upset. When there is a us-east-1 outage, it's headline news and I can link the news report to anyone who asks. That genuinely absolves us every time. When we're down, everybody else is down, too.
- 5mo ago
- dlenski 5mo agoThe idea that AWS's services are fully regionalized or isolated has always been a myth. All the identity and access services for the public cloud outside of China (aka "IAM for the aws partition" to employees) are centralized in us-east-1. This centralization is essentially necessary in order to have a cohesive view of an account, its billing, and its permissions. And IAM is not a wholly independent software stack: they rely on DynamoDB and a few other services, which in turn have a circular dependency on IAM. During us-east-1 outages it's sometimes possible to continue using existing auth tokens or sessions in other regions, while not possible to grant new ones. When I worked there, I remember at least one case where my team's on-calls were advised not to close ssh sessions or AWS console browser tabs, for fear that we'd be locked out until the outage was over.
- sidewndr46 5mo agoIsn't this kind of circular dependency what lead to extended downtime a while back?
- jethro_tell 5mo agoIt's basically what leads to extended downtime almost every time. There are just some things in the stack that are still single points of failure, and when they fail it's a mess.
- dlenski 5mo agoYes, I concur. Sometimes the circular dependencies get almost cartoonishly silly. Like, "One of the two guys who has the physical keys to the server cage in us-east-1 is on vacation. The other one can't get into his apartment because his smart lock runs into the AWS cloud. So he hires a locksmith, but the locksmith takes an extra two hours to do the job because his reference documents for this model of lock live on an S3 bucket." I made that example up, but only barely.
- MichaelZuo 5mo agoWhy would such a critical server even be accessible with only one set of keys? I’ve always thought mission critical stuff needs two independent key holders, with key holes placed far apart enough to make it impossible for 1 person to reach both.
- ohnei 5mo agoI've always been impressed by Amazon's ability to present the shittiest experience possible and imply the blame is with things like isolation that they don't really provide.
- y3ahd0g 5mo agoNo. This is nonsense. Some SaaS apps had issues. The Internet was fine. This is physical reality. The internet was designed to route around this. Just because some app devs do a lazy job doesn't mean the entire infrastructure as designed is garbage. Just because some app devs are over reliant on a single cloud service doesn't mean the Internet is broken.
- dehrmann 5mo ago> building across multiple regions and AZs is a thing If you do this for resiliency, be prepared to pay the capacity tax (2 regions means 2x capacity, 3 regions means 1.5x), have the machines already running in a multi-region setup (don't expect to be able to spin up instances or even get capacity during an outage), and ready to deal with the added complexity of multi-region hosting.
- coredog64 5mo agoThere’s all kinds of fun pitfalls with multi-AZ. Like you can create RDS subnets across multiple AZs but then you can’t remove an AZ. Which really sucks when your core database covers all 5 us-east-1 AZs and randomly can’t failover because you picked an instance type that use1-az4 can’t host.