10 ms·
Why is it always us-east-1 though? I have always stayed away from that region because it seems significantly less reliable than other regions.
by whoisjuan 3y ago
Why is it always us-east-1 though?
I have always stayed away from that region because it seems significantly less reliable than other regions.
- deleted 3y ago[deleted]
- JoshTriplett 3y agous-east-1 is AWS's oldest region, and has the most legacy infrastructure, in ways that many other regions do not.
- nicoffeine 3y agoI thought I read that this is where they deploy new changes first. Can anyone confirm?
- mparnisari 3y agoWhen i worked at aws, IIRC, us-east-1 was one of the last regions we deployed to. So this is very confusing to me
- throwaway019254 3y agoI don't believe it's true. I was working on one of the biggest AWS services and we always deployed to small regions first. @dijit is right: https://news.ycombinator.com/item?id=36315736 https://news.ycombinator.com/item?id=36315736
- foobiekr 3y agoI have a suspicion that AWS uses some regions as canaries. Because we control both ends of things, I have personally noted that certain AWS functions clearly break in Australia first.
- temp_praneshp 3y agoFrom observing my wife's teams over the years, they deploy new _products_ early to that region, but deploying code changes starts in smaller regions.
- shitlord 3y agoWhen I worked there, there were few hard and fast rules. Every team had its own release processes, so there was a lot of variance. It has been a couple of years, so this may have changed. Typically, a team would group their regions into batches and deploy their change to one batch at a time. Usually they follow a geometric progression, so the first batch has one region, the second batch has two regions, the third batch has four regions, and so on. This batching was performed for the sake of time; nobody wants to wait a month for a single change to finish rolling out. One reason not to deploy to us-east-1 in the first batch is so you don't blow up your biggest region. The fewer customers you break, the better. One reason not to deploy to us-east-1 in the last batch is that there are a lot of batches. If a problem is uncovered after deploying the last batch, then someone has to initiate rollbacks for every single region. Some teams tried to compromise and put us-east-1 in one of the earlier batches.
- shepherdjerred 3y agoNo definitely not. Usually pipelines deploy over 1-2 week periods, and they don't deploy on Fridays/holidays/high-traffic periods like December. Deployments start off very conservative, maybe 1-2 small regions on the first day of deployments. As you gain confidence, the pipeline deploys to more regions/bigger regions. A pipeline that deploys to 22 regions over one week might go from 2 small regions on monday, 4 small/medium regions on tuesday, 8 medium/large regions on wednesday, 8 regions on thursday. us-east-1 is usually going to be deployed to on the wednesday/thursday in this example, but that isn't always the case because sometimes deployments are accelerated for feature launches (especially around re:invent), or retried because of a failure. There are best practice guides within Amazon that very closely detail how you should deploy, although it is up to the teams to follow them, which they usually do an okay job of.
- arbitrage 3y agoBecause it was one of the first, and it shows its age and less than rigorous rollout compared to the other zones.
- gtirloni 3y agoI suspect it's where they concentrate a lot of their control plane.
- esotericimpl 3y ago[dead]
- dijit 3y agoIt's the: * Largest (DDoS'd most, most complex, scaling issues etc) * Oldest (More time for weird idiosyncrasies to take hold) * Where most testing happens * Where new products are deployed first
- Matthias247 3y ago1) and 2) certainly apply. 3) and 4) don't. Testing in the largest region is one of the biggest anti-patterns.
- grumple 3y agoAWS doesn't test there last I checked, they roll out to smaller regions first.
- jedberg 3y ago4 is still generally true. Most new features drop in us-east-1 on launch day.
- shepherdjerred 3y agoUsually us-east-1 is deployed to after several smaller regions. Usually it'll fall in the middle of the week depending on the pipeline. Just because a feature is there on launch day doesn't mean it was deployed to first. Features are often hidden behind flags that are switched for launch.
- jedberg 3y agoI'm well aware of that, but the point is that when the feature is ungated to the public, it's in us-east-1 and gets all that load, and more load than the rest because of the fact that a lot of big customers are based in us-east-1, including much of Amazon itself.
- mcast 3y agoMost AWS engineering is closest to (and tested in) us-west-2 (PDX) or us-east-2 (Ohio)
- mike_d 3y agous-east-1 is the largest region, so it is where changes meet scale. It is also a massively complex beast in itself spanning dozens of datacenters with massive amounts of fiber between them. Much more fragile than having everything in a single building and as you scale up the number of components you increase the rate of failure.
- sofixa 3y agoNo AWS region is in a single building, they aren't amateurs like Azure. Each region is at least 3 AZs, which is at least one physical DC.
- colinbartlett 3y agoI actually just wrote about this very thing. It's not just that it SEEMS less reliable, it absolutely is: https://statusgator.com/blog/is-north-virginia-aws-region-the-least-reliable-and-why/ https://statusgator.com/blog/is-north-virginia-aws-region-th...
- dubcee349 3y agoI don't think this article has any value. Are you only counting region wide outages? US east is probably 10x the size of any other region with more AZ's than any other region.