4 ms·
The unreliability claim is driven by two factors. 1. When aws deploys changes they run through a pipeline which pushes change to regions one at a time. Most se
by everfrustrated 1y ago
The unreliability claim is driven by two factors.
1. When aws deploys changes they run through a pipeline which pushes change to regions one at a time. Most services start with us-east-1 first.
2. us-east-1 is MASSIVE and considerably larger than the next largest region. There's no public numbers but I wouldn't be surprised if it was 50% of their global capacity. An outage in any other region never hits the news.
- iLoveOncall 1y ago> 1. When aws deploys changes they run through a pipeline which pushes change to regions one at a time. This is true. > Most services start with us-east-1 first. This is absolutely false. Almost every service will FINISH with the largest and most impactful regions.
- captainkrtek 1y agoAgreed. Most services start deployments on a small number of hosts in single AZs in small less-known regions, ramping up from there. In all my years there I don’t recall “us-east-1 first”.
- riknos314 1y agoEach AWS service may choose different pipeline ordering based on the risks specific to their architecture. In general: You don't deploy to the largest region first because of the large blast radius. You may not want to deploy to the largest region last because then if there's an issue that only shows up at that scale you may need to roll every single region back (divergent code across regions is generally avoided as much as possible). A middle ground is to deploy to the largest region second or third.
- kixiQu 1y ago> a pipeline which pushes change to regions one at a time > When AWS deploys updates to its services, deployments to Availability Zones in the same Region are separated in time to prevent correlated failure. https://docs.aws.amazon.com/whitepapers/latest/aws-fault-isolation-boundaries/availability-zones.html https://docs.aws.amazon.com/whitepapers/latest/aws-fault-iso...