5 ms·

I thought I read that this is where they deploy new changes first. Can anyone confirm?

mparnisari 3y ago
When i worked at aws, IIRC, us-east-1 was one of the last regions we deployed to. So this is very confusing to me
throwaway019254 3y ago
I don't believe it's true. I was working on one of the biggest AWS services and we always deployed to small regions first. @dijit is right: https://news.ycombinator.com/item?id=36315736 https://news.ycombinator.com/item?id=36315736
- foobiekr 3y ago
  I have a suspicion that AWS uses some regions as canaries. Because we control both ends of things, I have personally noted that certain AWS functions clearly break in Australia first.
temp_praneshp 3y ago
From observing my wife's teams over the years, they deploy new _products_ early to that region, but deploying code changes starts in smaller regions.
shitlord 3y ago
When I worked there, there were few hard and fast rules. Every team had its own release processes, so there was a lot of variance. It has been a couple of years, so this may have changed. Typically, a team would group their regions into batches and deploy their change to one batch at a time. Usually they follow a geometric progression, so the first batch has one region, the second batch has two regions, the third batch has four regions, and so on. This batching was performed for the sake of time; nobody wants to wait a month for a single change to finish rolling out. One reason not to deploy to us-east-1 in the first batch is so you don't blow up your biggest region. The fewer customers you break, the better. One reason not to deploy to us-east-1 in the last batch is that there are a lot of batches. If a problem is uncovered after deploying the last batch, then someone has to initiate rollbacks for every single region. Some teams tried to compromise and put us-east-1 in one of the earlier batches.
shepherdjerred 3y ago
No definitely not. Usually pipelines deploy over 1-2 week periods, and they don't deploy on Fridays/holidays/high-traffic periods like December. Deployments start off very conservative, maybe 1-2 small regions on the first day of deployments. As you gain confidence, the pipeline deploys to more regions/bigger regions. A pipeline that deploys to 22 regions over one week might go from 2 small regions on monday, 4 small/medium regions on tuesday, 8 medium/large regions on wednesday, 8 regions on thursday. us-east-1 is usually going to be deployed to on the wednesday/thursday in this example, but that isn't always the case because sometimes deployments are accelerated for feature launches (especially around re:invent), or retried because of a failure. There are best practice guides within Amazon that very closely detail how you should deploy, although it is up to the teams to follow them, which they usually do an okay job of.

When i worked at aws, IIRC, us-east-1 was one of the last regions we deployed to. So this is very confusing to me

throwaway019254 3y ago

I don't believe it's true. I was working on one of the biggest AWS services and we always deployed to small regions first. @dijit is right: https://news.ycombinator.com/item?id=36315736 https://news.ycombinator.com/item?id=36315736

foobiekr 3y ago
I have a suspicion that AWS uses some regions as canaries. Because we control both ends of things, I have personally noted that certain AWS functions clearly break in Australia first.

temp_praneshp 3y ago

From observing my wife's teams over the years, they deploy new _products_ early to that region, but deploying code changes starts in smaller regions.

shitlord 3y ago

When I worked there, there were few hard and fast rules. Every team had its own release processes, so there was a lot of variance. It has been a couple of years, so this may have changed. Typically, a team would group their regions into batches and deploy their change to one batch at a time. Usually they follow a geometric progression, so the first batch has one region, the second batch has two regions, the third batch has four regions, and so on. This batching was performed for the sake of time; nobody wants to wait a month for a single change to finish rolling out. One reason not to deploy to us-east-1 in the first batch is so you don't blow up your biggest region. The fewer customers you break, the better. One reason not to deploy to us-east-1 in the last batch is that there are a lot of batches. If a problem is uncovered after deploying the last batch, then someone has to initiate rollbacks for every single region. Some teams tried to compromise and put us-east-1 in one of the earlier batches.

shepherdjerred 3y ago

No definitely not. Usually pipelines deploy over 1-2 week periods, and they don't deploy on Fridays/holidays/high-traffic periods like December. Deployments start off very conservative, maybe 1-2 small regions on the first day of deployments. As you gain confidence, the pipeline deploys to more regions/bigger regions. A pipeline that deploys to 22 regions over one week might go from 2 small regions on monday, 4 small/medium regions on tuesday, 8 medium/large regions on wednesday, 8 regions on thursday. us-east-1 is usually going to be deployed to on the wednesday/thursday in this example, but that isn't always the case because sometimes deployments are accelerated for feature launches (especially around re:invent), or retried because of a failure. There are best practice guides within Amazon that very closely detail how you should deploy, although it is up to the teams to follow them, which they usually do an okay job of.