10 ms·

Does anyone know how often an AZ experiences an issue as compared to an entire region? AWS sells the redundancy of AZs pretty heavily, but it seems like a lot o

Does anyone know how often an AZ experiences an issue as compared to an entire region? AWS sells the redundancy of AZs pretty heavily, but it seems like a lot of the issues that happen end up being region-wide. I'm struggling to understand whether I should be replicating our service across regions or whether the AZ redundancy within a region is sufficient.

nemothekid 5y ago
I've been naively setting up our distributed databases in separate AZs for a couple years now, paying, sometimes, thousands of dollars per month in data replication bandwidth egress fees. As far as I can remember I've never never seen an AZ go down, and the only region that has gone down has been us-east-1.
- cyounkins 5y ago
  Is that separate AZs within the same region, or AZs across regions? I didn't think there were any bandwidth fees between AZs in the same region.
  - TheP1000 5y ago
    That is incorrect. Cross az fees are steep.
  - electroly 5y ago
    It's $0.01/GB for cross-AZ transfer within a region.
    - nemothekid 5y ago
      In reality it's more like $0.02/GB. You pay $0.01 on sending and $0.01 on receiving. I have no idea why ingress isn't free.
      - rfraile 5y ago
        Plus the support percentage, don’t forguet.
- treis 5y ago
  AZs definitely go down. It's usually due to a physical reason like fire or power issues.
- maximilianroos 5y ago
  > I've never never seen an AZ go down, and the only region that has gone down has been us-east-1. Doesn't the region going down mean that _all_ its AZs have gone down? Or is my mental model of this incorrect?
  - urthor 5y ago
    No. See https://aws.amazon.com/about-aws/global-infrastructure/regions_az/ https://aws.amazon.com/about-aws/global-infrastructure/regio... A region is a networking paradigm. An AZ is a group of 2-6 data centers in the same city more or less. If a region goes down or is otherwise impacted, its AZs are unavailable or similar. If an AZ goes down, your VMs in said centers are disrupted in the most direct sense. It's the difference between loss of service and actual data loss.
- wjossey 5y ago
  There was an AZ outage in Oregon a couple months back. You should definitely go multi AZ without hesitation for production workloads for systems that should be highly available. You can easily lose a system permanently in a single AZ setup if it’s not ephemeral.
deleted 5y ago
[deleted]
dijit 5y ago
The main issue tends to be a lot of AWS internal components tend to be in us-east-1; it’s also the oldest zone. So when failures happen in that region (and they happen more commonly than others due to age, scale, complexity) then they can be globally impacting.
- propter_hoc 5y ago
  I agree with you, but my services are actually in Canada (Central). There's only one region in Canada, so I don't really have an alternative. AWS justifies it by saying there are three AZs (distinct data centres) within Canada (Central), but I get scared when I see these region-wide issues. If the AZs were really distinct, you wouldn't really have region-wide issues.
  - post-it 5y ago
    Multiple AZs are moreso for earthquakes, fires[1], and similar disasters rather than software issues. [1] https://www.reuters.com/article/us-france-ovh-fire-idUSKBN2B20NU https://www.reuters.com/article/us-france-ovh-fire-idUSKBN2B...
  - gnabgib 5y ago
    Good news, a new region is coming to Canada in the west[0] eta 2023/24 [0]: https://aws.amazon.com/blogs/aws/in-the-works-aws-canada-west-calgary-region/ https://aws.amazon.com/blogs/aws/in-the-works-aws-canada-wes...
  - discodave 5y ago
    Take DynamoDB as an example. The AWS managed service takes care of replicating everything to multiple AZs for you, that's great! You're very unlikely to lose your data. But, the DynamoDB team is running a mostly-regional service. If they push bad code or fall over it's likely going to be a regional issue. Probably only the storage nodes are truly zonal. If you wanted to deploy something similar, like Cassandra across AZs, or even regions you're welcome to do that. But now you're on the hook for the availability of the system. Are you going to get higher availability running your own Cassandra implementation than the DynamoDB team? Maybe. DynamoDB had a pretty big outage in 2015 I think. But that's a lot more work than just using DynamoDB IMO.
banana_giraffe 5y ago
It can be a bit hard to know, since the AZ identifiers are randomized per account, so if you think you have problems in us-west-1a, I can't check on my side. You can get the AZ ID out of your account to de-randomize things, so we can compare notes, but people rarely bother, for whatever reason.
- lanstin 5y ago
  If you do a lot of VPC Endpoints to clients/thirdparties, you learn the AZIDs or you go to all AZIDs in a region by default.
- mnordhoff 5y ago
  Amazon seems to have stopped randomizing them in newer regions. Another reason to move to us-east-2. ;-)
codeduck 5y ago
the eu-central-1 datacenter fire earlier this year was purportedly just 1AZ, but it took down the entire region to all intents and purposes. Our SOP is to cut over to a second region the moment we see any AZ-level shenanigans. We've been burned too often.
ashtonkem 5y ago
The best bang for your buck isn’t deploying into multiple AZs, but relocating everything into almost any other region than us-east-1. My system is latency and downtime tolerant, but I’m thinking I should move all my Kafka processing over to us-west-2
nijave 5y ago
Over two years I think we'd see about 2-3 AZ issues but only once I would consider it an outage. Usually there would be high network error rates which were usually enough to make RDS Postgres fail over if it was in the impacted AZ The only real "outage" was DNS having extremely high error rates in a single us-east-1 AZ to the point most things there were barely working Lack of instance capacity, especially spot, especially for the NVMe types was common of CI (it used ASGs for builder nodes). It'd be pretty common for a single AZ to run out of spot instance types--especially the NVMe ([a-z]#d types)

nemothekid 5y ago

I've been naively setting up our distributed databases in separate AZs for a couple years now, paying, sometimes, thousands of dollars per month in data replication bandwidth egress fees. As far as I can remember I've never never seen an AZ go down, and the only region that has gone down has been us-east-1.

cyounkins 5y ago
Is that separate AZs within the same region, or AZs across regions? I didn't think there were any bandwidth fees between AZs in the same region.
- TheP1000 5y ago
  That is incorrect. Cross az fees are steep.
- electroly 5y ago
  It's $0.01/GB for cross-AZ transfer within a region.
  - nemothekid 5y ago
    In reality it's more like $0.02/GB. You pay $0.01 on sending and $0.01 on receiving. I have no idea why ingress isn't free.
    - rfraile 5y ago
      Plus the support percentage, don’t forguet.
treis 5y ago
AZs definitely go down. It's usually due to a physical reason like fire or power issues.
maximilianroos 5y ago
> I've never never seen an AZ go down, and the only region that has gone down has been us-east-1. Doesn't the region going down mean that _all_ its AZs have gone down? Or is my mental model of this incorrect?
- urthor 5y ago
  No. See https://aws.amazon.com/about-aws/global-infrastructure/regions_az/ https://aws.amazon.com/about-aws/global-infrastructure/regio... A region is a networking paradigm. An AZ is a group of 2-6 data centers in the same city more or less. If a region goes down or is otherwise impacted, its AZs are unavailable or similar. If an AZ goes down, your VMs in said centers are disrupted in the most direct sense. It's the difference between loss of service and actual data loss.
wjossey 5y ago
There was an AZ outage in Oregon a couple months back. You should definitely go multi AZ without hesitation for production workloads for systems that should be highly available. You can easily lose a system permanently in a single AZ setup if it’s not ephemeral.

deleted 5y ago

[deleted]

dijit 5y ago

The main issue tends to be a lot of AWS internal components tend to be in us-east-1; it’s also the oldest zone. So when failures happen in that region (and they happen more commonly than others due to age, scale, complexity) then they can be globally impacting.

propter_hoc 5y ago
I agree with you, but my services are actually in Canada (Central). There's only one region in Canada, so I don't really have an alternative. AWS justifies it by saying there are three AZs (distinct data centres) within Canada (Central), but I get scared when I see these region-wide issues. If the AZs were really distinct, you wouldn't really have region-wide issues.
- post-it 5y ago
  Multiple AZs are moreso for earthquakes, fires[1], and similar disasters rather than software issues. [1] https://www.reuters.com/article/us-france-ovh-fire-idUSKBN2B20NU https://www.reuters.com/article/us-france-ovh-fire-idUSKBN2B...
- gnabgib 5y ago
  Good news, a new region is coming to Canada in the west[0] eta 2023/24 [0]: https://aws.amazon.com/blogs/aws/in-the-works-aws-canada-west-calgary-region/ https://aws.amazon.com/blogs/aws/in-the-works-aws-canada-wes...
- discodave 5y ago
  Take DynamoDB as an example. The AWS managed service takes care of replicating everything to multiple AZs for you, that's great! You're very unlikely to lose your data. But, the DynamoDB team is running a mostly-regional service. If they push bad code or fall over it's likely going to be a regional issue. Probably only the storage nodes are truly zonal. If you wanted to deploy something similar, like Cassandra across AZs, or even regions you're welcome to do that. But now you're on the hook for the availability of the system. Are you going to get higher availability running your own Cassandra implementation than the DynamoDB team? Maybe. DynamoDB had a pretty big outage in 2015 I think. But that's a lot more work than just using DynamoDB IMO.

banana_giraffe 5y ago

It can be a bit hard to know, since the AZ identifiers are randomized per account, so if you think you have problems in us-west-1a, I can't check on my side. You can get the AZ ID out of your account to de-randomize things, so we can compare notes, but people rarely bother, for whatever reason.

lanstin 5y ago
If you do a lot of VPC Endpoints to clients/thirdparties, you learn the AZIDs or you go to all AZIDs in a region by default.
mnordhoff 5y ago
Amazon seems to have stopped randomizing them in newer regions. Another reason to move to us-east-2. ;-)

codeduck 5y ago

the eu-central-1 datacenter fire earlier this year was purportedly just 1AZ, but it took down the entire region to all intents and purposes. Our SOP is to cut over to a second region the moment we see any AZ-level shenanigans. We've been burned too often.

ashtonkem 5y ago

The best bang for your buck isn’t deploying into multiple AZs, but relocating everything into almost any other region than us-east-1. My system is latency and downtime tolerant, but I’m thinking I should move all my Kafka processing over to us-west-2

nijave 5y ago

Over two years I think we'd see about 2-3 AZ issues but only once I would consider it an outage. Usually there would be high network error rates which were usually enough to make RDS Postgres fail over if it was in the impacted AZ The only real "outage" was DNS having extremely high error rates in a single us-east-1 AZ to the point most things there were barely working Lack of instance capacity, especially spot, especially for the NVMe types was common of CI (it used ASGs for builder nodes). It'd be pretty common for a single AZ to run out of spot instance types--especially the NVMe ([a-z]#d types)