6 ms·

AWS didn’t “go down”. They had an outage in one AZ, which is why there are multiple AZs in each region. If your app went down then you should be blaming your de

by JCM9 5y ago

AWS didn’t “go down”. They had an outage in one AZ, which is why there are multiple AZs in each region. If your app went down then you should be blaming your developers on this one, not AWS. Those having issues are discovering gaps in their HA designs. Obviously it’s not good for an AZ to go down but it does happen and why any production workload should be architected to have seamless failover and recover to other AZs, typically by just dropping nodes in the down AZ. People commenting that servers shouldn’t go down ect don’t understand how true HA architectures work. You should expect and build for stuff to fail like this. Otherwise it’s like complaining that you lost data because a disk failed. Disks fail… build architecture where that won’t take you down.

bencoder 5y ago
Our API is just appsync (graphql) + lambdas + dynamoDB so, theoretically, we shouldn't have been affected. But about 1 in 3 requests was just hanging and timing out. As others have said, they are not being forthright about the severity of the issue, as is standard.
dkryptr 5y ago
100% agree. I'm actually surprised AWS hasn't built in a Chaos Monkey into their APIs/console so people can test their resiliency regularly if an AZ goes down. edit: of course, AWS does have this: AWS Fault Injection Simulator
- stingraycharles 5y ago
  Because then people would complain about AWS being less reliable than Azure / GCP.
- biohax2015 5y ago
  AWS Fault Injection Simulator does this.
  - lljk_kennedy 5y ago
    Is that what they call us-east-1 nowadays?
    - voidfunc 5y ago
      us-east-1 has always been the canary region hasn't it?
  - dkryptr 5y ago
    TIL. Thank you!
matharmin 5y ago
AWS is under-reporting the severity of the issue though. The primary outage may be in a single AZ, but there are parts of the AWS stack that affected all AZs in us-east-1, and potentially other regions as well. For example, even now I'm unable to create a new ElastiCache cluster in different AZs of us-east-1.
- zymhan 5y ago
  > I'm unable to create a new ElastiCache cluster in different AZs of us-east-1 Isn't that because Elasticache will distribute the cluster across AZs automatically? https://docs.aws.amazon.com/AmazonElastiCache/latest/red-ug/AutoFailover.html https://docs.aws.amazon.com/AmazonElastiCache/latest/red-ug/...
  - matharmin 5y ago
    In this case, this was specifically with a single-AZ setup, using an AZ that was supposed to be unaffected.
boudin 5y ago
Issues are across all us-east 1, not one AZ. Load balancers are not doing well at all. The only way in this case to avoid an outage is to be cross regions or cross cloud which is quite more complex to handle and require more resources to do well. And I hope that nobody is listening your blaming and pointing fingers advice, that's the worst way to solve anything. It's AWS job to ensure that things are reliable, that there is redundancy and that multi-AZ infra should be safe enough. The amount of issues in US-EAST-1 lately is really worrying.
- phamilton 5y ago
  Echoing this. We had to manually intervene and cut off the faulty AZ because our ASGs kept spinning up instances in it and our load balancers kept sending traffic to bad hosts. In the past I've seen both of those systems seamlessly handle an AZ failure. Today was different.
- acdha 5y ago
  Some load balancers may be having issues but I have multiple busy workloads showing no issues all morning. One big challenge can be that some people reporting multi-AZ issues are shifting traffic and competing with everyone else, while workloads which were already running in the other AZs were fine. It can be really hard to accurately tell how much the problems you’re seeing generalize to everyone else. I do agree that the end of this year has been a very bad period for AWS. I wonder whether there’s a connection to the pandemic conditions and the current job market – it feels like a lot of teams are running without much slack capacity, which could lead to both mistakes and longer recovery times.
  - boudin 5y ago
    I hope AWS will provide some explanation about those issues and what actions they will take to prevent those in the future On our side we saw some EC2 VM totally disconnected from the network in 3 AZs.
    - acdha 5y ago
      Yeah, definitely needs good visibility. They’re asking customers to trust them to a large degree.
tluyben2 5y ago
> People commenting that servers shouldn’t go down ect don’t understand how true HA architectures work. You should expect and build for stuff to fail like this. Otherwise it’s like complaining that you lost data because a disk failed. Disks fail… build architecture where that won’t take you down. Is that comparison fair? If you have 2 raid-5 mirrored raid 5 boxes in your room and all disks fail at the same time, you should complain. And that won't happen. These entire datacenter failures should be anticipated, but to expect them is a bit too easy I think. There are plenty of hosters who don't have this stuff even once for the last decade in their only datacenter. I do not find it strange to expect or even demand that level but to protect yourself if it happens in any case if that fits your specific project and budget. Edit; OK meant that raid-5 remark in the same context as the hosting; it can and does happen but it shouldn't; you should plan for a contingency but expect it goes far. We never had it (1000s of hard-drive, decades of hosting, millions of sites) and so we plan for it with backups; if it happens it will take some downtime but it costs next to nothing over time to do that. If we expected it, we would need to take far different measures. And we had less downtime in a decade than aws AZ had in the past months. I have a problem with the word 'expect'.
- dylan604 5y ago
  >And that won't happen HA! I had received new 16-bay chasis and all of the drives needed plus cold spares for each chasis. Set them up and started the RAID-5 init on a Friday. Left them running in the rack over the weekend. Returned on Monday to find multiple drives in each chasis had failed. Even with dedicated one of the 16 drives as a hot swap, the volumes would all have failed in an unrecoverable manner. All drives were purchased at the same time, and happened to all come from a single batch from the manufacture. The manufacture confirmed this via serial numbers, and admitted they had an issue during production. All drives were replaced and at a larger volume size. TL;DR: Drives will fail, and manufacturing issues happend. Don't buy all of your drives in an array from the same batch! It will happen. To say it won't is just pure inexeperience.
  - tluyben2 5y ago
    Guess I was lucky, we ran a lot of these over the decades when things were far more unreliable than now and never experienced anything like that. Manufacturing issues, sure, but we always had everything we bought run on stress for 48 hours and see if that killed it, if it didn't, it didn't usually break anymore (I have many of the machines from mid to end 2000s still and they don't have diskfailures now while they ran for many years).
deleted 5y ago
[deleted]
strunz 5y ago
Except many AWS services still route through us-east-1 anyway, which is why they have had huge outages recently. AWS isn't as redundant as people think it is.
tyingq 5y ago
>AWS didn’t “go down” The context of the parent seems to be that they intermittently couldn't get to the console. That seems fair to me. If we're blaming developers and finding gaps in HA design, then AWS should also figure out how to make the console url resilient. If it's not, then AWS does appear to be down. I imagine it's pretty hard to design around these failures, because it's not always clear what to do. You would think, for example, that load balancers would work properly during this outage. They aren't. Or that you could deploy an Elasticache cluster to the remaining AZs. You can't. And I imagine the problems vary based on the AWS outage type. Similarly, with the earlier broad us-east-1 outage, you couldn't update Route53 records. I don't think that was known beforehand by everyone that uses AWS. You can imagine changing DNS records might be useful during an outage.
deleted 5y ago
[deleted]
TameAntelope 5y ago
Here's a secret that's now saved me from three outages this month: Be in multiple AZs, and even multiple regions but if you're going to be in only one AZ or one region, make it us-east-2.

by JCM9 5y ago

bencoder 5y ago

Our API is just appsync (graphql) + lambdas + dynamoDB so, theoretically, we shouldn't have been affected. But about 1 in 3 requests was just hanging and timing out. As others have said, they are not being forthright about the severity of the issue, as is standard.

dkryptr 5y ago

100% agree. I'm actually surprised AWS hasn't built in a Chaos Monkey into their APIs/console so people can test their resiliency regularly if an AZ goes down. edit: of course, AWS does have this: AWS Fault Injection Simulator

stingraycharles 5y ago
Because then people would complain about AWS being less reliable than Azure / GCP.
biohax2015 5y ago
AWS Fault Injection Simulator does this.
- lljk_kennedy 5y ago
  Is that what they call us-east-1 nowadays?
  - voidfunc 5y ago
    us-east-1 has always been the canary region hasn't it?
- dkryptr 5y ago
  TIL. Thank you!

matharmin 5y ago

AWS is under-reporting the severity of the issue though. The primary outage may be in a single AZ, but there are parts of the AWS stack that affected all AZs in us-east-1, and potentially other regions as well. For example, even now I'm unable to create a new ElastiCache cluster in different AZs of us-east-1.

zymhan 5y ago
> I'm unable to create a new ElastiCache cluster in different AZs of us-east-1 Isn't that because Elasticache will distribute the cluster across AZs automatically? https://docs.aws.amazon.com/AmazonElastiCache/latest/red-ug/AutoFailover.html https://docs.aws.amazon.com/AmazonElastiCache/latest/red-ug/...
- matharmin 5y ago
  In this case, this was specifically with a single-AZ setup, using an AZ that was supposed to be unaffected.

boudin 5y ago

Issues are across all us-east 1, not one AZ. Load balancers are not doing well at all. The only way in this case to avoid an outage is to be cross regions or cross cloud which is quite more complex to handle and require more resources to do well. And I hope that nobody is listening your blaming and pointing fingers advice, that's the worst way to solve anything. It's AWS job to ensure that things are reliable, that there is redundancy and that multi-AZ infra should be safe enough. The amount of issues in US-EAST-1 lately is really worrying.

phamilton 5y ago
Echoing this. We had to manually intervene and cut off the faulty AZ because our ASGs kept spinning up instances in it and our load balancers kept sending traffic to bad hosts. In the past I've seen both of those systems seamlessly handle an AZ failure. Today was different.
acdha 5y ago
Some load balancers may be having issues but I have multiple busy workloads showing no issues all morning. One big challenge can be that some people reporting multi-AZ issues are shifting traffic and competing with everyone else, while workloads which were already running in the other AZs were fine. It can be really hard to accurately tell how much the problems you’re seeing generalize to everyone else. I do agree that the end of this year has been a very bad period for AWS. I wonder whether there’s a connection to the pandemic conditions and the current job market – it feels like a lot of teams are running without much slack capacity, which could lead to both mistakes and longer recovery times.
- boudin 5y ago
  I hope AWS will provide some explanation about those issues and what actions they will take to prevent those in the future On our side we saw some EC2 VM totally disconnected from the network in 3 AZs.
  - acdha 5y ago
    Yeah, definitely needs good visibility. They’re asking customers to trust them to a large degree.

tluyben2 5y ago

> People commenting that servers shouldn’t go down ect don’t understand how true HA architectures work. You should expect and build for stuff to fail like this. Otherwise it’s like complaining that you lost data because a disk failed. Disks fail… build architecture where that won’t take you down. Is that comparison fair? If you have 2 raid-5 mirrored raid 5 boxes in your room and all disks fail at the same time, you should complain. And that won't happen. These entire datacenter failures should be anticipated, but to expect them is a bit too easy I think. There are plenty of hosters who don't have this stuff even once for the last decade in their only datacenter. I do not find it strange to expect or even demand that level but to protect yourself if it happens in any case if that fits your specific project and budget. Edit; OK meant that raid-5 remark in the same context as the hosting; it can and does happen but it shouldn't; you should plan for a contingency but expect it goes far. We never had it (1000s of hard-drive, decades of hosting, millions of sites) and so we plan for it with backups; if it happens it will take some downtime but it costs next to nothing over time to do that. If we expected it, we would need to take far different measures. And we had less downtime in a decade than aws AZ had in the past months. I have a problem with the word 'expect'.

dylan604 5y ago
>And that won't happen HA! I had received new 16-bay chasis and all of the drives needed plus cold spares for each chasis. Set them up and started the RAID-5 init on a Friday. Left them running in the rack over the weekend. Returned on Monday to find multiple drives in each chasis had failed. Even with dedicated one of the 16 drives as a hot swap, the volumes would all have failed in an unrecoverable manner. All drives were purchased at the same time, and happened to all come from a single batch from the manufacture. The manufacture confirmed this via serial numbers, and admitted they had an issue during production. All drives were replaced and at a larger volume size. TL;DR: Drives will fail, and manufacturing issues happend. Don't buy all of your drives in an array from the same batch! It will happen. To say it won't is just pure inexeperience.
- tluyben2 5y ago
  Guess I was lucky, we ran a lot of these over the decades when things were far more unreliable than now and never experienced anything like that. Manufacturing issues, sure, but we always had everything we bought run on stress for 48 hours and see if that killed it, if it didn't, it didn't usually break anymore (I have many of the machines from mid to end 2000s still and they don't have diskfailures now while they ran for many years).

deleted 5y ago

[deleted]

strunz 5y ago

Except many AWS services still route through us-east-1 anyway, which is why they have had huge outages recently. AWS isn't as redundant as people think it is.

tyingq 5y ago

>AWS didn’t “go down” The context of the parent seems to be that they intermittently couldn't get to the console. That seems fair to me. If we're blaming developers and finding gaps in HA design, then AWS should also figure out how to make the console url resilient. If it's not, then AWS does appear to be down. I imagine it's pretty hard to design around these failures, because it's not always clear what to do. You would think, for example, that load balancers would work properly during this outage. They aren't. Or that you could deploy an Elasticache cluster to the remaining AZs. You can't. And I imagine the problems vary based on the AWS outage type. Similarly, with the earlier broad us-east-1 outage, you couldn't update Route53 records. I don't think that was known beforehand by everyone that uses AWS. You can imagine changing DNS records might be useful during an outage.

deleted 5y ago

[deleted]

TameAntelope 5y ago

Here's a secret that's now saved me from three outages this month: Be in multiple AZs, and even multiple regions but if you're going to be in only one AZ or one region, make it us-east-2.