5 ms·
Issues are across all us-east 1, not one AZ. Load balancers are not doing well at all. The only way in this case to avoid an outage is to be cross regions or c
by boudin 5y ago
Issues are across all us-east 1, not one AZ.
Load balancers are not doing well at all. The only way in this case to avoid an outage is to be cross regions or cross cloud which is quite more complex to handle and require more resources to do well.
And I hope that nobody is listening your blaming and pointing fingers advice, that's the worst way to solve anything.
It's AWS job to ensure that things are reliable, that there is redundancy and that multi-AZ infra should be safe enough. The amount of issues in US-EAST-1 lately is really worrying.
- phamilton 5y agoEchoing this. We had to manually intervene and cut off the faulty AZ because our ASGs kept spinning up instances in it and our load balancers kept sending traffic to bad hosts. In the past I've seen both of those systems seamlessly handle an AZ failure. Today was different.
- acdha 5y agoSome load balancers may be having issues but I have multiple busy workloads showing no issues all morning. One big challenge can be that some people reporting multi-AZ issues are shifting traffic and competing with everyone else, while workloads which were already running in the other AZs were fine. It can be really hard to accurately tell how much the problems you’re seeing generalize to everyone else. I do agree that the end of this year has been a very bad period for AWS. I wonder whether there’s a connection to the pandemic conditions and the current job market – it feels like a lot of teams are running without much slack capacity, which could lead to both mistakes and longer recovery times.
- boudin 5y agoI hope AWS will provide some explanation about those issues and what actions they will take to prevent those in the future On our side we saw some EC2 VM totally disconnected from the network in 3 AZs.
- acdha 5y agoYeah, definitely needs good visibility. They’re asking customers to trust them to a large degree.
- bcrosby95 5y agoBut if AWS can't handle the shifting traffic in response to an AZ going down then at that point you're just gambling on whether or not you're in the lucky AZ.