4 ms·
AWS Glue/Athena/Redshift outage
- nknealk 7y agoThankfully the redshift outage was just on APIs, not existing machines. Our cluster was fine today, but external schema which rely on glue/athena did time out.
- wgjordan 7y agoSummary: > Between 9:21 AM and 2:36 PM PDT we experienced increased query failures and latency in the US-EAST-1 Region. The issue has been resolved and the service is operating normally. > The issue with the Data Catalog APIs started with a software update in the US-EAST-1 Region that completed at 9:21 AM PDT. The software update was immediately rolled back[...]
- nnx 7y agoShould edit title to add “in us-east-1”. Other regions are unimpacted.
- thinkingkong 7y agoThere are so many outages in us-east-1. I've heard the reason is because that's where they roll out maintenance first or something along those lines. Just look at this list of outages on Wikipedia [1] and scan for US-east-1, North Virginia, or "Northeast" (all the same places). Just don't use US-EAST-1 as your region. 1. https://en.wikipedia.org/wiki/Timeline_of_Amazon_Web_Services#Amazon_Web_Services_Outages https://en.wikipedia.org/wiki/Timeline_of_Amazon_Web_Service...
- chocolatkey 7y agoIt is usually the cheapest region though. Maybe this is why
- empath75 7y agoIt’s also full of legacy infrastructure since it was the first region.
- Johnny555 7y agoI've heard the reason is because that's where they roll out maintenance first That doesn't make sense - why would they do maintenance in their largest (and oldest) region first? I'd expect them to roll out changes to smaller regions first so problems will affect fewer users. I think the more likely explanation is that it's their largest (and oldest) region.
- swasheck 7y agoan aws tam once told me the same thing. us-east-1a gets the new stuff first. i never validated it against anything other than this one person's statement.
- ecnahc515 7y ago"1a" in this context means nothing. The AZ assignments each account gets is random. us-east-1a is probably a different data center for you than me.
- deleted 7y ago[deleted]
- discodave 7y agoIt's the oldest region, which means: * It's the largest region (ever had an unexpected scaling bug?). * It has more legacy stuff lying around. For example, old regions have EC2 Classic, while new regions are VPC only. * There are more customers there. More whales, more use cases. Most AWS teams explicitly try not to deploy to us-east-1 first, but because us-east-1 is so different on so many dimensions, it is more likely to have issues that dont manifest elsewhere. (Source: An AWS Engineer)
- teej 7y agoRedshift changes roll out in us-east-1 after other regions though, so I imagine the root cause is something else.
- kache_ 7y agoFailovers, man.
- nullwasamistake 7y agoCloud services go down more often than my old WordPress sites. Avoiding vendor lock in and doing multi-provider deployments should be par the course.