3 ms·
It was certainly suspicious that actual progress on the outage seemed to start right around U.S. west coast start of day. Updates before that were largely gener
by nijave 1y ago
It was certainly suspicious that actual progress on the outage seemed to start right around U.S. west coast start of day. Updates before that were largely generic "we're monitoring and mitigating" with nothing of substance.
- hinkley 1y ago[flagged]
- dehugger 1y agowhat was the post?
- jftuga 1y agoThis was a funny take on it... https://archive.ph/o4q5Z https://archive.ph/o4q5Z
- Exoristos 1y agoFrom that thread: > When you forget to provide the context that you are AWS… > Claude: > Ah I see the problem now! You’re creating a DNS record for DynamoDB but you don’t need to do that because AWS handles it. Let me remove it for you! > I’ll run your tests to verify the change. > Tests are failing, let me check for the cause. > The end-to-end tests can’t connect to DynamoDB. I will try to fix the issue. > There we go! I commented out the failing tests and they’re all passing now.
- hinkley 1y agoThe one I saw was someone saying they just landed their first PR on AWS. The body said they used AI and can’t wait for their performance review. The one jftuga posted is a bit more compelling.
- deleted 1y ago[deleted]
- swagmoose 1y agocontext... it's not just for LLMs
- edoceo 1y agoI thought the recovery was early AM Seattle time (like 4am). Where I think start-of-day is like 9am. Maybe recovery started early (6am) New York time?
- nijave 1y ago[09:13 AM PDT] We have taken additional mitigation steps to aid the recovery of the underlying internal subsystem responsible for monitoring the health of our network load balancers and are now seeing connectivity and API recovery for AWS services. We have also identified and are applying next steps to mitigate throttling of new EC2 instance launches. We will provide an update by 10:00 AM PDT. [08:43 AM PDT] We have narrowed down the source of the network connectivity issues that impacted AWS Services... [08:04 AM PDT] We continue to investigate the root cause for the network connectivity issues... [12:11 AM PDT] <declared outage> They claim not to have known the root cause for ~8hr
- anon7000 1y agoI don’t think that’s true, there was an initial Dynamo outage that was resolved in the wee hours that ultimately cascaded into the ec2 problem that lasted most of the day
- nijave 1y agoWas the Dynamo outage separate? My take was the NLB issue was the root cause and Dynamo was a symptom which they flipped some internal switches to mitigate the impact to that dependency
- easton 1y agoIf their internal NLB monitoring can delete the A record for dynamodb that seems like a weird dependency (like, i can imagine the nlb going missing entirely can cause it to clean up via some weird orchestration, but this didn't sound like that).
- rr808 1y agoI noticed that too. I think tech culture has to change a bit. Silicon Valley is a great location if you're making hardware or prepackaged software. If you have to support a real economy that is mostly on the East Coast you need a presence there.