34 ms·
Their status page (https://health.aws.amazon.com/health/status https://health.aws.amazon.com/health/status) says the only disrupted service is DynamoDB, but it'
by emrodre 1y ago
Their status page (https://health.aws.amazon.com/health/status https://health.aws.amazon.com/health/status) says the only disrupted service is DynamoDB, but it's impacting 37 other services. It is amazing to see how big a blast radius a single service can have.
- thmpp 1y agoAWS engineers are trained to use their internal services for each new system. They seem to like using DynamoDB. Dependencies like this should be made transparent.
- Nextgrid 1y agoNot sure why this is downvoted - this is absolutely correct. A lot of AWS services under the hood depend on others, and especially us-east-1 is often used for things that require strong consistency like AWS console logins/etc (where you absolutely don't want a changed password or revoked session to remain valid in other regions because of eventual consistency).
- bsjaux628 1y agoNot "like using", they are mandated from the top to use DynamoDB for any storage. At my org in the retail page, you needed director approval if you wanted to use a relational DB for a production service.
- myroon5 1y ago> Dependencies like this should be made transparent even internally, Amazon's dependency graph became visually+logically incomprehensible a long time ago
- stevepotter 1y agoEx employee here who built an aws service. Dynamo is basically mandated. You need like VP approval to use a relational database because of some scaling stuff they ran into historically. That sucks because we really needed a relational database and had to bend over backwards to use dynamo and all the nonsense associated with not having sql. It was super low traffic too
- littlecranky67 1y agoThe same page now says 58 services - just 23 minutes after your post. Seems this is becoming a larger issue.
- kalleboo 1y agoWhen I first visited the page it said like 23 services, now it says 65
- oneeyedpigeon 1y ago74 now. This is an extreme way of finding out just how many AWS services there really are!
- misswaterfairy 1y ago82 now. Looks like AWS detonated six sticks of dynamite under a house of cards...
- Steeeve 1y agoUp to 104 now, with 33 services reported as having issues that have been resolved.
- nevada_scout 1y agoIt's now listing 58 impacted services, so the blast radius is growing it seems
- jamesbelchamber 1y agoIt's not surprising that it's impacting other services in the region because DynamoDB is one of those things that lots of other services build on top of. It is a little bit surprising that the blast radius seems to extend beyond us-east-1, mind. In the coming hours/days we'll find out if AWS still have significant single points of failure in that region, or if _so many companies_ are just not bothering to build in redundancy to mitigate regional outages. I'm looking forward to the RCA!
- XorNot 1y agoI'm real curious how much of AWS GovCloud has continued through this actually. But even if it's fine, from a strategic perspective how much damage did we just discover you could do with a targeted disruption at the right time?
- Asparagirl 1y agoThe US federal government is in yet another shutdown right now, so how would some sub-agency even know if there were an unplanned outage, who would report it, and who would try to fix it?
- standardly 1y agoGov't IT is mostly contractors, and IME they do 99% of the work, so... That might be your answer.
- tonymet 1y agoi'm surprised / bothered that the history log shows the issues starting AM 10/20 -- when they seemed to have started around midnight 10/19