17 ms·

Summary of the Amazon DynamoDB Service Disruption in US-East-1 Region

551 points by meetpateltech 1y ago

Recent and related: AWS multiple services outage in us-east-1 - https://news.ycombinator.com/item?id=45640838 https://news.ycombinator.com/item?id=45640838 (2045 comments)

deleted 1y ago
[deleted]
deleted 1y ago
[deleted]
joeyhage 1y ago
> as is the case with the recently launched IPv6 endpoint and the public regional endpoint It isn't explicitly stated in the RCA but it is likely these new endpoints were the straw that broke the camel's back for the DynamoDB load balancer DNS automation
yla92 1y ago
So the root cause is basically race condition 101 stale read ?
- philipwhiuk 1y ago
  Race condition and bad data validation.
galaxy01 1y ago
Would conditional read/write solve this? looks like some kind of stale read
lazystar 1y ago
> Since this situation had no established operational recovery procedure, engineers took care in attempting to resolve the issue with DWFM without causing further issues. interesting.
shayonj 1y ago
I was kinda surprised the lack of CAS on per-endpoint plan version or rejecting stale writes via 2PC or single-writer lease per endpoint like patterns. Definitely a painful one with good learnings and kudos to AWS for being so transparent and detailed :hugops:
- donavanm 1y ago
  See https://news.ycombinator.com/item?id=45681136 https://news.ycombinator.com/item?id=45681136. The actual DNS mutation API does, effectively, CAS. They had multiple unsynchronized writers who raced without logical constraints or ordering to teh changes. Without thinking much they _might_ have been able to implement something like a vector either through updating the zone serial or another "sentinel record" that was always used for ChangeRRSets affecting that label/zone; like a TXT record containing a serialized change set number or a "checksum" of the old + new state. Im guessing the "plans" aspect skipped that and they were just applying intended state, without trying serialize them. And last-write-wins, until it doesnt.
  - cyberax 1y ago
    Oh, I can see it from here. AWS internally has a problem with things like task orchestration. I bet that the enactor can be rewritten as a goroutine/thread in the planner, with proper locking and ordering. But that's too complicated and results in more code. So they likely just used an SQS queue with consumers reading from it.
jasode 1y ago
So the DNS records if-stale-then-needs-update it was basically a variation of the "2 Hard Things In Computer Science - cache invalidation". Excerpt from the giant paragraph: >[...] Right before this event started, one DNS Enactor experienced unusually high delays needing to retry its update on several of the DNS endpoints. As it was slowly working through the endpoints, several other things were also happening. First, the DNS Planner continued to run and produced many newer generations of plans. Second, one of the other DNS Enactors then began applying one of the newer plans and rapidly progressed through all of the endpoints. The timing of these events triggered the latent race condition. When the second Enactor (applying the newest plan) completed its endpoint updates, it then invoked the plan clean-up process, which identifies plans that are significantly older than the one it just applied and deletes them. At the same time that this clean-up process was invoked, the first Enactor (which had been unusually delayed) applied its much older plan to the regional DDB endpoint, overwriting the newer plan. The check that was made at the start of the plan application process, which ensures that the plan is newer than the previously applied plan, was stale by this time due to the unusually high delays in Enactor processing. [...] It outlines some of the mechanics but some might think it still isn't a "Root Cause Analysis" because there's no satisfying explanation of _why_ there were "unusually high delays in Enactor processing". Hardware problem?!? Human error misconfiguration causing unintended delays in Enactor behavior?!? Either the previous sequence of events leading up to that is considered unimportant, or Amazon is still investigating what made Enactor behave in an unpredictable way.
- mcmoor 1y ago
  Also, I don't know if I missed it, but they don't establish anything to prevent outage if there's unusually high delay again?
  - mattcrox 1y ago
    It’s at the end, they disabled the DDB DNS automations around this to fix before they re-enable them
    - deleted 1y ago
      [deleted]
gslin 1y ago
I believe a report with timezone not using UTC is a crime.
- exogenousdata 1y ago
  An epoch fail?
- cheeze 1y ago
  My guess is that PT was chosen to highlight the fact that this happened in the middle of the night for most of the responding ops folks. (I don't know anything here, just spitballing why that choice would be made)
  - throitallaway 1y ago
    Their headquarters is in Seattle (Pacific Time.) But yeah, I hate time zones.
- tguedes 1y ago
  I think it makes sense in this instance. Because this occurred in us-east-1, the vast majority of affected customers are US based. For most people, it's easier to do the timezone conversion from PT than UTC.
  - trenchpilgrim 1y ago
    us-east-1 is an exceptional Amazon region; it hosts many global services as well as services which are not yet available in other regions. Most AWS customers worldwide probably have an indirect dependency on us-east-1.
  - thayne 1y ago
    But us-east-1 is in Eastern Time, so if you aren't going to use UTC, why not use that? I'm guessing PT was chosen because the people writing this report are in PT (where Amazon headquarters is).
alexband2206 1y ago
[dead]
ericpauley 1y ago
Interesting use of the phrase “Route53 transaction” for an operation that has no hard transactional guarantees. Especially given the lack of transactional updates are what caused the outage…
- donavanm 1y ago
  I think you misunderstnad the failure case. The ChangeResourceRecordSet is transactional (or was when I worked on the service) https://docs.aws.amazon.com/Route53/latest/APIReference/API_ChangeResourceRecordSets.html https://docs.aws.amazon.com/Route53/latest/APIReference/API_.... The fault was two different clients with divergent goal states: - one ("old") DNS Enactor experienced unusually high delays needing to retry its update on several of the DNS endpoints - the DNS Planner continued to run and produced many newer generations of plans [Ed: this is key: its producing "plans" of desired state, the does not include a complete transaction like a log or chain with previous state + mutations] - one of the other ("new") DNS Enactors then began applying one of the newer plans - then ("new") invoked the plan clean-up process, which identifies plans that are significantly older than the one it just applied and deletes them [Ed: the key race is implied here. The "old" Enactor is reading _current state_, which was the output of "new", and applying its desired "old" state on top. The discrepency is because apparently Planer and Enactor aren't working with a chain/vector clock/serialized change set numbers/etc] - At the same time the first ("old") Enactor ... applied its much older plan to the regional DDB endpoint, overwriting the newer plan. [Ed: and here is where "old" Enactor creates the valid ChangeRRSets call, replacing "new" with "old"] - The check that was made at the start of the plan application process, which ensures that the plan is newer than the previously applied plan, was stale by this time [Ed: Whoops!] - The second Enactor’s clean-up process then deleted this older plan because it was many generations older than the plan it had just applied. Ironically Route 53 does have strong transactions of API changes _and_ serializes them _and_ has closed loop observers to validate change sets globally on every dataplane host. So do other AWS services. And there are even some internal primitives for building replication or change set chains like this. But its also a PITA and takes a bunch of work and when it _does_ fail you end up with global deadlock and customers who are really grumpy that they dont see their DNS changes going in to effect.
everfrustrated 1y ago
>Services like DynamoDB maintain hundreds of thousands of DNS records to operate a very large heterogeneous fleet of load balancers in each Region Does that mean a DNS query for dynamodb.us-east-1.amazonaws.com can resolve to one of a hundred thousand IP address? That's insane! And also well beyond the limits of route53. I'm wondering if they're constantly updating route53 with a smaller subset of records and using a low ttl to somewhat work around this.
- supriyo-biswas 1y ago
  DNS-based CDNs are also effectively this: collect metrics from a datastore regarding system usage metrics, packet loss, latency etc and compute a table of viewer networks and preferred PoPs. Unfortunately hard documentation is difficult to provide but that’s how a CDN worked at a place I used to work for, there’s also another CDN[1] which talks about the same thing in fancier terms. [1] https://bunny.net/network/smartedge/ https://bunny.net/network/smartedge/
  - donavanm 1y ago
    Akamai talked about it in the early 2000s. Facebook content folks had a decent paper describing the latency collection and realtime routing around 2011ish, something like “pinpoint” I want to say. Though as you say was industry practice before then.
- thayne 1y ago
  I haven't tested with dynamodb, but I once ran a loop of doing DNS lookups for s3, and I in a couple seconds I got hundreds of distinct ip addresses. And that was just for a single region, from a single source ip.
- donavanm 1y ago
  Some details, but yeah that's basically how all AWS DNS works. I think youre missing how labels, zones, and domains are related but distinct. And that R53 operates in resource record SETS. And there are affordances in the set relationships to build trees and logic for selecting an appropriate set (eg healthcheck, latency). > And also well beyond the limits of route53 Ipso facto, R53 can do this just fine. Where do you think all of your public EC2, ELB, RDS, API Gateway, etc etc records are managed and served?
LaserToy 1y ago
TLDR: A DNS automation bug removed all the IP addresses for the regional endpoints. The tooling that was supposed to help with recovery depends on the system it needed to recover. That’s a classic “we deleted prod” failure mode at AWS scale.
qrush 1y ago
Sounds like DynamoDB is going to continue to be a hard dependency for EC2, etc. I at least appreciate the transparency and hearing about their internal systems names.
- offmycloud 1y ago
  I think it's time for AWS to pull the curtain back a bit and release a JSON document that shows a list of all internal service dependencies for each AWS service.
  - throitallaway 1y ago
    Would it matter? Would you base decisions on whether or not to use one of their products based on the dependency graph?
    - withinboredom 1y ago
      Yes.
      - bdangubic 1y ago
        if so, I hate to tell you this but you would not use AWS (or any other cloud provider)!
        withinboredom 1y ago
        I don’t use AWS or any other cloud provider. I use bare metal since 2012. See, in 2012 (IIRC), one fateful day, we turned off our bare metal machines and went full AWS. That afternoon, AWS had its first major outage. Prior to that day, the owner could walk in and ask what we were doing about it. That day, all we could do was twiddle our thumbs or turn on a now outdated database replica. Surely AWS won’t be out for hours, right? Right? With bare metal, you might be out for hours, but you can quickly get back to a degraded state, no matter what happens. With AWS, you’re stuck with whatever they happen to fix first.
WaitWaitWha 1y ago
I gather, the root cause was a latent race condition in the DynamoDB DNS management system that allowed an outdated DNS plan to overwrite the current one, resulting in an empty DNS record for the regional endpoint. Correct?
- tptacek 1y ago
  I think you have to be careful with ideas like "the root cause". They underwent a metastable congestive collapse. A large component of the outage was them not having a runbook to safely recover an adequately performing state for their droplet manager service. The precipitating event was a race condition with the DynamoDB planner/enactor system. https://how.complexsystems.fail/ https://how.complexsystems.fail/
  - 1970-01-01 1y ago
    Why can't a race condition bug be seen as the single root cause? Yes, there were other factors that accelerated collapse, but those are inherent to DNS, which is outside the scope of a summary.
    - tptacek 1y ago
      Because the DNS race condition is just one flaw in the system. The more important latent flaw† is probably the metastable failure mode for the droplet manager, which, when it loses connectivity to Dynamo, gradually itself loses connectivity with the Droplets, until a critical mass is hit where the Droplet manager has to be throttled and manually recovered. Importantly: the DNS problem was resolved (to degraded state) in 1hr15, and fully resolved in 2hr30. The Droplet Manager problem took much longer! This is the point of complex failure analysis, and why that school of thought says "root causing" is counterproductive. There will always be other precipitating events! † which itself could very well be a second-order effect of some even deeper and more latent issue that would be more useful to address!
      - 1970-01-01 1y ago
pelagicAustral 1y ago
Had no idea Dynamo was so intertwined with the whole AWS stack.
- freedomben 1y ago
  Yeah, for better or worse, AWS is a huge dogfooder. It's nice to know they trust their stuff enough to depend on it themselves, but it's also scary to know that the blast radius of a failure in any particular service can be enormous
ecnahc515 1y ago
Seems like the enactor should be checking the version/generation of the current record before it applies the new value, to ensure it never applies an old plan on top of an record updated by a new plan. It wouldn't be as efficient, but that's just how it is. It's a basic compare and swap operation, so it could be handled easily within dynamodb itself where these records are stored.
shrubble 1y ago
The Bind resolver required each zone to have an increasing serial number for the zone. So if you made a change you had to increase the number, usually a timestamp like 20250906114509 which would be older / lower numbered than 20250906114702; making it easier to determine which zone file had the newest data. Seems like they sort of had the same setup but with less rigidity in terms of refusing to load older files.
stefan_bobev 1y ago
I appreciate the details this went through, especially laying out the exact timelines of operations and how overlaying those timelines produces unexpected effects. One of my all time favourite bits about distributed systems comes from the (legendary) talk at GDC - I Shot You First[1] - where the speaker describes drawing sequence diagrams with tilted arrows to represent the flow of time and asking "Where is the lag?". This method has saved me many times, all throughout my career from making games, to livestream and VoD services to now fintech. Always account for the flow of time when doing a distributed operation - time's arrow always marches forward, your systems might not. But the stale read didn't scare me nearly as much as this quote: > Since this situation had no established operational recovery procedure, engineers took care in attempting to resolve the issue with DWFM without causing further issues Everyone can make a distributed system mistake (these things are hard). But I did not expect something as core as the service managing the leases on the physical EC2 nodes to not have recovery procedure. Maybe I am reading too much into it, maybe what they meant was that they didn't have a recovery procedure for "this exact" set of circumstances, but it is a little worrying even if that were the case. EC2 is one of the original services in AWS. At this point I expect it to be so battle hardened that very few edge cases would not have been identified. It seems that the EC2 failure was more impactful in a way, as it cascaded to more and more services (like the NLB and Lambda) and took more time to fully recover. I'd be interested to know what gets put in place there to make it even more resilient. [1] https://youtu.be/h47zZrqjgLc?t=1587 https://youtu.be/h47zZrqjgLc?t=1587
- tptacek 1y ago
  It shouldn't scare you. It should spark recognition. This meta-failure-mode exists in every complex technological system. You should be, like, "ah, of course, that makes sense now". Latent failures are fractally prevalent and have combinatoric potential to cause catastrophic failures. Yes, this is a runbook they need to have, but we should all understand there are an unbounded number of other runbooks they'll need and won't have, too!
  - lazystar 1y ago
    the thing that scares me is that AI will never be able to diagnose an issue that it has never seen before. If there are no runbooks, there is no pattern recognition. this is something Ive been shouting about for 2 years now; hopefully this issue makes AWS leadership understand that current gen AI can never replace human engineering.
tptacek 1y ago
I'm a tedious broken record about this (among many other things) but if you haven't read this Richard Cook piece, I strongly recommend you stop reading this postmortem and go read Cook's piece first. It won't take you long. It's the single best piece of writing about this topic I have ever read and I think the piece of technical writing that has done the most to change my thinking: https://how.complexsystems.fail/ https://how.complexsystems.fail/ You can literally check off the things from Cook's piece that apply directly here. Also: when I wrote this comment, most of the thread was about root-causing the DNS thing that happened, which I don't think is the big story behind this outage. (Cook rejects the whole idea of a "root cause", and I'm pretty sure he's dead on right about why.)
- yabones 1y ago
  Another great lens to see this is "Normal Accidents" theory, where the argument is made that the most dangerous systems are ones where components are very tightly coupled, interactions are complex and uncontrollable, and consequences of failure are serious. https://en.wikipedia.org/wiki/Normal_Accidents https://en.wikipedia.org/wiki/Normal_Accidents
- dosnem 1y ago
  How does knowing this help you avoid these problems? It doesn’t seem to provide any guidance on what to do in the face of complex systems
  - tptacek 1y ago
    He's literally writing about Three Mile Island. He doesn't have anything to tell you about what concurrency primitives to use for your distributed DNS management system. But: given finite resources, should you respond to this incident by auditing your DNS management systems (or all your systems) for race conditions? Or should you instead figure out how to make the Droplet Manager survive (in some degraded state) a partition from DynamoDB without entering congestive collapse? Is the right response an identification of the "most faulty components" and a project plan to improve them? Or is it closing the human expertise/process gap that prevented them from throttling DWFM for 4.5 hours? Cook isn't telling you how to solve problems; he's asking you to change how you think about problems, so you don't rathole in obvious local extrema instead of being guided by the bigger picture.
__turbobrew__ 1y ago
From a meta analysis level: bugs will always happen, formal verification is hard, and sometimes it just takes a number of years to have some bad luck (I have hit bugs which were over 10 years old but due to low probability of them occurring they didn’t happen for a long time). If we assume that the system will fail, I think the logical thing to think about is how to limit the effects of that failure. In practice this means cell based architecture, phased rollouts, and isolated zones. To my knowledge AWS does attempt to implement cell based architecture, but there are some cross region dependencies specifically with us-east-1 due to legacy. The real long term fix for this is designing regions to be independent of each other. This is a hard thing to do, but it is possible. I have personally been involved in disaster testing where a region was purposely firewalled off from the rest of the infrastructure. You find out very quick where those cross region dependencies lie, and many of them are in unexpected places. Usually this work is not done due to lack of upper level VP support and funding, and it is easier to stick your head in the sand and hope bad things don’t happen. The strongest supporters of this work are going to be the share holders who are in it for the long run. If the company goes poof due to improper disaster testing, the shareholders are going to be the main bag holders. Making the board aware of the risks and the estimated probability of fundamentally company ending events can help get this work funded.
bithavoc 1y ago
does DynamoDB run on EC2? if I read it right, EC2 depends on DynamoDB.
- dokument 1y ago
  There are circular dependencies within AWS, but also systems to account for this (especially for cold starting). Also there really is no one AWS, each region is its own (Now more then ever before, where some systems weren't built to support this).
Velocifyer 1y ago
This is unreadable and terribly formatted.
- citizenpaul 1y ago
  Yeah for real thats what an "industry leading" company puts out for their post mortem? They should be red in the face embarrassed. Jeeze, paragraphs? Punctuation? Looks like Amazon is starting to show cracks in the foundation.
- citizenpaul 1y ago
  Yeah for real thats what an "industry leading" company puts out for their post mortem? They should be red in the face embarrassed. Jeeze paragraphs? Punctuation? I put more effort into my internet comments that won't be read by millions of people.
dilyevsky 1y ago
Sounds like they went with Availability over Correctness with this design but the problem is that if your core foundational config is not correct you get no availability either.
alexnewman 1y ago
Is it the internal dynamodb that other people use?
deleted 1y ago
[deleted]
827a 1y ago
I made it about ten lines into this before realizing that, against all odds, I wasn't reading a postmortem, I was reading marketing material designed to sell AWS. > Many of the largest AWS services rely extensively on DNS to provide seamless scale, fault isolation and recovery, low latency, and locality...
- Aeolun 1y ago
  I didn’t get 10 lines in before I realized that this wall of text couldn’t possibly contain the actual reason. Somewhere behind all of that is an engineer saying “We done borked up and deleted the dynamodb DNS records”
danpalmer 1y ago
776 word paragraph and 28 word screen width, this is practically unreadable.
- scottatron 1y ago
  yeah that is some pretty reader hostile formatting -_- I asked Claude to reformat it for readability for me: https://claude.ai/public/artifacts/958c4039-d2f1-45eb-9dfe-bee60154ca27 https://claude.ai/public/artifacts/958c4039-d2f1-45eb-9dfe-b... Obvs do your own cross-checking with the original if 100% accuracy is required.
al_be_back 1y ago
Postmortem all you want - the internet is breaking, hard. The internet was born out of the need for Distributed networks during the cold war - to reduce central points of failure - a hedging mechanism if you will. Now it has consolidated into ever smaller mono nets. A simple mistake in on one deployment could bring banking, shopping and travel to a halt globally. This can only get much worse when cyber warfare gets involved. Personally, I think the cloud metaphor has overstretched and has long burst. For R&D, early stage start-ups and occasional/seasonal computing, cloud works perfectly (similar to how time-sharing systems used to work). For well established/growth businesses and gov, you better become self-reliant and tech independent: own physical servers + own cloud + own essential services (db, messaging, payment). There's no shortage of affordable tech, know-how or workforce.
- anyonecancode 1y ago
  > The internet was born out of the need for Distributed networks during the cold war - to reduce central points of failure - a hedging mechanism if you will. I don't think the idea was that in the event of catastrophe, up to and including nuclear attack, the system would continue working normally, but that it would keep working. And the internet -- as a system -- certainly kept working during this AWS outage. In a degraded state, yes, but it was working, and recovered. I'm more concerned with the way the early public internet promised a different kind of decentralization -- of economics, power, and ideas -- and how _that_ has become heavily centralized. In which case, AWS, and Amazon, indeed do make a good example. The internet, as a system, is certainly working today, but arguably in a degraded state.
  - al_be_back 1y ago
    preventing a catastrophe was ARPA's mitigation strategy. the point is where it's heading, not where it is. It's not about AWS per se, or any one company, it's the way it is consolidating. AWS came about by accident - cleverly utilizing spare server capacity from amazon.com. In it's conception, the internet (not www), was not envisaged as a economical medium - it's success was a lovely side-effect.
- protocolture 1y ago
rr808 1y ago
https://newsletter.pragmaticengineer.com/p/what-caused-the-large-aws-outage https://newsletter.pragmaticengineer.com/p/what-caused-the-l... has a better explanation instead of the wall of text from AWS
- cowsandmilk 1y ago
  Except just in the DNS section, I’ve already found one place where he gets it wrong…
JCM9 1y ago
Good to see a detailed summary. The frustration from a customer perspective is that AWS continues to have these cross-region issues and they continue to be very secretive about where these single points of failure exist. The region model is a lot less robust if core things in other regions require US-East-1 to operate. This has been an issue in previous outages and appears to have struck again this week. It is what it is, but AWS consistently oversells the robustness of regions as fully separate when events like Monday reveal they’re really not.
- Arainach 1y ago
  >about where these single points of failure exist. In general, when you find one you work to fix it, and one of the most common ways to find more is when one of them fails. Having single points of failure and letting them live isn't the standard practice at this scale.
grogers 1y ago
> As this plan was deleted, all IP addresses for the regional endpoint were immediately removed. I feel like I am missing something here... They make it sound like the DNS enactor basically diffs the current state of DNS with the desired state, and then submits the adds/deletes needed to make the DNS go to the desired state. With the racing writers, wouldn't that have just made the DNS go back to an older state? Why did it remove all the IPs entirely?
- Aeolun 1y ago
  1. Read state, oh, I need to delete all this. 2. Read state, oh, I need to write all this. 2. Writes 1. Deletes Or some variant of that anyway. It happens in any system that has concurrent reader/writers and no locks.
martythemaniak 1y ago
It's not DNS There's no way it's DNS It was DNS
giamma 1y ago
Interesting analysis from The Register https://www.theregister.com/2025/10/20/aws_outage_amazon_brain_drain_corey_quinn/ https://www.theregister.com/2025/10/20/aws_outage_amazon_bra...
- mellosouls 1y ago
  Discussed the other day: Today is when the Amazon brain drain sent AWS down the spout (644 comments) https://news.ycombinator.com/item?id=45649178 https://news.ycombinator.com/item?id=45649178
baalimago 1y ago
Did they intentionally make it dense and complicated to discourage anyone from actually reading it..? 776 words in a single paragraph
- valbaca 1y ago
  Amazon writing “culture”
polyglotfacto2 1y ago
Use TLA+ (which I thought they did)
deleted 1y ago
[deleted]
JohnMakin 1y ago
Was still seeing SQS latency affecting my systems a full day after they gave the “all clear.” There are red flags all over this summary to me, particularly the case where they had no operational procedure for recovery. That seems to me impossible in a hyperscaler - you never considered this failure scenario, ever? Or did you lose engineers that did know? Anyway appreciate that this seems pretty honest and descriptive.
asim 1y ago
On the one hand it's an incentive to shift away to smaller self management where you don't need AWS e.g as an individual I just run a single DigitalOcean VPS. But on the other hand if you're a large business the evaluation process is basically, can I tolerate this kind of incident once in a while versus the massive operational cost of doing it myself. It's really going to be a case by case study of who stays, who moves and who tries some multicloud failover. It's not one of those situations where you can blanket just say oh this is terrible, stupid, should never happen, let's get off AWS. This is the slow build up of dependency on something people value. That's not going to change quickly. It might never change. The too big to fail mantra of banks applies. What happens next is essentially very anticlimactic which is to say, nothing.
- tonymet 1y ago
  multi-region AWS would have been adequate for this outage.

551 points by meetpateltech 1y ago

Recent and related: AWS multiple services outage in us-east-1 - https://news.ycombinator.com/item?id=45640838 https://news.ycombinator.com/item?id=45640838 (2045 comments)

deleted 1y ago

[deleted]

deleted 1y ago

[deleted]

joeyhage 1y ago

> as is the case with the recently launched IPv6 endpoint and the public regional endpoint It isn't explicitly stated in the RCA but it is likely these new endpoints were the straw that broke the camel's back for the DynamoDB load balancer DNS automation

yla92 1y ago

So the root cause is basically race condition 101 stale read ?

philipwhiuk 1y ago
Race condition and bad data validation.

galaxy01 1y ago

Would conditional read/write solve this? looks like some kind of stale read

lazystar 1y ago

> Since this situation had no established operational recovery procedure, engineers took care in attempting to resolve the issue with DWFM without causing further issues. interesting.

shayonj 1y ago

I was kinda surprised the lack of CAS on per-endpoint plan version or rejecting stale writes via 2PC or single-writer lease per endpoint like patterns. Definitely a painful one with good learnings and kudos to AWS for being so transparent and detailed :hugops:

donavanm 1y ago
See https://news.ycombinator.com/item?id=45681136 https://news.ycombinator.com/item?id=45681136. The actual DNS mutation API does, effectively, CAS. They had multiple unsynchronized writers who raced without logical constraints or ordering to teh changes. Without thinking much they _might_ have been able to implement something like a vector either through updating the zone serial or another "sentinel record" that was always used for ChangeRRSets affecting that label/zone; like a TXT record containing a serialized change set number or a "checksum" of the old + new state. Im guessing the "plans" aspect skipped that and they were just applying intended state, without trying serialize them. And last-write-wins, until it doesnt.
- cyberax 1y ago
  Oh, I can see it from here. AWS internally has a problem with things like task orchestration. I bet that the enactor can be rewritten as a goroutine/thread in the planner, with proper locking and ordering. But that's too complicated and results in more code. So they likely just used an SQS queue with consumers reading from it.

jasode 1y ago

So the DNS records if-stale-then-needs-update it was basically a variation of the "2 Hard Things In Computer Science - cache invalidation". Excerpt from the giant paragraph: >[...] Right before this event started, one DNS Enactor experienced unusually high delays needing to retry its update on several of the DNS endpoints. As it was slowly working through the endpoints, several other things were also happening. First, the DNS Planner continued to run and produced many newer generations of plans. Second, one of the other DNS Enactors then began applying one of the newer plans and rapidly progressed through all of the endpoints. The timing of these events triggered the latent race condition. When the second Enactor (applying the newest plan) completed its endpoint updates, it then invoked the plan clean-up process, which identifies plans that are significantly older than the one it just applied and deletes them. At the same time that this clean-up process was invoked, the first Enactor (which had been unusually delayed) applied its much older plan to the regional DDB endpoint, overwriting the newer plan. The check that was made at the start of the plan application process, which ensures that the plan is newer than the previously applied plan, was stale by this time due to the unusually high delays in Enactor processing. [...] It outlines some of the mechanics but some might think it still isn't a "Root Cause Analysis" because there's no satisfying explanation of _why_ there were "unusually high delays in Enactor processing". Hardware problem?!? Human error misconfiguration causing unintended delays in Enactor behavior?!? Either the previous sequence of events leading up to that is considered unimportant, or Amazon is still investigating what made Enactor behave in an unpredictable way.

mcmoor 1y ago
Also, I don't know if I missed it, but they don't establish anything to prevent outage if there's unusually high delay again?
- mattcrox 1y ago
  It’s at the end, they disabled the DDB DNS automations around this to fix before they re-enable them
  - deleted 1y ago
    [deleted]

gslin 1y ago

I believe a report with timezone not using UTC is a crime.

exogenousdata 1y ago
An epoch fail?
cheeze 1y ago
My guess is that PT was chosen to highlight the fact that this happened in the middle of the night for most of the responding ops folks. (I don't know anything here, just spitballing why that choice would be made)
- throitallaway 1y ago
  Their headquarters is in Seattle (Pacific Time.) But yeah, I hate time zones.
tguedes 1y ago
I think it makes sense in this instance. Because this occurred in us-east-1, the vast majority of affected customers are US based. For most people, it's easier to do the timezone conversion from PT than UTC.
- trenchpilgrim 1y ago
  us-east-1 is an exceptional Amazon region; it hosts many global services as well as services which are not yet available in other regions. Most AWS customers worldwide probably have an indirect dependency on us-east-1.
- thayne 1y ago
  But us-east-1 is in Eastern Time, so if you aren't going to use UTC, why not use that? I'm guessing PT was chosen because the people writing this report are in PT (where Amazon headquarters is).

alexband2206 1y ago

[dead]

ericpauley 1y ago

Interesting use of the phrase “Route53 transaction” for an operation that has no hard transactional guarantees. Especially given the lack of transactional updates are what caused the outage…

donavanm 1y ago
I think you misunderstnad the failure case. The ChangeResourceRecordSet is transactional (or was when I worked on the service) https://docs.aws.amazon.com/Route53/latest/APIReference/API_ChangeResourceRecordSets.html https://docs.aws.amazon.com/Route53/latest/APIReference/API_.... The fault was two different clients with divergent goal states: - one ("old") DNS Enactor experienced unusually high delays needing to retry its update on several of the DNS endpoints - the DNS Planner continued to run and produced many newer generations of plans [Ed: this is key: its producing "plans" of desired state, the does not include a complete transaction like a log or chain with previous state + mutations] - one of the other ("new") DNS Enactors then began applying one of the newer plans - then ("new") invoked the plan clean-up process, which identifies plans that are significantly older than the one it just applied and deletes them [Ed: the key race is implied here. The "old" Enactor is reading _current state_, which was the output of "new", and applying its desired "old" state on top. The discrepency is because apparently Planer and Enactor aren't working with a chain/vector clock/serialized change set numbers/etc] - At the same time the first ("old") Enactor ... applied its much older plan to the regional DDB endpoint, overwriting the newer plan. [Ed: and here is where "old" Enactor creates the valid ChangeRRSets call, replacing "new" with "old"] - The check that was made at the start of the plan application process, which ensures that the plan is newer than the previously applied plan, was stale by this time [Ed: Whoops!] - The second Enactor’s clean-up process then deleted this older plan because it was many generations older than the plan it had just applied. Ironically Route 53 does have strong transactions of API changes _and_ serializes them _and_ has closed loop observers to validate change sets globally on every dataplane host. So do other AWS services. And there are even some internal primitives for building replication or change set chains like this. But its also a PITA and takes a bunch of work and when it _does_ fail you end up with global deadlock and customers who are really grumpy that they dont see their DNS changes going in to effect.

everfrustrated 1y ago

>Services like DynamoDB maintain hundreds of thousands of DNS records to operate a very large heterogeneous fleet of load balancers in each Region Does that mean a DNS query for dynamodb.us-east-1.amazonaws.com can resolve to one of a hundred thousand IP address? That's insane! And also well beyond the limits of route53. I'm wondering if they're constantly updating route53 with a smaller subset of records and using a low ttl to somewhat work around this.

supriyo-biswas 1y ago
DNS-based CDNs are also effectively this: collect metrics from a datastore regarding system usage metrics, packet loss, latency etc and compute a table of viewer networks and preferred PoPs. Unfortunately hard documentation is difficult to provide but that’s how a CDN worked at a place I used to work for, there’s also another CDN[1] which talks about the same thing in fancier terms. [1] https://bunny.net/network/smartedge/ https://bunny.net/network/smartedge/
- donavanm 1y ago
  Akamai talked about it in the early 2000s. Facebook content folks had a decent paper describing the latency collection and realtime routing around 2011ish, something like “pinpoint” I want to say. Though as you say was industry practice before then.
thayne 1y ago
I haven't tested with dynamodb, but I once ran a loop of doing DNS lookups for s3, and I in a couple seconds I got hundreds of distinct ip addresses. And that was just for a single region, from a single source ip.
donavanm 1y ago
Some details, but yeah that's basically how all AWS DNS works. I think youre missing how labels, zones, and domains are related but distinct. And that R53 operates in resource record SETS. And there are affordances in the set relationships to build trees and logic for selecting an appropriate set (eg healthcheck, latency). > And also well beyond the limits of route53 Ipso facto, R53 can do this just fine. Where do you think all of your public EC2, ELB, RDS, API Gateway, etc etc records are managed and served?

LaserToy 1y ago

TLDR: A DNS automation bug removed all the IP addresses for the regional endpoints. The tooling that was supposed to help with recovery depends on the system it needed to recover. That’s a classic “we deleted prod” failure mode at AWS scale.

qrush 1y ago

Sounds like DynamoDB is going to continue to be a hard dependency for EC2, etc. I at least appreciate the transparency and hearing about their internal systems names.

offmycloud 1y ago
I think it's time for AWS to pull the curtain back a bit and release a JSON document that shows a list of all internal service dependencies for each AWS service.
- throitallaway 1y ago
  Would it matter? Would you base decisions on whether or not to use one of their products based on the dependency graph?
  - withinboredom 1y ago
    Yes.
    - bdangubic 1y ago
      if so, I hate to tell you this but you would not use AWS (or any other cloud provider)!
      - withinboredom 1y ago
        I don’t use AWS or any other cloud provider. I use bare metal since 2012. See, in 2012 (IIRC), one fateful day, we turned off our bare metal machines and went full AWS. That afternoon, AWS had its first major outage. Prior to that day, the owner could walk in and ask what we were doing about it. That day, all we could do was twiddle our thumbs or turn on a now outdated database replica. Surely AWS won’t be out for hours, right? Right? With bare metal, you might be out for hours, but you can quickly get back to a degraded state, no matter what happens. With AWS, you’re stuck with whatever they happen to fix first.

WaitWaitWha 1y ago

I gather, the root cause was a latent race condition in the DynamoDB DNS management system that allowed an outdated DNS plan to overwrite the current one, resulting in an empty DNS record for the regional endpoint. Correct?

tptacek 1y ago
I think you have to be careful with ideas like "the root cause". They underwent a metastable congestive collapse. A large component of the outage was them not having a runbook to safely recover an adequately performing state for their droplet manager service. The precipitating event was a race condition with the DynamoDB planner/enactor system. https://how.complexsystems.fail/ https://how.complexsystems.fail/
- 1970-01-01 1y ago
  Why can't a race condition bug be seen as the single root cause? Yes, there were other factors that accelerated collapse, but those are inherent to DNS, which is outside the scope of a summary.
  - tptacek 1y ago
    Because the DNS race condition is just one flaw in the system. The more important latent flaw† is probably the metastable failure mode for the droplet manager, which, when it loses connectivity to Dynamo, gradually itself loses connectivity with the Droplets, until a critical mass is hit where the Droplet manager has to be throttled and manually recovered. Importantly: the DNS problem was resolved (to degraded state) in 1hr15, and fully resolved in 2hr30. The Droplet Manager problem took much longer! This is the point of complex failure analysis, and why that school of thought says "root causing" is counterproductive. There will always be other precipitating events! † which itself could very well be a second-order effect of some even deeper and more latent issue that would be more useful to address!
    - 1970-01-01 1y ago

pelagicAustral 1y ago

Had no idea Dynamo was so intertwined with the whole AWS stack.

freedomben 1y ago
Yeah, for better or worse, AWS is a huge dogfooder. It's nice to know they trust their stuff enough to depend on it themselves, but it's also scary to know that the blast radius of a failure in any particular service can be enormous

ecnahc515 1y ago

Seems like the enactor should be checking the version/generation of the current record before it applies the new value, to ensure it never applies an old plan on top of an record updated by a new plan. It wouldn't be as efficient, but that's just how it is. It's a basic compare and swap operation, so it could be handled easily within dynamodb itself where these records are stored.

shrubble 1y ago

The Bind resolver required each zone to have an increasing serial number for the zone. So if you made a change you had to increase the number, usually a timestamp like 20250906114509 which would be older / lower numbered than 20250906114702; making it easier to determine which zone file had the newest data. Seems like they sort of had the same setup but with less rigidity in terms of refusing to load older files.

stefan_bobev 1y ago

I appreciate the details this went through, especially laying out the exact timelines of operations and how overlaying those timelines produces unexpected effects. One of my all time favourite bits about distributed systems comes from the (legendary) talk at GDC - I Shot You First[1] - where the speaker describes drawing sequence diagrams with tilted arrows to represent the flow of time and asking "Where is the lag?". This method has saved me many times, all throughout my career from making games, to livestream and VoD services to now fintech. Always account for the flow of time when doing a distributed operation - time's arrow always marches forward, your systems might not. But the stale read didn't scare me nearly as much as this quote: > Since this situation had no established operational recovery procedure, engineers took care in attempting to resolve the issue with DWFM without causing further issues Everyone can make a distributed system mistake (these things are hard). But I did not expect something as core as the service managing the leases on the physical EC2 nodes to not have recovery procedure. Maybe I am reading too much into it, maybe what they meant was that they didn't have a recovery procedure for "this exact" set of circumstances, but it is a little worrying even if that were the case. EC2 is one of the original services in AWS. At this point I expect it to be so battle hardened that very few edge cases would not have been identified. It seems that the EC2 failure was more impactful in a way, as it cascaded to more and more services (like the NLB and Lambda) and took more time to fully recover. I'd be interested to know what gets put in place there to make it even more resilient. [1] https://youtu.be/h47zZrqjgLc?t=1587 https://youtu.be/h47zZrqjgLc?t=1587

tptacek 1y ago
It shouldn't scare you. It should spark recognition. This meta-failure-mode exists in every complex technological system. You should be, like, "ah, of course, that makes sense now". Latent failures are fractally prevalent and have combinatoric potential to cause catastrophic failures. Yes, this is a runbook they need to have, but we should all understand there are an unbounded number of other runbooks they'll need and won't have, too!
- lazystar 1y ago
  the thing that scares me is that AI will never be able to diagnose an issue that it has never seen before. If there are no runbooks, there is no pattern recognition. this is something Ive been shouting about for 2 years now; hopefully this issue makes AWS leadership understand that current gen AI can never replace human engineering.

tptacek 1y ago

I'm a tedious broken record about this (among many other things) but if you haven't read this Richard Cook piece, I strongly recommend you stop reading this postmortem and go read Cook's piece first. It won't take you long. It's the single best piece of writing about this topic I have ever read and I think the piece of technical writing that has done the most to change my thinking: https://how.complexsystems.fail/ https://how.complexsystems.fail/ You can literally check off the things from Cook's piece that apply directly here. Also: when I wrote this comment, most of the thread was about root-causing the DNS thing that happened, which I don't think is the big story behind this outage. (Cook rejects the whole idea of a "root cause", and I'm pretty sure he's dead on right about why.)

yabones 1y ago
Another great lens to see this is "Normal Accidents" theory, where the argument is made that the most dangerous systems are ones where components are very tightly coupled, interactions are complex and uncontrollable, and consequences of failure are serious. https://en.wikipedia.org/wiki/Normal_Accidents https://en.wikipedia.org/wiki/Normal_Accidents
dosnem 1y ago
How does knowing this help you avoid these problems? It doesn’t seem to provide any guidance on what to do in the face of complex systems
- tptacek 1y ago
  He's literally writing about Three Mile Island. He doesn't have anything to tell you about what concurrency primitives to use for your distributed DNS management system. But: given finite resources, should you respond to this incident by auditing your DNS management systems (or all your systems) for race conditions? Or should you instead figure out how to make the Droplet Manager survive (in some degraded state) a partition from DynamoDB without entering congestive collapse? Is the right response an identification of the "most faulty components" and a project plan to improve them? Or is it closing the human expertise/process gap that prevented them from throttling DWFM for 4.5 hours? Cook isn't telling you how to solve problems; he's asking you to change how you think about problems, so you don't rathole in obvious local extrema instead of being guided by the bigger picture.

__turbobrew__ 1y ago

From a meta analysis level: bugs will always happen, formal verification is hard, and sometimes it just takes a number of years to have some bad luck (I have hit bugs which were over 10 years old but due to low probability of them occurring they didn’t happen for a long time). If we assume that the system will fail, I think the logical thing to think about is how to limit the effects of that failure. In practice this means cell based architecture, phased rollouts, and isolated zones. To my knowledge AWS does attempt to implement cell based architecture, but there are some cross region dependencies specifically with us-east-1 due to legacy. The real long term fix for this is designing regions to be independent of each other. This is a hard thing to do, but it is possible. I have personally been involved in disaster testing where a region was purposely firewalled off from the rest of the infrastructure. You find out very quick where those cross region dependencies lie, and many of them are in unexpected places. Usually this work is not done due to lack of upper level VP support and funding, and it is easier to stick your head in the sand and hope bad things don’t happen. The strongest supporters of this work are going to be the share holders who are in it for the long run. If the company goes poof due to improper disaster testing, the shareholders are going to be the main bag holders. Making the board aware of the risks and the estimated probability of fundamentally company ending events can help get this work funded.

bithavoc 1y ago

does DynamoDB run on EC2? if I read it right, EC2 depends on DynamoDB.

dokument 1y ago
There are circular dependencies within AWS, but also systems to account for this (especially for cold starting). Also there really is no one AWS, each region is its own (Now more then ever before, where some systems weren't built to support this).

Velocifyer 1y ago

This is unreadable and terribly formatted.

citizenpaul 1y ago
Yeah for real thats what an "industry leading" company puts out for their post mortem? They should be red in the face embarrassed. Jeeze, paragraphs? Punctuation? Looks like Amazon is starting to show cracks in the foundation.
citizenpaul 1y ago
Yeah for real thats what an "industry leading" company puts out for their post mortem? They should be red in the face embarrassed. Jeeze paragraphs? Punctuation? I put more effort into my internet comments that won't be read by millions of people.

dilyevsky 1y ago

Sounds like they went with Availability over Correctness with this design but the problem is that if your core foundational config is not correct you get no availability either.

alexnewman 1y ago

Is it the internal dynamodb that other people use?

deleted 1y ago

[deleted]

827a 1y ago

I made it about ten lines into this before realizing that, against all odds, I wasn't reading a postmortem, I was reading marketing material designed to sell AWS. > Many of the largest AWS services rely extensively on DNS to provide seamless scale, fault isolation and recovery, low latency, and locality...

Aeolun 1y ago
I didn’t get 10 lines in before I realized that this wall of text couldn’t possibly contain the actual reason. Somewhere behind all of that is an engineer saying “We done borked up and deleted the dynamodb DNS records”

danpalmer 1y ago

776 word paragraph and 28 word screen width, this is practically unreadable.

scottatron 1y ago
yeah that is some pretty reader hostile formatting -_- I asked Claude to reformat it for readability for me: https://claude.ai/public/artifacts/958c4039-d2f1-45eb-9dfe-bee60154ca27 https://claude.ai/public/artifacts/958c4039-d2f1-45eb-9dfe-b... Obvs do your own cross-checking with the original if 100% accuracy is required.

al_be_back 1y ago

Postmortem all you want - the internet is breaking, hard. The internet was born out of the need for Distributed networks during the cold war - to reduce central points of failure - a hedging mechanism if you will. Now it has consolidated into ever smaller mono nets. A simple mistake in on one deployment could bring banking, shopping and travel to a halt globally. This can only get much worse when cyber warfare gets involved. Personally, I think the cloud metaphor has overstretched and has long burst. For R&D, early stage start-ups and occasional/seasonal computing, cloud works perfectly (similar to how time-sharing systems used to work). For well established/growth businesses and gov, you better become self-reliant and tech independent: own physical servers + own cloud + own essential services (db, messaging, payment). There's no shortage of affordable tech, know-how or workforce.

anyonecancode 1y ago
> The internet was born out of the need for Distributed networks during the cold war - to reduce central points of failure - a hedging mechanism if you will. I don't think the idea was that in the event of catastrophe, up to and including nuclear attack, the system would continue working normally, but that it would keep working. And the internet -- as a system -- certainly kept working during this AWS outage. In a degraded state, yes, but it was working, and recovered. I'm more concerned with the way the early public internet promised a different kind of decentralization -- of economics, power, and ideas -- and how _that_ has become heavily centralized. In which case, AWS, and Amazon, indeed do make a good example. The internet, as a system, is certainly working today, but arguably in a degraded state.
- al_be_back 1y ago
  preventing a catastrophe was ARPA's mitigation strategy. the point is where it's heading, not where it is. It's not about AWS per se, or any one company, it's the way it is consolidating. AWS came about by accident - cleverly utilizing spare server capacity from amazon.com. In it's conception, the internet (not www), was not envisaged as a economical medium - it's success was a lovely side-effect.
protocolture 1y ago

rr808 1y ago

https://newsletter.pragmaticengineer.com/p/what-caused-the-large-aws-outage https://newsletter.pragmaticengineer.com/p/what-caused-the-l... has a better explanation instead of the wall of text from AWS

cowsandmilk 1y ago
Except just in the DNS section, I’ve already found one place where he gets it wrong…

JCM9 1y ago

Good to see a detailed summary. The frustration from a customer perspective is that AWS continues to have these cross-region issues and they continue to be very secretive about where these single points of failure exist. The region model is a lot less robust if core things in other regions require US-East-1 to operate. This has been an issue in previous outages and appears to have struck again this week. It is what it is, but AWS consistently oversells the robustness of regions as fully separate when events like Monday reveal they’re really not.

Arainach 1y ago
>about where these single points of failure exist. In general, when you find one you work to fix it, and one of the most common ways to find more is when one of them fails. Having single points of failure and letting them live isn't the standard practice at this scale.

grogers 1y ago

> As this plan was deleted, all IP addresses for the regional endpoint were immediately removed. I feel like I am missing something here... They make it sound like the DNS enactor basically diffs the current state of DNS with the desired state, and then submits the adds/deletes needed to make the DNS go to the desired state. With the racing writers, wouldn't that have just made the DNS go back to an older state? Why did it remove all the IPs entirely?

Aeolun 1y ago
1. Read state, oh, I need to delete all this. 2. Read state, oh, I need to write all this. 2. Writes 1. Deletes Or some variant of that anyway. It happens in any system that has concurrent reader/writers and no locks.

martythemaniak 1y ago

It's not DNS There's no way it's DNS It was DNS

giamma 1y ago

Interesting analysis from The Register https://www.theregister.com/2025/10/20/aws_outage_amazon_brain_drain_corey_quinn/ https://www.theregister.com/2025/10/20/aws_outage_amazon_bra...

mellosouls 1y ago
Discussed the other day: Today is when the Amazon brain drain sent AWS down the spout (644 comments) https://news.ycombinator.com/item?id=45649178 https://news.ycombinator.com/item?id=45649178

baalimago 1y ago

Did they intentionally make it dense and complicated to discourage anyone from actually reading it..? 776 words in a single paragraph

valbaca 1y ago
Amazon writing “culture”

polyglotfacto2 1y ago

Use TLA+ (which I thought they did)

deleted 1y ago

[deleted]

JohnMakin 1y ago

Was still seeing SQS latency affecting my systems a full day after they gave the “all clear.” There are red flags all over this summary to me, particularly the case where they had no operational procedure for recovery. That seems to me impossible in a hyperscaler - you never considered this failure scenario, ever? Or did you lose engineers that did know? Anyway appreciate that this seems pretty honest and descriptive.

asim 1y ago

On the one hand it's an incentive to shift away to smaller self management where you don't need AWS e.g as an individual I just run a single DigitalOcean VPS. But on the other hand if you're a large business the evaluation process is basically, can I tolerate this kind of incident once in a while versus the massive operational cost of doing it myself. It's really going to be a case by case study of who stays, who moves and who tries some multicloud failover. It's not one of those situations where you can blanket just say oh this is terrible, stupid, should never happen, let's get off AWS. This is the slow build up of dependency on something people value. That's not going to change quickly. It might never change. The too big to fail mantra of banks applies. What happens next is essentially very anticlimactic which is to say, nothing.

tonymet 1y ago
multi-region AWS would have been adequate for this outage.