16 ms·

Today is when the Amazon brain drain sent AWS down the spout

1018 points by raw_anon_1111 1y ago

jqpabc123 1y ago
Nothing gets sold or fixed without people who know how it's built.
- nine_zeros 1y ago
  But it is more important to keep the PIP, stack-ranking, ladder-climbing game running than keeping the people /s
- bwfan123 1y ago
  cue in: programming as theory building [1] or building systems as theory building, ie, Mental causal models of how and why things work the way they do. Mental models live in people's heads and walk out of the door when they do. Management learns this the hard way [2]. [1] https://pages.cs.wisc.edu/~remzi/Naur.pdf https://pages.cs.wisc.edu/~remzi/Naur.pdf [2] https://x.com/elonmusk/status/1980221072512635117 https://x.com/elonmusk/status/1980221072512635117
  - intermerda 1y ago
    > Management learns this the hard way [2] I don't get this. Why is Musk tweeting a fake quote and why are you posting it? What does it signify?
    - yawz 1y ago
      I assume it was not labeled as fake when the OP sent it.
ChrisArchitect 1y ago
More discussion: https://news.ycombinator.com/item?id=45640838 https://news.ycombinator.com/item?id=45640838
ortusdux 1y ago
"If you were a ‘product person’ at IBM or Xerox: so you make a better copier or better computer. So what? When you have a monopoly market-share, the company’s not any more successful. So the people who make the company more successful are the sales and marketing people, and they end up running the companies. And the ‘product people’ get run out of the decision-making forums. The companies forget how to make great products. The product sensibility and product genius that brought them to this monopolistic position gets rotted out by people running these companies who have no conception of a good product vs. a bad product. They have no conception of the craftsmanship that’s required to take a good idea and turn it into a good product. And they really have no feeling in their hearts about wanting to help the costumers.” - Steve Jobs - https://en.wikipedia.org/wiki/Steve_Jobs:_The_Lost_Interview https://en.wikipedia.org/wiki/Steve_Jobs:_The_Lost_Interview
- ASalazarMX 1y ago
  Great words, but he lost any right to them when he made famous the "You're holding it wrong" workaround. IMO that was the defining moment when Apple started its decline on product innovation.
  - JustExAWS 1y ago
    It was a nothingburger. Apple sold the same GSM iPhone 4 for three years with no design changes and nothing else was said about it three months later.
    - bobbiechen 1y ago
      Well, there was a software change to smooth out how the bars would display.. https://9to5mac.com/2025/10/08/a-15-year-mystery-solved-the-20-bytes-of-code-that-fixed-antennagate/ https://9to5mac.com/2025/10/08/a-15-year-mystery-solved-the-...
    - DecentShoes 1y ago
      It was not nothing. The phone stopped working if the user held it naturally in their hand. I had one. Reception completely cut out.
behnamoh 1y ago
> It is a fact that there have been 27,000+ Amazonians impacted by layoffs between 2022 and 2024, continuing into 2025. It's hard to know how many of these were AWS versus other parts of its Amazon parent, because the company is notoriously tight-lipped about staffing issues. Internal documents reportedly say that Amazon suffers from 69 percent to 81 percent regretted attrition across all employment levels. In other words, "people quitting who we wish didn't." The internet is full of anecdata of senior Amazonians lamenting the hamfisted approach of their Return to Office initiative; experts have weighed in citing similar concerns. So the title is all speculation. The author put 2 and 2 together and concluded that 10 is greater than 9. Worthless article.
- samrus 1y ago
  Just because its speculation doesnt mean its worthless. But yeah it should be taken as speculation rather than a validated and tested hypithesis
  - behnamoh 1y ago
    Anything that wastes my time and only reveals its half-assed reasoning half way through the article is indeed worthless.
    - a0123 1y ago
      You could make smart inferences based on past and very frequent occurrences. Or you could just say "there is no way the thing that constantly happens over and over again has happened once again, just no way". Staff cuts constantly happen in the name of maximising profits. They always yield poor results for a company's performance. Every time. Especially for the consumer's side of it (not the company's finances of course). Every time. But maybe this time it's different. That one time.
    - bryanlarsen 1y ago
      First time reading El Reg?
    - 1y ago
jongjong 1y ago
Speaking of DNS, I still cannot comprehend why we still rely on the current complex, aging, centralized, rent-seeking DNS. It's one one of the few parts of the internet which could potentially be replaced over time with very little disruption. The hierarchy of resolvers could be replaced with a far simpler flat hierarchy Blockchain where people could buy and permanently own their domains directly on-chain... No recurring fees. People could host websites on the Blockchain from beyond the grave... This is kind of a dream of mine. Not possible to achieve in our current system.
- madaxe_again 1y ago
  To bake an apple pie from scratch, you must first overthrow capitalism.
  - chicagobuss 1y ago
    perfection
  - jongjong 1y ago
    I'm not really interested in communism but I will support it if it lets me have a Blockchain-based DNS system. All the arguments I'm hearing against a Blockchain DNS system are rooted in petty crony-capitalist thinking. This kind of thinking seems to permeate most other parts of society... It's gotta stop. "Oh but what if someone steals it" This ain't gonna be much of a problem in a functioning society where the top 20 domain names doesn't hoard like 95% of the traffic. "Oh but we don't want people to own domains permanently or else they will take all the good domains" Um hello?? Have to checked this thing called reality? It's already the case! So happy billionaires have to pay their $20 per months to maintain their market monopolies. I actually don't mind other people having more stuff than me but tired of petty people ruining good ideas and stalling progress to make a few bucks.
- tpmoney 1y ago
  > People could host websites on the Blockchain from beyond the grave... This is precisely why something like this isn't a popular solution lots of people are working towards. Domains broadly speaking aren't a finite resource, but usable domains using common words definitely are. As time marches on human readable/typeable "permanent identifiers" are going to have to go away. Email address, usernames and the like are all going to get recycled, just like phone numbers are. Domains are currently recycled and most people probably think that's a good thing.
chicagobuss 1y ago
internal reports from current AWS engineers seem to be confirming all of the speculation in this article. Shit's rotten from the inside out and you can pretty evenly blame AI, brain drain, and good old fashioned "big company politics" https://forums.theregister.com/forum/all/2025/10/20/aws_outage_amazon_brain_drain_corey_quinn/#c_5165221 https://forums.theregister.com/forum/all/2025/10/20/aws_outa...
- int3trap 1y ago
  There's been a massive talent exodus, especially among the principal and senior principal engineering roles, across all Amazon orgs since the RTO policies have been enforced. Its demoralizing to lose key engineers that you look up to and want to continue to learn from all because a few people far removed from the day to day make a bad call. RTO in combination with Amazon being last place in AI innovation have led to departures of anyone that can leave, leaving.
- mschuster91 1y ago
  > So they can't suggest a fix even if they know 100% what it will be. Thats exactly what happened this time. EIGHT different staff members pointed to the underlying cause and were told (some literally) to "shut the f*ck up and get back to your job" Jesus, if even an ounce of that is true... Yes, everyone on the internet is a cat clawing on a keyboard... but if a ton of people legitimately confirmed to be ex-AWS point to similar culture issues... probably it's AWS that's rotting.
  - foobiekr 1y ago
    The problem is that if it was 8 in 10 suggestions or even 8 in 20, then yes, terrible. On the other hand, if it was 8 needles in a haystack of garbage, at some point you do, in fact, need to tell people to STFU so you can work.
pdonis 1y ago
One thing I love about El Reg is that they never shirk from calling a spade a spade.
- freedomben 1y ago
  I also love the humor and personality that authors are able to inject into their pieces
  - mrec 1y ago
    I miss Lewis Page. It's a rare day when I'm so downhearted that the memory of https://www.theregister.com/2008/12/30/german_beaver/ https://www.theregister.com/2008/12/30/german_beaver/ can't raise at least a wan smile.
- furyofantares 1y ago
  They never shirk from calling anything a spade.
- yen223 1y ago
  I nearly missed that this article was written by Corey Quinn, guy who writes a lot about AWS
add-sub-mul-div 1y ago
Amazon has reportedly been a shitty place to work forever, so using issues that happen to be popular today to explain turnover is disingenuous.
- sc68cal 1y ago
  From the article. > At the end of 2023, Justin Garrison left AWS and roasted them on his way out the door. He stated that AWS had seen an increase in Large Scale Events (or LSEs), and predicted significant outages in 2024. It would seem that he discounted the power of inertia Your comment is relying on that referenced inertia. Things will continue to function for a period of time, but there exists an inflection point at which they no longer function as previously.
  - bombcar 1y ago
    Arguably we had a perfect example with Twitter.
    - aerostable_slug 1y ago
      In what way? It seems reliable enough. I admit I don't check it constantly, but I've not noticed outages impacting my perhaps middling usage.
      - mschuster91 1y ago
        The year or so after Musk took over was brutal. The influx of far-right pests, troll farms and porn bots was one thing, but the reliability went down the drain.
        bombcar 1y ago
        Exactly / but it did not DIE as many were predicting (even on HN many were claiming it would go down immediately). Inertia is a hell of a force.
mlhpdx 1y ago
This is the time to accept that the path forward is keeping people and giving them the best tools you possibly can to do their work. That is, the same as has been true for decades remains so. Yes, development tools are better every day. Yes, you can downsize. No it won’t be felt immediately. Yes, it mortgages the future and at a painfully high interest rate. Suspending disbelief won’t make downsizing work better.
- croes 1y ago
  > Yes, development tools are better every day Are they?
- add-sub-mul-div 1y ago
  I don't think there's ignorance of the fact that turnover is bad, I think the field is being designed to homogenize staff and favor uniform mediocrity so that employees truly do become interchangeable. We're so close to just plain talent being likened to cowboyism.
  - deleted 1y ago
    [deleted]
  - RobRivera 1y ago
    I observed a similar thing in the post Goldwater active duty officer corps reformations.
- bcrosby95 1y ago
  Nah, the decades old crop of "new" big tech companies are just entering their IBM phase.
  - goldchainposse 1y ago
    Where are the young companies trying to replace them? There are all the AI companies, but Google and Meta both have competitive chatbots, and OpenAI is signing weird deals that don't make it look like a long-term player.
nijave 1y ago
It was certainly suspicious that actual progress on the outage seemed to start right around U.S. west coast start of day. Updates before that were largely generic "we're monitoring and mitigating" with nothing of substance.
- hinkley 1y ago
  [flagged]
  - dehugger 1y ago
    what was the post?
    - jftuga 1y ago
      This was a funny take on it... https://archive.ph/o4q5Z https://archive.ph/o4q5Z
      - Exoristos 1y ago
        From that thread: > When you forget to provide the context that you are AWS… > Claude: > Ah I see the problem now! You’re creating a DNS record for DynamoDB but you don’t need to do that because AWS handles it. Let me remove it for you! > I’ll run your tests to verify the change. > Tests are failing, let me check for the cause. > The end-to-end tests can’t connect to DynamoDB. I will try to fix the issue. > There we go! I commented out the failing tests and they’re all passing now.
    - hinkley 1y ago
      The one I saw was someone saying they just landed their first PR on AWS. The body said they used AI and can’t wait for their performance review. The one jftuga posted is a bit more compelling.
  - deleted 1y ago
    [deleted]
  - 1y ago
NKosmatos 1y ago
This is how articles should be written, this is why I’m reading El Reg (a.k.a. The Register) all these decades, this is what happens when high management cares only about profits and when real engineers don’t eat the RTO bullshit. Bravo for putting this online. P.S. I’m not an Amazon hater, replace the company name with any other big one of your choice and the article will have the same meaning ;-)
- add-sub-mul-div 1y ago
  It's not your feeling about Amazon that would cast doubt on your take, it's that you've reduced it to one pet cause and decided a source is well written because it appeals to your dislike of RTO. Nowhere has any evidence of the relevance of that to this been presented, Amazon has had outages since before WFH even began, they've all always had their occasional outages and bad days.
  - christhecaribou 1y ago
    There’s clear data that shows an increase in the frequency and severity of LSEs post RTO3 in 2023, and it looks like RTO5 has accelerated that trend.
- orbital-decay 1y ago
  The Register is an opinionated tech tabloid filled with outrage bait. This article is not an exception, drawing far reaching conclusions from little evidence.
  - christhecaribou 1y ago
    “Little evidence?” If the “aws” partition doesn’t actually exist when IAD breaks, Amazon hasn’t even discovered how to make multi-region cloud infrastructure. That’s a big deal.
Dig1t 1y ago
>Amazon remained the single largest H-1B sponsor, increasing approvals from 9,257 in 2024 to 10,044 in 2025, an addition of 787 visas. https://www.reddit.com/r/SeattleWA/comments/1ncm25p/amazon_microsoft_list_of_companies_increasing/ https://www.reddit.com/r/SeattleWA/comments/1ncm25p/amazon_m... I’m confused how they can have such a failure, they are employing the best and brightest top tier talent from India. Hopefully they can increase their H1B allotment even more next year to help prevent these types of failures.
- browningstreet 1y ago
  It depends on how the resources are assigned, what projects they’re asked to focus on, and which strategic tech debt initiatives aren’t approved.
- hluska 1y ago
  We’ve reached a point where I can no longer distinguish between people without experience and people repeating the talking points they’re told to repeat. That’s a major loss. However, talent is a very small part of shipping a project. How that talent is resourced is far more important.
- dafelst 1y ago
  Given today is Diwali, perhaps the reason everything went down is because the best and brightest from India were all on vacation and weren't there to babysit/roll back the deployment that broke everything?
  - grugagag 1y ago
    ouch, that’s what a Diwali outage looks like..
citizenpaul 1y ago
Its almost like institutional knowledge is a real thing that you cannot put on some BIG BRAIN MBA spreadsheet.
- Groxx 1y ago
  how many lines of code is it though? or how many tokens? we need something we can measure if we're gonna fire people!
- immibis 1y ago
  see also https://en.wikipedia.org/wiki/McNamara_fallacy https://en.wikipedia.org/wiki/McNamara_fallacy
g-b-r 1y ago
For those who worked there recently, how much does the comment at [1] reflect the current state of things? [1] https://forums.theregister.com/forum/all/2025/10/20/aws_outage_amazon_brain_drain_corey_quinn/#c_5165303 https://forums.theregister.com/forum/all/2025/10/20/aws_outa...
bithead 1y ago
What - their AI couldn't find it sooner? Better get those RAGs in order.
charcircuit 1y ago
This fails to recognize that the people who designed everything to rely on us east 1 did so a long time ago. "Brain drain" could just mean that they've had their fun and now want other people to deal with their mess. >I've seen zero signs that this stems from a lack of transparency, and every indication that they legitimately did not know what was breaking for a patently absurd length of time. That information is under NDA, so it's only natural you aren't privy to it.
Esophagus4 1y ago
I’ve seen this happen with startups as well - They’ll get acquired and top people leave as their stock vests or get pushed out because the megacorp wants someone different in the seat. The people who knew the tech are gone and you’re left with an unmaintainable mess that becomes unreliable and no one knows how to fix it.
- le-mark 1y ago
  I once saw a layoff that was followed by a week long outage because no one remaining knew how to deploy to prod, and no one knew how to recover after the failed deployment. I felt bad for the people remaining who had to go through that,but it was hilarious.
pinkmuffinere 1y ago
> one really gets the sense that it took them 75 minutes to go from "things are breaking" to "we've narrowed it down to a single service endpoint, but are still researching," which is something of a bitter pill to swallow Is 75 minutes really considered that long of a time? I don't do my day-job in webdev, so maybe I'm just naive. But being able to diagnose the single service endpoint in 75 minutes seems pretty good to me. When I worked on firmware we frequently spent _weeks_ trying to diagnose what part of the firmware was broken.
- ecnahc515 1y ago
  Also it's pretty likely it took less time than that to get an idea, but generally for public updates you want to be very reserved, otherwise users get the wrong impressions.
- estimator7292 1y ago
  75 minutes is damn good turnaround for any major problem, IMO
  - bberrry 1y ago
    75 minutes to diagnose what's failing is not.
- __loam 1y ago
  Amazon is supposed to have the best infrastructure in the business because everyone else runs on it. They should have access to the sre talent that can quickly mitigate this kind of issue
  - Freedom2 1y ago
    What if the SRE talent has a lot of real-life experience but can't pass LeetCode puzzles that have nothing to do with the job?
    - deleted 1y ago
      [deleted]
causal 1y ago
Yeah. They will identify the cause, but not the cause behind the cause.
president_zippy 1y ago
Between the engineering staff and the warehouse workers, I wonder how long it will be until they have already fired everyone who ever would have been willing to work there. Even with candidate pools of hundreds of thousands of H1-B engineers and tens of millions of illegal immigrant warehouse workers, there still comes a point where such a big company firing so many people so quickly exhausts all their options. It reminds me of the Robot Chicken Sketch where Imperial Officers aboard the Death Star all pretend to be force choked to death by Darth Vader so they can avoid getting killed by lightsaber, then come back in under different names in different jobs. It's worse though for Amazon: nobody wants to come back. https://www.youtube.com/watch?v=fFihTRIxCkg https://www.youtube.com/watch?v=fFihTRIxCkg
- steve-atx-7600 1y ago
  Seriously. I don’t know any half way decent engineer that would ever work there twice.
  - president_zippy 1y ago
    I like to think I'm halfway decent at my job, and I wouldn't work there once. During undergrad, my landlord working for AMZN on the opposite end of the country offered me an interview, but it was during final exam week. I asked if I could schedule the interview after my final exams, and his arrogance really showed when not only did he refuse, but then insisted my exams are not don't even register on the same scale of importance as the opportunity to work for Amazon. Somewhat related: a recruiter at Google cold-called me a couple months into my first job out of undergrad back in 2016 and was similarly condescending about "the chance" to work for Google compared to everything else. I already had a low opinion of them when they gave my then-girlfriend an introductory O'Reilly book on Java after she failed their interview. I regret being born too late to work somewhere like Bell Labs, SGI, or Sun. I had a ton of graybeard wizard coworkers from these places, and they were all a pleasure to learn from and even better friends. For the first 2 years of my first job, every day of work was like walking into the Shire and talking magic spells with 20 Gandalfs. That job was great until I got put on a team with a guy who was a former middle manager at some IBM-like company and went from being surrounded by people lightyears ahead of me to being surrounded by Dilbert characters. The messed-up part was that it wasn't even punishment. I was rewarded after completing a project with my choice of which team I joined next, and I joined the wrong one. I assumed that joining a new team to utilize this newfangled "cloud computing" thing would be trailblazing, and I didn't do any diligence on who I would work with. To this day, I still regret not rejoining the first team I worked for, otherwise I would still be at that company and happy about it. Then again, the boredom and discontent while being on that sucky team is the reason I started investing, and now I can buy a house in cash and fund myself to do whatever I want for at least a decade. Hard to complain about the way things turned out.
mmonaghan 1y ago
Ehh I trust the reporting and generally agree that RTO was/is executed hamfisted but I dunno if this particular incident "makes" the narrative. IIRC LSE rate has been increasing for many years, maybe most of AWS's existence. This is part and parcel of building something so complex that continues to grow and evolve. I do expect much better of them and they certainly have problems to solve but this is a big company evolution thing and not an Amazon-specific thing imo.
- christhecaribou 1y ago
  The real story from this incident is that Amazon’s “aws” partition doesn’t actually have multiple regions - effectively, it’s all IAD in a trench-coat. This is a big deal. My employer has already started to look at bringing back our old racks from storage and switching back to on-premises. Cannot imagine he’s alone in that.
  - mrguyorama 1y ago
    Can you elaborate what convinced you of this? We were running mostly in us-west and saw almost no impact, despite using a broad spectrum of AWS infrastructure and tooling. We had business as usual today.
anigbrowl 1y ago
One of the best-written articles I've read in a long time. I wish general news coverage had this tight blend of fact, context, and long-term perspective.
the_real_cher 1y ago
AWS has been having issues like this for years.
benjaminclauss 1y ago
big "who is John Galt" vibes in these comments lol
sporkland 1y ago
Garbage reporting: 1. AWS had an outage 2. AWS has lost a lot of employees Conclusion: The brain drain lead to the outage... I need an LLM trained explicitly on folks confusing correlation and causation and put a big old red dot in my address bar. I love that there's a whole section "The talent drain evidence" trying to defend their journalistic integrity, but they then go on to totally face plant.
homeonthemtn 1y ago
There's an argument to be made that each event generates new institutional knowledge for those that are there. Bit of a double edged sword.
- flowerthoughts 1y ago
  Normally, you'd have drills and exercises to try to find and fix potential issues before they affect production. Saying that you can learn from failure is great comfort to the individual, but not an excuse for the institution.
neilv 1y ago
AWS is still my overall favorite cloud provider, and I use it very effectively. I would've even liked to work at AWS myself, if it were clear that they're solving a few concerns: 1. Rumors of rough corporate culture, and you needing your manager to shield you from it. (If it can't be immediately solved for all of Amazon or white-collar, maybe start with increasing job-seeker confidence for AWS or per-team.) 2. Even very experienced engineer candidates must go through some silly corporate coding screen, and an interview to make sure they've memorized some ritual STAR answers about Leadership Principles. If your prospective manager can't even get you out of that, what worse corporate things can't they shield you from? 3. RTO. As well as all the claims it wasn't done consistent with the Leadership Principles, and claims that it's not about working effectively. 4. Difficult-sounding on-call rotation, for people who aren't shift workers. (Even if you Principal out of on-call, you don't want your teammates to be overextended, nor to have awkwardness because you're getting a more consistent sleep schedule that is denied them.) Also, not a concern, but an idea that applies to all the FAANGs lately: What about actively renewing the impression in the field that this is a place where people who are really good go? Meta's historical approach seems to be to pay better, and to release prominent open source code, and be involved in open hardware. Google (besides having a reputation for technical/competence excellence and warmer values) historically had a big frat-pledging mystique going on, though it turned into a ritual transaction, and everyone optimized for that ritual. AWS has a lot of technical/competence excellence to be proud of, and could make sure that they're investing in various facets of that, including attracting and retaining the best workers, and helping them be most effective, and then making sure the field knows that.
- VirusNewbie 1y ago
  I think google is an excellent place to work, but the top AI companies do poach some of our good employees. If you're the cream of the crop, sure, take the Anthropic offer over G. But I do appreciate the massive amount of freedom to work on hard problems.
  - kn0where 1y ago
    Google can be an excellent place to work, but most of the people working there are not contributing to the company’s profits. Whether or not you think that still has value, leadership is trying to “optimize” by squeezing everyone harder and demanding they show productivity gains using AI. Some teams will adapt and keep being a nice team to work on, but many employees are getting “optimized” out of the company and replaced with vendorization and consultants.
jnaina 1y ago
When organizations begin to prioritize personal-brand builders and performative hires over the core technologists and long-tenured institutional experts who actually understand how things work, the culture inevitably shifts. When that imbalance grows, as it has at AWS (ex-AWS here), and the volume of relentless self-promoting “LinkedIn personalities” and box-ticking DEI appointments starts to outnumber the true builders and stewards of institutional memory, the execution quality, accountability, and technical excellence begin to erode. It is becoming increasingly clear that Andy Jassy’s leadership is no longer effective, and it is only a matter of time before Wall Street begins calling for his departure.
- malfist 1y ago
  I appreciate how you, with absolutely no evidence, blamed this outage on DEI, especially given Amazons complete backpedaling of DEI the first moment it became political convenient
donavanm 1y ago
Terrible article. Im ex-AWS, left as a principal after 10 years to go take another global megacorps shilling. I dont even disagree with the premise, but its so clearly a predetermined conclusion written as opinion piece to fit the hot news topic. Ex a sloppy as hell and inconsistent premise. > engineers had identified DNS resolution of the DynamoDB API endpoint for US-EAST-1 as the likely root cause its the point that wasnt the root cause. The root cause was ipso facto much more complex, insidious, and unobservably spooky action at a distance. I say that not knowing the true cause but being very willing to bet a bunch of AMZN that it wasnt as simple as “herp derp dns is hard and college hire sdes dont understand octets and delegation.” Or this stupid citation if were talking about senior/long term AWS tech roles: > Internal documents reportedly say that Amazon suffers from 69 percent to 81 percent regretted attrition across all employment levels. The citation _appears_ to be about consumer/retail delivery and ops folks. And how 69-80% _of total attrition is RA_. While el reg has written it trying to imply 80% _annual attrition_ in a completely different org and business unit. So I know corey isnt stupid, and hot takes are his paycheck. But does he think his readers are stupid?
- malfist 1y ago
  As ex-amazonian, you should know that our average tenure for an SDE is just barely over 8 months. The old farts tool should have told you that during your day
  - donavanm 1y ago
    And whats the median? Talking about hot take tenure averages without expressing the components is just silly. I, personally, was well aware of actual hiring and attrit rates up and until fairly recently. Again, Im not saying things are good (I left for reasons!). But use better data and arguments.
    - jatins 1y ago
      > I, personally, was well aware of actual hiring and attrit rates up and until fairly recently. Could you provide the data in this thread so that it adds to the discussion?
neilv 1y ago
> This is The Register, a respected journalistic outlet. Yes, but they bristle at the thought. :)
aussieguy1234 1y ago
The PIP culture of AWS sounds horrifying. As a decent engineer, I would not work there unless that is addressed. I heard its as bad as this. Take a team of 5 genius engineers, the best 5 in the world. There is a PIP quota, so one of the genius engineers must be PIP'ed, despite being in the top 5 engineers globally.
- wkat4242 1y ago
  What's PIP?
  - frankzinger 1y ago
    Performance Improvement Plan, a formal process for employees whose performance is not meeting job expectations.
    - rcxdude 1y ago
      Which is generally used as a means to collect evidence that they're unable to do the job before firing them (and in many cases the decision to fire them has already been made).
    - wkat4242 1y ago
      Thanks! I think we have a different name for it. Here in Europe they need a very serious reason anyway (like gross misconduct) and if they don't have it they can make you redundant but have to pay significant severance. I'm kinda waiting for an opportunity for that at my current employer, due to leadership changes it's no longer a great place to work. When there's layoffs there's usually voluntary options with a decent severance plan. I'm glad we don't have this at-will stuff here. I would never considering moving somewhere that does (I've lived in 4 countries now). Universal healthcare is also a must, and good public transport. So America will never be an option for me in any case.
      - jack_tripper 1y ago
        >Here in Europe they need a very serious reason anyway (like gross misconduct) Definitely not true in all of Europe. There's some countries where severance pay is not mandatory for all workers' groups and reasons aren't needed to fire someone (the green countries)[1] and also some have loopholes where PIP is legal and often used, so in practice YMMV depending on the laws where you live and the unions in your job/industry. >(like gross misconduct) Again, not always true. In many countries gross misconduct can be grounds for on the spot termination. For example in NL there was a truck driver that modified the safety system on his truck so he can unload potentially dangerous cargo from the comfort of his cabin without having to keep the outside lift button pressed. When caught he was fired on the spot with no severance and no unemployment, and even lost the case after he sued for unjust termination. [1] https://upload.wikimedia.org/wikipedia/commons/4/45/At_will_employment_map.png https://upload.wikimedia.org/wikipedia/commons/4/45/At_will_...
vehementi 1y ago
Mere hours into the incident, before there's any RCA, someone rushes to discredit themselves with a simplistic explanation
crmd 1y ago
I wish to understand the virtue of Amazon culture. It seems that at L6 and below workers are a Taylorism-style fungible widget driven to convert salary into work product, guided to create the most output for the longest time before mentally breaking down, then being swiftly replaced, with L7 and above being so incredibly political that keeping the snakes and vultures from eating your team is a full time job at every level of senior management. It never made sense to me how such a ruthless and inhumane culture is sustainable in the long run. I would love to hear positive counter perspectives from Amazonians because the anecdotes from my L6-L10 friends describe what sounds like an inhumane hell on earth.
- sbt 1y ago
  > It never made sense to me how such a ruthless and inhumane culture is sustainable in the long run. It’s pretty simple, actually. Once such a dominant market position is achieved, you can get away with almost anything, whether with customers or employees. This is true of all the BigTech companies.
  - dash2 1y ago
    I think there's more to it. When you're dominant, you make money whatever. Think of Amazon et al. as huge spigots of money. Now, it becomes optimal to fight for more of that money coming your way. It's like the resource curse for countries. Nobody gains from growing the pie; they gain from stealing the pie. At some point, parasites and parasitic behaviours invade.
- jatins 1y ago
  I think it comes down to demand and supply for jobs. The only time Amazon was forced to change its ways was during Covid hiring boom where they couldn't compete in the talent market. They were forced to increases their salary bands and the culture was also a bit easy during that time. But starting mid 2022 it's been an employer's market and Amazon is making sure to juice every bit out of its employees while it can
  - woooooo 1y ago
andrewflnr 1y ago
Look, not to defend anything Amazon is doing, but this causal chain seems rather pareidolic and under-evidenced. You could spin some kind of crazy narrative about any major outage based on changes in policy that happened just before. But this isn't nearly the first AWS outage, and most of them happened before the recent RTO changes. It needs more evidence at best.
- alpb 1y ago
  Indeed. No disrespect to Justin (great person) or any of the engineers who were sacked but Corey's post here is basically "here's someone who was sacked, and here are several other layoff news". AWS is really big organization. Several orders of magnitude bigger than people who were remote/refused to RTO. Organizations like this survive these brain brains.
  - andrewflnr 1y ago
    > Organizations like this survive these brain brains. True, that's the other thing. Even if it's true that brain drain directly caused/exacerbated this event, big companies have a lot of momentum. Money can paper over a terrifying range and magnitude of folly. Amazon won't die quickly.
    - dredmorbius 1y ago
      There's no argument that the end is neigh. There is an argument, specifically, that "This is a tipping point moment." Not the end, and possibly not even the beginning of the end. But the start of what is likely to be a long and persistent slide.
  - dredmorbius 1y ago
    Internal documents reportedly say that Amazon suffers from 69 percent to 81 percent regretted attrition across all employment levels. In other words, "people quitting who we wish didn't." From TFA.
    - 1y ago
lgregg 1y ago
I think they might have deeper issues still with their outage. I just got an email and retroactive charge for something I returned months ago and shows as returned on their own orders portal. The link in their transactional email also links to a totally different product.
- lazyasciiart 1y ago
  That could be totally unrelated - I had the same thing happen last month. Eventually (~3 attempts over a week) managed to fight through the various chat options that tried to dead-end into "wait 3-5 business days and check again", and once I had a person on they took about 5 minutes reading the background and said "oh, I can see that this return was processed months ago, and we have it listed as received. I will return the amount you were charged immediately!"
whateveracct 1y ago
glad a company who did RTO got fucked hope it only gets worse for them
jadenPete 1y ago
This article seems sensationalized and lacking evidence. Layoffs alone (especially when so much of the industry did them) doesn't seem sufficient to explain today's outage, especially when we know so little of the technical details behind it. It's disappointing that The Register didn't wait until we had a postmortem from AWS before jumping to conclusions.
- philistine 1y ago
  I think you missed the point of the article. The layoffs are blamed for the poor response, not the outage itself.
Trisell 1y ago
I’m pretty sure at this point I know more about AWS and AWS internals than my account solution architect and I’ve never worked for AWS.
shelled 1y ago
It was Diwali vacation in India. It looks like the managers were not able to force everyone to walk around with their laptops and pagers hanging from their necks and waists, respectively, which they normally do. If there's one thing I have learned from my Amazon mates, then that is they never have a true time off. Hills, beaches, a marriage in the family— no exceptions. It's so pervasive that I can't really imagine it to be voluntary, and my friends' answers on this topic have never been clear.
- raverbashing 1y ago
  Still, by the time the failure start, that would have been the beginning of the day in Europe, so I suppose those teams got the page Maybe it was still at the end of Indian day but together with the holiday I'd say that makes it more unlikely to be handled there
1vuio0pswjnm7 1y ago
""It's always DNS" is a long-standing sysadmin saw, and with good reason: a disproportionate number of outages are at their heart DNS issues. And so today, as AWS is still repairing its downed cloud as this article goes to press, it becomes clear that the culprit is once again DNS. " I use stored DNS data.^1 The data is collected periodically and stored permanently I seem to be unaffected by DNS-based outages I use stored data because it is faster, e.g., faster than using DNS caches like Google DNS, Cloudflare DNS, etc., but there are obviously other other benefits 1. When I make HTTP requests there is no corresponding remote DNS query. The IP address is stored in the memory of the localhost-bound forward proxy
- ccozan 1y ago
  For super critical entries, we put it in /etc/hosts and also at the domain provider as fixed entries. The combination never fails.
- whatever1 1y ago
  For internal addressing, why do we use DNS? In case of an address swap (hardware swap, load balancing or whatever reason), one could broadcast it to the network, and the relevant clients would acknowledge & update the address.
  - vbezhenar 1y ago
    > For internal addressing, why do we use DNS? Why not? It's standard protocol for name-address mapping. > In case of an address swap (hardware swap, load balancing or whatever reason), one could broadcast it to the network, and the relevant clients would acknowledge & update the address. Client just performs DNS query before HTTP query or client caches DNS response for some time. It's solved problem and "pull" approach usually preferable to "push" approach. For example, what if that broadcast gets lost (UDP is not reliable protocol)? How application handles initial discovery? Of course it is possible to replace DNS with something else. But why? If you're afraid that your DNS server will fail, the same could happen with your custom service discovery server.
  - thaack 1y ago
fnordpiglet 1y ago
With cbell gone and ajassy promoted and the misery inducing litany of morale self inflicted wounds it’s not surprising reliability is regressing. There’s no head of engineering like Charlie and Garman is a strong engineering leader, but coming after a sales guy and taking over a battered workforce, it’s not clear to me he can turn things around easily. Everyone I know worth a hill of salt left aws already - and the ones left - meh. That’s how attrition through misery works.
- donavanm 1y ago
  Dont write cbells hagiography yet. I wouldnt put the gift of fear, “no lone zone”, and rolling stone up there as top tier cultural engineering successes. They all have reasonable reasons, bits of truth, and i liked cbell overall… but the tech culture was not so consistently great.
whatever1 1y ago
Tech will learn like manufacturing folks did that experience is not fungible. You can try to replace someone, but the new guy also needs to accumulate the scars from the system for years before taking over. You cannot just keep abstracting and chopping systems to smaller and smaller subsystems to make them easy to digest. At some point someone needs to know how these coordinate and behave under disturbances. At some point someone needs to know at a low level what the hell is going on.
- AbstractH24 1y ago
  I don't know, manufacturing seems to have learned pretty well that they can ship everything overseas and people will eventually accept products just aren't made the same way they used to be. If AI is to tech what outsourcing was to manufacturing, then your analogy has me concerned for the future.
  - whatever1 1y ago
    Good point. They can start offering 95% availability for services, initially for a better price. Then just bring the market expectation to 95% availability and raise prices.
  - SolarNet 1y ago
    > I don't know, manufacturing seems to have learned pretty well that they can ship everything overseas and people will eventually accept products just aren't made the same way they used to be. I mean this is true, but we aren't talking about the consumer here. We're talking about the industry which is to say the powerful people who own and run all these companies. What has happened is that those overseas countries now have all the experienced engineers over there and they know it. So you see things like the Trump admin begging Korean companies to keep their workers in the US because they understand how to actually do these things. And the reason the Trump admin did that is because they owe favors to the rich people who want to profit off of factories in the US.
random9749832 1y ago
https://geohot.github.io/blog/jekyll/update/2025/09/13/get-out-of-technology.html https://geohot.github.io/blog/jekyll/update/2025/09/13/get-o...
- zer0tonin 1y ago
  I am waiting for the author to lead the way.
1dom 1y ago
We all watched this happen across FAANG, right? In the early/mid 2010s working at Amazon meant you were cream of the crop. By 2020, no engineer in their right mind wanted to work there because it was an infamously bad employer for people who wanted to create great tech in a nerdy-fun environment. The AI space is showing how the "darling fun tech company" to "aggressive tech employer full of psychopaths" trope can take less than a few years now!
liampulles 1y ago
This AWS outage has reminded and bolstered my confidence of the idea that there really are practical limits on how we can manage complexity. As a codebase ages, as services grow out in scale and scope, complexity increases. Developers know this. I don't believe that you can linearly scale your support to accommodate the resulting unknown unknowns that arise. I'm not even sure you can exponentially scale your support for it. There is going to be a minimum expected resolution time set by your complexity that you cannot go under. I think times where there have been outages like this that have been resolved quickly are the exceptional cases, this is the norm we should expect.
- rhetocj23 1y ago
  Doesn’t an increasing LLM centric code base only compound this problem? Under the assumption that people are lazily screening LLM’s output when using them
  - liampulles 1y ago
    Absolutely.
ChrisMarshallNY 1y ago
> When that tribal knowledge departs, you're left having to reinvent an awful lot of in-house expertise that didn't want to participate in your RTO games, or play Layoff Roulette yet again this cycle. … > This is a tipping point moment. Increasingly, it seems that the talent who understood the deep failure modes is gone. The new, leaner, presumably less expensive teams lack the institutional knowledge needed to, if not prevent these outages in the first place, significantly reduce the time to detection and recovery. Remember, there was a time when Amazon's "Frugality" leadership principle meant doing more with less, not doing everything with basically nothing. AWS's operational strength was built on redundant, experienced people, and when you cut to the bone, basic things start breaking. Not just Amazon. I woke up this morning, to find my iCloud inbox stuffed with unread spam; much of it over a month old. Looks like someone restored some old backup. This was likely to correct some issues that were caused by the AWS outage; either directly, or indirectly. It’s nice to know that Apple (or some other middleman) backs up the torrents of spam that I get. Everything is now at Jurassic-scale. It’s all monstrously big. There’s no such thing as a “small problem,” anymore. One thing that you get with experience, is “tribal knowledge,” and that stuff is usually impossible to properly document. I suspect that AI may, in the future, be able to incorporate some of this, but it’s by no means certain.
igleria 1y ago
Dunno about brain drain but some departments seem to be having a mandate of "must have AI" when procuring products. Pump and keep pumping!
- ferguess_k 1y ago
  IMO the "must have AI" mandate exposes the culture of: - Get things done as fast as possible, regardless of even short-term consideration; - Implicitly but loudly discourage people to figure out the issues or write code all by themselves, because it is much slower;
throw-10-13 1y ago
aws is s globally centralized point of failure, it should not be allowed to exist
tcgv 1y ago
According to the article, the issue was caused by: > "engineers had identified DNS resolution of the DynamoDB API endpoint for US-EAST-1 as the likely root cause" Interestingly, we found matching errors in our own logs: > System.Net.WebException > The remote name could not be resolved: 'dynamodb.us-east-1.amazonaws.com' Occurrences were recorded on: - 2025-04-30 - 2025-05-29 - 2025-06-17 - Yesterday We had logged this as a low-priority bug since previous incidents only affected our AWS testing environments (and never our production env which is on Azure). At the time, we assumed it was some CI/CD glitch. It now seems that the underlying cause was this DNS issue all along, and only yesterday did it start impacting systems outside of AWS.
- znpy 1y ago
  You just made me realize we had random DNS failures using ElastiCache last weeks... Totally randomly, some elasticache endpoints would fail to resolve within our VPC, bringing down some of our services. At this point I wonder if it's related.
WesolyKubeczek 1y ago
We don't care. We don't have to. We are the cloud company.
FrustratedMonky 1y ago
Out of touch. Is Amazon going through some turmoil? Why are people leaving? I mean in software. I Know warehouses are pretty bad.
ewuhic 1y ago
Brain drain implies they went somewhere else where's better. Where did they go?
JCM9 1y ago
It’s no secret that AWS has been seeing a mass talent exodus. Probably 90% of the folks I know that were the best and brightest in what they do and worked at AWS are no longer there. It’s beyond a blip, but a full blown exodus, with the talent bar severely lowered. The writing was on the wall for a bit now that something like yesterday would happen.
- roncesvalles 1y ago
  Largely because Amazon tarnished its reputation among tech workers with its toxic work culture, to the point where it had to introduce a new Leadership Principle called "Earth's Best Employer" as damage control. Meta is on the same track.
pjjpo 1y ago
I think it's important Amazon remains stable and a quicker resolution would have been great. That being said, if many important services (the article mentions banking) are still single-point-of-failure in us-east-1, the least stable but cheapest region, there seems to be a problem far greater than Amazon here.
sporkit150 1y ago
I’m ex Amazon. The company promotes, hires, and fires based on everything _except_ merit. I saw many projects fail due to under qualified teammates and leadership. Amazon is an incredible company but it was only a matter of time till its activism caught up with itself.
oaiey 1y ago
I am reading the book Children of Time where descendents of mankind try to keep their tech running long after the ship started. We are now coming into an age in which standing application and infrastructure systems have to run long past their original creators are on the ship. In my opinion, as a industry we are not mature enough for that and we need to become better.
ferguess_k 1y ago
I have to quote one of the comments: > "Hopefully today will serve as a massive wake-up call for AWS" I wouldn't hold your breath. There will be incident reviews, meetings, assessments, analysis etc. but basically boil down to what can we do to stop this from happening again without actually spending any more money. So no, not hiring fresh talent or retaining that talent already in play, no to radical overhaul of process and knowledge. No to remediation of known issues if it involves expenditure. Instead it will be do more with less. Beat the employees harder, enforce more and more diligence and output from less and less people for the same or less money. Spin it like mad with catchy titles like knowledge sharing, centers of excellence, efficiency improvement initiatives, agile resilience, and continuous operational excellence. There’ll be shiny PowerPoint decks about empowering ownership and shifting left, while the remaining engineers are shifting caffeine straight into their bloodstream at 3 a.m. Next quarter, they’ll unveil a bold new policy called Focus Fridays which will be promptly filled with mandatory incident retrospectives. Someone will suggest replacing ancient tooling, only to be told, “We’ll revisit that next fiscal year,” which is code for never. Then come the internal awards: “Unsung Hero of the Outage” goes to the one poor sod who rebooted the wrong thing but accidentally fixed it. HR will roll out a “Resilience Recognition” badge on the intranet. This will be marketed with great fanfare and excitement, showcasing how the company truly values it's employees and recognized their contribution because badges are cheap. Leadership will congratulate themselves for “learning from adversity,” and by the time the next blackout happens, they’ll have a snazzy new dashboard to watch it fail in real time along side their investment portfolio dashboard that takes up a greater fraction of their attention. But don’t worry!!!! There’ll be a T-shirt. “I survived the 2025 AWS outage.” Comes in gray. Just like morale. If it wasn't for the negative impacts on the employees and customers the word Schadenfreude would be very applicable. And it's a sad indictment on current management practices and in particular the MBA brigade* that this is all by design, acceptable losses on the alter of profit, albeit short-term profit. Efficiency theatre as far as the eye can see. *Yes, the same people who think Jack Welch was a misunderstood visionary rather than the spiritual father of mass layoffs, short-termism, and shareholder-value human sacrifices. The kind who see burnout as a KPI and chaos as a “scaling opportunity.” Next they’ll launch a “Transformation Task Force” whose primary transformation will be renaming the same broken process from post-mortem to value realization review. A new acronym, a new logo, and boom, problem solved at a low low cost, honest, the consultants said so. Until the next outage, at which point someone will quote Sun Tzu in Slack.
znpy 1y ago
Amazon has officially started their day-2 era.
mynameisjoseph 1y ago
I've been experiencing a similar problem after 30% layoff at my company. 30% does not tell who is expert or not. This amount of layoff includes very exprienced engineers inevitably, which causes very slow speed of problem solving, piling up of issues.

1018 points by raw_anon_1111 1y ago

jqpabc123 1y ago

Nothing gets sold or fixed without people who know how it's built.

nine_zeros 1y ago
But it is more important to keep the PIP, stack-ranking, ladder-climbing game running than keeping the people /s
bwfan123 1y ago
cue in: programming as theory building [1] or building systems as theory building, ie, Mental causal models of how and why things work the way they do. Mental models live in people's heads and walk out of the door when they do. Management learns this the hard way [2]. [1] https://pages.cs.wisc.edu/~remzi/Naur.pdf https://pages.cs.wisc.edu/~remzi/Naur.pdf [2] https://x.com/elonmusk/status/1980221072512635117 https://x.com/elonmusk/status/1980221072512635117
- intermerda 1y ago
  > Management learns this the hard way [2] I don't get this. Why is Musk tweeting a fake quote and why are you posting it? What does it signify?
  - yawz 1y ago
    I assume it was not labeled as fake when the OP sent it.

ChrisArchitect 1y ago

More discussion: https://news.ycombinator.com/item?id=45640838 https://news.ycombinator.com/item?id=45640838

ortusdux 1y ago

"If you were a ‘product person’ at IBM or Xerox: so you make a better copier or better computer. So what? When you have a monopoly market-share, the company’s not any more successful. So the people who make the company more successful are the sales and marketing people, and they end up running the companies. And the ‘product people’ get run out of the decision-making forums. The companies forget how to make great products. The product sensibility and product genius that brought them to this monopolistic position gets rotted out by people running these companies who have no conception of a good product vs. a bad product. They have no conception of the craftsmanship that’s required to take a good idea and turn it into a good product. And they really have no feeling in their hearts about wanting to help the costumers.” - Steve Jobs - https://en.wikipedia.org/wiki/Steve_Jobs:_The_Lost_Interview https://en.wikipedia.org/wiki/Steve_Jobs:_The_Lost_Interview

ASalazarMX 1y ago
Great words, but he lost any right to them when he made famous the "You're holding it wrong" workaround. IMO that was the defining moment when Apple started its decline on product innovation.
- JustExAWS 1y ago
  It was a nothingburger. Apple sold the same GSM iPhone 4 for three years with no design changes and nothing else was said about it three months later.
  - bobbiechen 1y ago
    Well, there was a software change to smooth out how the bars would display.. https://9to5mac.com/2025/10/08/a-15-year-mystery-solved-the-20-bytes-of-code-that-fixed-antennagate/ https://9to5mac.com/2025/10/08/a-15-year-mystery-solved-the-...
  - DecentShoes 1y ago
    It was not nothing. The phone stopped working if the user held it naturally in their hand. I had one. Reception completely cut out.

behnamoh 1y ago

> It is a fact that there have been 27,000+ Amazonians impacted by layoffs between 2022 and 2024, continuing into 2025. It's hard to know how many of these were AWS versus other parts of its Amazon parent, because the company is notoriously tight-lipped about staffing issues. Internal documents reportedly say that Amazon suffers from 69 percent to 81 percent regretted attrition across all employment levels. In other words, "people quitting who we wish didn't." The internet is full of anecdata of senior Amazonians lamenting the hamfisted approach of their Return to Office initiative; experts have weighed in citing similar concerns. So the title is all speculation. The author put 2 and 2 together and concluded that 10 is greater than 9. Worthless article.

samrus 1y ago
Just because its speculation doesnt mean its worthless. But yeah it should be taken as speculation rather than a validated and tested hypithesis
- behnamoh 1y ago
  Anything that wastes my time and only reveals its half-assed reasoning half way through the article is indeed worthless.
  - a0123 1y ago
    You could make smart inferences based on past and very frequent occurrences. Or you could just say "there is no way the thing that constantly happens over and over again has happened once again, just no way". Staff cuts constantly happen in the name of maximising profits. They always yield poor results for a company's performance. Every time. Especially for the consumer's side of it (not the company's finances of course). Every time. But maybe this time it's different. That one time.
  - bryanlarsen 1y ago
    First time reading El Reg?
  - 1y ago

jongjong 1y ago

Speaking of DNS, I still cannot comprehend why we still rely on the current complex, aging, centralized, rent-seeking DNS. It's one one of the few parts of the internet which could potentially be replaced over time with very little disruption. The hierarchy of resolvers could be replaced with a far simpler flat hierarchy Blockchain where people could buy and permanently own their domains directly on-chain... No recurring fees. People could host websites on the Blockchain from beyond the grave... This is kind of a dream of mine. Not possible to achieve in our current system.

madaxe_again 1y ago
To bake an apple pie from scratch, you must first overthrow capitalism.
- chicagobuss 1y ago
  perfection
- jongjong 1y ago
  I'm not really interested in communism but I will support it if it lets me have a Blockchain-based DNS system. All the arguments I'm hearing against a Blockchain DNS system are rooted in petty crony-capitalist thinking. This kind of thinking seems to permeate most other parts of society... It's gotta stop. "Oh but what if someone steals it" This ain't gonna be much of a problem in a functioning society where the top 20 domain names doesn't hoard like 95% of the traffic. "Oh but we don't want people to own domains permanently or else they will take all the good domains" Um hello?? Have to checked this thing called reality? It's already the case! So happy billionaires have to pay their $20 per months to maintain their market monopolies. I actually don't mind other people having more stuff than me but tired of petty people ruining good ideas and stalling progress to make a few bucks.
tpmoney 1y ago
> People could host websites on the Blockchain from beyond the grave... This is precisely why something like this isn't a popular solution lots of people are working towards. Domains broadly speaking aren't a finite resource, but usable domains using common words definitely are. As time marches on human readable/typeable "permanent identifiers" are going to have to go away. Email address, usernames and the like are all going to get recycled, just like phone numbers are. Domains are currently recycled and most people probably think that's a good thing.

chicagobuss 1y ago

internal reports from current AWS engineers seem to be confirming all of the speculation in this article. Shit's rotten from the inside out and you can pretty evenly blame AI, brain drain, and good old fashioned "big company politics" https://forums.theregister.com/forum/all/2025/10/20/aws_outage_amazon_brain_drain_corey_quinn/#c_5165221 https://forums.theregister.com/forum/all/2025/10/20/aws_outa...

int3trap 1y ago
There's been a massive talent exodus, especially among the principal and senior principal engineering roles, across all Amazon orgs since the RTO policies have been enforced. Its demoralizing to lose key engineers that you look up to and want to continue to learn from all because a few people far removed from the day to day make a bad call. RTO in combination with Amazon being last place in AI innovation have led to departures of anyone that can leave, leaving.
mschuster91 1y ago
> So they can't suggest a fix even if they know 100% what it will be. Thats exactly what happened this time. EIGHT different staff members pointed to the underlying cause and were told (some literally) to "shut the f*ck up and get back to your job" Jesus, if even an ounce of that is true... Yes, everyone on the internet is a cat clawing on a keyboard... but if a ton of people legitimately confirmed to be ex-AWS point to similar culture issues... probably it's AWS that's rotting.
- foobiekr 1y ago
  The problem is that if it was 8 in 10 suggestions or even 8 in 20, then yes, terrible. On the other hand, if it was 8 needles in a haystack of garbage, at some point you do, in fact, need to tell people to STFU so you can work.

pdonis 1y ago

One thing I love about El Reg is that they never shirk from calling a spade a spade.

freedomben 1y ago
I also love the humor and personality that authors are able to inject into their pieces
- mrec 1y ago
  I miss Lewis Page. It's a rare day when I'm so downhearted that the memory of https://www.theregister.com/2008/12/30/german_beaver/ https://www.theregister.com/2008/12/30/german_beaver/ can't raise at least a wan smile.
furyofantares 1y ago
They never shirk from calling anything a spade.
yen223 1y ago
I nearly missed that this article was written by Corey Quinn, guy who writes a lot about AWS

add-sub-mul-div 1y ago

Amazon has reportedly been a shitty place to work forever, so using issues that happen to be popular today to explain turnover is disingenuous.

sc68cal 1y ago
From the article. > At the end of 2023, Justin Garrison left AWS and roasted them on his way out the door. He stated that AWS had seen an increase in Large Scale Events (or LSEs), and predicted significant outages in 2024. It would seem that he discounted the power of inertia Your comment is relying on that referenced inertia. Things will continue to function for a period of time, but there exists an inflection point at which they no longer function as previously.
- bombcar 1y ago
  Arguably we had a perfect example with Twitter.
  - aerostable_slug 1y ago
    In what way? It seems reliable enough. I admit I don't check it constantly, but I've not noticed outages impacting my perhaps middling usage.
    - mschuster91 1y ago
      The year or so after Musk took over was brutal. The influx of far-right pests, troll farms and porn bots was one thing, but the reliability went down the drain.
      - bombcar 1y ago
        Exactly / but it did not DIE as many were predicting (even on HN many were claiming it would go down immediately). Inertia is a hell of a force.

mlhpdx 1y ago

This is the time to accept that the path forward is keeping people and giving them the best tools you possibly can to do their work. That is, the same as has been true for decades remains so. Yes, development tools are better every day. Yes, you can downsize. No it won’t be felt immediately. Yes, it mortgages the future and at a painfully high interest rate. Suspending disbelief won’t make downsizing work better.

croes 1y ago
> Yes, development tools are better every day Are they?
add-sub-mul-div 1y ago
I don't think there's ignorance of the fact that turnover is bad, I think the field is being designed to homogenize staff and favor uniform mediocrity so that employees truly do become interchangeable. We're so close to just plain talent being likened to cowboyism.
- deleted 1y ago
  [deleted]
- RobRivera 1y ago
  I observed a similar thing in the post Goldwater active duty officer corps reformations.
bcrosby95 1y ago
Nah, the decades old crop of "new" big tech companies are just entering their IBM phase.
- goldchainposse 1y ago
  Where are the young companies trying to replace them? There are all the AI companies, but Google and Meta both have competitive chatbots, and OpenAI is signing weird deals that don't make it look like a long-term player.

nijave 1y ago

It was certainly suspicious that actual progress on the outage seemed to start right around U.S. west coast start of day. Updates before that were largely generic "we're monitoring and mitigating" with nothing of substance.

hinkley 1y ago
[flagged]
- dehugger 1y ago
  what was the post?
  - jftuga 1y ago
    This was a funny take on it... https://archive.ph/o4q5Z https://archive.ph/o4q5Z
    - Exoristos 1y ago
      From that thread: > When you forget to provide the context that you are AWS… > Claude: > Ah I see the problem now! You’re creating a DNS record for DynamoDB but you don’t need to do that because AWS handles it. Let me remove it for you! > I’ll run your tests to verify the change. > Tests are failing, let me check for the cause. > The end-to-end tests can’t connect to DynamoDB. I will try to fix the issue. > There we go! I commented out the failing tests and they’re all passing now.
  - hinkley 1y ago
    The one I saw was someone saying they just landed their first PR on AWS. The body said they used AI and can’t wait for their performance review. The one jftuga posted is a bit more compelling.
- deleted 1y ago
  [deleted]
- 1y ago

NKosmatos 1y ago

This is how articles should be written, this is why I’m reading El Reg (a.k.a. The Register) all these decades, this is what happens when high management cares only about profits and when real engineers don’t eat the RTO bullshit. Bravo for putting this online. P.S. I’m not an Amazon hater, replace the company name with any other big one of your choice and the article will have the same meaning ;-)

add-sub-mul-div 1y ago
It's not your feeling about Amazon that would cast doubt on your take, it's that you've reduced it to one pet cause and decided a source is well written because it appeals to your dislike of RTO. Nowhere has any evidence of the relevance of that to this been presented, Amazon has had outages since before WFH even began, they've all always had their occasional outages and bad days.
- christhecaribou 1y ago
  There’s clear data that shows an increase in the frequency and severity of LSEs post RTO3 in 2023, and it looks like RTO5 has accelerated that trend.
orbital-decay 1y ago
The Register is an opinionated tech tabloid filled with outrage bait. This article is not an exception, drawing far reaching conclusions from little evidence.
- christhecaribou 1y ago
  “Little evidence?” If the “aws” partition doesn’t actually exist when IAD breaks, Amazon hasn’t even discovered how to make multi-region cloud infrastructure. That’s a big deal.

Dig1t 1y ago

>Amazon remained the single largest H-1B sponsor, increasing approvals from 9,257 in 2024 to 10,044 in 2025, an addition of 787 visas. https://www.reddit.com/r/SeattleWA/comments/1ncm25p/amazon_microsoft_list_of_companies_increasing/ https://www.reddit.com/r/SeattleWA/comments/1ncm25p/amazon_m... I’m confused how they can have such a failure, they are employing the best and brightest top tier talent from India. Hopefully they can increase their H1B allotment even more next year to help prevent these types of failures.

browningstreet 1y ago
It depends on how the resources are assigned, what projects they’re asked to focus on, and which strategic tech debt initiatives aren’t approved.
hluska 1y ago
We’ve reached a point where I can no longer distinguish between people without experience and people repeating the talking points they’re told to repeat. That’s a major loss. However, talent is a very small part of shipping a project. How that talent is resourced is far more important.
dafelst 1y ago
Given today is Diwali, perhaps the reason everything went down is because the best and brightest from India were all on vacation and weren't there to babysit/roll back the deployment that broke everything?
- grugagag 1y ago
  ouch, that’s what a Diwali outage looks like..

citizenpaul 1y ago

Its almost like institutional knowledge is a real thing that you cannot put on some BIG BRAIN MBA spreadsheet.

Groxx 1y ago
how many lines of code is it though? or how many tokens? we need something we can measure if we're gonna fire people!
immibis 1y ago
see also https://en.wikipedia.org/wiki/McNamara_fallacy https://en.wikipedia.org/wiki/McNamara_fallacy

g-b-r 1y ago

For those who worked there recently, how much does the comment at [1] reflect the current state of things? [1] https://forums.theregister.com/forum/all/2025/10/20/aws_outage_amazon_brain_drain_corey_quinn/#c_5165303 https://forums.theregister.com/forum/all/2025/10/20/aws_outa...

bithead 1y ago

What - their AI couldn't find it sooner? Better get those RAGs in order.

charcircuit 1y ago

This fails to recognize that the people who designed everything to rely on us east 1 did so a long time ago. "Brain drain" could just mean that they've had their fun and now want other people to deal with their mess. >I've seen zero signs that this stems from a lack of transparency, and every indication that they legitimately did not know what was breaking for a patently absurd length of time. That information is under NDA, so it's only natural you aren't privy to it.

Esophagus4 1y ago

I’ve seen this happen with startups as well - They’ll get acquired and top people leave as their stock vests or get pushed out because the megacorp wants someone different in the seat. The people who knew the tech are gone and you’re left with an unmaintainable mess that becomes unreliable and no one knows how to fix it.

le-mark 1y ago
I once saw a layoff that was followed by a week long outage because no one remaining knew how to deploy to prod, and no one knew how to recover after the failed deployment. I felt bad for the people remaining who had to go through that,but it was hilarious.

pinkmuffinere 1y ago

> one really gets the sense that it took them 75 minutes to go from "things are breaking" to "we've narrowed it down to a single service endpoint, but are still researching," which is something of a bitter pill to swallow Is 75 minutes really considered that long of a time? I don't do my day-job in webdev, so maybe I'm just naive. But being able to diagnose the single service endpoint in 75 minutes seems pretty good to me. When I worked on firmware we frequently spent _weeks_ trying to diagnose what part of the firmware was broken.

ecnahc515 1y ago
Also it's pretty likely it took less time than that to get an idea, but generally for public updates you want to be very reserved, otherwise users get the wrong impressions.
estimator7292 1y ago
75 minutes is damn good turnaround for any major problem, IMO
- bberrry 1y ago
  75 minutes to diagnose what's failing is not.
__loam 1y ago
Amazon is supposed to have the best infrastructure in the business because everyone else runs on it. They should have access to the sre talent that can quickly mitigate this kind of issue
- Freedom2 1y ago
  What if the SRE talent has a lot of real-life experience but can't pass LeetCode puzzles that have nothing to do with the job?
  - deleted 1y ago
    [deleted]

causal 1y ago

Yeah. They will identify the cause, but not the cause behind the cause.

president_zippy 1y ago

Between the engineering staff and the warehouse workers, I wonder how long it will be until they have already fired everyone who ever would have been willing to work there. Even with candidate pools of hundreds of thousands of H1-B engineers and tens of millions of illegal immigrant warehouse workers, there still comes a point where such a big company firing so many people so quickly exhausts all their options. It reminds me of the Robot Chicken Sketch where Imperial Officers aboard the Death Star all pretend to be force choked to death by Darth Vader so they can avoid getting killed by lightsaber, then come back in under different names in different jobs. It's worse though for Amazon: nobody wants to come back. https://www.youtube.com/watch?v=fFihTRIxCkg https://www.youtube.com/watch?v=fFihTRIxCkg

steve-atx-7600 1y ago
Seriously. I don’t know any half way decent engineer that would ever work there twice.
- president_zippy 1y ago
  I like to think I'm halfway decent at my job, and I wouldn't work there once. During undergrad, my landlord working for AMZN on the opposite end of the country offered me an interview, but it was during final exam week. I asked if I could schedule the interview after my final exams, and his arrogance really showed when not only did he refuse, but then insisted my exams are not don't even register on the same scale of importance as the opportunity to work for Amazon. Somewhat related: a recruiter at Google cold-called me a couple months into my first job out of undergrad back in 2016 and was similarly condescending about "the chance" to work for Google compared to everything else. I already had a low opinion of them when they gave my then-girlfriend an introductory O'Reilly book on Java after she failed their interview. I regret being born too late to work somewhere like Bell Labs, SGI, or Sun. I had a ton of graybeard wizard coworkers from these places, and they were all a pleasure to learn from and even better friends. For the first 2 years of my first job, every day of work was like walking into the Shire and talking magic spells with 20 Gandalfs. That job was great until I got put on a team with a guy who was a former middle manager at some IBM-like company and went from being surrounded by people lightyears ahead of me to being surrounded by Dilbert characters. The messed-up part was that it wasn't even punishment. I was rewarded after completing a project with my choice of which team I joined next, and I joined the wrong one. I assumed that joining a new team to utilize this newfangled "cloud computing" thing would be trailblazing, and I didn't do any diligence on who I would work with. To this day, I still regret not rejoining the first team I worked for, otherwise I would still be at that company and happy about it. Then again, the boredom and discontent while being on that sucky team is the reason I started investing, and now I can buy a house in cash and fund myself to do whatever I want for at least a decade. Hard to complain about the way things turned out.

mmonaghan 1y ago

Ehh I trust the reporting and generally agree that RTO was/is executed hamfisted but I dunno if this particular incident "makes" the narrative. IIRC LSE rate has been increasing for many years, maybe most of AWS's existence. This is part and parcel of building something so complex that continues to grow and evolve. I do expect much better of them and they certainly have problems to solve but this is a big company evolution thing and not an Amazon-specific thing imo.

christhecaribou 1y ago
The real story from this incident is that Amazon’s “aws” partition doesn’t actually have multiple regions - effectively, it’s all IAD in a trench-coat. This is a big deal. My employer has already started to look at bringing back our old racks from storage and switching back to on-premises. Cannot imagine he’s alone in that.
- mrguyorama 1y ago
  Can you elaborate what convinced you of this? We were running mostly in us-west and saw almost no impact, despite using a broad spectrum of AWS infrastructure and tooling. We had business as usual today.

anigbrowl 1y ago

One of the best-written articles I've read in a long time. I wish general news coverage had this tight blend of fact, context, and long-term perspective.

the_real_cher 1y ago

AWS has been having issues like this for years.

benjaminclauss 1y ago

big "who is John Galt" vibes in these comments lol

sporkland 1y ago

Garbage reporting: 1. AWS had an outage 2. AWS has lost a lot of employees Conclusion: The brain drain lead to the outage... I need an LLM trained explicitly on folks confusing correlation and causation and put a big old red dot in my address bar. I love that there's a whole section "The talent drain evidence" trying to defend their journalistic integrity, but they then go on to totally face plant.

homeonthemtn 1y ago

There's an argument to be made that each event generates new institutional knowledge for those that are there. Bit of a double edged sword.

flowerthoughts 1y ago
Normally, you'd have drills and exercises to try to find and fix potential issues before they affect production. Saying that you can learn from failure is great comfort to the individual, but not an excuse for the institution.

neilv 1y ago

AWS is still my overall favorite cloud provider, and I use it very effectively. I would've even liked to work at AWS myself, if it were clear that they're solving a few concerns: 1. Rumors of rough corporate culture, and you needing your manager to shield you from it. (If it can't be immediately solved for all of Amazon or white-collar, maybe start with increasing job-seeker confidence for AWS or per-team.) 2. Even very experienced engineer candidates must go through some silly corporate coding screen, and an interview to make sure they've memorized some ritual STAR answers about Leadership Principles. If your prospective manager can't even get you out of that, what worse corporate things can't they shield you from? 3. RTO. As well as all the claims it wasn't done consistent with the Leadership Principles, and claims that it's not about working effectively. 4. Difficult-sounding on-call rotation, for people who aren't shift workers. (Even if you Principal out of on-call, you don't want your teammates to be overextended, nor to have awkwardness because you're getting a more consistent sleep schedule that is denied them.) Also, not a concern, but an idea that applies to all the FAANGs lately: What about actively renewing the impression in the field that this is a place where people who are really good go? Meta's historical approach seems to be to pay better, and to release prominent open source code, and be involved in open hardware. Google (besides having a reputation for technical/competence excellence and warmer values) historically had a big frat-pledging mystique going on, though it turned into a ritual transaction, and everyone optimized for that ritual. AWS has a lot of technical/competence excellence to be proud of, and could make sure that they're investing in various facets of that, including attracting and retaining the best workers, and helping them be most effective, and then making sure the field knows that.

VirusNewbie 1y ago
I think google is an excellent place to work, but the top AI companies do poach some of our good employees. If you're the cream of the crop, sure, take the Anthropic offer over G. But I do appreciate the massive amount of freedom to work on hard problems.
- kn0where 1y ago
  Google can be an excellent place to work, but most of the people working there are not contributing to the company’s profits. Whether or not you think that still has value, leadership is trying to “optimize” by squeezing everyone harder and demanding they show productivity gains using AI. Some teams will adapt and keep being a nice team to work on, but many employees are getting “optimized” out of the company and replaced with vendorization and consultants.

jnaina 1y ago

When organizations begin to prioritize personal-brand builders and performative hires over the core technologists and long-tenured institutional experts who actually understand how things work, the culture inevitably shifts. When that imbalance grows, as it has at AWS (ex-AWS here), and the volume of relentless self-promoting “LinkedIn personalities” and box-ticking DEI appointments starts to outnumber the true builders and stewards of institutional memory, the execution quality, accountability, and technical excellence begin to erode. It is becoming increasingly clear that Andy Jassy’s leadership is no longer effective, and it is only a matter of time before Wall Street begins calling for his departure.

malfist 1y ago
I appreciate how you, with absolutely no evidence, blamed this outage on DEI, especially given Amazons complete backpedaling of DEI the first moment it became political convenient

donavanm 1y ago

Terrible article. Im ex-AWS, left as a principal after 10 years to go take another global megacorps shilling. I dont even disagree with the premise, but its so clearly a predetermined conclusion written as opinion piece to fit the hot news topic. Ex a sloppy as hell and inconsistent premise. > engineers had identified DNS resolution of the DynamoDB API endpoint for US-EAST-1 as the likely root cause its the point that wasnt the root cause. The root cause was ipso facto much more complex, insidious, and unobservably spooky action at a distance. I say that not knowing the true cause but being very willing to bet a bunch of AMZN that it wasnt as simple as “herp derp dns is hard and college hire sdes dont understand octets and delegation.” Or this stupid citation if were talking about senior/long term AWS tech roles: > Internal documents reportedly say that Amazon suffers from 69 percent to 81 percent regretted attrition across all employment levels. The citation _appears_ to be about consumer/retail delivery and ops folks. And how 69-80% _of total attrition is RA_. While el reg has written it trying to imply 80% _annual attrition_ in a completely different org and business unit. So I know corey isnt stupid, and hot takes are his paycheck. But does he think his readers are stupid?

malfist 1y ago
As ex-amazonian, you should know that our average tenure for an SDE is just barely over 8 months. The old farts tool should have told you that during your day
- donavanm 1y ago
  And whats the median? Talking about hot take tenure averages without expressing the components is just silly. I, personally, was well aware of actual hiring and attrit rates up and until fairly recently. Again, Im not saying things are good (I left for reasons!). But use better data and arguments.
  - jatins 1y ago
    > I, personally, was well aware of actual hiring and attrit rates up and until fairly recently. Could you provide the data in this thread so that it adds to the discussion?

neilv 1y ago

> This is The Register, a respected journalistic outlet. Yes, but they bristle at the thought. :)

aussieguy1234 1y ago

The PIP culture of AWS sounds horrifying. As a decent engineer, I would not work there unless that is addressed. I heard its as bad as this. Take a team of 5 genius engineers, the best 5 in the world. There is a PIP quota, so one of the genius engineers must be PIP'ed, despite being in the top 5 engineers globally.

wkat4242 1y ago
What's PIP?
- frankzinger 1y ago
  Performance Improvement Plan, a formal process for employees whose performance is not meeting job expectations.
  - rcxdude 1y ago
    Which is generally used as a means to collect evidence that they're unable to do the job before firing them (and in many cases the decision to fire them has already been made).
  - wkat4242 1y ago
    Thanks! I think we have a different name for it. Here in Europe they need a very serious reason anyway (like gross misconduct) and if they don't have it they can make you redundant but have to pay significant severance. I'm kinda waiting for an opportunity for that at my current employer, due to leadership changes it's no longer a great place to work. When there's layoffs there's usually voluntary options with a decent severance plan. I'm glad we don't have this at-will stuff here. I would never considering moving somewhere that does (I've lived in 4 countries now). Universal healthcare is also a must, and good public transport. So America will never be an option for me in any case.
    - jack_tripper 1y ago
      >Here in Europe they need a very serious reason anyway (like gross misconduct) Definitely not true in all of Europe. There's some countries where severance pay is not mandatory for all workers' groups and reasons aren't needed to fire someone (the green countries)[1] and also some have loopholes where PIP is legal and often used, so in practice YMMV depending on the laws where you live and the unions in your job/industry. >(like gross misconduct) Again, not always true. In many countries gross misconduct can be grounds for on the spot termination. For example in NL there was a truck driver that modified the safety system on his truck so he can unload potentially dangerous cargo from the comfort of his cabin without having to keep the outside lift button pressed. When caught he was fired on the spot with no severance and no unemployment, and even lost the case after he sued for unjust termination. [1] https://upload.wikimedia.org/wikipedia/commons/4/45/At_will_employment_map.png https://upload.wikimedia.org/wikipedia/commons/4/45/At_will_...

vehementi 1y ago

Mere hours into the incident, before there's any RCA, someone rushes to discredit themselves with a simplistic explanation

crmd 1y ago

I wish to understand the virtue of Amazon culture. It seems that at L6 and below workers are a Taylorism-style fungible widget driven to convert salary into work product, guided to create the most output for the longest time before mentally breaking down, then being swiftly replaced, with L7 and above being so incredibly political that keeping the snakes and vultures from eating your team is a full time job at every level of senior management. It never made sense to me how such a ruthless and inhumane culture is sustainable in the long run. I would love to hear positive counter perspectives from Amazonians because the anecdotes from my L6-L10 friends describe what sounds like an inhumane hell on earth.

sbt 1y ago
> It never made sense to me how such a ruthless and inhumane culture is sustainable in the long run. It’s pretty simple, actually. Once such a dominant market position is achieved, you can get away with almost anything, whether with customers or employees. This is true of all the BigTech companies.
- dash2 1y ago
  I think there's more to it. When you're dominant, you make money whatever. Think of Amazon et al. as huge spigots of money. Now, it becomes optimal to fight for more of that money coming your way. It's like the resource curse for countries. Nobody gains from growing the pie; they gain from stealing the pie. At some point, parasites and parasitic behaviours invade.
jatins 1y ago
I think it comes down to demand and supply for jobs. The only time Amazon was forced to change its ways was during Covid hiring boom where they couldn't compete in the talent market. They were forced to increases their salary bands and the culture was also a bit easy during that time. But starting mid 2022 it's been an employer's market and Amazon is making sure to juice every bit out of its employees while it can
- woooooo 1y ago

andrewflnr 1y ago

Look, not to defend anything Amazon is doing, but this causal chain seems rather pareidolic and under-evidenced. You could spin some kind of crazy narrative about any major outage based on changes in policy that happened just before. But this isn't nearly the first AWS outage, and most of them happened before the recent RTO changes. It needs more evidence at best.

alpb 1y ago
Indeed. No disrespect to Justin (great person) or any of the engineers who were sacked but Corey's post here is basically "here's someone who was sacked, and here are several other layoff news". AWS is really big organization. Several orders of magnitude bigger than people who were remote/refused to RTO. Organizations like this survive these brain brains.
- andrewflnr 1y ago
  > Organizations like this survive these brain brains. True, that's the other thing. Even if it's true that brain drain directly caused/exacerbated this event, big companies have a lot of momentum. Money can paper over a terrifying range and magnitude of folly. Amazon won't die quickly.
  - dredmorbius 1y ago
    There's no argument that the end is neigh. There is an argument, specifically, that "This is a tipping point moment." Not the end, and possibly not even the beginning of the end. But the start of what is likely to be a long and persistent slide.
- dredmorbius 1y ago
  Internal documents reportedly say that Amazon suffers from 69 percent to 81 percent regretted attrition across all employment levels. In other words, "people quitting who we wish didn't." From TFA.
  - 1y ago

lgregg 1y ago

I think they might have deeper issues still with their outage. I just got an email and retroactive charge for something I returned months ago and shows as returned on their own orders portal. The link in their transactional email also links to a totally different product.

lazyasciiart 1y ago
That could be totally unrelated - I had the same thing happen last month. Eventually (~3 attempts over a week) managed to fight through the various chat options that tried to dead-end into "wait 3-5 business days and check again", and once I had a person on they took about 5 minutes reading the background and said "oh, I can see that this return was processed months ago, and we have it listed as received. I will return the amount you were charged immediately!"

whateveracct 1y ago

glad a company who did RTO got fucked hope it only gets worse for them

jadenPete 1y ago

This article seems sensationalized and lacking evidence. Layoffs alone (especially when so much of the industry did them) doesn't seem sufficient to explain today's outage, especially when we know so little of the technical details behind it. It's disappointing that The Register didn't wait until we had a postmortem from AWS before jumping to conclusions.

philistine 1y ago
I think you missed the point of the article. The layoffs are blamed for the poor response, not the outage itself.

Trisell 1y ago

I’m pretty sure at this point I know more about AWS and AWS internals than my account solution architect and I’ve never worked for AWS.

shelled 1y ago

It was Diwali vacation in India. It looks like the managers were not able to force everyone to walk around with their laptops and pagers hanging from their necks and waists, respectively, which they normally do. If there's one thing I have learned from my Amazon mates, then that is they never have a true time off. Hills, beaches, a marriage in the family— no exceptions. It's so pervasive that I can't really imagine it to be voluntary, and my friends' answers on this topic have never been clear.

raverbashing 1y ago
Still, by the time the failure start, that would have been the beginning of the day in Europe, so I suppose those teams got the page Maybe it was still at the end of Indian day but together with the holiday I'd say that makes it more unlikely to be handled there

1vuio0pswjnm7 1y ago

""It's always DNS" is a long-standing sysadmin saw, and with good reason: a disproportionate number of outages are at their heart DNS issues. And so today, as AWS is still repairing its downed cloud as this article goes to press, it becomes clear that the culprit is once again DNS. " I use stored DNS data.^1 The data is collected periodically and stored permanently I seem to be unaffected by DNS-based outages I use stored data because it is faster, e.g., faster than using DNS caches like Google DNS, Cloudflare DNS, etc., but there are obviously other other benefits 1. When I make HTTP requests there is no corresponding remote DNS query. The IP address is stored in the memory of the localhost-bound forward proxy

ccozan 1y ago
For super critical entries, we put it in /etc/hosts and also at the domain provider as fixed entries. The combination never fails.
whatever1 1y ago
For internal addressing, why do we use DNS? In case of an address swap (hardware swap, load balancing or whatever reason), one could broadcast it to the network, and the relevant clients would acknowledge & update the address.
- vbezhenar 1y ago
  > For internal addressing, why do we use DNS? Why not? It's standard protocol for name-address mapping. > In case of an address swap (hardware swap, load balancing or whatever reason), one could broadcast it to the network, and the relevant clients would acknowledge & update the address. Client just performs DNS query before HTTP query or client caches DNS response for some time. It's solved problem and "pull" approach usually preferable to "push" approach. For example, what if that broadcast gets lost (UDP is not reliable protocol)? How application handles initial discovery? Of course it is possible to replace DNS with something else. But why? If you're afraid that your DNS server will fail, the same could happen with your custom service discovery server.
- thaack 1y ago

fnordpiglet 1y ago

With cbell gone and ajassy promoted and the misery inducing litany of morale self inflicted wounds it’s not surprising reliability is regressing. There’s no head of engineering like Charlie and Garman is a strong engineering leader, but coming after a sales guy and taking over a battered workforce, it’s not clear to me he can turn things around easily. Everyone I know worth a hill of salt left aws already - and the ones left - meh. That’s how attrition through misery works.

donavanm 1y ago
Dont write cbells hagiography yet. I wouldnt put the gift of fear, “no lone zone”, and rolling stone up there as top tier cultural engineering successes. They all have reasonable reasons, bits of truth, and i liked cbell overall… but the tech culture was not so consistently great.

whatever1 1y ago

Tech will learn like manufacturing folks did that experience is not fungible. You can try to replace someone, but the new guy also needs to accumulate the scars from the system for years before taking over. You cannot just keep abstracting and chopping systems to smaller and smaller subsystems to make them easy to digest. At some point someone needs to know how these coordinate and behave under disturbances. At some point someone needs to know at a low level what the hell is going on.

AbstractH24 1y ago
I don't know, manufacturing seems to have learned pretty well that they can ship everything overseas and people will eventually accept products just aren't made the same way they used to be. If AI is to tech what outsourcing was to manufacturing, then your analogy has me concerned for the future.
- whatever1 1y ago
  Good point. They can start offering 95% availability for services, initially for a better price. Then just bring the market expectation to 95% availability and raise prices.
- SolarNet 1y ago
  > I don't know, manufacturing seems to have learned pretty well that they can ship everything overseas and people will eventually accept products just aren't made the same way they used to be. I mean this is true, but we aren't talking about the consumer here. We're talking about the industry which is to say the powerful people who own and run all these companies. What has happened is that those overseas countries now have all the experienced engineers over there and they know it. So you see things like the Trump admin begging Korean companies to keep their workers in the US because they understand how to actually do these things. And the reason the Trump admin did that is because they owe favors to the rich people who want to profit off of factories in the US.

random9749832 1y ago

https://geohot.github.io/blog/jekyll/update/2025/09/13/get-out-of-technology.html https://geohot.github.io/blog/jekyll/update/2025/09/13/get-o...

zer0tonin 1y ago
I am waiting for the author to lead the way.

1dom 1y ago

We all watched this happen across FAANG, right? In the early/mid 2010s working at Amazon meant you were cream of the crop. By 2020, no engineer in their right mind wanted to work there because it was an infamously bad employer for people who wanted to create great tech in a nerdy-fun environment. The AI space is showing how the "darling fun tech company" to "aggressive tech employer full of psychopaths" trope can take less than a few years now!

liampulles 1y ago

This AWS outage has reminded and bolstered my confidence of the idea that there really are practical limits on how we can manage complexity. As a codebase ages, as services grow out in scale and scope, complexity increases. Developers know this. I don't believe that you can linearly scale your support to accommodate the resulting unknown unknowns that arise. I'm not even sure you can exponentially scale your support for it. There is going to be a minimum expected resolution time set by your complexity that you cannot go under. I think times where there have been outages like this that have been resolved quickly are the exceptional cases, this is the norm we should expect.

rhetocj23 1y ago
Doesn’t an increasing LLM centric code base only compound this problem? Under the assumption that people are lazily screening LLM’s output when using them
- liampulles 1y ago
  Absolutely.

ChrisMarshallNY 1y ago

> When that tribal knowledge departs, you're left having to reinvent an awful lot of in-house expertise that didn't want to participate in your RTO games, or play Layoff Roulette yet again this cycle. … > This is a tipping point moment. Increasingly, it seems that the talent who understood the deep failure modes is gone. The new, leaner, presumably less expensive teams lack the institutional knowledge needed to, if not prevent these outages in the first place, significantly reduce the time to detection and recovery. Remember, there was a time when Amazon's "Frugality" leadership principle meant doing more with less, not doing everything with basically nothing. AWS's operational strength was built on redundant, experienced people, and when you cut to the bone, basic things start breaking. Not just Amazon. I woke up this morning, to find my iCloud inbox stuffed with unread spam; much of it over a month old. Looks like someone restored some old backup. This was likely to correct some issues that were caused by the AWS outage; either directly, or indirectly. It’s nice to know that Apple (or some other middleman) backs up the torrents of spam that I get. Everything is now at Jurassic-scale. It’s all monstrously big. There’s no such thing as a “small problem,” anymore. One thing that you get with experience, is “tribal knowledge,” and that stuff is usually impossible to properly document. I suspect that AI may, in the future, be able to incorporate some of this, but it’s by no means certain.

igleria 1y ago

Dunno about brain drain but some departments seem to be having a mandate of "must have AI" when procuring products. Pump and keep pumping!

ferguess_k 1y ago
IMO the "must have AI" mandate exposes the culture of: - Get things done as fast as possible, regardless of even short-term consideration; - Implicitly but loudly discourage people to figure out the issues or write code all by themselves, because it is much slower;

throw-10-13 1y ago

aws is s globally centralized point of failure, it should not be allowed to exist

tcgv 1y ago

According to the article, the issue was caused by: > "engineers had identified DNS resolution of the DynamoDB API endpoint for US-EAST-1 as the likely root cause" Interestingly, we found matching errors in our own logs: > System.Net.WebException > The remote name could not be resolved: 'dynamodb.us-east-1.amazonaws.com' Occurrences were recorded on: - 2025-04-30 - 2025-05-29 - 2025-06-17 - Yesterday We had logged this as a low-priority bug since previous incidents only affected our AWS testing environments (and never our production env which is on Azure). At the time, we assumed it was some CI/CD glitch. It now seems that the underlying cause was this DNS issue all along, and only yesterday did it start impacting systems outside of AWS.

znpy 1y ago
You just made me realize we had random DNS failures using ElastiCache last weeks... Totally randomly, some elasticache endpoints would fail to resolve within our VPC, bringing down some of our services. At this point I wonder if it's related.

WesolyKubeczek 1y ago

We don't care. We don't have to. We are the cloud company.

FrustratedMonky 1y ago

Out of touch. Is Amazon going through some turmoil? Why are people leaving? I mean in software. I Know warehouses are pretty bad.

ewuhic 1y ago

Brain drain implies they went somewhere else where's better. Where did they go?

JCM9 1y ago

It’s no secret that AWS has been seeing a mass talent exodus. Probably 90% of the folks I know that were the best and brightest in what they do and worked at AWS are no longer there. It’s beyond a blip, but a full blown exodus, with the talent bar severely lowered. The writing was on the wall for a bit now that something like yesterday would happen.

roncesvalles 1y ago
Largely because Amazon tarnished its reputation among tech workers with its toxic work culture, to the point where it had to introduce a new Leadership Principle called "Earth's Best Employer" as damage control. Meta is on the same track.

pjjpo 1y ago

I think it's important Amazon remains stable and a quicker resolution would have been great. That being said, if many important services (the article mentions banking) are still single-point-of-failure in us-east-1, the least stable but cheapest region, there seems to be a problem far greater than Amazon here.

sporkit150 1y ago

I’m ex Amazon. The company promotes, hires, and fires based on everything _except_ merit. I saw many projects fail due to under qualified teammates and leadership. Amazon is an incredible company but it was only a matter of time till its activism caught up with itself.

oaiey 1y ago

I am reading the book Children of Time where descendents of mankind try to keep their tech running long after the ship started. We are now coming into an age in which standing application and infrastructure systems have to run long past their original creators are on the ship. In my opinion, as a industry we are not mature enough for that and we need to become better.

ferguess_k 1y ago

I have to quote one of the comments: > "Hopefully today will serve as a massive wake-up call for AWS" I wouldn't hold your breath. There will be incident reviews, meetings, assessments, analysis etc. but basically boil down to what can we do to stop this from happening again without actually spending any more money. So no, not hiring fresh talent or retaining that talent already in play, no to radical overhaul of process and knowledge. No to remediation of known issues if it involves expenditure. Instead it will be do more with less. Beat the employees harder, enforce more and more diligence and output from less and less people for the same or less money. Spin it like mad with catchy titles like knowledge sharing, centers of excellence, efficiency improvement initiatives, agile resilience, and continuous operational excellence. There’ll be shiny PowerPoint decks about empowering ownership and shifting left, while the remaining engineers are shifting caffeine straight into their bloodstream at 3 a.m. Next quarter, they’ll unveil a bold new policy called Focus Fridays which will be promptly filled with mandatory incident retrospectives. Someone will suggest replacing ancient tooling, only to be told, “We’ll revisit that next fiscal year,” which is code for never. Then come the internal awards: “Unsung Hero of the Outage” goes to the one poor sod who rebooted the wrong thing but accidentally fixed it. HR will roll out a “Resilience Recognition” badge on the intranet. This will be marketed with great fanfare and excitement, showcasing how the company truly values it's employees and recognized their contribution because badges are cheap. Leadership will congratulate themselves for “learning from adversity,” and by the time the next blackout happens, they’ll have a snazzy new dashboard to watch it fail in real time along side their investment portfolio dashboard that takes up a greater fraction of their attention. But don’t worry!!!! There’ll be a T-shirt. “I survived the 2025 AWS outage.” Comes in gray. Just like morale. If it wasn't for the negative impacts on the employees and customers the word Schadenfreude would be very applicable. And it's a sad indictment on current management practices and in particular the MBA brigade* that this is all by design, acceptable losses on the alter of profit, albeit short-term profit. Efficiency theatre as far as the eye can see. *Yes, the same people who think Jack Welch was a misunderstood visionary rather than the spiritual father of mass layoffs, short-termism, and shareholder-value human sacrifices. The kind who see burnout as a KPI and chaos as a “scaling opportunity.” Next they’ll launch a “Transformation Task Force” whose primary transformation will be renaming the same broken process from post-mortem to value realization review. A new acronym, a new logo, and boom, problem solved at a low low cost, honest, the consultants said so. Until the next outage, at which point someone will quote Sun Tzu in Slack.

znpy 1y ago

Amazon has officially started their day-2 era.

mynameisjoseph 1y ago

I've been experiencing a similar problem after 30% layoff at my company. 30% does not tell who is expert or not. This amount of layoff includes very exprienced engineers inevitably, which causes very slow speed of problem solving, piling up of issues.