7 ms·
Amazon holds engineering meeting following AI-related outages
https://archive.ph/wXvF3 https://archive.ph/wXvF3
- potetoooooo 8mo agonice domain
- o10449366 8mo agoPaywalled
- techterrier 8mo agopaste headline into google, click first link
- kqr 8mo agoHuh, it has to be Google, specifically, too! There used to be a shortcut for this action on HN (a link under the submission saying "web" or something?), but it seems that has been removed.
- deleted 8mo ago[deleted]
- shablulman 8mo ago[flagged]
- shablulman 8mo ago[flagged]
- mediumsmart 8mo agoIs it only 45 dollars for the subscription? Does that cover the AI-related outages too or just the engineering meeting
- palmotea 8mo ago> Amazon’s ecommerce business has summoned a large group of engineers to a meeting on Tuesday for a “deep dive” into a spate of outages, including incidents tied to the use of AI coding tools. > The online retail giant said there had been a “trend of incidents” in recent months, characterised by a “high blast radius” and “Gen-AI assisted changes” among other factors, according to a briefing note for the meeting seen by the FT. > Under “contributing factors” the note included “novel GenAI usage for which best practices and safeguards are not yet fully established”. > “Folks, as you likely know, the availability of the site and related infrastructure has not been good recently,” Dave Treadwell, a senior vice-president at the group, told employees in an email, also seen by the FT.
- VirusNewbie 8mo agoGenAI at fault, and nothing to do with amazon laying off 30k people and having an overall shitty culture where people mostly don’t want to stay?
- jiggawatts 8mo agoAlso, managers are incentivised to force AI onto the remaining staff to “boost productivity” but of course they won’t accept any of the responsibility or blame for that decision.
- zihotki 8mo agoJust tell the employees to make AI fully adopted in SDLC and make it secure and reliable. Don't make mistakes. If it works for models, why not humans? /s
- aerhardt 8mo agoMaybe both, and possibly other causes too, but allow us a moment to revel in the schadenfreude of AI code slop at hyperscale, will you?
- deleted 8mo ago[deleted]
- wiseowise 8mo agoHold a meeting?! No way! That’s a news worthy material! Seriously, who even cares? It’s probably going to be “guys be careful but also continue to push slop kthx”.
- kerim-ca 8mo agoFull Article Amazon’s ecommerce business has summoned a large group of engineers to a meeting on Tuesday for a “deep dive” into a spate of outages, including incidents tied to the use of AI coding tools. The online retail giant said there had been a “trend of incidents” in recent months, characterised by a “high blast radius” and “Gen-AI assisted changes” among other factors, according to a briefing note for the meeting seen by the FT. Under “contributing factors” the note included “novel GenAI usage for which best practices and safeguards are not yet fully established”. “Folks, as you likely know, the availability of the site and related infrastructure has not been good recently,” Dave Treadwell, a senior vice-president at the group, told employees in an email, also seen by the FT. The note ahead of Tuesday’s meeting did not specify which particular incidents the group planned to discuss. Amazon’s website and shopping app went down for nearly six hours this month in an incident the company said involved an erroneous “software code deployment”. The outage left customers unable to complete transactions or access functions such as checking account details and product prices. Treadwell, a former Microsoft engineering executive, told employees that Amazon would focus its weekly “This Week in Stores Tech” (TWiST) meeting on a “deep dive into some of the issues that got us here as well as some short immediate term initiatives” the group hopes will limit future outages. He asked staff to attend the meeting, which is normally optional. Junior and mid-level engineers will now require more senior engineers to sign off any AI-assisted changes, Treadwell added. Amazon said the review of website availability was “part of normal business” and it aims for continual improvement. “TWiST is our regular weekly operations meeting with a specific group of retail technology leaders and teams where we review operational performance across our store,” the company said. Separately, the company’s cloud computing arm — Amazon Web Services — has suffered at least two incidents linked to the use of AI coding assistants, which the company has been actively rolling out to its staff. AWS suffered a 13-hour interruption to a cost calculator used by customers in mid-December after engineers allowed the group’s Kiro AI coding tool to make certain changes, and the AI tool opted to “delete and recreate the environment”, the FT previously reported. Amazon previously said the incident in December was an “extremely limited event” affecting only a single service in parts of mainland China. Amazon added that the second incident did not have an impact on a “customer facing AWS service”. The FT previously reported multiple Amazon engineers said their business units had to deal with a higher number of “Sev2s” — incidents requiring a rapid response to avoid product outages — each day as a result of job cuts. Amazon has undertaken multiple rounds of lay-offs in recent years, most recently eliminating 16,000 corporate roles in January. The group has disputed the claim that headcount cuts were responsible for an increase in recent outages.
- jqpabc123 8mo agoSummary: AWS has voluteered to serve as a crash test dummy for vibe coding. But don't tell anyone --- and if you do, don't blame AI because it's all the humans fault for not shaping their questions in the "right way".
- arjie 8mo agoFor this particular experiment, regardless of phrasing, I think the guys with the most appetite for risk have to be Cloudflare. They're shipping at an astonishing pace but I think there have been far more outages than there were before in jgc era. Perhaps Anthropic's application side teams are faster and more cowboy[0] but they are super AI-native so that makes sense. 0: I think this is the eras cowboys win so they're (unsurprisingly) smart about doing this
- Rohunyyy 8mo agoI am surprised we haven't had an actual Y2K crash with these AI codes. Like how do you review a 1000 lines of Claude generated PR?
- krilcebre 8mo agoYou don't. I can guarantee that 90% of the generated code will never receive a detailed review, simply because there's too much of a cognitive overhead, and too little time, everything moves too fast. I remember having to do such a code review before an AI in a highly complex component, and it would take a full day of work to do it. In this day and age, most of the people i know take like half an hour and are mostly scanning for obvious mistakes, where the bigger problem are those sneaky non obvious ones.
- kakacik 8mo agoExactly. Its same for reviewing somebody else's code. How many companies did this perfectly before llms came? I know mine didn't. But these days people that aren't senior enough do reviews of llm output, and do a quick mental path through the code, see the success and approve it. What could work - llm creating a very good test suite, for their own code changes and overall app (as much as feasible), and those tests need a hardcore review. Then actual code review doesn't have to be that deep. But if everybody is shipping like there is no tomorrow, edge cases will start biting hard and often.
- urban_winter 8mo agohttps://archive.ph/wXvF3 https://archive.ph/wXvF3
- dang 8mo agoThanks! added to toptext now.
- andyjohnson0 8mo agohttps://archive.ph/wXvF3 https://archive.ph/wXvF3
- rhubarbtree 8mo agoSome engineers will point to this and say, hey, AI is not gonna work. It doesn’t reason very well and it leads to these problems. But what they’re missing is all code quality is going to tank, and we are just going to accept that. Just as artisanal goods were replaced in the Industrial Revolution with mass produced inferior ones. People will accept bad code if it is cheap enough. We’ve gotten used to aiming for great, even if we often only hit functional. The new bar is going to be so much lower. Welcome to the era of cheap bad code. Lots more software, lots more value overall, but much worse reliability. Every day the apps I use get buggier.
- gtsop 8mo agoYou are almost right. As I say since the beginning of this ai circus, this is the equivalent of flipping mcdonalds burgers (no insult intended for those workers). It is a thing, and people buy and eat them. But high quality burgers made by talented chefs will always be out there. That's my analogy, and i dont intend to be on the side of flipping mcdonalds burgers
- rhubarbtree 8mo agoThere are a lot of McDonalds and very few Michelin starred restaurants. Safety critical engineering and infrastructure layers will (eventually again) be rigorous. Everything else is headed to slop. My craft died. I’m sad. Time to move on.
- kakacik 8mo agoWhere I live, gourmet high quality burger joints definitely, and massively overwhelm McDonalds in number (Geneva, Switzerland). Even if I count in burger king. Shows that sometimes people pay for the quality even if they don't desperately need it. And its trivial to make better burgers than mcd, heck I can surpass them trivially at home with every ingredient, they are really the lowest level of quality, taste, looks, or (lack of) healthy components. You don't need Michelin * for that, far from it. Plus food is often cold outside of peak hours, something that never happened to me in proper restaurant. Also, mcd ain't at the end much cheaper, just marginally, the choice of drinks is pathetic, usually no beer. The main reason folks go there because its easier/faster than getting table in real restaurant. But also the environment in mcd is absolute soulless cheap fugly shit. (there are kids corners to be fair, but they are often disgustingly dirty). Its a very good analogy at the end IMHO, maybe just not tilting the way you intended, at least not here.
- jcgrillo 8mo ago> Junior and mid-level engineers will now require more senior engineers to sign off any AI-assisted changes, Treadwell added. Lol. Lmao. You have got to be joking. Seniors leaving in droves is how that plays out.
- Rohunyyy 8mo agoNope. NO ONE is quitting in the current market because they got asked to review extra PRs.
- kakacik 8mo agoTop people definitely do if they feel like it, why the heck shouldn't they. There is no shortage of work for those. But its fine if company, via its actions, claims it doesn't want to even retain its top talent. Just market forces and all that.
- rsynnott 8mo agoIf you’re a senior at Amazon and your whole job becomes reviewing slop, well, you can likely get another job which does not revolve around reviewing slop. The current market is not great, but it’s disproportionately painful for juniors.
- onei 8mo agoI read that line and thought "so, the solution is code review?". What has to happen to your processes that code review is not only missing, but unironically claimed to be the solution? I know there are some companies that never did code review, but this is Amazon. They should know better.
- rendaw 8mo agoIt's _more_ code review. They already had senior code review.
- wrxd 8mo agoThis is going to end either with seniors rubber-stamping absolutely everything without even reading or with seniors blocking most of the slop for no overall productivity gain
- nwmcsween 8mo ago> Junior and mid-level engineers will now require more senior engineers to sign off any AI-assisted changes, Treadwell added. Beatings will continue until senior engineers leave?
- deleted 8mo ago[deleted]
- rsynnott 8mo agoI wonder what senior means here. Like, unless it’s fairly junior seniors, the ratios are going to make that impossible.
- abroszka33 8mo agoEven if they have an army of senior engineers, reviewing AI generated code is fundamentally different than reviewing code written without AI. The change is usually larger, looks good on the surface and there are stupid mistakes in it. It's like reviewing someone's code whose only goal is to get an approval on the change.
- lamontcg 8mo agoSo exactly like reviewing a code change from an outsourced development team in another country?
- abroszka33 8mo agoI'm not sure. How common is it to review the outsourced development team's code? My guess is that there is rarely any review. They usually ship the whole software and are responsible for it.
- lamontcg 8mo agoI spent years doing it.
- 8mo ago
- bravetraveler 8mo agoWhen you hear "left behind", remember: is 'it' going to places you want?
- MOSI3 8mo agoAnd if it's going to get easier and easier for my work to be performed by AI, then what does it mean for me to "keep up"? Do I just need to create more slop than anyone else?
- bravetraveler 8mo agoExcellent consideration, probably. Sounds like a lot to do for very little in return. I'll leave with this, a sort of sick joke given context: Quit when the work is done
- MOSI3 8mo agoFortunately my job is not based on generating plausible-sounding bs, so I should be safe.
- bravetraveler 8mo agoHear, hear. Wish you the best! There's a whole list of other silly games to avoid, unfortunately. Namely, "up or out". For instance, I want to engineer [more]. Closer to management or sales due to scope creep. At this rate, by career-end, I'll be operating a small country by myself.
- pinkmuffinere 8mo ago> The group has disputed the claim that headcount cuts were responsible for an increase in recent outages. It's a bit hard to believe this.
- jamiemallers 8mo ago[dead]
- alecco 8mo agoThey are trying to cure the symptom. The actual cause: one of the most toxic environments to work as a developer. Another case of AI as a scapegoat.
- shruubi 8mo agoAmazon - Where the beatings and layoffs will continue until AI usage improves.
- stavarotti 8mo agoI'm curious whether this is due to their insistence on using home grown tools ie, Kiro and not Codex/Cursor/Claude et all? I tried Kiro and quickly left.
- dang 8mo agoRelated ongoing thread: After outages, Amazon to make senior engineers sign off on AI-assisted changes - https://news.ycombinator.com/item?id=47323017 https://news.ycombinator.com/item?id=47323017 - March 2026 (194 comments) I'm not going to merge the current thread thither, because it's so bad. Interesting specimen of how much worse the comments get when there isn't a readable, substantive article to backstop the thread. (Not a criticism of the submitter! - ft.com was the original source for this story and there are workarounds available, like the archive link and the google trick described at https://news.ycombinator.com/item?id=47319643 https://news.ycombinator.com/item?id=47319643.)
- ChrisArchitect 8mo agoIf you're going to criticize a submitter, criticize the one who submitted that dupe of a story with multiple threads like 12 hours later. Why reward Ars and the submitter for being late? Shrug.
- dang 8mo agoHave I criticized a submitter?
- rendaw 8mo agoWhat's so bad about the comments here? They look qualitatively similar to the comments in the other submission.
- irishtel 8mo agoThe framing of "AI-related outages" is going to drive the wrong conversation. The question isn't whether the AI made a mistake — it's whether the review process, change controls, and blast radius gates were designed to handle agents acting at the speed and confidence of a senior engineer. 1,500 engineers flagged the tool wasn't ready. Leadership made adoption a KPI. Those two things together are the incident. The AI just executed. When the process hasn't caught up to the blast radius the tooling can create, this is what you get. It's not new — Knight Capital, CrowdStrike — the pattern is always the same. Automation moves faster than governance.