16 ms·

After outages, Amazon to make senior engineers sign off on AI-assisted changes

• andsoitis 7mo ago
  > Amazon’s website and shopping app went down for nearly six hours this month in an incident the company said involved an erroneous “software code deployment.” The outage left customers unable to complete transactions or access functions such as checking account details and product prices. The environment breathed a little.
  • 8note 7mo ago
    hardly. as an alternative, a bunch of people got into their one-person trucks and drove to the store to buy whatever thing would have been efficiently delivered
• ndr42 7mo ago
  I think the problem of responsibility will come for many more companies sooner than later. It is possible that some of the alleged efficacy gains by using ai are not so big anymore when someone has to be accountable for it.
• AlexeyBrin 7mo ago
  I wonder how this will work in practice. Say I'm a senior engineer and I produce myself thousands of lines of code per day with the help of LLMs as mandated by the company. I still need to presumably read and test the code that I push to production. When will I have time to read and evaluate similar amounts of code produced by a junior or a mid level engineer ?
  • quantified 7mo ago
    This is an important bottleneck. You can have LLM-based reviewers help you. But unless you yourself understood your thousands of lines, it's "somebody else's" code and that somebody else cannot be fired or taken to court. The presumably human mid-level or junior engineer has their own issues with this, but the point of the LLM is that you don't need that engineer. For productivity purposes, the dev org only needs the seniors to wrangle all the LLMs they can. That doesn't sustain, so a couple of more-junior engineers can do similar work to mature.
  • MichaelRo 7mo ago
    >> I wonder how this will work in practice. Say I'm a senior engineer and I produce myself thousands of lines of code per day with the help of LLMs as mandated by the company. LOL, it's the age old "responsibility without authority". The pressure to use AI will increase and basically you'll be fired for not using it. Simultaneously with the pressure to take the blame when AI fucks up and you can't keep up with the bullshit, leading you to get fired. One way or the other, get some training on how to stack shelves at the supermarket because that's how our future looks, one way or the other.
  • hrmtst93837 7mo ago
    [flagged]
    • jamiemallers 7mo ago
      [dead]
• kmg_finfolio 7mo ago
  The accountability problem is real but I think it's slightly different from what's being described. The issue isn't just "who signs off"; it's that the reasoning behind a change becomes invisible when AI generates it. A senior engineer can approve output they don't fully understand, and six months later when something breaks, nobody can reconstruct why that decision was made. Human review works when the reviewer can actually interrogate the logic. At LLM-assisted velocity, that bar gets harder to clear every month.
• CodingJeebus 7mo ago
  I'm at a small company struggling with this problem. Fundamentally, we have a limited context and AI is capable of generating tremendous amounts of output that exceed our ability to deeply process. I find myself context-switching all the time and it's pretty exhausting, while also finding that I'm not retaining as much deep application domain knowledge as I used to. On the surface, it's nice that I can give my LLM a well-written bug ticket and let it loose since it does a good job most of the time. But when it doesn't do a good job or it's making a change in an area of the codebase I'm not familiar with, auditing the change gets tiring really fast.
• AlotOfReading 7mo ago
  I'm not surprised by the outages, but I am surprised that they're leaning into human code review as a solution rather than a neverending succession of LLM PR reviewers. I wonder if it's an early step towards an apprenticeship system.
  • bilbo0s 7mo ago
    You shouldn't be surprised. How else would they train the LLM PR reviewers to their standards? I've never personally been in the position, because my entire career has been in startups, but I've had many friends be in the unenviable position of training their replacements. Here's the thing though, at least they knew they were training their replacements. We could be looking at a potential future where an employee or contractor doesn't realize s/he is actually just hired to generate training data for an LLM to replace them, and then be cut.
  • monarchwadia 7mo ago
    Interesting. How would it be an early step towards an apprenticeship system?
• josefritzishere 7mo ago
  The excessive exuberance of AI adoption is all part of the bubble.
  • deleted 7mo ago
    [deleted]
• Lalabadie 7mo ago
  I'm not sure the sustainable solution is to treat an excess of lower-quality code output as the fixed thing to work with, and operationalize around that, but sure.
  • gtowey 7mo ago
    It's the same as the offshoring episode of the early 2000's. There is such a massive financial incentive to somehow make the low quality code work. And they will try to resist the reality that it's a huge net negative for as long as they can.
• cobolcomesback 7mo ago
  This “mandatory meeting” is just the usual weekly company-wide meeting where recent operational issues are discussed. There was a big operational issue last week, so of course this week will have more attendance and discussion. This meeting happens literally every week, and has for years. Feels like the media is making a mountain out of a mole hill here.
  • otterley 7mo ago
    > Feels like the media is making a mountain out of a mole hill here. That's been their job ever since cable news was invented.
    • ses1984 7mo ago
      It’s been a bit longer than that. https://en.wikipedia.org/wiki/Yellow_journalism https://en.wikipedia.org/wiki/Yellow_journalism It probably goes back as long as they have been shouting news in the town square in Rome or before that even.
      • otterley 7mo ago
        True enough!
      • lukan 7mo ago
        Word around the campfire is, telling stories and exaggerating them to get people attention, is as old as humanity. But good journalism is still something else.
  • davidclark 7mo ago
    The article claims: >He asked staff to attend the meeting, which is normally optional. Is that false? It also discusses a new policy: >Junior and mid-level engineers will now require more senior engineers to sign off any AI-assisted changes, Treadwell added. Is that inaccurate? It is good context that this is a regularly scheduled meeting. But, regularly scheduled meetings can have newsworthy things happen at them.
• dude250711 7mo ago
  I knew this would happen. Take a perfectly productive senior developer and instead make him be responsible for output of a bunch of AI juniors with the expectation of 10x output.
  • frogperson 7mo ago
    makes me want to vomit. I am not spending more time reviewing code than the "author" spent creating it. Ill just leave the industry if that happens.
    • hard24 7mo ago
      I think as long as having to review code stays around, the 'artistry' of writing code isn't going away. Think about it - how do you increase the speed at which one can review code? Well first it must be attractive to look at - the more attractive the faster you review/understand and move through the review. Now this won't be the case everywhere - e.g. in outsourced regions the conditions will force people to operate a certain way. Im not a SWE by trade, I just try to look at things from a pragmatic stand-point of how org's actually make incremental progress faster.
      • bombdailer 7mo ago
        The better looking the code, the less effort people will put into reviewing it due to the ease of reading it - the assumption being that what is beautiful is good. Just as a beautiful facade of a building can hide a cheap structure behind it, the same is true with code. Beauty itself is not a good signal for goodness as in excess it is in effect a rhetoric device that aims to mislead and draw ones eyes towards itself and away from what lies beneath it. A beautiful building is only as good as the correctness of its foundation, framework, materials, and construction. Those qualities can only be assessed by those with expertise enough to understand their importance. Beauty in its proper place is the output of the intersection between a craftsman and a engineer. Beauty is optional, but it makes life more worth living. The same is true for code - attractive code is optional, but it makes being a SWE more rewarding.
• ritlo 7mo ago
  The only way to see the kinds of speed-up companies want from these things, right now, is to do way too little review. I think we're going to see a lot of failures in a lot of sectors where companies set goals for reduced hours on various things they do, based on what they expected from LLM speed-ups, and it will have turned out the only way to hit those goals was by spending way too little time reviewing LLM output. They're torn between "we want to fire 80% of you" and "... but if we don't give up quality/reliability, LLMs only save a little time, not a ton, so we can only fire like 5% of you max". (It's the same in writing, these things are only a huge speed-up if it's OK for the output to be low-quality, but good output using LLMs only saves a little time versus writing entirely by-hand—so far, anyway, of course these systems are changing by the day, but this specific limitation has remained true for about four years now, without much improvement)
  • hard24 7mo ago
    My prediction is a concorde-like incident is going to shatter trust and make people re-think their expectations of the capabilities of LLMs and their abilities of the present. Essentially something big has to happen that affects the revenue/trust of a large provider of goods, stemming from LLM-use. They wont go away entirely. But this idea that they can displace engineers at a high-rate will.
    • Terr_ 7mo ago
      Assuming you mean this crash [0], it reads to me more like a confluence of bad events versus a big fundamental design flaw in the THERAC-25 mold. I feel the current proliferation of LLMs is going to resemble asbestos problem: Cheap miracle thingy, overused in several places, with slow gradual regret and chronic harms/costs. Although I suppose the "undocumented nasty surprise" aspect would depend on adoption of local LLMs. If it's a monthly subscription to cloud-stuff, people are far less-likely to lose track of where the systems are and what they're doing. [0] https://en.wikipedia.org/wiki/Air_France_Flight_4590 https://en.wikipedia.org/wiki/Air_France_Flight_4590
    • _wire_ 7mo ago
      Like bombing a building full of little kids? Oops too late...
• sethops1 7mo ago
  > The response for now? Junior and mid-level engineers can no longer push AI-assisted code without a senior signing off. So basically, kill the productivity of senior engineers, kill the ability for junior engineers to learn anything, and ensure those senior engineers hate their jobs. Bold move, we'll see how that goes.
  • dragonelite 7mo ago
    Accelerate a person speed toward being burned out..
    • altairprime 7mo ago
      ..and you lower overall engineering salary spend by rotating out seniority-paid engineers for newly-promoted AI reviewers with lower specs
  • almostdeadguy 7mo ago
    I'm sorry what? Junior engineers can't learn anything without using AI assistants (or is the implication that having seniors review their code makes them incapable of learning?) and senior engineer would hate their jobs reviewing more code from their teammates? What reality do people live in now?
    • ritlo 7mo ago
      > senior engineer would hate their jobs reviewing more code from their teammates Jesus, yes. Maybe I'm an oddball but there's a limit to how much PR reviewing I could do per week and stay sane. It's not terribly high, either. I'd say like 5 hours per week max, and no more than one hour per half-workday, before my eyes glaze over and my reviews become useless. Reviewing code is important and is part of the job but if you're asking me to spend far more of my time on it, and across (presumably) a wider set of projects or sections of projects so I've got more context-switching to figure out WTF I'm even looking at, yes, I would hate my job by the end of day 1 of that.
      • 7mo ago
• happytoexplain 7mo ago
  >Junior and mid-level engineers can no longer push AI-assisted code without a senior signing off Review by a senior is one of the biggest "silver bullet" illusions managers suffer from. For a person (senior or otherwise) to examine code or configuration with the granularity required to verify that it even approximates the result of their own level of experience, even only in terms of security/stability/correctness, requires an amount of time approaching the time spent if they had just done it themselves. I.e. senior review is valuable, but it does not make bad code good. This is one major facet of probably the single biggest problem of the last couple decades in system management: The misunderstanding by management that making something idiot proof means you can now hire idiots (not intended as an insult, just using the terminology of the phrase "idiot proof").
  • belval 7mo ago
    The unwritten thing is that if you need seniors to review every single change from junior and mid-level engineers, and those engineers are mostly using Kiro to write their CRs, then what stops the senior from just writing the CRs with Kiro themselves?
  • steveBK123 7mo ago
    Right, code reviews should already have been happening with human written junior code. If AI is a productivity boost and juniors are going to generate 10x the PRs, do you need 10x the seniors (expensive) or 1/10th the juniors (cost save). A reminder that in many situations, pure code velocity was never the limiting factor. Re: idiot prooofing I think this is a natural evolution as companies get larger they try to limit their downside & manage for the median rather than having a growth mindset in hiring/firing/performance.
  • marginalia_nu 7mo ago
    Expert reviews are just about the only thing that makes AI generated code viable, though doing them after the fact is a bit sketchy, to be efficient you kinda need to keep an eye on what the model is doing as its working. Unchecked, AI models output code that is as buggy as it is inefficient. In smaller green field contexts, it's not so bad, but in a large code base, it's performs much worse as it will not have access to the bigger picture. In my experience, you should be spending something like 5-15X the time the model takes to implement a feature on reviewing and making it fix its errors and inefficiencies. If you do that (with an expert's eye), the changes will usually have a high quality and will be correct and good. If you do not do that due dilligence, the model will produce a staggering amount of low quality code, at a rate that is probably something like 100x what a human could output in a similar timespan. Unchecked, it's like having a small army of the most eager junior devs you can find going completely fucking ape in the codebase.
• dragonelite 7mo ago
  Expect a shitload of AI powered code review products the next 18 months.
  • AlexeyBrin 7mo ago
    You mean like what Anthropic announced yesterday ? Code Review can review your code for $15 - $25 per review. /s So now, you can speed up using Claude Code and use Code Review to keep it in check.
  • gdulli 7mo ago
    "Why don't they just make the plane out of the black box?"
    • wiseowise 7mo ago
      “You wouldn’t write a code by hand, would you?”
  • daheza 7mo ago
    Create the problem and then create the solution.
    • 0x500x79 7mo ago
      Sell the solution. The Claude code review system is 15-25 dollars per-review!
    • recursive 7mo ago
      The TSA Pre-check monetization model
  • hard24 7mo ago
    This is incredibly circular lol...
  • booleandilemma 7mo ago
    Emphasis on shit.
• lokar 7mo ago
  If this is true, it misunderstands the primary goals of code review. Code review should not be (primarily) about catching serious errors. If there are always a lot of errors, you can’t catch most of them with review. If there are few it’s not the best use of time. The goal is to ensure the team is in sync on design, standards, etc. To train and educate Jr engineers, to spread understanding of the system. To bring more points of view to complex and important decisions. These goals help you reduce the number of errors going into the review process, this should be the actual goal.
  • rossdavidh 7mo ago
    As Deming once said in regard to manufacturing inspections: "Inspection does not improve the quality, nor guarantee quality. Inspection is too late. The quality, good or bad, is already in the product." The fact that software is "soft" makes it seem like this doesn't apply, but it does, not least because of the fact that once you have gone down the wrong path with software design, it is very difficult to pull back and realize you need to go down an entirely different one.
    • lokar 7mo ago
      I agree, but it's worse. Even a "simple" coding error (so, no long term arch issues) is a problem, if the review that catches it does not educate the author. The analogy to manufacturing would be something like if the parts coming out a machine are all bad, just sending them to re-work is not a solution, you need to re-calibrate the machine.
• bigbuppo 7mo ago
  Ugh. The Great Oops has never been closer.
• skeledrew 7mo ago
  > the affected tool served customers in mainland China Thought this blurb most interesting. What's the between-lines subtext here? Are they deliberately serving something they know to be faulty to the Chinese? Or is it the case that the Chinese use it with little to no issue/complaint? Or...?
• MDGeist 7mo ago
  A former colleague of mine recently took a role that has largely turned out to be "greybeard that reviews the AI slop of the junior engineers". In theory it sounds workable, but the volume of slop makes thoughtful review impossible. Seems like most orgs will just put pressure on the slop generators to do more and put pressure on the approvers and then scape goat the slop approvers if necessary?
  • dboreham 7mo ago
    This was always the case in the before times with humans. AI just pulls back the curtain on the delusion that code can and is being meaningful reviewed.
• oxqbldpxo 7mo ago
  Not fun to work at amazon.com it seems.
• mattschaller 7mo ago
  Anyone work with Kiro before? As I understood, it was held as an INTERNAL USE ONLY tool for much longer than expected.
  • daheza 7mo ago
    I used Kiro IDE and really liked it. The all you can eat model of LLM usage is very tempting compared to say Cursor. The features in the editor are basically the same. Haven't tried Kiro CLI.
  • riknos314 7mo ago
    The technique of creating specs before implementation that Kiro embodies was used widely internally before Kiro's release, but as a (now former) employee I gained access to the Kiro tool at the same time as the public. Others may have had internal access earlier but I'm not aware of them.
  • wenc 7mo ago
    I use Kiro IDE (≠ Kiro CLI) primarily as a spec generator. In my experience, it's high-quality for creating and iterating on specs. Tools like Cursor are optimized for human-driven vibing -- they have great autocomplete, etc. Kiro, by contrast, is optimized around spec, which ironically has been the most effective approach I've found for driving agents. I'd argue that Cursor, Antigravity, and similar tools are optimized for human steering, which explains their popularity, while Kiro is optimized for agent harnesses. That's also why it’s underused: it's quite opinionated, but very effective. Vibe-coding culture isn't sold on spec driven development (they think it's waterfall and summarily dismiss it -- even Yegge has this bias), so people tend to underrate it. Kiro writes specs using structured formats like EARS and INCOSE. It performs automated reasoning to check for consistency, then generates a design document and task list from the spec -- similar to what Beads does. I usually spend a significant amount of time pressure-testing the spec before implementing (often hours to days), and it pays off. Writing a good, consistent spec is essentially the computer equivalent of "writing as a tool of thought" in practice. Once the spec is tight, implementation tends to follow it closely. Kiro also generates property-based tests (PBTs) using Hypothesis in Python, inspired by Haskell's QuickCheck. These tests sweep the input domain and, when combined with traditional scenario-based unit tests, tend to produce code that adheres closely to the spec. I also add a small instruction "do red/green TDD" (I learned this from Simon Willison) and that one line alone improved the quality of all my tests. Kiro can technically implement the task list itself, but this is where agents come in. With the spec in hand, I use multiple headless CLI agents in tmux (e.g., Kiro CLI, Claude Code) for implementation. The results have been very good. With a solid Kiro spec and task list, agents usually implement everything end-to-end without stopping -- I haven’t found a need for Ralph loops. (agents sometimes tend to stop mid way on Claude plans, but I've never had that happen with Kiro, not sure why, maybe it's the checklist, which includes PBT tests as gates). Kiro didn't have the strongest start, but the Kiro IDE is one of the best spec generators I've used, and it integrates extremely well with agent-driven workflows.
• sdevonoes 7mo ago
  Reviewing AI generated code at PR time is a bottleneck. It cancels most of the benefits senior leadership thinks AI offers (delivery speed). There’s also this implicit imbalance engineers typically don’t like: it takes me 10 min to submit a complete feature thanks to Claude… but for the human reviewing my PR in a manual way it will take them 10-20 times that. Edit: at the end real engineers know that what takes effort is a) to know what to build and why, b) to verify that what was built is correct. Currently AI doesn’t help much with any of these 2 points. The inbetweens are needed but they are a byproduct. Senior leadership doesn’t know this, though.
  • beardedetim 7mo ago
    This is what I don't understand about this policy. There's no way a senior has enough spare capacity to be the gate keeper on every PR made by AI below them. So now we are just making it so the senior people use more AI to keep up but now they're to blame for letting it happen. It sounds like a piss poor deal for seniors unless senior engineer now means professional code reviewer.
    • malfist 7mo ago
      That's amazon in a nutshell though. Create conflicting metrics for performance, push credit up and responsibility down, punish everyone below you for not meeting the double standards
      • znpy 7mo ago
        > Create conflicting metrics for performance, push credit up and responsibility down, punish everyone below you for not meeting the double standards This resonates with my experience. The only thing you forgot is that you can also use the 12^H^H 14 leadership principles to argue whatever you want (and then the opposite of what you argued last month, still using the same leadership principles).
        • malfist 7mo ago
          Got a project finished early? Well, you didn't insist on the highest standards. Made sure things were held to a high standard? Well, you weren't biased for action. Were you a knowledge source for the entire team? Well, you weren't learning and being curious. Did you ask a lot of questions to learn everything? Well, then you weren't "are right a lot". Did you think big and come up with an architecture that saved Amazon a lot of money? Then you weren't inventing and simplifying. Build something simple to get out out the door quick? Well, you weren't thinking big. Did you act quickly without consulting others to fix an issue? Well you weren't earning trust. Did you consult people to make sure they were happy with the solution? Well you weren't biased for action. Thats just a few examples, there's so many more
• th2o34i3432897 7mo ago
  First Microsoft and now Amazon (eg. their RufusAI is useless compared to the old comment search!) Has Seattle now become the code-slop capital ? Or is SFO still on top ?
• Insanity 7mo ago
  It's only going to get worse with the brain drain as a result of the layoffs. Which will increase the use of AI assisted coding and increase the number of outages related to this. Imagine having to debug code that caused an outage when 80% is written by an LLM and you now have to start actually figuring out the codebase at 2am.. :)
  • 8note 7mo ago
    but thats what it was like when i started at amazon in 2016? i think the team i was on was a bit of an outlier in terms of owning 40 dumptser fires at once, and the first time reading any one of them was at 2AM because it was down. having an LLM give early passes on reading the godawful c++ code that you can tell at a glance that its not gonna work as expected, but you cant tell why, or what expected actually is would have been phenomenal, and gotten me back to sleep at 3 on those codebases rather than 5.
    • PessimalDecimal 7mo ago
      That's what it was like when you started out, but did you eventually learn that code? Imagine constantly getting out back into square one on understanding a legacy code base you just inherited, forever. This is what it's be like with constant LLM-induced churn on code repositories.
• throwaw12 7mo ago
  If Seniors are going to review every GenAI generated code, how do they keep up with the volume of changes? So you have 2 systems of engineers: Sr- and Sr+ 1. Both should write code to justify their work and impact 2. Sr- code must be reviewed by Sr+ What happens: a. Sr+ output drops because review takes their time more and more b. Sr+ just blindly accepts because of the volume is too high, and they should also do their own work c. Sr+ asks Sr- to slow-down, then Sr- can get bad reviews for the output, because on average Sr+ will produce more code I think (b) will happen
• LogicFailsMe 7mo ago
  For the good of the company's future, all code should be reviewed by L10s going forward before they are accepted. They're the only ones with enough skin in the game to know what really matters after all. And from their sagely reviews, we shall train a large language model to ultimately replace them because the most fungible thing at Amazon is the leadership.
• cmiles8 7mo ago
  The optics here are really bad for Amazon. The continuing mass departures of long tenured folks, second-rate AI products, and a string of bad outages paints a picture that current leadership is overseeing a once respected engineering train flying off the tracks. News from the inside makes it sound like things are getting pretty bad.
  • the_biot 7mo ago
    > The continuing mass departures of long tenured folks You mean senior programmers that have been there for ages don't want to spend their time reviewing AI slop? Who'd a thunk it!
• letitgo12345 7mo ago
  Worth noting that this is when they used Amazon's own AI product, not when using Claude Code or Codex.
• throw_m239339 7mo ago
  Yet another example of vibe coding at scale. You'll have to hire a lot of seniors out of retirement to fix that mess of gigantic proportions... and don't blame "the juniors" for that, they didn't make the decision to allow those tools at first place.
  • 10xDev 7mo ago
    A lot of juniors only graduated using these tools. Good luck taking it away from them.
    • throw_m239339 7mo ago
      Juniors don't set up these policies or even chose the tools they have to use professionally. If the higher ups are panicking it's fully of their own doing.
• smy20011 7mo ago
  An outage could cost Amazon ~millions to tens of millions. Most of the time, we want the junior to learn from the outage and fix the process. With AI agent, we can only update the agent.md and hope it will never happen again.
• butILoveLife 7mo ago
  Maybe its my 1 buddy that works at amazon, but they seemed extremely slow to adopt LLMs. Big ships take a long time to turn, but this seemed hostile. I am seeing this mindset still, with AI Agents. I imagine they will slowly realize they need to use this stuff to be competitive, but being slow to adopt AI seems like it could have been the source of this.
  • lmc 7mo ago
    LLMs have been garbage for real work until very recently. Doesn't this show they were adopted too soon at amazon?
    • butILoveLife 7mo ago
      Disagree, I've been using it for at least a year to write functions.
    • bigstrat2003 7mo ago
      They're still garbage for real work.
• fredgrott 7mo ago
  Curious question, how many Amazon Engineers flunk basic CS? If you know CS you know two things: 1. AI can not judge code either noise or signal, AI cannot tell. 2. CS-wise we use statistic analysis to judge good code from bad. How much time does it take to take AI output and run the basic statistic tools for most computer languages? Some juniors need firing outright
  • 8note 7mo ago
    that isnt computer science at all though? if it is, that would be phd research topics, rather than than basics. maybe as software engineering topics, but thats a different discipline
  • wiseowise 7mo ago
    “Haha, sure man. CSI, great tv series. Look, just review Jeff’s PR and move that button down, okay? Have a tennis lesson, kbye”
• AlexandrB 7mo ago
  "We want you to use AI for everything!" "No, not like that though!"
• prakhar897 7mo ago
  From the amazon I know, people only care about a. not getting fired and b. promotions. For devs, the matrix looks like this: 1. Shipping: deliver tickets or be pipped. 2. Having Less comments on their PRs: for some drastically dumb reason, having a PR thoroughly reviewed is a sign of bad quality. L7 and above use this metric to Pip folks. 3. Docs: write docs, get them reviewed to show you're high level. Without AI, an employee is worse off in all of the above compared to folks who will cheat to get ahead. I can't see how "requesting" folks for forego their own self-preservation will work. especially when you've spent years pitting people against each other.
  • malfist 7mo ago
    Not only is having too many comments on your PRs bad for you, but so is not leaving comments on other people's PRs. Both are metrics used
    • dude250711 7mo ago
      I'd leave lots of comments out of spite whenever I would feel my PRs had been treated unfairly. If I am going down, you all are coming with me.
      • malfist 7mo ago
        Welcome to Amazon, you'll fit right in.
      • grogenaut 7mo ago
        I specifically look at the quality / substance of the comments when I'm reviewing someone for promo/transfer/fire.
  • embedding-shape 7mo ago
    > 2. Having Less comments on their PRs: for some drastically dumb reason, having a PR thoroughly reviewed I'm very far away from liking Amazon's engineering culture and general work culture, but having PRs with countless of discussions and feedback on it does signal that you've done a lot of work without collaborating with others before doing the work. Generally in teams that work well together and build great software, the PRs tend to have very little on them, as most of the issues were resolved while designing together with others.
• julienchastang 7mo ago
  > best practices and safeguards are not yet fully established The way I am working with AI agents (codex) these days is have the AI generate a spec in a series of MD documents where the AI implementation of each document is a bite sized chunk that can be tested and evaluated by the human before moving to the next step and roughly matches a commit in version control. The version control history reflects the logical progression of the code. In this manner, I have a decent knowledge of the code, and one that I am more comfortable with than one-shotting.
  • danjl 7mo ago
    Yes, more time on up front spec and plan building. Bite sized specifically to fit within the context window of a single implementation session. Each step should have a verification process that includes new tests. Prior to each step, I prompt the AI to review the step and ask clarifying questions to fill any missing details. Then implement. Then prompt the AI after to review the changes for any fixes before moving on to the next step. Rinse, repeat. The specs and plans are actually better for sharing context with the rest of the team than a traditional review process. I find the code generated by this process to be better in general than the code I've generated over my previous 35+ years of coding. More robust, more complete, better tested. I used to "rush" through this process before, with less upfront planning, and more of a focus on getting a working scaffold up and running as fast as possible, with each step along the way implemented a bit quicker and less robustly, with the assumption I'd return to fix up the corner cases later.
• dedoussis 7mo ago
  How do they determine whether a PR is AI-assisted and therefore requires senior review? A junior engineer could still copy-paste AI-generated code and claim it as their own.
  • emotiveengine 7mo ago
    Right? If they're using some sort of tool, there's always another tool to fool the tool.
• tcbrah 7mo ago
  the funniest part is amazon literally started tying AI usage to performance reviews like 6 months ago and now theyre doing damage control. you cant simultaneously pressure every engineer to use more AI AND be shocked when AI-assisted code breaks prod. pick one lol
  • PessimalDecimal 7mo ago
    Why can't they?
    • tcbrah 7mo ago
      because the incentive structure is broken. if my performance review rewards me for using AI more, im going to use AI more even when i shouldn't. engineers will rubber stamp AI suggestions to hit their metrics instead of actually reviewing the code. you cant optimize for quantity of AI usage and quality of output at the same time IMO
• 10xDev 7mo ago
  With AI it makes sense to have leaner teams. Being able to go faster requires greater responsibility.
• mhogers 7mo ago
  .agentignore/.agentnotallowed file force agents to not touch mission critical things, fail in CI otherwise let it work on frontends and things at the frontier of the dependency tree, where it is worth the risk
  • readthemanual 7mo ago
    a) what happens if there is change that hasn't been encountered yet so it's not in .agentnotallowed? b) is there a guarantee that something described in these files won't be touched? I've seen examples when agents directly violate these rules, profusely apologising after they get caught on it.
    • mhogers 7mo ago
      allowlist instead of denylist, depending on your risk profile :)
• dlev_pika 7mo ago
  A few days ago, after some very weird failed purchase attempts I made (payment couldn’t be validated or Smth) I received an even weirder mail from Amazon saying they had detected suspicious activity, all my devices got logged out and I was forced to change my password. I did it, after verifying it was a legit email (even if it looked sketchy af, pure text, unstyled, but sender verified and confirmed with in-app behavior), and next I know all my orders and browsing history had disappeared - +15 yrs of history, done. Over the next few days my account history came back, except purchases made Q1 2026. Those are still missing. There are a few substantial purchases I made that are nowhere to be found anymore. I attributed this Iranian missiles hitting some of their infrastructure in EU, as it had been reported. Now I am not sure if it was blast radius from missiles or AI mishaps. Lmao - couldn’t happen to a worse company…
• zcw100 7mo ago
  I just met a guy from Amazon this past weekend who was bragging, "We've got unlimited access to LLMs and our developers have 10 agents going at a time.". I tried telling him it wasn't all unicorns and rainbows but I didn't get the impression he cared and just kept crapping out skittles.
  • agoodusername63 7mo ago
    Because he doesn’t. The impression I get from SWEs I’ve met throughout my life is that most of them don’t actually care about their job. They got in because it paid well and demand was plentiful.
    • booleandilemma 7mo ago
      It's rare when I meet one who actually likes writing code nowadays. At my last company everyone was trying to be an architect.
      • jacquesm 7mo ago
        I still like writing code. But not for others.
• m3kw9 7mo ago
  A year later, they will require AI to sign off engineer changes.
• adrien_dev 7mo ago
  [dead]
• rvz 7mo ago
  Hope this happens at GitHub since there are constant outages on the entire platform.
• newobj 7mo ago
  Speed of code-writing was never the issue at Amazon or AWS. It was always wrong-headed strategic directions, out to lunch PMs, dogshit testing environment, stakeholder soup, high turnover, bureaucracy, a pantheon of legacy systems, insane operational burdens, garbage tooling, and last but not last -- designing for inter-system failure modes, which let's be real, AI has no chance of having context for -- and so on... Imagine if the #1 problem of your woodworking shop is staff injuries, and the solution that management foists on you is higher RPM lathes.
• ChrisArchitect 7mo ago
  Previously: https://news.ycombinator.com/item?id=47319273 https://news.ycombinator.com/item?id=47319273
• teeray 7mo ago
  > Junior and mid-level engineers can no longer push AI-assisted code without a senior signing off So what incentive is there for juniors to look at the code at all? Seniors are now just another CI stage for their slop to pass.
• mikkupikku 7mo ago
  lgtm
• throwaway613746 7mo ago
  [dead]
• philip1209 7mo ago
  I think the deeper need is a "self-review" flow. People push AI-reviewed code like they wrote it. In the past, "wrote it" implies "reviewed it." With AI, that's no longer true. I advocate for GitHub and other code review systems to add a "Require self-review" option, where people must attest that they reviewed and approved their own code. This change might seem symbolic, but it clearly sets workflows and expectations.
  • Tyr42 7mo ago
    Heck, doing a self review when you wrote the code catches stuff like forgetting debug prints.
    • therealdrag0 7mo ago
      Self review should also include adding guiding comments for other reviewers.
      • fireant 7mo ago
        Do you add these into the code or into the review itself? I sometimes write these into the review, but I wonder if it's a useful information that should actually be inside the code that will get lost when the PR is merged
        • therealdrag0 7mo ago
          Into the review is what I’m talking about. The diff is often a scattered collection of files missing context, and may have refactors that obscure behavioral changes. So there is reason to add comments that address a different readers understanding than the code rest.
    • nothrabannosir 7mo ago
      (tangent of the decade : prefixing your debug printfs with NOCOMMIT helps catching them before commit :) sample precommit hook and GitHub ci action I wrote is at https://github.com/nobssoftware/nocommit https://github.com/nobssoftware/nocommit but it’s just a grep)
• znpy 7mo ago
  "Make senior engineer sign off ai-assisted changes" sounds incredibly weird. First thing that comes to mind is: reminds me of those movie where some dictatorship starts to crumble and the dictator start being tougher and tougher on generals, not realizing the whole endeavor is doomed, not just the current implementation. Then again, as a former amazon (aws) engineer: this is just not going to work. Depending how you define "senior engineer" (L5? L6? L7?) this is less and less feasible. L5 engineers are already supposed to work pretty much autonomously, maybe with L6 sign-off when changes are a bit large in scope. L6 engineers already have their own load of work, and a fairly large amount of engineers "under" them (anywhere from 5 to 8). Properly reviewing changes from all them, and taking responsibility for that, is going to be very taxing on such people. L7 engineers work across teams and they might have anywhere from 12 to 30 engineers (L4/5/6) "under" them (or more). They are already scarce in number and they already pretty much mostly do reviews (which is proving not sufficient, it seems). Mandating sign-off and mandating assumption of responsibility for breaking changes means these people basically only do reviews and will be stricter and stricter[1] with engineers under them. L8 engineers, they barely do any engineering at all, from what I remember. They mostly review design documents, in my experience not always expressing sound opinions or having proper understanding of the issues being handled. In all this, considering the low morale (layoffs), the reduced headcount (layoffs) and the rise in expectations (engineers trying harder to stay afloat[2] due to... layoffs)... It's a dire situation. I'm going to tell you, this stinks A LOT like rotting day 2 mindset. ---- 1. keep in mind you can't, in general, determine the absence of bugs 2. Also cranking out WAY MUCH MORE code due to having gen-ai tools at their fingertips...
• rglover 7mo ago
  The amount of time and money being wasted chasing this dragon is unreal.
  • alexyz12 7mo ago
    there's nothing else left to chase apparently
  • dana321 7mo ago
    Its useful but it makes wrong assumptions, not checking the code is essentially gambling.
• mentos 7mo ago
  What are we going to do about software for critical infrastructure in the coming decade? Feels inevitable that code for aviation will slowly rot from the same forces at play but with lethal results.
  • rhubarbtree 7mo ago
    It will be written much the same as now, but using AI to improve quality through code inspection etc. Just because nearly all software is going to be written by AI, does not mean critical infrastructure will be.
    • bandrami 7mo ago
      At some point people are going to start asking why software that doesn't need to be right should exist in the first place
      • rhubarbtree 7mo ago
        I’m not sure I follow, but I don’t hear anyone asking those questions of the bad software we already have today.
        • deleted 7mo ago
          [deleted]
  • deleted 7mo ago
    [deleted]
  • franktankbank 7mo ago
    Cabin in the woods.
• secondcoming 7mo ago
  Has Amazon's advertising TAM product been affected by AI?
• fmajid 7mo ago
  If they do not also increase the senior devs’ allotted time for code reviews to make up for the increased volume of changes due to increased productivity of the junior to mid level devs, or hire more seniors, this will just lead to burnout (on top of Amazon’s already high levels) and scapegoating seniors for having waved through a change because they materially can’t review them fast enough.
  • 8note 7mo ago
    Id maybe consider spinning out a different job role for like "review engineer" who's not busy making strategic decisions and near term planning, just making sure the code is actually good still within the engineering IC role, but on a different track
    • deleted 7mo ago
      [deleted]
    • fmajid 7mo ago
      I like to describe this as being "the person who know what good is". Sounds supiciously like the return of the SDET role...
• daxfohl 7mo ago
  You could create an agent template for each incident you've ever had, with context pre-cached with the postmortem report, full code change, and any other information about the incident. Then for every new PR you could clone agents from all those templates and ask whether the PR could cause something similar to the pre-loaded incident. If any of them say yes, reject the PR unless there's a manual override. You'd never have a repeat incident. Obviously it's probably cost-prohibitive to do an all to all analysis for every PR, but I imagine with some intelligent optimizations around likelihood and similarity analysis something along those lines would be possible and practical.
  • iLoveOncall 7mo ago
    You vastly underestimate the complexity of systems in a company like Amazon. COEs and Operation Readiness Reviews are already the documents that you mention, but they are largely useless in preventing incidents.
    • senderista 7mo ago
      Actually CoEs would make an amazing training corpus for code reviews.
  • 8note 7mo ago
    code review is too late to give some of that feedback, and design/requirements documents dont have nearly the standardization of presentation and feedback tools for that to be useable. Amazon does have those things, and has fine tuning on models based on those postmortems. Noisy reviews are also a problem causer. the PR doesnt know what scale a chunk of code is running at, without having access to 20 more packages and other details.
• monster_truck 7mo ago
  Have they tried simply not writing bugs? I've found that works best for me personally
• softwaredoug 7mo ago
  Getting junior / mid-level people to slop cannon PRs at seniors will just burn out seniors. The team might be better having fewer developers using AI more thoughtfully.
• VorpalWay 7mo ago
  I'm bewildered by Amazon here. I would assume every change require code review by another enigneer already, as is standard practice in the industry I work in (industrial equipment). Is the change just that it has to be a senior engineer specifically, rather than any engineer? Or did Amazon really not have mandatory code review before?
  • shepherdjerred 7mo ago
    Code review is mandatory at Amazon
• andai 7mo ago
  So the take-away here is maybe we should read the code that "we" wrote? :) (Before injecting it into global infra...)
• sizzzzlerz 7mo ago
  Who fixes code that gets rejected? Do you simply try again and hope or does someone go into this computer-generated code that they didn't write and do the equivalent of battlefield triage? And what are they going to do when they've fired all the senior engineers because they make too much money, leaving just juniors and AI?
  • throwaway613746 7mo ago
    [dead]
  • rhubarbtree 7mo ago
    AI will fix it, same way AI wrote it. At the behest of a human. When they fire everyone, juniors will fix it with AI. This is in general. I wouldn’t recommend this at critical services like AWS.
    • sizzzzlerz 7mo ago
      Or in airplanes, nuclear power plants, spacecraft, CAT scanners, ECGs, traffic control systems, navigation devices, warehouse management systems, banking. Feel free to add your own.
      • rhubarbtree 7mo ago
        Doubt warehouse management systems will fit in there, and only critical systems at banking. But yes agree with the rest, which probably makes up a tiny tiny fraction of the software created today, and will be orders of magnitude smaller as a fraction in the future.
• xodn348 7mo ago
  very expected outcomes.
• ihsw 7mo ago
  [dead]
• rushabh 7mo ago
  Unfortunately you can’t just yell at the AI so it learns never to do this again. Humans take such a large range of feedback that LLMs can’t.
• sailfast 7mo ago
  I anticipate they will fix this by adding better AI evaluation tools that work better to test their infra and changes. In the meantime they will be quite a bit slower I’d imagine. Also wonder if those seniors will ever get to actually do any engineering themselves now that they’re the bottleneck. :)
• varenc 7mo ago
  digression: the long twitter urls make this entire page wider and the text smaller on iOS for me. Feels like a minor bug. Maybe a `overflow-wrap: anywhere` CSS rules needs to be added to URLs.
• desireco42 7mo ago
  So essentially they will be blamed, everything will stay the same. I do consulting and use AI a lot. You just have to take responsibility for the code. We are delivering like never before, but have a lot of experience into how to do it as safe as possible. And we are learning along the way. They say you need a year to build up experience fyi. I feel bad for those engineers who will have to sign off for things they will most likely not have enough time to review. Kiro is nice and all.
• luxuryballs 7mo ago
  They weren’t already signing off on them? o.O
• recallingmemory 7mo ago
  .. So our jobs aren't going away?
• testbjjl 7mo ago
  Then what’s the point of AI? Pay for the code gen, pay a human to review the code gen, when the senior can train a junior and coordinate output with their incentives and performance reviews, problems largely solved. Seems to me too low level in everyone’s stack to not have humans doing the work, especially at this stage. But what do I know, I certainly am not at the helm of a multibillion dollar operation.
  • booleandilemma 7mo ago
    To make the AI companies money and prop up the American economy, obviously.
• aplomb1026 7mo ago
  [flagged]
• paxys 7mo ago
  Someone should teach the decision makers how pipelines work. If AI-created diffs are being churned out at 10x the previous rate but manual reviews are the bottleneck then the overall system is producing at the exact same rate as before. The only thing you have added is cost, uncertainty and engineers being less familiar with the system.
  • booleandilemma 7mo ago
    The next thing these geniuses will think of will be to have AI review the diffs.
  • Herring 7mo ago
    Check back in a year or two. Just the other day we had Claude finding significant bugs in Firefox. https://news.ycombinator.com/item?id=47273854 https://news.ycombinator.com/item?id=47273854
• moomoo11 7mo ago
  If you don't use ~crypto~ AI you will go broke!
• jmspring 7mo ago
  "After outages due to outsourcing the economically convenient developers with no skin in what your building or care, company X requires all senior engineers to review all code from outsourcing company".
• locopati 7mo ago
  use AI! no! not that way!
• adamzwasserman 7mo ago
  I'm sure they are going to have a ball reading through thousands of lines of AI slop.
• 827a 7mo ago
  > Company that lays-off 20% of its staff every year in an attempt to "reduce inefficiency" and "remain agile in the adoption of new technologies and workflows" finds they cannot run a stable service, have more inefficiency than ever, and have also failed to establish leadership in the adoption of any new technologies or workflows. They plan to solve these problems by introducing more inefficiency (making your most expensive employees review the work of others). We love this for Amazon, they're a very strong company making bold decisions.
• zouhair 7mo ago
  Some years of AI Technical Debt will be something to behold.
• captainkrtek 7mo ago
  One challenge with code review as an antidote to poor quality gen-AI code, is that we largely see only the code itself, not the process or inputs. In the pre-gen-AI days, if an engineer put up a PR, it implied (somewhat) they wrote their code, reviewed it implicitly as they wrote it, and made choices (ie: why is this the best approach). If Claude is just the new high level programming language, in terms of prompting in natural language, the challenge is that we're not reviewing the natural language, we're reviewing the machine code without knowing what the inputs were. I'm not sure of a solution to this, but something along the lines of knowing the history of the prompting that ultimately led to the PR, the time/tokens involved, etc. may inform the "quality" or "effort" spent in producing the PR. A one-shotted feature vs. a multi-iteration feature may produce the same lines of code and general shape, but one is likely to be higher "quality" in terms of minimal defects. Along the same lines, when I review some gen-AI produced PR, it feels like I'm reading assembly and having to reverse how we got here. It may be code that runs and is perfectly fine, but I can't tell what the higher level inputs were that produced it, and if they were sufficient.
• rubyrfranklin2 7mo ago
  We ran into something similar at heyvid.ai — shipped AI-generated code without a proper review gate and ended up with a subtle bug in our rendering pipeline that took the team a week to trace. Not catastrophic, but it seriously eroded trust in the tooling for a while. Amazon's approach makes total sense at their scale. The honest reality is that LLMs are great at producing plausible-looking code and genuinely bad at knowing when they're wrong. Senior sign-off isn't overhead — it's what makes AI-assisted development actually sustainable.
• oliver_dr 7mo ago
  [dead]
• jacknews 7mo ago
  This looks like a blame allocation exercise to me. The seniors will now be directly responsible for all the AI slop that goes in. But how can they possibly properly review reams of code to a sufficient degree they can personally vouch for it?
• jwpapi 7mo ago
  How much of damage is 6 hours offline for Amazon?
• fud101 7mo ago
  This is what humans will become, on call to take the blame for AI. It will be less about skill and confidence and more about being on the hook to take the fall for when things go wrong.
• joeyguerra 7mo ago
  I’m wonder how many sr engineers are going to quit because they don’t want to read a bunch of code?
• 33MHz-i486 7mo ago
  In case it isn’t completely obvious from this, it is indeed hellish to work there. Most of AWS has a 2 reviewer requirement. If AI is writing most of the code (and it is because most Amazon code is copypasta boilerplate) you need 3 developers to sign off to ship anything. But of course due to headcount attrition, managers have ~1.5 developers to a project. Meanwhile the L8 manager is doing nothing except stack ranking each level of engineers according to number of commits merged & customer facing features shipped, and firing 15% of the bottom at the end of each year. There is no notion of subject matter expertise or technical depth, theyre happy to replace whoever with fresh-grads (theyre all just cogs anyway right!). Between that and voluntary departures, teams having 80-100% turnover every 5 years is basically par. Also while this is happening most developers are getting constantly hammered by operational issues and critical security tasks because 1) the legacy toolchain imports 6 different language package ecosystems and 2)no one ever pays down tech debt in legacy code until its a high severity ticket count in a KPI dashboard visible to the senior management.
  • mikert89 7mo ago
    The thing is, this management philosophy worked when AWS knew what they needed to build and just needed to execute with top notch operations. But now with AI, they are getting disrupted. Most AWS services might become obsolete, why does an ai need these janky higher levels abstractions AWS piles on. So now they need innovation, but the company isn’t set up for it. They are forcing short deadlines for product launches that don’t matter
    • 33MHz-i486 7mo ago
      its not even AI. most of the cloud offerings are commodities now. the marginal technological direction is determined by middle managers whoes primary motivation is “what new customer facing feature can I launch at this years re:invent and build a little empire” (of course this is a shrinking offering as tech debt and complexity pile up) junior engineers are burned and churned on execution, seniors are project managers, principals just do high level reviews & high level fire fighting (note not actually leading the tech) director and above just their spend time on “what to kill” or “who to fire” as priorities change every 6 months
• nickvec 7mo ago
  So Amazon senior SWEs now have to review every single PR for all intents and purposes? I didn't think Amazon could get worse.
• camillomiller 7mo ago
  Such fun. On top of your already strapped schedule, now you have to bet your career on vibe code that you will now have to spend time reading and debugging. All that instead of a chain of accountability that has people in place instead of stupid bots with fake agency. This is beyond corporate satire. There was never before a technology capable of convincing leadership of its usefulness despite its constant blunders and despite the low quality of its output. This feels like a corporate mass delusion of unprecedented scale.
• petterroea 7mo ago
  I feel bad for the seniors who have to take on this workload. The general pattern I am seeing is that seniors at "AI-first" companies are being held back from doing their work by reviewing junior PRs, who are now able to ship much more code they don't understand the badness of. Mentoring Juniors is an important part of the job and crucial service to the industry, but juniors equipped with LLMs make the deal a bit more sour. Anecdotally, they don't really remember the feedback as well, because they weren't involved in writing the code. Its burnout-inducing to see your hard work and feedback go in one ear and out another. I personally know people looking to jump ship because they waste too much time at their current employer on this.
  • wiseowise 7mo ago
    > Mentoring Juniors is an important part of the job and crucial service to the industry Not really.
• tracerbulletx 7mo ago
  The way we used to build confidence in what we shipped was by beating our head against it for a week figuring it all out. You really can't have the same confidence with code reviews unless you basically do the same work you'd do to write it by hand for a lot of these things.
• vetrom 7mo ago
  AI seems to be the whipping boy, but to me, it really seems more of a symptom than a cause. At its root, isn't this an issue of a decline in critical thinking? I do think AI adoption exacerbates said falloff.
• qxxx 7mo ago
  so, seniors will review now the AI slop code.. I am also doing this task and reviewing this kind of code takes time as the code is often overengineered. Code works but will have potential bugs. I am not able to find every bug or implication quickly. But I am also using ai to review the ai slop lol, because why now. After that I am also quickly reviewing by myself.
• PeterStuer 7mo ago
  There's going to be some serious acceleration in senior engineer burnout. Maybe they can use more genAI to support their increased workload.
• mancerayder 7mo ago
  Is the rebuttal posted anywhere? I collapsed the huge first few threads but nothing is there. True or false, Amazon are saying it's not true that it's due to AI, and in fact their change in operational processes to add review is broader: https://www.aboutamazon.com/news/aws/aws-service-outage-ai-bot-kiro https://www.aboutamazon.com/news/aws/aws-service-outage-ai-b...
• HeavyStorm 7mo ago
  This is so fucking ridiculous.
• d0mine 7mo ago
  16000 layoffs cost just $180 million—it is a win so far https://x.com/gothburz/status/2031778265958842541 https://x.com/gothburz/status/2031778265958842541