7 ms·
>Junior and mid-level engineers can no longer push AI-assisted code without a senior signing off Review by a senior is one of the biggest "silver bullet" illus
by happytoexplain 8mo ago
>Junior and mid-level engineers can no longer push AI-assisted code without a senior signing off
Review by a senior is one of the biggest "silver bullet" illusions managers suffer from. For a person (senior or otherwise) to examine code or configuration with the granularity required to verify that it even approximates the result of their own level of experience, even only in terms of security/stability/correctness, requires an amount of time approaching the time spent if they had just done it themselves.
I.e. senior review is valuable, but it does not make bad code good.
This is one major facet of probably the single biggest problem of the last couple decades in system management: The misunderstanding by management that making something idiot proof means you can now hire idiots (not intended as an insult, just using the terminology of the phrase "idiot proof").
- belval 8mo agoThe unwritten thing is that if you need seniors to review every single change from junior and mid-level engineers, and those engineers are mostly using Kiro to write their CRs, then what stops the senior from just writing the CRs with Kiro themselves?
- steveBK123 8mo agoRight, code reviews should already have been happening with human written junior code. If AI is a productivity boost and juniors are going to generate 10x the PRs, do you need 10x the seniors (expensive) or 1/10th the juniors (cost save). A reminder that in many situations, pure code velocity was never the limiting factor. Re: idiot prooofing I think this is a natural evolution as companies get larger they try to limit their downside & manage for the median rather than having a growth mindset in hiring/firing/performance.
- marginalia_nu 8mo agoExpert reviews are just about the only thing that makes AI generated code viable, though doing them after the fact is a bit sketchy, to be efficient you kinda need to keep an eye on what the model is doing as its working. Unchecked, AI models output code that is as buggy as it is inefficient. In smaller green field contexts, it's not so bad, but in a large code base, it's performs much worse as it will not have access to the bigger picture. In my experience, you should be spending something like 5-15X the time the model takes to implement a feature on reviewing and making it fix its errors and inefficiencies. If you do that (with an expert's eye), the changes will usually have a high quality and will be correct and good. If you do not do that due dilligence, the model will produce a staggering amount of low quality code, at a rate that is probably something like 100x what a human could output in a similar timespan. Unchecked, it's like having a small army of the most eager junior devs you can find going completely fucking ape in the codebase.
- locusofself 8mo agoIf you spend 5-15x the time reviewing what the LLM is doing, are you saving any time by using it?
- bluGill 8mo agoSome, but not very much. Writing code is hard. Ai will do a lot of tedious code that you procrastinate writing.
- hard24 8mo agoAlso when you are writing code yourself you are implicitly checking it whilst at the back of your mind retaining some form of the entire system as a whole. People seem to gloss over this... As a CEO if people don't function like this I'd be awake at night sweating.
- bluGill 8mo agoSortof. I work on a system too large for anyone to know the whole thing. Often people who don't know each other do something that will break the other. (Often because of the number of different people - most individuals go years between this)
- bonesss 8mo agoThat’s the reverse-centaur issue I see: humans are not great at repetitive nuanced similar seeming tasks, putting the onus on humans to retroactively approve high volumes of critical code has them managing a critical failure mode at their weakest and worst. Automated reviews should be enhancing known good-faith code, manual reviews of high volume superficially sound but subversive code is begging for issues over time. Which results the software engineering issue I’m not seeing addressed by the hype: bugs cost tens to hundreds of times their coding cost to resolve if they require internal or external communication to address. Even if everyone has been 10x’ed, the math still strongly favours not making mistakes in the first place. An LLM workflow that yields 10x an engineer but psychopathically lies and sabotages client facing processes/resources once a quarter is likely a NNPP (net negative producing programmer), once opportunity and volatility costs are factored in.
- yifanl 8mo agoSenior reviews are useful, but as I understand it, Amazon has a fairly high turnover rate, so I wonder just how many seniors with deep knowledge of the codebase they could possibly have.
- tartoran 8mo agoFrom engineers are interchangeable to high turnover are decisions that the company took. The payback time always comes at some point.
- grvdrm 8mo agoWhat a statement at the end. You are absolutely right. I hear “x tool doesn’t really work well” and then I immediately ask: “does someone know how to use it well?” The answer “yes” is infrequent. Even a yes is often a maybe. The problem is pervasive in my world (insurance). Number-producing features need to work in a UX and product sense but also produce the right numbers, and within range of expectations. Just checking the UX does what it’s supposed to do is one job, and checking the numbers an entirely separate task. I don’t many folks that do both well.
- RamblingCTO 8mo agoWho said PR reviews need to solve all the things and result in proof against idiots? So you're saying that peer reviews are a waste of time and only idiots would use/propose them?
- happytoexplain 8mo agoNone of that, sorry if I wasn't clear. To partially clarify: "Idiot proof" is a broad concept that here refers specifically to abstraction layers, more or less (e.g. a UI framework is a little "idiot proof"; a WYSIWYG builder is more "idiot proof"). With AI, it's complicated, but bad leadership is over-interpreting the "idiot proof" aspects of it. It's a phrase, not an insult to users of these tools.
- jetrink 8mo agoIt could create the right sort of incentives though. If I'm a junior and I suddenly have to take my work to a senior every time I use AI, I'm going to be much more selective about how I use it and much more careful when I do use it. AI is dangerous because it is so frictionless and this is a way to add friction. Maybe I don't have the correct mental model for how the typical junior engineer thinks though. I never wanted to bug senior people and make demands on their time if I could help it.
- devonbleak 8mo agoWhat you're actually going to see is seniors inundated by slop and burning out and quitting because what used to be enjoyable solving of problems has become wading through slop that took 10 minutes to generate and submit but 30+ minutes to understand and write up a critique for it.
- SpicyLemonZest 8mo agoIn my experience, Claude and the juniors piloting it are usually receptive to quick feedback along the lines of "This is unreasonably hard to understand, please try refactoring it this way and let me know when it's cleaner".
- suzzer99 8mo agoCan I interest you in a bunch of emoji-laden comments?
- 8note 8mo agonot even that. implementing this requirement could be a general work stoppage whenever the senior engineer is in all day meetings or on vacation With a layout of 4 juniors, 5 intermediates, and 0-1 senior per team, putting all the changes through senior engineer review means you mostly wont be able to get CRs approved. I guess it could result in forcing everyone who's sandbagging as intermediate instead of going to senior to have to get promoted?
- js8 8mo ago> requires an amount of time approaching the time spent if they had just done it themselves It's actually often harder to fix something sloppy than to write it from scratch. To fix it, you need to hold in your head both the original, the new solution, and calculate the difference, which can be very confusing. The original solution can also anchor your thinking to some approach to the problem, which you wouldn't have if you solve it from scratch.
- bluGill 8mo agoSloppy code that has been around for a while works. It likely has support for edge cases you forgot about. Often the sloppyness is because of those edge cases.
- js8 8mo agoThat's the incidental (necessary) vs accidental (avoidable) complexity distinction. But I don't think it makes it any easier to deal with.
- bluGill 8mo agothose are different things. Often you don't plan for all the necessary things and so it doesn't fit in - even though a better design evists that would have it fit in neater - but only years latter do you see it and getting there is now a massive effort you can't afford. The result looks sloppy because on hindsight right is obvious
- ummonk 8mo agoIn fairness though, it does give you good practice for the essential skill of maintaining / improving an old codebase.
- ardeaver 8mo agoWhen I was really early in my career, a mentor told me that code review is not about catching bugs but spreading context (i.e. increasing bus factor.) Catching bugs is a side effect, but unless you have a lot of people review each pull request, it's basically just gambling. The more expensive and less sexy option is to actually make testing easier (both programmatically and manually), write more tests and more levels of tests, and spend time reducing code complexity. The problem, I think, is people don't get promoted for preventing issues.
- bluGill 8mo ago> people don't get promoted for preventing issues. they do - but only after a company has been burned hard. They also can be promoted for their area being enough better that everyone notices. still the best way to a promotion is write a major bug that you can come in at the last moment and be the hero for fixing.
- tartoran 8mo agoThat could work but plenty of quiet heros weren’t promoted for fixing critical bugs.
- recursive 8mo agoThey fixed it too soon. You have to wait until the effect is visible on someone's dashboard somewhere.
- bluGill 8mo agoYou have to make sure it doesn't arrive at you before it is on the dashboard. Otherwise you are why it is blowing up the time to fix a bug metric. Unless you can make the problem so obscure other smart people asked to help you can't figure it out thus making you look bad.
- marcta 8mo agoGoodhart's Law strikes again... "When a measure becomes a target, it ceases to be a good measure."
- femiagbabiaka 8mo agothe outcome of the review isn't just that the code gets shipped, it's knowledge transfer from the senior engineer to the junior engineers that then creates more senior engineers
- mrbonner 8mo agoWhat stops the senior from using AI to review the AI generated code the junior published?
- tartoran 8mo agoThat’s something that the junior can do. What companies want to do is put responsibility on someone who has more knowledge and skin in the game
- lokar 8mo agoThe goal of Sr code review is not to make the code better, it's to make the author better.
- skeeter2020 8mo agoAgree but even broader: authors. I always viewed reviews as targeting Brook's less famous findings about the optimal team size being one, and asking how can we get better at building systems too big for the individual. I think code review is about shared, consistent understanding with catching bugs a nice side effect (or justification for the bean counters).
- lokar 8mo agoI agree, made (mostly) that point in my top level comment. Code reviews (both in the normal GitHub flow, but also small meetings, design reviews, etc) all help to tie the team together and improve quality.
- sumeno 8mo agoThat's not going to work when the author is an LLM
- raw_anon_1111 8mo agoWhy only AI generated code? I wouldn’t let a junior or mid level developer’s code go into production without at least verifying the known hotspots - concurrency, security, database schema, and various other non functional requirements that only bite you in production. I’m probably not going to review a random website built by someone except for usability, requirements and security.
- happytoexplain 8mo agoI didn't restrict my opinion to genAI code. I'm expressing a general thought that was relevant before AI. AI is just salient in relation to it. I also said senior review is valuable, but I'm not 100% sure if you're implying I didn't.
- OrangeDelonge 8mo agoI’ve seen hundreds of PR’s produced by a junior and reviewed by a mid lvl go into prod. I don’t see any problem with that
- bs7280 8mo agoThis is also why I think we will enter a world without Jr's. The time it takes for a Senior to review the Jr's AI code is more expensive than if the Sr produced their own AI code from scratch. Factor in the lack of meetings from a Sr only team, and the productivity gains will appear to be massive. Whether or not these productivity gains are realized is another question, but spreadsheet based decision makers are going to try.
- czscout 8mo agoIn this scenario, how might one become a senior without first being a junior? Seniors just pop into existence?
- simplyluke 8mo agoThe bet from various industry leaders appears to be that the current generation of engineers will be the last who will ever need to think about complex systems and engineering, as the AI will just get good enough to do all of that by the time they retire.
- lovich 8mo agoI think it’s deeper than that because it’s affected more industries than software and already started pre AI. American corporate culture has decided that training costs are someone else’s problem. Since every corporation acts this way it means all training costs have been pushed onto the labor market. Combine that with the past few decades of “oops, looks like you picked the wrong career that took years of learning and/or 10 to 100s of thousands of dollars to acquire but we’ve obsoleted that field” and new entrants into the labor market are just choosing not to join. Take trucking for example. For the past decade I’ve heard logistics companies bemoan the lack of CDL holders, while simultaneously gleefully talk about how the moment self driving is figured out they are going to replace all of them. We’re going to be outpaced by countries like China at some point because we’re doing the industrial equivalent of eating our seed corn and there is seemingly no will to slow that trend down, much less reverse it.
- napolux 8mo agoLGTM
- hnthrow0287345 8mo ago>requires an amount of time approaching the time spent if they had just done it themselves. I would actually say having at least 2 people on any given work item should probably be the norm at Amazon's size if you also want to churn through people as Amazon does and also want quality. Doing code reviews are not as highly valued in terms of incentives to the employees and it blocks them working on things they would get more compensation for.
- qnleigh 8mo agoI seriously doubt that they think senior reviewers will meticulously hunt down and fix all the AI bugs. Even if they could, they surely don't have the time. But it offers other benefits here: 1. They can assess whether the use of AI is appropriate without looking in detail. E.g. if the AI changed 1000 lines of code to fix a minor bug, or changed code that is essential for security. 2. To discourage AI use, because of the added friction.
- kaffekaka 8mo agoPoint 1 is important. Seniors (or any developer really) with experience in the code base in question can judge pretty quickly if a CR seems reasonable.
- onion2k 8mo agoI.e. senior review is valuable, but it does not make bad code good. I suspect that isn't the goal. Review by more senior people shifts accountability from the Junior to a Senior, and reframes the problem from "Oh dear, the junior broke everything because they didn't know any better" to "Ah, that Senior is underperforming because they approved code that broke everything."
- AgentOrange1234 8mo agoSeniors are going to need to hold Juniors to a high bar for understanding and explaining what they are committing. Otherwise it will become totally soul destroying to have a bunch of juniors submitting piles of nonsense and claiming they are blocked on you all the time.
- deleted 8mo ago[deleted]
- sethops1 8mo agoThis was challenging enough pre AI. Now that everybody has an AI slop button, the life of an effective code reviewer just got so much more miserable.
- esafak 8mo agoMake them first go through an AI reviewer that is informed by the code base's standards.
- kaffekaka 8mo agoDue to recent events at work this is a venue I will start exploring. I didn't grasp how quickly this would become a problem, but "juniors with a slop button" is indeed a reality now. And I lack the ability to let things slide.
- remarkEon 8mo agoOther than “don’t hire idiots”, what is the solution to this problem? I agree with you, and this particular systems management issue is not constrained to software.
- happytoexplain 8mo agoI don't know. We need smart people at every layer. If leadership isn't in that category, it spreads to all layers. I don't know how we defeat capitalism to incentivize smart leadership. It's fundamentally opposed to market forces.
- radiator 8mo agoDeming's point 3 (of 14): Cease dependence on inspection to achieve quality. Eliminate the need for massive inspection by building quality into the product in the first place.
- mrothroc 8mo agoSenior review can definitely help, regardless if the code comes from a junior or an LLM. We've done this since the dawn of time. However, it doesn't scale and since LLM volume far exceeds what juniors can do, you end up overwhelming the seniors, who are normally overbooked anyway. The other problem is that the type of errors LLMs make are different than juniors. There are huge sections of genuinely good code. So the senior gets "review fatigue" because so much looks good they just start rubber stamping. I use an automated pipeline to generate code (including terraform, risking infrastructure nukes), and I am the senior reviewer. But I have gates that do a whole range of checks, both deterministic and stochastic, before it ever gets to me. Easy things are pushed back to the LLM for it to autofix. I only see things where my eyes can actually make a difference. Amazon's instinct is right (add a gate), but the implementation is wrong (make it human). Automated checks first, humans for what's left.
- hintymad 8mo ago> Review by a senior is one of the biggest "silver bullet" illusions managers suffer from Especially in a big co like Amazon, most senior engineers are box drawers, meeting goers, gatekeepers, vision setters, org lubricants, VP's trustees, glorified product managers, and etc. They don't necessarily know more context than the more junior engineers, and they most likely will review slowly while uncovering fewer issues.
- munk-a 8mo agoReviewing code changes (generally) takes more time than writing code changes for a pretty significant chunk of engineers. If we're optimizing slop code writing at the expense of more senior's time being funneled into detailed reviews then we're _doing it wrong_.
- zamalek 8mo ago> Review by a senior is one of the biggest "silver bullet" illusions managers suffer from. My manager has been urging us to truly vibe code, just yesterday saying that "language is irrelevant because we've reached the point where it works - so you don't need to see it." This article is a godsend; I'll take this flawed silver bullet any day of the week.
- unshavedyak 8mo ago> For a person (senior or otherwise) to examine code or configuration with the granularity required to verify that it even approximates the result of their own level of experience, even only in terms of security/stability/correctness, requires an amount of time approaching the time spent if they had just done it themselves. Hell, often it feels slower/worse. Foreign code is easily confusing at first, which slows you down - and bad code quickly gets bewildering and sends you down paths of clarifications that waste time.
- SchemaLoad 8mo agoSo many times I get AI generated PRs from juniors where I don't feel comfortable with the code, I wouldn't do it like this myself, but I can't strictly find faults that I can reject the PR with. Usually it's just a massive amount of code being generated which is extremely difficult to review, much harder than it was for the submitter to generate and send it for review. Then often it blows up in production. Makes me almost want to blanket reject PRs for being too difficult to understand. Hand written code almost has an aversion to complexity, you'd search around for existing examples, libraries, reusable components, or just a simpler idea before building something crazy complex. While with AI you can spit out your first idea quickly no matter how complex or flawed the original concept was.
- MarkSweep 8mo agoRejecting a PR for being overly complicated or difficult to understand is valid. Breaking a large change into understandable pieces is an important skill both for making changes reviewable as well as helping the author understand the problem.
- yalogin 8mo agoDon’t forget that this auto generated code will have subtle bugs and feels complete at the outset
- rco8786 8mo agoGoing to systemically turn off your senior staff over time also. Most Senior Engineers aren't that interested in doing even more code review.
- mmcconnell1618 8mo agoAlso, have massive layoffs every few months just to keep people on edge. AWS wants people to leave with RTO and badging policies, comp range shifts lower unless you have year over year ratings, and an obsessive push to force AI into every process. Top talent is leaving and will continue to leave AWS.
- deleted 8mo ago[deleted]
- rafaelmn 8mo agoEventually they'll get rid of juniors and mid level devs because realistically it's easier to review when you're the one doing the prompting.
- strogonoff 8mo agoI would argue that the amount of time needed for a proper review exceeds the amount of time needed to just do it yourself. When reviewing, you need to go through every step of implementing it yourself (understand the problem, solve the problem, etc.), but you additionally need to 1) understand someone else’s solution and 2) diff your solution against theirs to provide meaningful feedback. Review could take roughly equivalent time, but only if I am allowed to reject/approve in a binary way (“my solution would not be the same, therefore denied”) which is not considered appropriate in most places. This is why I am not a fan of being the reviewer.
- lionkor 8mo ago> [...] requires an amount of time approaching the time spent if they had just done it themselves. Yes, but with the caveat that the junior learns and eventually can become the senior.
- montymaty 8mo agodfasdsaasdasdasd