11 ms·
Productivity gains from AI coding assistants haven’t budged past 10% – survey
- chvid 8mo agoAs far as I can tell from my workplace the total impact on productivity is neutral to negative.
- nasretdinov 8mo agoI think that over time people will start looking at AI-assisted coding the same way we now look at loosely typed code, or at (heavy) frameworks: it saves time in the short term, but may cause significant problems down the line. Whether or not this tradeoff makes sense in a specific situation is a matter of debate, and there's usually no obviously right or wrong answer.
- doomslayer999 8mo agoOnce the free money runs out, the AI cos may shift to making heavily verified code snippets with more direct language control. This will heavily simplify a lot of boilerplate instead of fairytales of some AGI coding wiz.
- co_king_5 8mo ago[dead]
- doomslayer999 8mo agoNo to the first question, and maybe with a lot of money for the second question.
- mattmanser 8mo agoIn the 20 years I've been in the industry, boiler plate has dropped dramatically in the backend. Right now, front end has tons of boiler plate. It's one of the reasons AI hassle such a wow factor for FE, trivial tasks require a lot of code. But even that is much better than it was 10 years ago. That was a long way of saying I disagree with your no.
- skydhash 8mo agoFE has a lot of boilerplate only if you’re starting from scratch every single time. That’s why we had template systems and why we invented view libraries. Once you’ve defined your libraries, you just copy-paste stuff.
- matthewbauer 8mo agoIt seems like they should be able to “overweight” newer training data. But the risk is the newer training data is going to skew more towards AI slop than older training data.
- otabdeveloper4 8mo agoThere won't ever be newer training data. The OG data came from sites like Stackoverflow. These sites will stop existing once LLMs become better and easier to use. Game over.
- esclerofilo 8mo agoEvery time claude code runs tests or builds after a change, it's collecting training data.
- co_king_5 8mo ago[dead]
- esclerofilo 8mo agoI can't pretend to know how things work internally, but I would expect it to be involved in model updates.
- otabdeveloper4 8mo agoYou need human language programming-related questions to train on too, not just the code.
- 8note 8mo agothats what the related chats are for?
- otabdeveloper4 8mo ago
- jimbokun 8mo agoI mean if people continue checking open source code into GitHub using those new features then they should be able to learn them just fine.
- danaris 8mo agoThis is only true if there continues to be tremendous amounts of money/hardware/power available to perform the training, in perpetuity.
- moffkalast 8mo agoIt really depends on the situation. I think there's an argument for generating in a lower level strongly typed language, where most of the work of writing the pointlessly verbose parts is eliminated, any errors are found by the compiler immediately, but it still leaves the option for handwritten optimizations when needed. Sort of how one can drop down to C in python for the parts that need more performance.
- gedy 8mo agoI can see where productivity could be higher if all I did was type in programs to some spec, or bootstrapping new apps all day - but that's like not the reality of "programming", at least for me past 25 years. Sorting through what to even make and interpreting "requirements" is what takes the most time
- downrightmike 8mo agoYeah, industry has told them that devs aren't valuable and AI can do their job. Who TF has motivation after that?
- co_king_5 8mo ago[flagged]
- havefunbesafe 8mo agoPeople getting paid >$400k TC
- overgard 8mo agoYou're only as fast as your biggest bottleneck. Adding AI to an existing organization is just going to show you where your bottlenecks are, it's not going to magically make them go away. For most companies, the speed of writing code probably wasn't the bottleneck in the first place.
- vorticalbox 8mo agoone thing that aways slowed me down was writing jsdocs and testing. Now i can write one example of a pass and then get codex to read the code and write a test for all the branches in that section saves time as it can type a lot faster than i can and its mostly copying the example i already have but changing the input to hit all the branches.
- otabdeveloper4 8mo ago> let's have LLMs check our code for correctness Lmao. Rofl even. (Testing is the one thing you would never outsource to AI.)
- idle_zealot 8mo agoOutsourcing testing to AI makes perfect sense if you assume that tests exist out of an obligation to meet some code coverage requirements, rather than to ensure correctness. Often I'll write a module and a few tests that cover its functionality, only for CI to complain that line coverage has decreased and reject my merge! AI to the rescue! A perfect job for a bullshit generator.
- 8note 8mo agooutsourcing testing the AI also gets its code to be connected to deterministic results, and show let the agent interact with the code to speculate expectations and check them against the actual code. it could still speculate wrong things, but it wont speculate that the code is supposed to crash on the first line of code
- 8mo ago
- bluejekyll 8mo agoI found the title for this post misleading. To clarify it a bit, AI has only improved productivity by 10% even though 93% of devs are using it.
- dandanua 8mo agoYeah, the title may suggest that productivity is still 10% out of 100% after CEOs fired half of developers believing that the rest will do all the job with the help of AI.
- mytailorisrich 8mo agoBlunt opinion: Most devs are not that good and really only execute what they are told to do. The threat of AI for devs, and the way to drastically improve productivity is there: keep the better devs who can think systemically, who can design solutions, who can solve issues themselves and give them all the AI help available, cut the rest.
- tyleo 8mo agoThat’s how I feel too. When I was an architect at a ~300-person company, a big chunk of my job shifted to reviews, technical design docs, and guidance. I’m getting great results by feeding context like that into Claude Code, then reviewing and steering what it produces. It really does feel like a multiplier on me and I understand things enough to get my hands dirty where Claude struggles. Lately I’ve been wondering if that role evolves into a more hierarchical review system: senior engineers own independent modules end-to-end, and architects focus on integration, interfaces, and overall coherence. Honestly, the best parts of our product already worked like that even before AI.
- jimbokun 8mo agoI would collapse the senior engineer and architect roles into a single role and only hire people capable of doing both. Owning the modules requires understanding the overall architecture and how the module’s API needs to fit into it, when a new module is required, when the architecture needs to change, and how to pragmatically migrate from one design to another in a live system. All of that is easier if everyone is capable of operating at any level from overall architecture to micro optimizations.
- moralestapia 8mo agoAI adoption has reduced productivity at my workplace, and by a noticeable amount!
- otabdeveloper4 8mo agoThat's expected for any new "low-code" solution du jour.
- randomtoast 8mo agoThis will lead to natural selection. As AI becomes increasingly integrated into all areas, companies that manage it less effectively than others will face greater selection pressure.
- emp17344 8mo agoOr, AI will turn out to just not be that useful.
- moralestapia 8mo agoIt's such a weird effect. At a personal level, AI has made non-trivial improvements to my life. I can clearly see the value in there. At an organizational level, it tends to get in the way much more than helping out. I do not yet see the value in there.
- ilovetux 8mo agoI think some AI companies are just now starting to feel the pressure to profit. Soon, I predict we will see a pretty significant jump in price that will make a 10% productivity gain seem tiny compared to the associated bills. For now, these companies are trying to reach critical mass so their users are so dependant on their tech that they have to keep paying at least in the short term.
- xeiotos 8mo agoUnsurprising for multiple reasons. Most organizations have other bottlenecks and limiting factors than “how fast can you develop”. Regardless, if you’re a dev who is now 2x as productive in terms of work completed per day, and quality remains stable, why should this translate to 2x the output? Most people are paid by the hour and not for outcomes. And yes, I am suggesting that if you complete in 4 hours that which took you 8 hours in 2019, that you should consider calling it a day.
- ihsw 8mo ago[dead]
- ptx 8mo agoApparently "AI is speeding up the onboarding process", they say. But isn't that because the onboarding process is about learning, and by having an AI regurgitate the answers you can complete the process without learning anything, which might speed it up but completely defeats the purpose?
- OptionOfT 8mo agoCorrect. Reading code is important. The details are in the minutia, and the way code works is that the minutia are important. Summarizing this with AI makes you lose that context.
- snsjzhhz 8mo agoThis has been my experience as a dev, and it always confuses me when people say they prefer to work at a “higher level”. The minutiae are often just as important as some of the higher level decisions. Not everything, but not an insignificant portion either. This applies to basic things like correctness, performance, and security - craft, style, and taste are not involved.
- co_king_5 8mo ago[dead]
- raphman 8mo agoYes, that's how I'd interpret it, too. According to the article, onboarding speed is measured as “time to the 10th Pull Request (PR).” As we have seen on public GitHub projects, LLMs have made it really easy to submit a large number of low-effort pull requests without having any understanding of a project. Obviously, such a kind of higher onboarding speed is not necessarily good for an organization.
- jimbokun 8mo agoYeah it should only count ACCEPTED pull requests.
- jdlshore 8mo agoThis is self-reported productivity, in that devs are saying AI saves them about 4 hours per week. But let’s not forget the METR study that found a 20% increase in self-reported productivity but a 19% decrease in actual measured productivity. (It used a clever and rigorous technique for measuring productivity differences, BTW, for anyone as skeptical of productivity measures as I am.)
- williamcotton 8mo agoHas the METR study been replicated?
- jdlshore 8mo agoI haven’t heard about any similar studies, no. I’m planning to conduct one at my workplace but we’re still deciding exactly which uses of AI to test.
- lunar_mycroft 8mo agoNot a scientific study, but someone did replicate the experiment on themselves [0] and found that in their case, any effect from LLM use wasn't detectable in their sample. Notably they almost certainly had more experience with LLMs than most of the METR participants did. [0] https://mikelovesrobots.substack.com/p/wheres-the-shovelware-why-ai-coding https://mikelovesrobots.substack.com/p/wheres-the-shovelware...
- samuelknight 8mo agohttps://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/ https://metr.org/blog/2025-07-10-early-2025-ai-experienced-o... That info is from mid 2025, talking about models released in Oct 2024 and Feb 2025. It predates tools like Claude Code and Codex, Lovable was 1/3 current ARR, etc. This might still be true but we desperately need new data.
- lunar_mycroft 8mo agoNone of those changes address the issue jdlshore is pointing out: self assessed developers productivity increases from LLMs are not a reliable indication of actual productivity increases. It's true that modern LLMs might have less of a negative impact on productivity or increase it, but you won't be able to tell by asking developers if they feel more productive. (Also, Anthropic released Claude Code in Febuary of 2025, which was near the start of the period the study ran).
- agentifysh 8mo agoI read this article as the CTO being the bottleneck if he's only seeing 10% productivity boost at his organization. I dont think this is a purely AI problem more with the legacy costs of maintaining many minds that can't be solved by just giving people AI tools until the AI comes for the CTO role (but not CEO or revenue generating roles) too and whichever manager is bottlenecking. I imagine a future where we have Nasdaq listed companies run by just a dozen people with AI agents running and talking to each other so fast that text becomes a bottleneck and they need another medium that can only be understood by an AI that will hold humans hand This shift would also be reflected by new hardware shifts...perhaps photonic chips or anything that lets AI scale up crazy without the energy cost.... Exciting times are ahead AI but it's also accelerating digital UBI....could be good and bad.
- Nezteb 8mo ago> it's also accelerating digital UBI Do you have sources for this claim?
- rcfox 8mo agoThe title is misleading. Productivity isn't at 10%, it's at 110%.
- onion2k 8mo agoA 10% uplift in productivity for the cost of probably 0.001% of the salary budget is an incredible success.
- arctic-true 8mo agoThis is exactly right. And assuming organizations use the gains to cut headcount rather than boost total productivity, a 10% reduction in white collar employment would still be an era-defining systemic shock to the economy.
- emp17344 8mo agoProductivity improvements from automation actually result in an increase in jobs, not fewer jobs. Basic economics.
- kiernanmcgowan 8mo agoMy biggest road blocks as an engineer has almost never been the authorship of code but everything else around it. * Getting code reviewed * Making sure its actually solving the problem * Communicating to the rest of the team whats happening * Getting tests to pass * Getting it deployed * Verifying that the fix is implemented in production * Starting it all over when there is a misunderstanding Slinging more code faster is great and getting unit testing more-or-less for free is awesome but the separation between a good and great engineer is one of communication and management. AI is causing us to regress to thinking that code velocity is a good metric to use when comparing engineers.
- wewewedxfgdf 8mo agoHow are CTO's so out of touch and yet loud and proud about it.
- keeda 8mo agoThe real takeaway here -- also corroborated by the DORA 2025 report https://dora.dev/research/2025/ https://dora.dev/research/2025/ -- is that more than anything, AI amplifies your current development culture. Organizations with strong quality control discipline enjoy more velocity, those with weak practices suffer more outages. Expecting AI to magically overcome your development culture is like expecting consultants to magically fix your business culture. Furthermore, by various estimates, engineers only spend 10 - 60% of their time on actual code. So, given that currently AI is largely used only for coding activities, 10% is actually considerable savings. Also this is the result of retro-fitting AI into existing workflows; actual "AI-native" workflows would probably look very different, likely having refactored in other parts of software engineering. Spotify's "Honk" workflow is probably just a starting point.
- orwin 8mo agoI'm pretty sure it has to do with the individual as well as the culture. Juniors/new hire use AI to multiply by two their wrong/unsafe output, and seniors then have to spend more time correcting it. I'll be honest: I piss poor code, each time I come back to an old project I see where I could have done better. New hires are worse, but before AI (and especially Opus) they didn't produce that much code before spending like 6 months learning (I'm on a netsec tooling team). Now, they start producing code after two weeks or less, and every line have to be checked because they don't understand what they are doing. I think my personal output was increased by 15% on average (maybe 5 on difficult projects), but our team output decreased overall.
- keeda 8mo agoYes, we as a society urgently have to figure out how to learn and educate with AI. There are even studies showing that students who use AI to do their work do not learn the necessary skills. And I'm also hearing grumblings about entry level talent that is absolutely clueless without AI, which does not help the junior hiring scene at all. At this point it seems clear that people wishing to learn a discipline should restrict their usage of AI until they have "built the muscles", but none of our educational, testing, recruitment and upskilling practices are conducive to that.
- aaroninsf 8mo agoXimm's Law applies to the "plateau" of 10% In other words: notionally, if not literally, by the time trailing numbers are collected they are out of date. This is of course axiomatic, but, that staleness is a serious matter in this particular moment. It's a cliché that six months can be a lifetime on the bleeding edge of tech. This is the first time in my career that is more or less literally true. Humans reason poorly with non-linear change. This entire article is a demonstration of that.
- ai_tools_daily 8mo ago10% feels right for the median case but wildly wrong for specific workflows. Where I see massive gains (50%+ time savings): boilerplate generation, test writing, regex/config syntax I'd otherwise have to look up, and exploring unfamiliar codebases. These are high-frequency, low-creativity tasks where AI genuinely excels. Where I see near-zero gains: debugging complex distributed systems, architectural decisions, understanding why code exists the way it does (not what it does), and navigating ambiguous requirements. These require context that doesn't fit in a prompt. The 10% average likely masks a bimodal distribution. Developers who restructured their workflow around AI assistance probably see 30-40%. Developers who use it as a fancy autocomplete see 5%. Measuring "productivity" as a single number hides all the interesting variation. Also worth noting: the biggest unlock isn't code generation — it's using AI to quickly evaluate multiple approaches before committing to one. That's hard to measure but extremely valuable.