4 ms·
The gauge broke: devs felt 20% faster with AI, measured 19% slower (2025)
- jiggawatts 3mo ago"19 August 2025" This may as well have been written in the stone ages, when we were banging AI rocks together. I just did a ~6 month project in ~2 weeks using a frontier model. I wouldn't even have attempted this kind work a year ago, with or without the AIs available at the time!
- suddenlybananas 3mo agoWhat was the project? Could you share the code?
- koe123 3mo agoPersonal or work? Used by anyone or for fun?
- loveparade 3mo agoUsed by Anthropic to sell tokens
- deleted 3mo ago[deleted]
- ImprobableTruth 3mo ago>I just did a ~6 month project in ~2 weeks using a frontier model. Claims like this are hard for me to take seriously because 'good' models have been available since the start of the year. So, if they really 10x one's productivity, then people should be able to have gotten done 5 years worth of work since then, but I've never actually seen anybody show any project like this.
- deleted 3mo ago[deleted]
- IshKebab 3mo agoMy guess at what's happening is that people are mostly using the tools on low impact or speculative projects. Notice that he said he wouldn't have attempted it without AI. That's been my experience too. I had an idea that I didn't need so I hadn't bothered doing it, but AI made it easier to just have a go. I suspect people aren't using AI as much on their main profit-making projects (which also are going to be bigger, more complex and not greenfield - which is all harder for AI). Also give it a chance - as you said "good" models have only been available very recently and you wouldn't expect everyone to start using them instantly.
- ImprobableTruth 3mo agoSure, but "10x faster, but only applies on small greenfield, throwaway projects" is a major caveat. In fact, there's a good chance this doesn't disprove the original blog post, you could be way faster on small projects but slower on 'real' projects. >Also give it a chance - as you said "good" models have only been available very recently and you wouldn't expect everyone to start using them instantly. But I'm not expecting everyone to have built something like that, but surely among millions of users someone should have, especially the people proclaiming insane productivity gains? There are no super impressive open source projects done using AI and all the companies boasting about how all their code is AI written now don't show much improvement either.
- jiggawatts 3mo ago> 'good' models have been available since the start of the year today: https://www.anthropic.com/news/redeploying-fable-5 https://www.anthropic.com/news/redeploying-fable-5 35 days ago: https://www.anthropic.com/news/claude-opus-4-8 https://www.anthropic.com/news/claude-opus-4-8 70 days ago: https://openai.com/index/introducing-gpt-5-5/ https://openai.com/index/introducing-gpt-5-5/ <-- first model I've found useful 77 days ago: https://www.anthropic.com/news/claude-opus-4-7 https://www.anthropic.com/news/claude-opus-4-7 119 days ago: https://openai.com/index/introducing-gpt-5-4/ https://openai.com/index/introducing-gpt-5-4/ 182 days ago: The start of the year
- ImprobableTruth 3mo agoOpus 4.5/4.6 are what many people consider the first 'good' models and it's from last year/start of this year. But fine, let's say everything before gpt 5.5 was unusable crap. Then there should still be projects that would normally have previously taken ~2 years done in just two months. Where are they?
- deleted 3mo ago[deleted]
- jiggawatts 3mo agoI’m starting to see huge solo projects with 100 commits per day turning up on GitHub. Last year these were unusable slop. Now they’re “getting there”. Not quite as good as hand crafted code written by humans, but usable.
- Shitty-kitty 3mo agoThat's amazing. What's even more incredible is that somehow you managed to do a real code review and testing in that time-frame.
- tombot 3mo ago“16 developers across 246 tasks”
- bfjvibybd6cuvu6 3mo agoWith barely any experience or training with AI.
- bitwize 3mo agoThis study was shown to be flawed at the time; METR has retracted it. And it doesn't take into account current frontier models. AI makes you more productive. This is no longer up for debate. The energy you spend arguing last year's talking points is better spent knuckling down and learning the tools.
- idle_zealot 3mo agoWhat tools? The ones that will be outmoded in 6 months, or the ones that will be banned in 6 months?
- bitwize 3mo agoWhatever tools come along, step one is getting into the mindset that typing code in is no longer part of your job. You are a designer and director, not a coder. As Steve Yegge says, if you still have an IDE open entering code by hand, you're one of the crappy engineers. You need to be getting into the habit of understanding the strengths and weaknesses of your model and agentic harness and using those to produce the results you want. When those change in six months, you adapt along with them. Adjusting to the new mindset is the biggest hurdle, and there'll be plenty of devs who can't, and won't make the cut or stay in the field for very long. Just like there were plenty of devs who couldn't adjust to anything beyond COBOL on punched cards.
- fxwin 3mo agoHave they retracted it? My understanding was simply that they released results with more recent data, not that this study itself was flawed (and their website doesn't mention a retraction either)
- croes 3mo agoSo where are the profit jumps from all the productivity gains? Or do you just produce more code but not more productive value?
- wisty 3mo agoPartial agree ... I suspect: If you know what you are doing it is a power tool. If you don't know what you are doing it's also a power tool - if you measure a lot of devs then the bad ones (or anyone having a bad day, or the wrong fit for a project) can make work for everyone else at an outrageous pace.
- shaky-carrousel 3mo agoThere was a study that people using the keyboard instead of the mouse felt they were working faster but in fact they were working slower. A perception thing. Users were more engaged when using a keyboard.
- shawabawa3 3mo agoI'm convinced this is what causes people to feel productive with vim
- 100ms 3mo agoTriggered by both of these comments.. interaction mode dictates a style of thinking. I have to use a mouse, I'm forced to use my eyes, which also means I probably have to use a massive screen. I have to pay attention to some hyperactive Intellisense-like feature, I'm forced to remove my attention from the problem. It's like saying you're convinced people reporting they feel more productive in a mauve-coloured room are liars, or those that drive automatic vs manual. Maybe they just find muave a restful colour?
- Avshalom 3mo ago>> I'm forced to use my eyes, which also means I probably have to use a massive screen. I have to pay attention to some hyperactive Intellisense-like feature What the hell are you talking about?
- 100ms 3mo agoI can quite easily (and often do) use a basic editor while staring at the wall. I've yet to use an IDE where there wasn't some idiotic race between keystrokes and whatever random latency language server just told it to insert parens or a newline after you already typed them, assuming the text is even visible on a 13" screen buried in sidebars and "essential" extensions. They're full attention tools which is a completely different mode of work than is otherwise possible. It's not to say an IDE isn't a useful thing, they just have their place like anything else. I personally find autocompletion useful for a couple of weeks going into a new language or project after which it's very often more a distraction than a productivity enhancer. Same goes with e.g. Git integration. I wouldn't presume to say a Git integration user simply needs to learn Git in much the same way I wouldn't expect someone to tell me that I can't use Git just because I don't use the IDE Git integration. They're just tools
- loveparade 3mo agoThese studies are meaningless because speedup is heavily dependent on the kind of work you're doing. No doubt that you can do mechanical refactors 100x faster with AI, and also no doubt that using AI will be slower for tasks where it's less about writing code and more about context/world knowledge or building understanding. Averaging across these tasks doesn't make sense because everyone's work consists of a different distribution of tasks. A frontend dev doing tailwind integration for his day job is gonna see very different speedups than someone working in a niche scientific codebase. Taking the average makes about as much sense as taking the average of the speedup from calculators for a mathematician, a farmer, and an elementary school student.
- dakolli 3mo agoI have found llms to be utterly useless for frontend (tailwind included). That is, unless you're building a single page app/landing page that is the typical center column with a hero and below that a 3x3 feature grid with those same 3 colors that all the sloppers show off. I'm not a frontend dev, but these statements are starting to get outright disrespectful to those that are. Do you people understand how much "world", customer and product knowledge is required to design and implement great UX/UI? I promise you are not going to be able to translate all this internalized understanding to an LLM and have it do your "tailwind integration" It actually sucks at all frontend outside of the 3 types of page layouts it understand.. Shitty landing pages, generic dashboards and shitty blog layouts. Ya'll yearn for slop though so maybe everything will just become shit anyways.
- loveparade 3mo agoFair point, I was more trying to make a statement about the amount of training data available, not the "difficulty" of the task. I just used Tailwind as an example because it is so ubiquitous with so much training data for LLMs to learn from, while any niche application doesn't have that.
- Zababa 3mo agoTraining data has stopped being a good predictor of LLM abilities ever since they started doing heavy RL runs. I'm not sure how much corporate dashboards/I can't believe it's not excel stuff were in the training data, I guess not that many considering that stuff is almost always corporate and kept inside companies, and LLMs are still great at it, good enough to make people that used excel and used it well daily for 10+ years stop using it for lots of stuff.
- deleted 3mo ago[deleted]
- ianbutler 3mo ago2025 is such old news that this just isn't relevant. METR already redid the study at a later date and now finds a likely 18% speedup "For the subset of the original developers who participated in the later study, we now estimate a speedup of -18% with a confidence interval between -38% and +9%" (note their use of - and + here could be slightly confusing but they do mean 18% faster per the post) https://metr.org/blog/2026-02-24-uplift-update/ https://metr.org/blog/2026-02-24-uplift-update/
- krige 3mo agoEven not touching the laughable sample size for both studies - almost halved sample size between 2025 and 2026? Sounds like a massive selection bias, and not in the way they're implying.
- markbao 3mo agoAnd to be specific, the METR study was using the Cursor harness with Claude Sonnet 3.5/3.7, along with other models of that era of the participant’s choosing. Which is ancient at this point, and half a year older than the November 2025 inflection point when agentic coding got really good. The original article is from August 2025, and the overall message to not trust ‘how it feels’ and rather measure outcomes seems right to me despite the outdated figures. On my team at least, we are seeing a noticeable inflection in work shipped with AI according to Weave.
- verzali 3mo agoThat post literally says the results are unreliable...
- lars512 3mo ago...and in particular it says that one of the reasons is that developers are refusing to participate in the non-AI branch, and when they do, changing what tasks they select to those where AI would be less useful. Overall this suggests to them that the current speedup is likely greater than what the study could measure.
- wewewedxfgdf 3mo agoMy two bosses are anti-AI. Whenever I tell them about how awesome AI is, they come back with stories about how they used AI and it couldn't even do anything basic and what it did do had errors. People will always create a world narrative that matches what they already believe. Anti AI people are always quoting these "facts" about how AI reduces productivity even when developers feel it increases productivity - it reinforces their world view.
- croes 3mo agoBut this isn’t just a narrative but a study. Limited but a study
- jdkoeck 3mo agoA study from 2025. Might as well be from five years ago.
- croes 3mo agoSo where are all those profits from the higher productivity?
- mschuetz 3mo agoHaving participated in many studies, I lost much faith in studies. You could study the same thing, and get opposite results depending on how you build the study, which people participate (friends and colleagues will be reluctant to speak against your results), and the bias with which you set everything up. Also, many tasks require learning the tools, and some tools will start to be more productive with expertise than others.
- deleted 3mo ago[deleted]
- Almondsetat 3mo ago>feel Productivity is not a feeling though. Either you show an increased productivity or it doesn't exist
- onion2k 3mo agoGeneration got cheap. Verification got expensive. That proves AI is capable of doing one part of the software engineering process. The 16 devs in the study trusted AI to write the code. Once we trust AI to do the verification as well we'll realise the gains we feel we're getting now. Essentially we're intentionally going slower on the second half because the trust is missing. Alternatively, rather than trusting AI to do the validation, we could follow the vibe-coder approach by skipping the validation entirely, and trust that the generation stage is good enough not to need it. Historically that's come with some small downsides, like the code being a broken mess of security holes, but with time AI might fix that.
- alfalfasprout 3mo agosmall downsides like security holes? Those aren't small. Neither is creating a codebase that's an inextensible mess that even LLMs begin to struggle with. The reality is making good decisions and thinking about approaches take time. AI can absolutely make us faster at it but it's not magic and these speedups come with effort.
- onion2k 3mo agosmall downsides like security holes? Those aren't small I'm British. I've been taught to turn understatement into an art.
- hnlmorg 3mo agoI wouldn’t even trust experienced developers to merge code without peer review.
- make_it_sure 3mo agowhy is this on the front page. It's an old debunked study lol, not relevant at all today. Read more than just the title
- kumiko_studio 3mo ago[dead]
- jdkoeck 3mo ago> The honest counter, and it matters here more than usual. This is most likely the dip in a J-curve, not the destination. Oh, the irony of this post being AI-generated.
- titanomachy 3mo agoI don’t understand how these things still write so annoyingly. Eliminating just a handful of tells would make a big difference.
- luckilydiscrete 3mo agohttps://www.faros.ai/blog/ai-software-engineering https://www.faros.ai/blog/ai-software-engineering The actual study with the data, minus the "I was right all along" commentary
- vachina 3mo agoIt will feel slower because I finish my task in 1/20th the time and spend the remaining time browsing HN.
- raincole 3mo agoYeah, when people who are not familiar with AI and use Cursor with Sonnet 3.7 they are only 19% slower. In retrospect that research was very bullish for AI.
- orphereus 3mo agoAs I read the blog post, I thought that it was released today. Maybe point out that it is almost a year old. It feels like it is manipulating HN users. And this is coming from an AI sceptic.
- deleted 3mo ago[deleted]
- DonHopkins 3mo agoLuc Barthelet, who I worked with at EA, is a Mathematica whiz (he later worked at Wolframe Research on Wolfram|Alpha), and he would prototype game ideas in Mathematica, which would render out web pages with animations. He came up with a fun idea for a racing game renderer: it distorted the perspective transformation a bit, grading depth on a curve, so far away things would linger in the distance a bit longer, then speed up and WHOOSH past you, seeming even faster than they would be photorealisticly! https://www.mobygames.com/person/29352/luc-barthelet/ https://www.mobygames.com/person/29352/luc-barthelet/ https://community.wolfram.com/web/luc https://community.wolfram.com/web/luc
- miika 3mo agoWell.. feelings have never been a good way to measure quantities.
- bezier-curve 3mo agoOne thing I see missing in a lot of these discussions is whether or not the metric is solely based on speed. I think AI just allows you to look at your code in different ways and provides more chances to catch mistakes. I am definitely slower with AI assistance, but that is because I use it to increase the quality of my work.
- sigmoid10 3mo agoIf you linked to the actual source of the study [1] instead of a random blog only talks about the result, you would see the big banner that the authors put there noting that the study is horribly outdated. Current models do make developers faster. [1] https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/ https://metr.org/blog/2025-07-10-early-2025-ai-experienced-o...
- arisAlexis 3mo agoSo why are all the top labs using the tools internally? They are lying or they are stupid? Devs wish that was true but it isn't and it will get better.
- suddenlybananas 3mo agoConversely, why do they have so many open bugs?
- dude250711 3mo agoAI psychosis is neither a lie nor a stupidity.
- xg15 3mo ago> AI speeds up typing, which was never the bottleneck for an expert in a codebase they already know. For me as a dev, that's not the whole truth. Where I've found actual value in AI (and I think were some of that "perceived speedup" is coming from) is looking up things. Unless you know the codebase and used libraries extremely well, you will have to do lots of "micro-lookups" during coding, where you have to find the specific APIs or library functions for your problem, then figure out how exactly you have to call them, how to handle the result, etc. That's lots of "research" work interleaved with actually writing the code. AIs seem to be good enough to have a lot of that knowledge already baked into their weights, at least for popular platforms, so if you prompt it something, you can skip all that low-level lookup work or at least defer it until code review. Even during review, it's easier, because you don't have to come up with the appropriate library function from scratch, you only have to verify that the ones the AI used make sense and are used correctly.
- michaelt 3mo agoOne thing I've noticed with generative AI is it's now easier than ever to write more lines of code. Before, a backend guy asked to add an intranet page would make an austere page -bare html with barely any styling or javascript. Today, the same guy given the same task can turn in something with styling, javascript, internationalisation, interactive form validation, progress spinner, minification build stage, linting, maybe even automated browser tests. And I have to code review it. Now the bottleneck of writing the code has been removed, I now find code review is the bottleneck - and a bottleneck facing much higher flow must either let more through, or start applying back pressure. Sometimes I think an evil genie granted my wish for better tested code by trying to drown me in it.
- globular-toast 3mo agoI mean they could have turned in one of the countless HTML "boilerplate" projects before LLMs too. It hasn't been necessary to start completely from scratch for quite a while now. I'm surprised any professional web developer wouldn't maintain their own boilerplate as templates that they can quickly roll off the production line. Tools like cookiecutter[0] have been available for a long time. Sometimes I feel like LLMs are just allowing people to somewhat catch up to where others were decades ago. [0] https://github.com/cookiecutter/cookiecutter https://github.com/cookiecutter/cookiecutter
- intrepidkarthi 3mo ago[flagged]
- larodi 3mo agoSo are you trying to imply that I've somehow accidentally stumbled upon more than 100k lines of new high-perf working code, done in less than 6 months, which is like not 20%, but 200% my actual output, and this code, already generating revenue for me and my employer, is something of the ordinary, and actually I can do 20% better typing it manually? ROFL sorry
- DanielNeveux 3mo ago[dead]
- jdjdjrbrbrb 3mo agoI honestly don't care if I am faster or slower ... I am sooooooooo much less burnt out... And that metric is infinitely better for me and my company.