2 ms·
> It's not a bad paper, but it's also turning into a fantastic illustration of how much thirst there is out there for anything that shows that AI productivity d
by timr 1y ago
> It's not a bad paper, but it's also turning into a fantastic illustration of how much thirst there is out there for anything that shows that AI productivity doesn't work.
Maybe. I think it's a fantastic illustration of anyone doing anything to provide something other than hype around the subject. An actual RCT? Too good to be true! The thirst is for fact vs. speculation, influencer blogposts and self-promotion.
That this RCT provided evidence opposing the hype, is, of course, irresistible.
- simonw 1y agoThat's what I liked about the paper: the methodology felt better to me than many other studies. I noted that when I blogged about it (second-to-last paragraph https://simonwillison.net/2025/Jul/12/ai-open-source-productivity/ https://simonwillison.net/2025/Jul/12/ai-open-source-product... )
- pydry 1y agoThe other papers ive seen so far all used toy projects. This paper was the first ive seen that deliberately used realistic projects. I think that part is understated, while the part about the "steep learning curve" has been considerably overstated. Ive watched plenty of good developers with hours of cursor practice, using plenty of relevant context giving clear instructions wasting chunks of 5-10 minutes at a time because 4/5 times claude produced crap that needed to be undone. That story doesnt sell though. Nobody will pay me consulting fees to say it. Hype has more $$$ attached.