2 ms·
> Could you expand on your criticism? Sure! The study focused on experienced devs working in complex codebases they already knew thoroughly. This was and is
by mwigdahl 1y ago
> Could you expand on your criticism?
Sure! The study focused on experienced devs working in complex codebases they already knew thoroughly. This was and is the worst case for using AI tooling from a cold start, _particularly_ AI tooling as it existed at the time.
There were also only 16 developers involved in the study.
Time has passed since the study and we've had an entirely new class of tool introduced (the agentic CLI a la Claude Code) as well as two subsequent generations of model improvement (Sonnet 3.7 to Sonnet 4 to Sonnet 4.5). Given that the results of the METR study were stated as an eternal, unqualified truth, the fact that tooling and models are much superior now compared to when the study was conducted is worth noting as well.
- immibis 1y agoWhich would be non-news if the developers also thought it wasn't going to be helpful because they already knew their codebases thoroughly. Or at least if they did the task, and then reported that AI made it harder. But in reality, they expected it to be faster, and then after doing it slower, said they'd done it faster. That's weird.
- monooso 1y agoI appreciate the clarification. From my perspective, the most striking observation was the gap between perception and reality. Whether the recent model advances have widened or narrowed that gap is unclear.