3 ms·

I feel like all you need to know about how seriously to take this is that they cite that ancient early-2025 METR study, and describe it in the text as "recently

I feel like all you need to know about how seriously to take this is that they cite that ancient early-2025 METR study, and describe it in the text as "recently one even found..."

katzgrau 2mo ago
Same thought - 80% through reading it occurred to me to check the citations. A few items from 2025 and most well before that. So much has changed since late 2025 one can’t really draw any conclusions from this. In fact, I’m guessing things will continue to move so fast that by the time one were to execute a survey of developers, many of the responses and findings are no longer relevant.
- joshuastuden 2mo ago
  Exactly. I saw them using things from 2025... AI sorta sucked then and didn't really "take off" until that Opus drop in December or whatever it was.
- greenhat76 2mo ago
  Your point really goes both ways, we really don't know anything about how LLM usage is affecting anything. No one knows, it's the wild wild west, which is whatever. But I think no one can really draw conclusions from what's happening in tech right now. Reminds me of COVID and how everyone was fighting over early trends during that time.
CompoundEyes 2mo ago
I felt the same and why didn’t the authors look over METR’s recent material? https://metr.org/blog/2026-05-11-ai-usage-survey/ https://metr.org/blog/2026-05-11-ai-usage-survey/
- Izkata 2mo ago
  The whole point of the 2025 one is that they found the self-reporting to be significantly inflated, which is why self-reported surveys like this one are hard to trust.
  - mkozlows 2mo ago
    Yes, but their newer write-up discusses that (and shows that the self-reported numbers have gone up radically, in a way that suggests that even if there is some inflation, the numbers are almost certainly positive if you deflate). They also have an update -- linked from the original study! -- explaining that it's out of date and no longer reliable, and explaining why they had to cancel a follow-up study because it was understating productivity gains (but also was showing wins for the people who carried over from their previous study): https://metr.org/blog/2026-02-24-uplift-update/ https://metr.org/blog/2026-02-24-uplift-update/ The authors of this paper decided to ignore all of METR's follow-up data and discussion, and to report only the ancient number from early 2025 (a time when Windsurf was state of the art). And then, rather than apologizing for it, and caveating it as a number not to be taken seriously, they described it as a study done "recently." That's either shockingly dishonest or incredibly out-of-touch.
    - jurgenburgen 2mo ago
      > shows that the self-reported numbers have gone up radically, in a way that suggests that even if there is some inflation, the numbers are almost certainly positive if you deflate Or the hype machine is even stronger, making people inflate those numbers even more.
    - IsTom 2mo ago
      If I wanted to be uncharitable, the question of "how much slower you'd be without AI" compares against people's AI-less productivity from 2026, which could have become much less than 2025 AI-less productivity, if hypothetically AI use makes people helpless and deskilled.

I feel like all you need to know about how seriously to take this is that they cite that ancient early-2025 METR study, and describe it in the text as "recently one even found..."

katzgrau 2mo ago

Same thought - 80% through reading it occurred to me to check the citations. A few items from 2025 and most well before that. So much has changed since late 2025 one can’t really draw any conclusions from this. In fact, I’m guessing things will continue to move so fast that by the time one were to execute a survey of developers, many of the responses and findings are no longer relevant.

joshuastuden 2mo ago
Exactly. I saw them using things from 2025... AI sorta sucked then and didn't really "take off" until that Opus drop in December or whatever it was.
greenhat76 2mo ago
Your point really goes both ways, we really don't know anything about how LLM usage is affecting anything. No one knows, it's the wild wild west, which is whatever. But I think no one can really draw conclusions from what's happening in tech right now. Reminds me of COVID and how everyone was fighting over early trends during that time.

CompoundEyes 2mo ago

I felt the same and why didn’t the authors look over METR’s recent material? https://metr.org/blog/2026-05-11-ai-usage-survey/ https://metr.org/blog/2026-05-11-ai-usage-survey/

Izkata 2mo ago
The whole point of the 2025 one is that they found the self-reporting to be significantly inflated, which is why self-reported surveys like this one are hard to trust.
- mkozlows 2mo ago
  Yes, but their newer write-up discusses that (and shows that the self-reported numbers have gone up radically, in a way that suggests that even if there is some inflation, the numbers are almost certainly positive if you deflate). They also have an update -- linked from the original study! -- explaining that it's out of date and no longer reliable, and explaining why they had to cancel a follow-up study because it was understating productivity gains (but also was showing wins for the people who carried over from their previous study): https://metr.org/blog/2026-02-24-uplift-update/ https://metr.org/blog/2026-02-24-uplift-update/ The authors of this paper decided to ignore all of METR's follow-up data and discussion, and to report only the ancient number from early 2025 (a time when Windsurf was state of the art). And then, rather than apologizing for it, and caveating it as a number not to be taken seriously, they described it as a study done "recently." That's either shockingly dishonest or incredibly out-of-touch.
  - jurgenburgen 2mo ago
    > shows that the self-reported numbers have gone up radically, in a way that suggests that even if there is some inflation, the numbers are almost certainly positive if you deflate Or the hype machine is even stronger, making people inflate those numbers even more.
  - IsTom 2mo ago
    If I wanted to be uncharitable, the question of "how much slower you'd be without AI" compares against people's AI-less productivity from 2026, which could have become much less than 2025 AI-less productivity, if hypothetically AI use makes people helpless and deskilled.