4 ms·

Every study I've read says nobody is seeing productivity gains from AI use. Here's an AI vendor saying the opposite. Funny.

by pluc 1y ago

azdle 1y ago
It's not even claiming that. It's only claiming that people who responded to the survey feel more productive. (Unless you assume that people taking this survey have an objective measure for their own productivity.) > Significant productivity gains: Over 80% of respondents indicate that AI has enhanced their productivity. _Feeling_ more productive is inline with the one proper study I've seen.
- Foobar8568 1y ago
  Well I feel and I am more productive, now on coding activities, I am not convinced, it basically replaced SO and google, but at the end of the day, I always need and want to check reference material that I may have known or not existed. Plenty of time, Google couldn't even find them. So in my case, yes but not on activities these sellers are usually claiming.
- thebigspacefuck 1y ago
  The METR study showed even though people feel more productive they weren’t https://arxiv.org/abs/2507.09089 https://arxiv.org/abs/2507.09089
  - knes 1y ago
    the MTR study is a joke. it surveyed only 16 devs. in the era of Sonnet 3.5 Can we stop citing this study I'm not saying the DORA study is more accurate, but at least it surveyed 5000 developers, globally and more recently (between June 13 and July 21, 2025) which means using the most recent SOTA models
    - bopbopbop7 1y ago
      Yea cite the study funded by a company that invested billions into AI instead, that will surely be non biased and accurate.
    - capnrefsmmat 1y ago
      It didn't "survey" devs. It paid them to complete real tasks while they were randomly assigned to use AI or not, and measured the actual time taken to complete the tasks vs. just the perception. It is much higher quality evidence than a convenience sample of developers who just report their perceptions.
    - rsynnott 1y ago
      > I'm not saying the DORA study is more accurate, but at least it surveyed 5000 developers, globally and more recently It's asking a completely different question; it is a survey of peoples' _perceptions of their own productivity_. That's basically useless; people are notoriously bad at self-evaluating things like that.
Pannoniae 1y ago
There's a few explanations for this, and it's not necessarily contradictory. 1. AI doesn't improve productivity and people just have cognitive biases. (logical, but I also don't think it's true from what I know...) 2. AI does improve productivity, but only if you find your own workflow and what tasks it's good for, and many companies try to shoehorn it into things which just don't work for it. 3. AI does improve productivity, but people aren't incentivised to improve their productivity because they don't see returns from it. Hence, they just use it to work less and have the same output. 4. The previous one but instead of working less, they work at a more leisurely pace. 5. AI doesn't improve producivity, people just feel it's more productive because it requires less cognitive effort to use than actually doing the task. Any of these is plausible, yet they have massively different underlying explanations.... studies don't really show why that's the case. I personally think it's mostly 2. and 3., but it could really be any of these.
- mlinhares 1y ago
  Why not all? I've seen them all play out. There's also the people that are downstream of AI slop that feel less productive because now they have to clean up the shit other people produced.
  - Pannoniae 1y ago
    You're right, it kinda depends on the situation itself! And the downstream effects. Although, I'd argue that the one you're talking about isn't really caused by AI itself, that's squarely a "I can't say no to the slop because they'll take my head off" problem. In healthy places, you would just say "hell no I'm not merging slop", just as you have previously said "no I'm not merging shit copypasted from stackoverflow".
- welshwelsh 1y ago
  I think it's 5. I was very impressed when I first started using AI tools. Felt like I could get so much more done. A couple of embarrassing production incidents later, I no longer feel that way. I always tell myself that I will check the AI's output carefully, but then end up making mistakes that wouldn't have happened if I wrote the code myself.
rsynnott 1y ago
This seems to be a poll of _users_. "Do people think it has improved _their_ productivity?" is a very different question to "Has it empirically improved aggregate productivity of a team/company/industry." People think _all_ sorts of snake oil improve their productivity; you can't trust people to self-report on things like this.

Every study I've read says nobody is seeing productivity gains from AI use. Here's an AI vendor saying the opposite. Funny.

by pluc 1y ago

Every study I've read says nobody is seeing productivity gains from AI use. Here's an AI vendor saying the opposite. Funny.

azdle 1y ago

It's not even claiming that. It's only claiming that people who responded to the survey feel more productive. (Unless you assume that people taking this survey have an objective measure for their own productivity.) > Significant productivity gains: Over 80% of respondents indicate that AI has enhanced their productivity. _Feeling_ more productive is inline with the one proper study I've seen.

Foobar8568 1y ago
Well I feel and I am more productive, now on coding activities, I am not convinced, it basically replaced SO and google, but at the end of the day, I always need and want to check reference material that I may have known or not existed. Plenty of time, Google couldn't even find them. So in my case, yes but not on activities these sellers are usually claiming.
thebigspacefuck 1y ago
The METR study showed even though people feel more productive they weren’t https://arxiv.org/abs/2507.09089 https://arxiv.org/abs/2507.09089
- knes 1y ago
  the MTR study is a joke. it surveyed only 16 devs. in the era of Sonnet 3.5 Can we stop citing this study I'm not saying the DORA study is more accurate, but at least it surveyed 5000 developers, globally and more recently (between June 13 and July 21, 2025) which means using the most recent SOTA models
  - bopbopbop7 1y ago
    Yea cite the study funded by a company that invested billions into AI instead, that will surely be non biased and accurate.
  - capnrefsmmat 1y ago
    It didn't "survey" devs. It paid them to complete real tasks while they were randomly assigned to use AI or not, and measured the actual time taken to complete the tasks vs. just the perception. It is much higher quality evidence than a convenience sample of developers who just report their perceptions.
  - rsynnott 1y ago
    > I'm not saying the DORA study is more accurate, but at least it surveyed 5000 developers, globally and more recently It's asking a completely different question; it is a survey of peoples' _perceptions of their own productivity_. That's basically useless; people are notoriously bad at self-evaluating things like that.

Pannoniae 1y ago

There's a few explanations for this, and it's not necessarily contradictory. 1. AI doesn't improve productivity and people just have cognitive biases. (logical, but I also don't think it's true from what I know...) 2. AI does improve productivity, but only if you find your own workflow and what tasks it's good for, and many companies try to shoehorn it into things which just don't work for it. 3. AI does improve productivity, but people aren't incentivised to improve their productivity because they don't see returns from it. Hence, they just use it to work less and have the same output. 4. The previous one but instead of working less, they work at a more leisurely pace. 5. AI doesn't improve producivity, people just feel it's more productive because it requires less cognitive effort to use than actually doing the task. Any of these is plausible, yet they have massively different underlying explanations.... studies don't really show why that's the case. I personally think it's mostly 2. and 3., but it could really be any of these.

mlinhares 1y ago
Why not all? I've seen them all play out. There's also the people that are downstream of AI slop that feel less productive because now they have to clean up the shit other people produced.
- Pannoniae 1y ago
  You're right, it kinda depends on the situation itself! And the downstream effects. Although, I'd argue that the one you're talking about isn't really caused by AI itself, that's squarely a "I can't say no to the slop because they'll take my head off" problem. In healthy places, you would just say "hell no I'm not merging slop", just as you have previously said "no I'm not merging shit copypasted from stackoverflow".
welshwelsh 1y ago
I think it's 5. I was very impressed when I first started using AI tools. Felt like I could get so much more done. A couple of embarrassing production incidents later, I no longer feel that way. I always tell myself that I will check the AI's output carefully, but then end up making mistakes that wouldn't have happened if I wrote the code myself.

rsynnott 1y ago

This seems to be a poll of _users_. "Do people think it has improved _their_ productivity?" is a very different question to "Has it empirically improved aggregate productivity of a team/company/industry." People think _all_ sorts of snake oil improve their productivity; you can't trust people to self-report on things like this.