13 ms·

Well, there are two possible interpretations here of 75% of participants (all of whom had some experience using LLMs) being slower using generative AI: LLMs ha

Well, there are two possible interpretations here of 75% of participants (all of whom had some experience using LLMs) being slower using generative AI: LLMs have a v. steep and long learning curve as you posit (though note the points from the paper authors in the other reply). Current LLMs just are not as good as they are sold to be as a programming assistant and people consistently predict and self-report in the wrong direction on how useful they are.

Terr_ 1y ago
> people consistently predict and self-report in the wrong direction I recall an adage about work-estimation: As chunks get too big, people unconsciously substitute "how possible does the final outcome feel" with "how long will the work take to do." People asked "how long did it take" could be substituting something else, such as "how alone did I feel while working on it."
- sandinmyjoints 1y ago
  That’s an interesting adage. Any ideas of its source?
  - Dilettante_ 1y ago
    It might have been in Kahneman's "Thinking, Fast and Slow"
    - Terr_ 1y ago
      I'm not sure, but something involving Kahneman et al. seems very plausible: The relevant term is probably "Attribute Substitution." https://en.wikipedia.org/wiki/Attribute_substitution https://en.wikipedia.org/wiki/Attribute_substitution
steveklabnik 1y ago
> Current LLMs One thing that happened here is that they aren't using current LLMs: > Most issues were completed in February and March 2025, before models like Claude 4 Opus or Gemini 2.5 Pro were released. That doesn't mean this study is bad! In fact, I'd be very curious to see it done again, but with newer models, to see if that has an impact.
- blibble 1y ago
  > One thing that happened here is that they aren't using current LLMs I've been hearing this for 2 years now the previous model retroactively becomes total dogshit the moment a new one is released convenient, isn't it?
  - simonw 1y ago
    The previous model retroactively becomes not as good as the best available models. I don't think that's a huge surprise.
    - cwillu 1y ago
      The surprise is the implication that the crossover between net-negative and net-positive impact happened to be in the last 4 months, in light of the initial release 2 years ago and sufficient public attention for a study to be funded and completed. Yes, it might make a difference, but it is a little tiresome that there's always a “this is based on a model that is x months old!” comment, because it will always be true: an academic study does not get funded, executed, written up, and published in less time.
      - Ntrails 1y ago
        Some of it is just that (probably different) people said the same damn things 6 months ago. "No, the 2.8 release is the first good one. It massively improves workflows" Then, 6 months later, the study comes out. "Ah man, 2.8 was useless, 3.0 really crossed the threshold on value add" At some point, you roll your eyes and assume it is just snake oil sales
        Filligree 1y ago
        Or you accept that different people have different skill levels, workflows and goals, and therefore the AIs reach usability at different times.
burnte 1y ago
> Current LLMs just are not as good as they are sold to be as a programming assistant and people consistently predict and self-report in the wrong direction on how useful they are. I would argue you don't need the "as a programming assistant" phrase as right now from my experience over the past 2 years, literally every single AI tool is massively oversold as to its utility. I've literally not seen a single one that delivers on what it's billed as capable of. They're useful, but right now they need a lot of handholding and I don't have time for that. Too much fact checking. If I want a tool I always have to double check, I was born with a memory so I'm already good there. I don't want to have to fact check my fact checker. LLMs are great at small tasks. The larger the single task is, or the more tasks you try to cram into one session, the worse they fall apart.
- deleted 1y ago
  [deleted]
atiedebee 1y ago
Let me bring you a third (not necessarily true) interpretation: The developer who has experience using cursor saw a productivity increase not because he became better at using cursor, but because he became worse at not using it.
- card_zero 1y ago
  Or, one person in 16 has a particular personality, inclined to LLM dependence.
  - runarberg 1y ago
    Invoking personality is to the behavioral science as invoking God is to the natural sciences. One can explain anything by appealing to personality, and as such it explains nothing. Psychologists have been trying to make sense of personality for over a century without much success (the best efforts so far have been a five factor model [Big 5] which has ultimately pretty minor predictive value), which is why most behavioral scientists have learned to simply leave personality to the philosophers and concentrate on much simpler theoretical framework. A much simpler explanation is what your parent offered. And to many behavioralists it is actually the same explanation, as to a true scotsm... [cough] behavioralist personality is simply learned habits, so—by Occam’s razor—you should omit personality from your model.
    - card_zero 1y ago
      Fair comment, but I'm not down with behavioralism, and people have personalities, regrettably.
      - runarberg 1y ago
        This is still ultimately a research within the field of the behavior sciences, and as such the laws of human behavior apply, where behaviorism offers a far more successful theoretical framework than personality psychology. Nobody is denying that people have personalities btw. Not even true behavioralists do that, they simply argue from reductionism that personality can be explained with learning contingencies and the reinforcement history. Very few people are true behavioralists these days though, but within the behavior sciences, scientists are much more likely to borrow missing factors (i.e. things that learning contingencies fail to explain) from fields such as cognitive science (or even further to neuroscience) and (less often) social science. What I am arguing here, however, is that the appeal to personality is unnecessary when explaining behavior. As for figuring out what personality is, that is still within the realm of philosophy. Maybe cognitive science will do a better job at explaining it than psychometricians have done for the past century. I certainly hope so, it would be nice to have a better model of human behavior. But I think even if we could explain personality, it still wouldn’t help us here. At best we would be in a similar situation as physics, where one model can explain things traveling at the speed of light, while another model can explain things at the sub-atomic scale, but the two models cannot be applied together.
robwwilliams 1y ago
Or a sampling artifact. 4 vs 12 does seem significant within a study, but consider a set of N such studies. I assume that many large companies have tested efficiency gains and losses of there programmers much more extensively than the authors of this tiny study. A survey of companies and their evaluation and conclusions would carry more weight—-excluding companies selling AI products, of course.
- rs186 1y ago
  If you use binomial test, P(X<=4) is about 0.105 which means p = 0.21.
giantg2 1y ago
The third option is that the person who used Cursor before had some sort of skill atrophy that led to lower unassisted speed. I think an easy measure to help identify why a slow down is happening would be to measure how much refactoring happened on the AI generated code. Often times it seems to be missing stuff like error handling, or adds in unnecessary stuff. Of course this assumes it even had a working solution in the first place.

Terr_ 1y ago

> people consistently predict and self-report in the wrong direction I recall an adage about work-estimation: As chunks get too big, people unconsciously substitute "how possible does the final outcome feel" with "how long will the work take to do." People asked "how long did it take" could be substituting something else, such as "how alone did I feel while working on it."

sandinmyjoints 1y ago
That’s an interesting adage. Any ideas of its source?
- Dilettante_ 1y ago
  It might have been in Kahneman's "Thinking, Fast and Slow"
  - Terr_ 1y ago
    I'm not sure, but something involving Kahneman et al. seems very plausible: The relevant term is probably "Attribute Substitution." https://en.wikipedia.org/wiki/Attribute_substitution https://en.wikipedia.org/wiki/Attribute_substitution

steveklabnik 1y ago

> Current LLMs One thing that happened here is that they aren't using current LLMs: > Most issues were completed in February and March 2025, before models like Claude 4 Opus or Gemini 2.5 Pro were released. That doesn't mean this study is bad! In fact, I'd be very curious to see it done again, but with newer models, to see if that has an impact.

blibble 1y ago
> One thing that happened here is that they aren't using current LLMs I've been hearing this for 2 years now the previous model retroactively becomes total dogshit the moment a new one is released convenient, isn't it?
- simonw 1y ago
  The previous model retroactively becomes not as good as the best available models. I don't think that's a huge surprise.
  - cwillu 1y ago
    The surprise is the implication that the crossover between net-negative and net-positive impact happened to be in the last 4 months, in light of the initial release 2 years ago and sufficient public attention for a study to be funded and completed. Yes, it might make a difference, but it is a little tiresome that there's always a “this is based on a model that is x months old!” comment, because it will always be true: an academic study does not get funded, executed, written up, and published in less time.
    - Ntrails 1y ago
      Some of it is just that (probably different) people said the same damn things 6 months ago. "No, the 2.8 release is the first good one. It massively improves workflows" Then, 6 months later, the study comes out. "Ah man, 2.8 was useless, 3.0 really crossed the threshold on value add" At some point, you roll your eyes and assume it is just snake oil sales
      - Filligree 1y ago
        Or you accept that different people have different skill levels, workflows and goals, and therefore the AIs reach usability at different times.

burnte 1y ago

> Current LLMs just are not as good as they are sold to be as a programming assistant and people consistently predict and self-report in the wrong direction on how useful they are. I would argue you don't need the "as a programming assistant" phrase as right now from my experience over the past 2 years, literally every single AI tool is massively oversold as to its utility. I've literally not seen a single one that delivers on what it's billed as capable of. They're useful, but right now they need a lot of handholding and I don't have time for that. Too much fact checking. If I want a tool I always have to double check, I was born with a memory so I'm already good there. I don't want to have to fact check my fact checker. LLMs are great at small tasks. The larger the single task is, or the more tasks you try to cram into one session, the worse they fall apart.

deleted 1y ago
[deleted]

atiedebee 1y ago

Let me bring you a third (not necessarily true) interpretation: The developer who has experience using cursor saw a productivity increase not because he became better at using cursor, but because he became worse at not using it.

card_zero 1y ago
Or, one person in 16 has a particular personality, inclined to LLM dependence.
- runarberg 1y ago
  Invoking personality is to the behavioral science as invoking God is to the natural sciences. One can explain anything by appealing to personality, and as such it explains nothing. Psychologists have been trying to make sense of personality for over a century without much success (the best efforts so far have been a five factor model [Big 5] which has ultimately pretty minor predictive value), which is why most behavioral scientists have learned to simply leave personality to the philosophers and concentrate on much simpler theoretical framework. A much simpler explanation is what your parent offered. And to many behavioralists it is actually the same explanation, as to a true scotsm... [cough] behavioralist personality is simply learned habits, so—by Occam’s razor—you should omit personality from your model.
  - card_zero 1y ago
    Fair comment, but I'm not down with behavioralism, and people have personalities, regrettably.
    - runarberg 1y ago
      This is still ultimately a research within the field of the behavior sciences, and as such the laws of human behavior apply, where behaviorism offers a far more successful theoretical framework than personality psychology. Nobody is denying that people have personalities btw. Not even true behavioralists do that, they simply argue from reductionism that personality can be explained with learning contingencies and the reinforcement history. Very few people are true behavioralists these days though, but within the behavior sciences, scientists are much more likely to borrow missing factors (i.e. things that learning contingencies fail to explain) from fields such as cognitive science (or even further to neuroscience) and (less often) social science. What I am arguing here, however, is that the appeal to personality is unnecessary when explaining behavior. As for figuring out what personality is, that is still within the realm of philosophy. Maybe cognitive science will do a better job at explaining it than psychometricians have done for the past century. I certainly hope so, it would be nice to have a better model of human behavior. But I think even if we could explain personality, it still wouldn’t help us here. At best we would be in a similar situation as physics, where one model can explain things traveling at the speed of light, while another model can explain things at the sub-atomic scale, but the two models cannot be applied together.

robwwilliams 1y ago

Or a sampling artifact. 4 vs 12 does seem significant within a study, but consider a set of N such studies. I assume that many large companies have tested efficiency gains and losses of there programmers much more extensively than the authors of this tiny study. A survey of companies and their evaluation and conclusions would carry more weight—-excluding companies selling AI products, of course.

rs186 1y ago
If you use binomial test, P(X<=4) is about 0.105 which means p = 0.21.

giantg2 1y ago

The third option is that the person who used Cursor before had some sort of skill atrophy that led to lower unassisted speed. I think an easy measure to help identify why a slow down is happening would be to measure how much refactoring happened on the AI generated code. Often times it seems to be missing stuff like error handling, or adds in unnecessary stuff. Of course this assumes it even had a working solution in the first place.