6 ms·

We don't just tell you they were imagined, we can provide receipts. https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/ https://metr.org/b

We don't just tell you they were imagined, we can provide receipts. https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/ https://metr.org/blog/2025-07-10-early-2025-ai-experienced-o...

nojito 1y ago
Cursor is an old way of using LLMs. Not to mention in the study less than 1/2 have ever used it before the study.
- roywiggins 1y ago
  The AI tooling churn is so fast that by the time a study comes out people will be able to say "well they were using an older tool" no matter what tool that the study used.
  - cratermoon 1y ago
    It's the eternal future. "AI will soon be able to...". There's an entire class of investment scammers that string along their marks, claiming that the big payoff is just around corner while they fleece the victim with the death of a thousand cuts.
  - nojito 1y ago
    Not really. Chatting with a llm was cutting edge for 3 years it’s only within the last 8-10 months with Claude code and Gemini cli do we have the next big change in how we interact with llms
    - roywiggins 1y ago
      Claude Code was released in May.
      - nojito 1y ago
        Yup. But they are improvements over what cursor was releasing over the last year or so.
        roywiggins 1y ago
        If there are paradigm-shattering improvements every six months, every single study that is ever released will be "behind" or "use an older tool." In six months when a study comes out using Claude Code, people dissatisfied with it will be able to point to the newest hotness, ad infinitum.
    - camdenreslink 1y ago
      How is Claude Code and Gemini CLI any different from using Cursor in agent mode? It's basically the same exact thing.
      - steveklabnik 1y ago
        I can't speak to how they're technically different, but in practice, Cursor was basically useless for me, and Claude Code works well. Even with Cursor using Claude's models.
  - bubblyworld 1y ago
    What is the problem with this, exactly? It's a valid criticism of the study (when applied to current agentic coding practices). That the pace of progress is so fast sucks for researchers, in some sense, but this is the reality right now.
  - deleted 1y ago
    [deleted]
steveklabnik 1y ago
> We do not provide evidence that: > AI systems do not currently speed up many or most software developers > We do not claim that our developers or repositories represent a majority or plurality of software development work
brokencode 1y ago
Certainly an interesting result, but remember that a single paper doesn’t prove anything. This will no doubt be something studied very extensively and change over time as tools develop. Personally, I find the current tools don’t work great for large existing codebases and complex tasks. But I’ve found they can help me quickly make small scripts to save me time. I know, it’s not the most glamorous application, but it’s what I find useful today. And I have confidence the tools will continue to improve. They hardly even existed a few years ago.

nojito 1y ago

Cursor is an old way of using LLMs. Not to mention in the study less than 1/2 have ever used it before the study.

roywiggins 1y ago
The AI tooling churn is so fast that by the time a study comes out people will be able to say "well they were using an older tool" no matter what tool that the study used.
- cratermoon 1y ago
  It's the eternal future. "AI will soon be able to...". There's an entire class of investment scammers that string along their marks, claiming that the big payoff is just around corner while they fleece the victim with the death of a thousand cuts.
- nojito 1y ago
  Not really. Chatting with a llm was cutting edge for 3 years it’s only within the last 8-10 months with Claude code and Gemini cli do we have the next big change in how we interact with llms
  - roywiggins 1y ago
    Claude Code was released in May.
    - nojito 1y ago
      Yup. But they are improvements over what cursor was releasing over the last year or so.
      - roywiggins 1y ago
        If there are paradigm-shattering improvements every six months, every single study that is ever released will be "behind" or "use an older tool." In six months when a study comes out using Claude Code, people dissatisfied with it will be able to point to the newest hotness, ad infinitum.
  - camdenreslink 1y ago
    How is Claude Code and Gemini CLI any different from using Cursor in agent mode? It's basically the same exact thing.
    - steveklabnik 1y ago
      I can't speak to how they're technically different, but in practice, Cursor was basically useless for me, and Claude Code works well. Even with Cursor using Claude's models.
- bubblyworld 1y ago
  What is the problem with this, exactly? It's a valid criticism of the study (when applied to current agentic coding practices). That the pace of progress is so fast sucks for researchers, in some sense, but this is the reality right now.
- deleted 1y ago
  [deleted]

steveklabnik 1y ago

> We do not provide evidence that: > AI systems do not currently speed up many or most software developers > We do not claim that our developers or repositories represent a majority or plurality of software development work

brokencode 1y ago

Certainly an interesting result, but remember that a single paper doesn’t prove anything. This will no doubt be something studied very extensively and change over time as tools develop. Personally, I find the current tools don’t work great for large existing codebases and complex tasks. But I’ve found they can help me quickly make small scripts to save me time. I know, it’s not the most glamorous application, but it’s what I find useful today. And I have confidence the tools will continue to improve. They hardly even existed a few years ago.