3 ms·

Show me your data. The only study I’ve seen so far on LLMs and productivity, showed that developers using an LLM were LESS productive than those who didn’t use

Show me your data. The only study I’ve seen so far on LLMs and productivity, showed that developers using an LLM were LESS productive than those who didn’t use them.

williamcotton 1y ago
Here's my blog with a work in progress article [0], written in a DSL I wrote called Web Pipe [1] that I started four days ago [2] with probably about 12 hours worth of work: [0] https://williamcotton.com/articles/introducing-web-pipe https://williamcotton.com/articles/introducing-web-pipe [1] https://github.com/williamcotton/williamcotton.com/blob/master/app.wp https://github.com/williamcotton/williamcotton.com/blob/mast... [2] https://github.com/williamcotton/webpipe-rs/commits/main/?after=bfda9be0a4453681f6aacaec41abd5024f4ea2be+69 https://github.com/williamcotton/webpipe-rs/commits/main/?af...
- satisfice 1y ago
  Who tested it? Was he competent? where’s his report?
ants_everywhere 1y ago
You could start with a basic literature review. https://scholar.google.com/scholar?hl=en&q=llm%20developer%20productivity https://scholar.google.com/scholar?hl=en&q=llm%20developer%2... > The only study I’ve seen so far on LLMs and productivity Why are you guys all citing the same study in this thread? There are several studies including a survey paper.
- didibus 1y ago
  Personally it's because it's the one that had the best methodology I've seen. Most of the other ones I've seen are kind of a joke. They're mostly self-surveyed, like just asking the participants to say if they were more productive, wrote better code, etc. Or you've got some that takes like beginner devs that never worked a job and have them run through some unrealistically simple task. Or a handful that are claiming productivity boosts, but they count things like faster than Google search, or fewer keystrokes. Or when they measure the "code quality" they just mean the tests passes. So when that study came out and I read the methodology, I was like, oh, this is interesting. And then the results were also quite surprising, and surprisingly the one with the best methodology had the opposite finding of all the crappy ones I've seen before. Food for thought.
  - deleted 1y ago
    [deleted]
didibus 1y ago
There's a dissonance I feel. The study for example looked at experienced developer working on existing open source projects. Lots of people we're now conversing with could be junior or mid-level, might have tried it for little prototypes/experiments, or for more trivial software like commissioned websites, and so on. They could all be benefiting from agentic coding workflows in ways that we don't. With the caveat that the study you talked about also showed even the experience devs felt more productive, so clearly the use of AI biases your perception of delivery speed. The large array of context I suspect is responsible for some of that dissonance on online discourse.
keeda 1y ago
There are more studies out there, but here are a couple I know of offhand, showing a 25% to 55% boost. https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4945566 https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4945566 https://arxiv.org/abs/2302.06590 https://arxiv.org/abs/2302.06590 The METR study that you're likely talking about had a lot of nuances that don't get talked about, not to mention outright concerns e.g. this one participant revealed he had a pretty damning selection bias: https://xcancel.com/ruben_bloom/status/1943536052037390531 https://xcancel.com/ruben_bloom/status/1943536052037390531

Show me your data. The only study I’ve seen so far on LLMs and productivity, showed that developers using an LLM were LESS productive than those who didn’t use them.

williamcotton 1y ago

Here's my blog with a work in progress article [0], written in a DSL I wrote called Web Pipe [1] that I started four days ago [2] with probably about 12 hours worth of work: [0] https://williamcotton.com/articles/introducing-web-pipe https://williamcotton.com/articles/introducing-web-pipe [1] https://github.com/williamcotton/williamcotton.com/blob/master/app.wp https://github.com/williamcotton/williamcotton.com/blob/mast... [2] https://github.com/williamcotton/webpipe-rs/commits/main/?after=bfda9be0a4453681f6aacaec41abd5024f4ea2be+69 https://github.com/williamcotton/webpipe-rs/commits/main/?af...

satisfice 1y ago
Who tested it? Was he competent? where’s his report?

ants_everywhere 1y ago

You could start with a basic literature review. https://scholar.google.com/scholar?hl=en&q=llm%20developer%20productivity https://scholar.google.com/scholar?hl=en&q=llm%20developer%2... > The only study I’ve seen so far on LLMs and productivity Why are you guys all citing the same study in this thread? There are several studies including a survey paper.

didibus 1y ago
Personally it's because it's the one that had the best methodology I've seen. Most of the other ones I've seen are kind of a joke. They're mostly self-surveyed, like just asking the participants to say if they were more productive, wrote better code, etc. Or you've got some that takes like beginner devs that never worked a job and have them run through some unrealistically simple task. Or a handful that are claiming productivity boosts, but they count things like faster than Google search, or fewer keystrokes. Or when they measure the "code quality" they just mean the tests passes. So when that study came out and I read the methodology, I was like, oh, this is interesting. And then the results were also quite surprising, and surprisingly the one with the best methodology had the opposite finding of all the crappy ones I've seen before. Food for thought.
- deleted 1y ago
  [deleted]

didibus 1y ago

There's a dissonance I feel. The study for example looked at experienced developer working on existing open source projects. Lots of people we're now conversing with could be junior or mid-level, might have tried it for little prototypes/experiments, or for more trivial software like commissioned websites, and so on. They could all be benefiting from agentic coding workflows in ways that we don't. With the caveat that the study you talked about also showed even the experience devs felt more productive, so clearly the use of AI biases your perception of delivery speed. The large array of context I suspect is responsible for some of that dissonance on online discourse.

keeda 1y ago

There are more studies out there, but here are a couple I know of offhand, showing a 25% to 55% boost. https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4945566 https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4945566 https://arxiv.org/abs/2302.06590 https://arxiv.org/abs/2302.06590 The METR study that you're likely talking about had a lot of nuances that don't get talked about, not to mention outright concerns e.g. this one participant revealed he had a pretty damning selection bias: https://xcancel.com/ruben_bloom/status/1943536052037390531 https://xcancel.com/ruben_bloom/status/1943536052037390531