3 ms·

It's remarkable that academia hasn't produced rigorous studies with statistically significant samples, with hundreds of developers across languages and skill le

by belter 1y ago

It's remarkable that academia hasn't produced rigorous studies with statistically significant samples, with hundreds of developers across languages and skill levels, to properly assess GenAI impact on productivity. The latest published research has a sample size of like...16 developers? We don't even need human subjects. GitHub contains a natural experiment: Millions of high-quality commits and PRs from before the GenAI explosion (pre-Jan 2021). Compare velocity, complexity, and quality metrics before and after, the data is sitting there, waiting to be analyzed. Instead, we're drowning in anecdotes while petabytes of empirical evidence remain untouched.

steveBK123 1y ago
As in every bubble, its the incentives. Everyone is selling AI or being sold AI.
CJefferson 1y ago
The main problem (and the reason I have no intention of doing any research involving AI programming), is even if you could get the research started, completed, and published in 4 months (and incredibly ambitious goal), 95% of the comments would just be "Oh, but you didn't consider FishGPT 5.2, and Pineapple AI, which came out 6 weeks ago, so any negative points are entirely out of date".
- bigtex 1y ago
  "You didn't use SOTA" which is what I see when people respond to critical statements about the productivity of AI on Twitter and Reddit.
lelanthran 1y ago
> It's remarkable that academia hasn't produced rigorous studies with statistically significant samples, with hundreds of developers across languages and skill levels, to properly assess GenAI impact on productivity. It's much more remarkable, to me, that the AI vendors have been unable to publish a study on the effects of agents on developer velocity. I mean, if it was really a 10x (or even 2x) boost to productivity, why aren't they seeing it? The returns on each generation have been diminishing compared to the cost. Shouldn't it be the other way around if AI was such a productivity boost?
AIPedant 1y ago
This has been the case in software development for a very long time, it is almost impossible to get reliable scientific data about these things because it's almost impossible to run a controlled experiment, and even if you did it's very difficult to decide what to measure and how to measure it. (It's easy to say "compare complexity and quality metrics" but how the fk are you going to define that? Does it really make sense to use the same complexity metric for Go and C++?) Software developers have had many flame wars over the decades because the lack of data forces the conversation to be anecdotal and ideological: - should we use dynamic/gradual typing and prioritize developer productivity, or static typing to help enforce correctness? - agile vs waterfall - OO vs procedural vs functional - is Rust's fussiness around memory management more trouble than it's worth for large projects? - when should you use a 3rd-party library vs doing it yourself? So this really is nothing new. You are badly underestimating the scientific challenges, instead just hoping big data will plow through. It won't.
- belter 1y ago
  Your point cuts both ways: If measuring GenAI impact is as difficult as you say, then the bold claims of dramatic productivity gains are just as fragile. How, exactly, do nine out of ten CEOs know it works?
  - AIPedant 1y ago
    Yes, that is correct. (I don't use LLMs at all for ethical reasons, so I don't have a dog in this specific fight.)

by belter 1y ago

steveBK123 1y ago

As in every bubble, its the incentives. Everyone is selling AI or being sold AI.

CJefferson 1y ago

The main problem (and the reason I have no intention of doing any research involving AI programming), is even if you could get the research started, completed, and published in 4 months (and incredibly ambitious goal), 95% of the comments would just be "Oh, but you didn't consider FishGPT 5.2, and Pineapple AI, which came out 6 weeks ago, so any negative points are entirely out of date".

bigtex 1y ago
"You didn't use SOTA" which is what I see when people respond to critical statements about the productivity of AI on Twitter and Reddit.

lelanthran 1y ago

> It's remarkable that academia hasn't produced rigorous studies with statistically significant samples, with hundreds of developers across languages and skill levels, to properly assess GenAI impact on productivity. It's much more remarkable, to me, that the AI vendors have been unable to publish a study on the effects of agents on developer velocity. I mean, if it was really a 10x (or even 2x) boost to productivity, why aren't they seeing it? The returns on each generation have been diminishing compared to the cost. Shouldn't it be the other way around if AI was such a productivity boost?

AIPedant 1y ago

This has been the case in software development for a very long time, it is almost impossible to get reliable scientific data about these things because it's almost impossible to run a controlled experiment, and even if you did it's very difficult to decide what to measure and how to measure it. (It's easy to say "compare complexity and quality metrics" but how the fk are you going to define that? Does it really make sense to use the same complexity metric for Go and C++?) Software developers have had many flame wars over the decades because the lack of data forces the conversation to be anecdotal and ideological: - should we use dynamic/gradual typing and prioritize developer productivity, or static typing to help enforce correctness? - agile vs waterfall - OO vs procedural vs functional - is Rust's fussiness around memory management more trouble than it's worth for large projects? - when should you use a 3rd-party library vs doing it yourself? So this really is nothing new. You are badly underestimating the scientific challenges, instead just hoping big data will plow through. It won't.

belter 1y ago
Your point cuts both ways: If measuring GenAI impact is as difficult as you say, then the bold claims of dramatic productivity gains are just as fragile. How, exactly, do nine out of ten CEOs know it works?
- AIPedant 1y ago
  Yes, that is correct. (I don't use LLMs at all for ethical reasons, so I don't have a dog in this specific fight.)