I don't think the study is flawed. It just seems rather narrow: "We conduct a randomized controlled trial (RCT) to understand how AI tools at the February-June

I don't think the study is flawed. It just seems rather narrow: "We conduct a randomized controlled trial (RCT) to understand how AI tools at the February-June 2025 frontier affect the productivity of experienced open-source developers. 16 developers with moderate AI experience complete 246 tasks in mature projects on which they have an average of 5 years of prior experience." So the question is what other kinds of software development tasks this result applies to. Moderate AI experience is fine. This applies to many other situations. But 5 years of experience with a single code base is an outlier. That said, they used relatively large repositories (1.1 million LOC) and the tasks were randomly assigned. So developers couldn't pick and choose tasks in areas of the codebase they already knew extremely well. I think the study does generalise to some degree, but I've seen conclusions drawn from this study that the methodology doesn't support. In my view, it doesn't generalise over all or even most software development taks. Personally, I'm a bit sceptical (but not hostile) about LLMs for coding (and some other thinking tasks), because the difference in quality between requests for which there are many examples and tasks for which there are only few examples is so extreme. Reasoning capabilities of LLMs still seem minimal.