2 ms·

I think the author misses a few points * METR was at best a flawed study. Repo-familiarity and tool-unfamiliarity being the biggest points of critique, but far

I think the author misses a few points * METR was at best a flawed study. Repo-familiarity and tool-unfamiliarity being the biggest points of critique, but far from the only one * they assume that all code gets shipped as a product. Meanwhile, AI code has (at least in my field of view) led to a proliferation of useful-but-never-shipped one-off tools. Random dashboards to visualize complex queries, scripts to drive refactors, or just sheer joy like "I want to generate an SVG of my vacation trip and consume 15 data sources and give it a certain look". * Their own self-experiment is not exactly statistically sound :) That does leave the fact that we aren't seeing AI shovelware. I'm still convinced that's because commercially viable software is beyond the AI complexity horizon, not because AI isn't an extremely useful tool

ModernMech 1y ago
> * METR was at best a flawed study. They didn't claim it was flawless, they had just brought it up because it caused them to question their internal narrative of their own productivity. > * Their own self-experiment is not exactly statistically sound :) They didn't claim it was. > * they assume that all code gets shipped as a product. The author did not assume this. They assumed that if AI is making developers more productive, that should apply to shovelware developers. That we don't see an increase in shovelware post-AI, makes it very unlikely AI brings an increase in productivity for more complex software.