7 ms·

Interesting question to study, but I'm extremely skeptical of the experimental design. They used Opus 4.6 to synthetically produce "degraded" or "cleaned" code

Interesting question to study, but I'm extremely skeptical of the experimental design. They used Opus 4.6 to synthetically produce "degraded" or "cleaned" code bases for relative comparison in the experiment. Worse, they don't control for breaking the application's tests. > Pass rate scores the agent’s final state against the hidden tests we wrote for each task. We do not check whether the agent broke unrelated tests already present in the repository, and a cleaner-side and messier-side solution that both pass the hidden test may still differ on tests they were not graded on. Any conclusions with respect to token consumption seems pretty meaningless if we're not controlling for the quality of the final output.

jwpapi 3mo ago
Also controlling input
- huflungdung 3mo ago
  [dead]
- rullelito 3mo ago
  Input matching distribution of a specific one-shotted LLM... Nice.
lloydatkinson 3mo ago
> they don't control for breaking the application's tests. Essentially it's an absolutely worthless study then. This AI fatigue got boring a long time ago, this is just painful now.
- geraltofrivia 3mo ago
  To be sure, we had hidden tests that validate whether agent implemented the task appropriately. And this is something we paid a lot of attention to (pass rate, in our paper.) What we didn't do (stupid oversight on my part) was to ensure that there are no regressions in the remaining tests in the repo (unrelated to the current task). In practice, when using Sonnet 4.6 IRL, I don't see a lot of regressions because often the agent runs the test before calling it done. But it could have gone either way. We don't know.
softwaredoug 3mo ago
At best the conclusions are a best case scenario for degraded code quality. That things remain functionally OK with more costly token usage to get work done.
bastawhiz 3mo ago
If you're measuring a sloppy project against a well structured one with regard to how many tests fail, that'll bias success towards the sloppy codebase, which likely has worse test coverage and less robust testing in the first place. You'd essentially need to write a single test suite that works for both projects in each pair to compare fairly. That's not to say the study is good, but I can respect their decision because the tests passing isn't necessarily correlated with the effectiveness of the agent.
- geraltofrivia 3mo ago
  > You'd essentially need to write a single test suite that works for both projects in each pair to compare fairly. That's pretty much what we did. We start with a repository, and - (vibeclean pipeline) make changes to clean analyzer issues (some of them involve moving big code chunks around to reduce cognitive complexity), and tweak the existing test sets when needed. - (slopify pipeline) inverse, add analyzer issues and increase complexity, add dead code etc. In both cases, we ensure that the test coverage remains the same, and the tests are passing, before we start our experiments.

jwpapi 3mo ago

Also controlling input

huflungdung 3mo ago
[dead]
rullelito 3mo ago
Input matching distribution of a specific one-shotted LLM... Nice.

lloydatkinson 3mo ago

> they don't control for breaking the application's tests. Essentially it's an absolutely worthless study then. This AI fatigue got boring a long time ago, this is just painful now.

geraltofrivia 3mo ago
To be sure, we had hidden tests that validate whether agent implemented the task appropriately. And this is something we paid a lot of attention to (pass rate, in our paper.) What we didn't do (stupid oversight on my part) was to ensure that there are no regressions in the remaining tests in the repo (unrelated to the current task). In practice, when using Sonnet 4.6 IRL, I don't see a lot of regressions because often the agent runs the test before calling it done. But it could have gone either way. We don't know.

softwaredoug 3mo ago

At best the conclusions are a best case scenario for degraded code quality. That things remain functionally OK with more costly token usage to get work done.

bastawhiz 3mo ago

If you're measuring a sloppy project against a well structured one with regard to how many tests fail, that'll bias success towards the sloppy codebase, which likely has worse test coverage and less robust testing in the first place. You'd essentially need to write a single test suite that works for both projects in each pair to compare fairly. That's not to say the study is good, but I can respect their decision because the tests passing isn't necessarily correlated with the effectiveness of the agent.

geraltofrivia 3mo ago
> You'd essentially need to write a single test suite that works for both projects in each pair to compare fairly. That's pretty much what we did. We start with a repository, and - (vibeclean pipeline) make changes to clean analyzer issues (some of them involve moving big code chunks around to reduce cognitive complexity), and tweak the existing test sets when needed. - (slopify pipeline) inverse, add analyzer issues and increase complexity, add dead code etc. In both cases, we ensure that the test coverage remains the same, and the tests are passing, before we start our experiments.