3 ms·

I love this train of thought. Code quality is critical, but I don’t think we’re correctly evaling it at the moment. If we could get solid benchmarks measuring t

I love this train of thought. Code quality is critical, but I don’t think we’re correctly evaling it at the moment. If we could get solid benchmarks measuring the quality of generated code, we might see the models climb those benches fast. I believe that the era of “ai writes tons of slop code” will be a stepping stone in the longer story, and is simply a current gap in the reward functions. Per the author - if we can get strong measurements of what good code is, we can train against it and close the gap fast. Excited to see more thinking in this area

loveparade 23d ago
If you could easily benchmark the quality of code then models would be trained on these benchmarks/metrics.
- yehoshuapw 23d ago
  that is true, but if the metric is what we want optimized, then that's fine. However it is more likely to be something which can be detached..
- datsci_est_2015 23d ago
  Code quality is probably isomorphic to the halting problem, or can be reduced to the halting problem in the simplest case. I.e. it’s intractable.
peder 23d ago
Sooooo much of what is considered "code quality" today is irrelevant when robots are writing the code. We've been largely optimizing for things like composability/unit testability in the past 15 or so years, and that's primarily a human concern that's unrelated to the final output. Totally agreed that we're not looking at the correct metric right now. Increasingly, code quality will be determined by outcomes.
- alecbz 23d ago
  > that's primarily a human concern that's unrelated to the final output. It's an LLM concern too. LLMs seem to do better with well-organized codebases, just like the humans they were trained on.
- stiiv 23d ago
  I think Martin F's team tied outcomes to factoring https://martinfowler.com/articles/exploring-gen-ai/refactoring-economic-benefit.html https://martinfowler.com/articles/exploring-gen-ai/refactori... which is crucial to code quality. Even grug brain developer agree. On the other hand, there are claims that the best languages for robots tend to be terse (allegedly); I tend to regard a rich domain model (static types, not terse) as a major facet of high-quality code.
- FuckButtons 23d ago
  I don’t agree with this. The things people care about with code quality if you really think about it actually match up surprisingly well with the metric which models are trained to emulate in pre training, namely compression and modularity. Those two ideas actually seem to be universal to intelligent systems. Writing verbose highly coupled code is I think provably stupid, though I don’t know that I could formalize it.

loveparade 23d ago

If you could easily benchmark the quality of code then models would be trained on these benchmarks/metrics.

yehoshuapw 23d ago
that is true, but if the metric is what we want optimized, then that's fine. However it is more likely to be something which can be detached..
datsci_est_2015 23d ago
Code quality is probably isomorphic to the halting problem, or can be reduced to the halting problem in the simplest case. I.e. it’s intractable.

peder 23d ago

Sooooo much of what is considered "code quality" today is irrelevant when robots are writing the code. We've been largely optimizing for things like composability/unit testability in the past 15 or so years, and that's primarily a human concern that's unrelated to the final output. Totally agreed that we're not looking at the correct metric right now. Increasingly, code quality will be determined by outcomes.

alecbz 23d ago
> that's primarily a human concern that's unrelated to the final output. It's an LLM concern too. LLMs seem to do better with well-organized codebases, just like the humans they were trained on.
stiiv 23d ago
I think Martin F's team tied outcomes to factoring https://martinfowler.com/articles/exploring-gen-ai/refactoring-economic-benefit.html https://martinfowler.com/articles/exploring-gen-ai/refactori... which is crucial to code quality. Even grug brain developer agree. On the other hand, there are claims that the best languages for robots tend to be terse (allegedly); I tend to regard a rich domain model (static types, not terse) as a major facet of high-quality code.
FuckButtons 23d ago
I don’t agree with this. The things people care about with code quality if you really think about it actually match up surprisingly well with the metric which models are trained to emulate in pre training, namely compression and modularity. Those two ideas actually seem to be universal to intelligent systems. Writing verbose highly coupled code is I think provably stupid, though I don’t know that I could formalize it.