2 ms·

I find this shocking though not unbelievable. Could you share how you measure this accurately? I'm interested in integrating such measurements into our services

I find this shocking though not unbelievable. Could you share how you measure this accurately? I'm interested in integrating such measurements into our services as well.

bigstrat2003 1mo ago
I find it unbelievable. I've seen the code LLMs write and it sucks compared to what a typical human produces. The only way an LLM is doing better than human programmers is if your human programmers were producing really terrible work.
- revetkn 1mo ago
  What LLM/harness are you using that the results are so terrible?
- tarun_anand 1mo ago
  Believe it or not, most programmers by definition are average. Hence, producing code better than them is not a hard feat to achieve for today's models.
- LeafItAlone 1mo ago
  I personally find it unbelievable that you have access to all of the public GitHub projects available and still think the typical developer writes good code.
- nharziro 1mo ago
  What exactly do you mean when you say llm generated code? Are people prompting llms for changes and features without reviewing the code or iterating on it and then comparing that to what human writes? Because if so it's not surprising that you're getting worse results. Humans also write code through iteration. You can definitely get llms to write good code by enforcing guardrails and constraints through tooling and agent.md, and iterative reviews to nudge towards what you want. The first pass will look nothing like the committed code. I don't expect the llm to one shot anything.
- pmarreck 25d ago
  That must be why literally every single AI-powered code review of completely-human-written work turns up something. >..< Also, regarding your accusation, you need to contextualize- what language, what use-case, what LLM, etc.?
LeafItAlone 1mo ago
We’ve been tracking performance and bugs for years. Including commits those bugs were introduced in. So when LLM-generated code started working its way into our codebases, we have the before and after. And even comparing human generated code today with LLM-generated code today.

I find this shocking though not unbelievable. Could you share how you measure this accurately? I'm interested in integrating such measurements into our services as well.

bigstrat2003 1mo ago

I find it unbelievable. I've seen the code LLMs write and it sucks compared to what a typical human produces. The only way an LLM is doing better than human programmers is if your human programmers were producing really terrible work.

revetkn 1mo ago
What LLM/harness are you using that the results are so terrible?
tarun_anand 1mo ago
Believe it or not, most programmers by definition are average. Hence, producing code better than them is not a hard feat to achieve for today's models.
LeafItAlone 1mo ago
I personally find it unbelievable that you have access to all of the public GitHub projects available and still think the typical developer writes good code.
nharziro 1mo ago
What exactly do you mean when you say llm generated code? Are people prompting llms for changes and features without reviewing the code or iterating on it and then comparing that to what human writes? Because if so it's not surprising that you're getting worse results. Humans also write code through iteration. You can definitely get llms to write good code by enforcing guardrails and constraints through tooling and agent.md, and iterative reviews to nudge towards what you want. The first pass will look nothing like the committed code. I don't expect the llm to one shot anything.
pmarreck 25d ago
That must be why literally every single AI-powered code review of completely-human-written work turns up something. >..< Also, regarding your accusation, you need to contextualize- what language, what use-case, what LLM, etc.?

LeafItAlone 1mo ago

We’ve been tracking performance and bugs for years. Including commits those bugs were introduced in. So when LLM-generated code started working its way into our codebases, we have the before and after. And even comparing human generated code today with LLM-generated code today.