3 ms·

Depends on the prior state. For one codebase I work in, LLM written code is noticeably and measurably (we have literally measured it for bugs, performance, etc

Depends on the prior state. For one codebase I work in, LLM written code is noticeably and measurably (we have literally measured it for bugs, performance, etc.) better than what existed before. So the LLM code is a welcome change.

w0m 21d ago
'ai slop' is hard for me to complain about when it's better than 95% of code prior. The issue is more people trusting it too much to go too far, and you end up with lots of extraneous bullcrap you don't need. Doing too much and too big of a change is still a problem in the agentic world.
dayvid 21d ago
It also documents itself extensively
- karlshea 21d ago
  But then you’ll use it to change something, and it won’t update any of the extensive documentation. So then it’s filled with lies.
  - nuancebydefault 21d ago
    Sounds like human generated code and docs. TBH
  - PepegaRoach 21d ago
    Or worse It'll mention the old version then the change, and the relating tickets.
  - snoman 21d ago
    That actually hasn’t been my experience. For all that I dislike about LLM code, it maintains comments well.
  - LeafItAlone 20d ago
    You need a better harness. LLMs are significantly better at this than humans. Add it as a step in your LLM process and your documentation will always be up to date.
- tempest_ 21d ago
  I fucking hate it. It encodes context it needs into doc comments and regular comments that make them in insufferable to read and so verbose because the comments contain info on other parts of the code base (that might have changed since that comment was written) but were useful to the model for its implementation. I have so far found no way to stop claude from doing this. It will sometimes do it with hand holding but the moment a task takes a slightly large amount of context its back to the word barf.
  - LeafItAlone 20d ago
    This is certainly a problem with out-of-the-box usage. But easily overcome. Spend time updating your harness. This is not a hard problem to solve and there are many helpful guides out there. We basically took our documented (human) SDLC process and applied it to the relevant harness hooks. Since our SDLC docs talked about what good comments looked like and what to avoid, we basically had the direction for the tools. Opus 5 did throw us a loop and increase the verbosity and decrease usefulness of the text, but refining our documentation cleared that hurdle within about a day.
eudamoniac 21d ago
I find this shocking though not unbelievable. Could you share how you measure this accurately? I'm interested in integrating such measurements into our services as well.
- bigstrat2003 21d ago
  I find it unbelievable. I've seen the code LLMs write and it sucks compared to what a typical human produces. The only way an LLM is doing better than human programmers is if your human programmers were producing really terrible work.
  - revetkn 21d ago
    What LLM/harness are you using that the results are so terrible?
  - tarun_anand 21d ago
    Believe it or not, most programmers by definition are average. Hence, producing code better than them is not a hard feat to achieve for today's models.
  - LeafItAlone 21d ago
    I personally find it unbelievable that you have access to all of the public GitHub projects available and still think the typical developer writes good code.
  - nharziro 20d ago
    What exactly do you mean when you say llm generated code? Are people prompting llms for changes and features without reviewing the code or iterating on it and then comparing that to what human writes? Because if so it's not surprising that you're getting worse results. Humans also write code through iteration. You can definitely get llms to write good code by enforcing guardrails and constraints through tooling and agent.md, and iterative reviews to nudge towards what you want. The first pass will look nothing like the committed code. I don't expect the llm to one shot anything.
  - 14d ago
wpietri 21d ago
Do you measure anything about complexity or brevity? I'm currently working on importing some LLM code and it's 5-10x bigger than it should be for the actual needs. One of my open questions is how much that increases over time. It could be that is a constant. But it also could be that, say, coding agents will infer false needs from the excess code/docs and elaborate further.
- pmarreck 14d ago
  You always must measure complexity (O-time), brevity, etc. LLM code that is written to a lower standard has not had the controls put in place.
mjevans 20d ago
I've only personally utilized it for code _review_ and always ignored the suggested result code instead to understand the source of the issue and try to arrive at a resolution I find satisfactory for my mental model of the issue and solutions I'd prefer. My take is that the main difference is the approach to problems the coding agents have. They optimize towards presuming a fully working invocation environment, yet checking everything anyway, and then rolling back any changes and re-testing that such changes worked. Which is great for normal types of software but incredibly tedious for anything aimed at less than a fully hands free automated environment. In other words, AI is very awesome at scope creep of assigned problems and targeting the validation prompts baked into the review system.

w0m 21d ago

'ai slop' is hard for me to complain about when it's better than 95% of code prior. The issue is more people trusting it too much to go too far, and you end up with lots of extraneous bullcrap you don't need. Doing too much and too big of a change is still a problem in the agentic world.

dayvid 21d ago

It also documents itself extensively

karlshea 21d ago
But then you’ll use it to change something, and it won’t update any of the extensive documentation. So then it’s filled with lies.
- nuancebydefault 21d ago
  Sounds like human generated code and docs. TBH
- PepegaRoach 21d ago
  Or worse It'll mention the old version then the change, and the relating tickets.
- snoman 21d ago
  That actually hasn’t been my experience. For all that I dislike about LLM code, it maintains comments well.
- LeafItAlone 20d ago
  You need a better harness. LLMs are significantly better at this than humans. Add it as a step in your LLM process and your documentation will always be up to date.
tempest_ 21d ago
I fucking hate it. It encodes context it needs into doc comments and regular comments that make them in insufferable to read and so verbose because the comments contain info on other parts of the code base (that might have changed since that comment was written) but were useful to the model for its implementation. I have so far found no way to stop claude from doing this. It will sometimes do it with hand holding but the moment a task takes a slightly large amount of context its back to the word barf.
- LeafItAlone 20d ago
  This is certainly a problem with out-of-the-box usage. But easily overcome. Spend time updating your harness. This is not a hard problem to solve and there are many helpful guides out there. We basically took our documented (human) SDLC process and applied it to the relevant harness hooks. Since our SDLC docs talked about what good comments looked like and what to avoid, we basically had the direction for the tools. Opus 5 did throw us a loop and increase the verbosity and decrease usefulness of the text, but refining our documentation cleared that hurdle within about a day.

eudamoniac 21d ago

I find this shocking though not unbelievable. Could you share how you measure this accurately? I'm interested in integrating such measurements into our services as well.

bigstrat2003 21d ago
I find it unbelievable. I've seen the code LLMs write and it sucks compared to what a typical human produces. The only way an LLM is doing better than human programmers is if your human programmers were producing really terrible work.
- revetkn 21d ago
  What LLM/harness are you using that the results are so terrible?
- tarun_anand 21d ago
  Believe it or not, most programmers by definition are average. Hence, producing code better than them is not a hard feat to achieve for today's models.
- LeafItAlone 21d ago
  I personally find it unbelievable that you have access to all of the public GitHub projects available and still think the typical developer writes good code.
- nharziro 20d ago
  What exactly do you mean when you say llm generated code? Are people prompting llms for changes and features without reviewing the code or iterating on it and then comparing that to what human writes? Because if so it's not surprising that you're getting worse results. Humans also write code through iteration. You can definitely get llms to write good code by enforcing guardrails and constraints through tooling and agent.md, and iterative reviews to nudge towards what you want. The first pass will look nothing like the committed code. I don't expect the llm to one shot anything.
- 14d ago

wpietri 21d ago

Do you measure anything about complexity or brevity? I'm currently working on importing some LLM code and it's 5-10x bigger than it should be for the actual needs. One of my open questions is how much that increases over time. It could be that is a constant. But it also could be that, say, coding agents will infer false needs from the excess code/docs and elaborate further.

pmarreck 14d ago
You always must measure complexity (O-time), brevity, etc. LLM code that is written to a lower standard has not had the controls put in place.

mjevans 20d ago

I've only personally utilized it for code _review_ and always ignored the suggested result code instead to understand the source of the issue and try to arrive at a resolution I find satisfactory for my mental model of the issue and solutions I'd prefer. My take is that the main difference is the approach to problems the coding agents have. They optimize towards presuming a fully working invocation environment, yet checking everything anyway, and then rolling back any changes and re-testing that such changes worked. Which is great for normal types of software but incredibly tedious for anything aimed at less than a fully hands free automated environment. In other words, AI is very awesome at scope creep of assigned problems and targeting the validation prompts baked into the review system.