3 ms·

LLM speedups have always been predicated on not understanding the code they produce as well as if you wrote it yourself. Reviewing even well written code to und

• default-kramer 2mo ago
  > the most valuable commodity is engineers knowing WTF is going on, and that loss of understanding of your codebase is the #1 blocker to actually getting things done Perfectly said. It was true before LLMs too. As I've been reviewing more and more LLM code, I've noticed that while good developers can use LLMs to produce good code maybe 1x-5x faster (depending on circumstances), bad developers can use LLMs to produce bad code 1000x faster. And if you are collecting metrics that only capture that 1000x number, I bet you feel great about what you're doing.
• locknitpicker 2mo ago
  > LLM speedups have always been predicated on not understanding the code they produce as well as if you wrote it yourself. Reviewing even well written code to understand it in depth, has always taken longer than simply writing it yourself - let alone sloppy LLM code that you have to fix I don't think this personal opinion holds well when contrasted with reality. The truth of the matter is that in moderately large codebases, which is the norm in any production setting, at best you have a T-shape understanding of the system: in specific areas you might have somewhat deep understanding, but for the vast majority of the system you have at best a high level understanding of the software architecture. This is why software engineering frames things like design patterns, principle of least surprise,single responsibility, etc as premium design and code quality traits: they allow developers to effectively extrapolate their high level understanding of the project onto components they know nothing about. In any software development setting,you will find that the most successful and even senior engineer is that which successfully and skillfully manages this uncertainty, and is able to hit the ground running on environments they never touched. I had a colleague who once was a senior software development engineer at a FANG that mastered this art, and called it JIT onboarding. So why are you pretending this is something new or novel? I think this blend of criticism only serves to allow inexperienced developers to stand out by complaining that a very mundane aspect of working on projects as a member of a sizeable team is somehow a novel development caused by LLMs. And don't get me started on the "slop" nonsense. Have you actually browsed through code written by people?
  • vector_spaces 2mo ago
    I will say that yes, sadly, I have browsed through code written by people, and sadly also have browsed through code written by LLMs. Who do you suppose wrote the code the LLM was trained on? Divinity? I'm being tongue and cheek, but as someone who uses these things extensively in work and personal projects, I want to point out that it's new or novel that generating lots of plausible code is now vastly cheaper than it once was, so the scales are quite different. And yes, slop is an apt moniker for something that requires lots of careful prompting and configuration and guardrails (which is extremely sensitive to small changes in these initial conditions in ways that can be fairly nonobvious) to produce something that will in most cases read at least a little bit worse than if a human had written it. And it will still go off the rails sometimes! To be clear I don't really agree that typing out the code generated by LLMs is a reasonable counterweight, it largely defeats the purpose. I agree that at a certain point you have to treat parts of your code base like a black box and enforce modularity. But this is still nontrivial, especially in more specialized domains with lots of nuance
    • locknitpicker 2mo ago
      > I will say that yes, sadly, I have browsed through code written by people, and sadly also have browsed through code written by LLMs. Who do you suppose wrote the code the LLM was trained on? Divinity? Training is a red herring. The root cause is constraints, or lack thereof. If you lock a junior dev in a basement and force him to deliver the same features that these LLMs output, you will see exactly the same type of slop. The root cause is that junior devs are inexperienced and oblivious to best practices and guidelines and even the team's internal standards. They output code unconstrained by these guidelines and thus output big balls of mud. > I'm being tongue and cheek, but as someone who uses these things extensively in work and personal projects, I want to point out that it's new or novel that generating lots of plausible code is now vastly cheaper than it once was, so the scales are quite different. I agree, but we need to be mindful of what is the actual root cause. I argue it's not AI coding agente but the diverse source of these code changes, which we also see in production settings in projects managed by large teams. In teams manned by junior devs you see very much the same slop building up to a big ball of mud in a few iterations. This is nothing new. What changed is that now ai coding assistants grant everyone access to what amounts to a large team of junior devs.
• tetha 2mo ago
  > People are, in heavy LLM systems, realising that the most valuable commodity is engineers knowing WTF is going on, and that loss of understanding of your codebase is the #1 blocker to actually getting things done. Any senior engineer knows this all too well I've been pushing against this "maximum LLM driven speed" in our infrastructure as well and been working on a middle ground: We can generate a change to Ansible or terraform code in half or a third of the time, sure. But we don't use this to make three times the changes in the same time frame. We rather use the freed up time to discuss the change, the context and the affected systems with 2-3 engineers maintaining them. And yes, at times this means us three are sitting around half a day discussing and drawing diagrams about our systems. I'm just happy to work in a company understanding the value of speeding up less than we could in the feature direction, but investing this time in the direction of control and understanding of the infrastructure.
  • embedding-shape 2mo ago
    Yup, this is how I see teams who care about quality, design and architecture seems to use LLMs, not to "produce more and faster" but to retain same speed but with a lot more confidence and reliability. Hoping this will spread eventually, some companies seem to take a more... hazardous approach to the whole thing.
• kriro 2mo ago
  """People are, in heavy LLM systems, realising that the most valuable commodity is engineers knowing WTF is going on, and that loss of understanding of your codebase is the #1 blocker to actually getting things done.""" Very much this. It is also true for non-LLM code bases. I still remember my first job at a company with a bigger codebase than my typical "university project code base". It was a full fledged ERP system and just learning the lay of the land and understanding what lives where and why took quite a bit of time. Interestingly, I think LLMs can be used for this "self-onboarding" and codebase-browsing/understanding. I've experimented a bit with treating an LLM as a guide of a bigger codebase it has "ingested" and having dialogues about it. Works reasonably well, I'm curious if there are companies out there that use this approach.
• veber-alex 2mo ago
  > LLM speedups have always been predicated on not understanding the code they produce as well as if you wrote it yourself. Reviewing even well written code to understand it in depth, has always taken longer than simply writing it yourself - let alone sloppy LLM code that you have to fix I am at least 3x more productive by letting an LLM write my code and I review and understand every single line it writes. Also, Fable 5 came up with some amazing solution and designs that I would have never been able to do myself. I can say with 100% surety that my current project at work is MUCH better with an LLM compared to if I wrote it myself.