4 ms·

Huh, that explains a lot about the F500, and their buzzword slogans like "culture of excellence". LLM code is still mostly absurdly bad, unless you tell it in

by _0ffh 7mo ago

Huh, that explains a lot about the F500, and their buzzword slogans like "culture of excellence". LLM code is still mostly absurdly bad, unless you tell it in painstaking detail what to do and what to avoid, and never ask it to do a bigger job at a time than a single function or very small class. Edit: I'll admit though that the detailed explanation is often still much less work than typing everything yourself. But it is a showstopper for autonomous "agentic coding".

jghn 7mo ago
> unless you tell it in painstaking detail what to do and what to avoid, and never ask it to do a bigger job at a time than a single function or very small class. This is hyperbolic, but the general sentiment is accurate enough, at least for now. I've noticed a bimodal distribution of quality when using these tools. The people who approach the LLM from the lens of a combo architect & PM, do all the leg work, set up the guard rails, define the acceptance criteria, these are the people who get great results. The people who walk up and say "sudo make me a sandwich" do not. Also the latter group complains that they don't see the point of the first group. Why would they put in all the work when they could just code? But what they don't see is that *someone* was always doing that work, it just wasn't them in the past. We're moving to a world where the mechanical part of grinding the code is not worth much, people who defined their existence as avoiding all the legwork will be left in the cold.
- _0ffh 7mo ago
  > This is hyperbolic Maybe a bit, but unfortunately sometimes not so much. I recently had an LLM write a couple of transforms on a tree in Python. The node class just had "kind" and "children" defined, nothing else. The LLM added new attributes to use in the new node kinds (Python allows to just do "foo.bar=baz" to add one). Apparently it saw a lot of code doing that during training. I corrected the code by hand and modified the Node class to raise an error when new attributes are added, with an emphatic source code comment to not add new attributes. A couple of sessions later it did it again, even adding it's own comment about circumventing the restriction! X-| Anyways, I think I mostly agree with your assessment. I might be dating myself here, but I'm not even sure what happened that made "coding" grunt work. It used to be every "coder" was an "architect" as well, and did their own legwork as needed. Maybe labor shortages changed that.
  - queenkjuul 7mo ago
    I'm with you, it's constantly doing stupid shit and ignoring instructions, and I've always been responsible for determining architecture and doing the "legwork." Unless the task is so small and well defined that it's less typing to tell the LLM (and clean up its output) then i may as well just do it myself
  - jghn 7mo ago
    > It used to be every "coder" was an "architect" as well, and did their own legwork as needed. I disagree. I remember in the days before "software engineer" became the rage that the standard job titles had a clear delineation between the people who thought the big thoughts with titles like "analyst" and the people who did the grunt work of coding who were "programmers". You'd also see roles in between like "programmer/analyst"
    - _0ffh 7mo ago
      Might be a big company thing then, but I'm not wholly convinced. There's a big gap between designing the outline of a big system and coding instructions that can be followed without having to make your own decisions. The question of how much of that gap is filled by the "design" vs "coding" levels is a spectrum.
      - jghn 7mo ago
        I think I see what you're saying and if so we're talking past each other a bit and I agree with what you're saying as well. The point I was raising is by the time an IC developer sees something, there's already been a process of curation that happens that frames the possible solutions & constrains branch points. This is different from saying that an IC makes 0 implementation decisions. The C-suite has set a direction. A product manager has defined the shape of the solution. A tech lead, architect, or whatever may have further limited scope. And any of these could just already be in effect at a global scale or on the specific problem at hand. Then the IC picks up the work and proceeds to make the last mile decisions. And it's turtles all the way up. At almost all levels on the career ladder, there are people above and/or upstream of you who are pre-curating your potential decision tree. As an analogy, I once had a fresh tech lead under me where they didn't understand this. Their team became a mess. They'd introduce raw tickets straight from the PM to their team without having thought about them at all and things ground to a halt due to decision paralysis. From their perspective that's how it was always done when they were an IC in that group. The team tackled the tickets together to work out how to accomplish their goals. It took a lot of effort to convince them that what they *didn't see* was their prior tech lead narrowing down the search space a bit, and then framing the problem in a way that that made it easier for the team to move forward.
- satvikpendem 7mo ago
  It's almost as if architecture and code quality mattered just as before and that those who don't know proper engineering principles and problem decomposition will not succeed with these new tools.
- gedy 7mo ago
  > The people who walk up and say "sudo make me a sandwich" do not. My personal beef is the human devs get "make me a sandwich", and the LLM superfans now suddenly know how to specify requirements. That's fine but don't look down your nose at people for not getting the same info. This is happening now at my company where leadership won't explain what they want, won't answer questions, but now type all day into Claude and ChatGPT. Like you could have Slacked me the same info last year knuckleheads...
  - jghn 7mo ago
    Absolutely. Merely being a member of the business class does not magically mean one has the ability to specify business requirements much less product specifications. These are *not* the people I'm talking about now having superpowers. I am picturing people who blend high level engineering and product skills, ideally with business sense.
    - godelski 7mo ago
      > Merely being a member of the business class does not magically mean one has the ability to specify business requirements much less product specifications Is this not why COBOL failed? Common Business-Oriented Language sure does look much more like natural language than a lot of other code, but it could never solve the abstraction needed to do the complex things. I don't think LLMs will ever get rid of coders. Business people can no more tell an LLM what to build than they can a team of programmers. I've long argued that the contention between the "business monkeys" and "coding monkeys" is a good one. That the former focuses on making money and the latter focuses on making a better product. The contention is good because they need each other (though I do not think the dependence is symmetric). Maybe one day AI will get there, but I don't see how it does without achieving AGI. To break down intent. To differentiate what was asked from what was intended. To understand the depth and all the context surrounding the many little parts. To understand the needs of the culture. The needs of the users. The needs of the business. This is all quite complex and it's why the number of employees typically grows quite rapidly. How do we move forward without asking how we got here? Why we got here? How optimizing for decades (or much longer) led us to these patters. Under what conditions makes these patterns (near) optimal? I've yet to see a good answer to how LLMs actually address this. If typing was the bottleneck I think we would have optimized in very different ways.
- valicord 7mo ago
  I agree with your first paragraph but not the second one. In many cases it's easier for me to directly write the code that satisfies the unwritten acceptance criteria I have in my head than to write those criteria down in English, have an LLM turn them into code, and then have to carefully review that code to see if I forgot some detail that changes everything.
  - Terretta 7mo ago
    > easier for me to directly write the code that satisfies the unwritten acceptance criteria I have in my head than to write those criteria down in English Yes, and for team or company code, "there's the problem". Those acceptance criteria are guardrails for the change that comes after, and getting those out of your head into English is more important over the long haul than your undocumented short-term solution to the criteria. Virtually all teams — because virtually all PgMs, PjMs, TLs, and Devs — miscalculate this. Easier for you, not better for team or firm. • • • FWIW, perpetuation of this problem isn't really a fault of culture or skill or education. It's largely thanks to "leadership" having no idea how to correctly incentivize what the outcome should holistically be, as they don't know enough to know what long-haul good looks like. FWIW, you can make that easier for them by having the LLM derive your acceptance criteria into English (based not only on code but on your entire conversation+iteration history) and write that up, which you can read and correct, after the countless little iterations you made since your head-spec wasn't as concrete as you imagined before you started iterating. Even if you refuse to do spec driven development, LLMs can do development-driven spec. You can review that, you must correct it, and then ... Change can come after more easily — thanks to that context.
    - valicord 7mo ago
      > Those acceptance criteria are guardrails for the change that comes after, and getting those out of your head into English is more important over the long haul than your undocumented short-term solution to the criteria. I have a lot of context about the system/codebase inside my head. 99.9% of it is not relevant to the specific task I need to do this week. The 0.1% that is relevant to this task is not relevant to other tasks that I or my teammates will need to do next week. You're suggesting that I write down this particular 0.1% in some markdown file so that LLM can write the code for me, instead of writing the code myself (which would have been faster). Chances are, nobody is going to touch that particular piece of code again for a long time. By the time they do, whatever I have written down is likely out of date, so the long term benefit of writing everything down disappears. > after the countless little iterations you made since your head-spec wasn't as concrete as you imagined before you started iterating. That's exactly the point. If I need to iterate on the spec anyway, why would I use an intermediary (LLM) instead of just writing the code myself?
theshrike79 7mo ago
If you're using words in English to "tell it in painstaking detail", you're doing it the hard way. You can provide it with a tool to do that. Agents run tools in a loop, give it good tools. We have linters, code analysers, fuzzers and everything else. Configure them correctly, tell the agent to use them (in painstaking detail) and it can't mess things up.

by _0ffh 7mo ago

jghn 7mo ago

> unless you tell it in painstaking detail what to do and what to avoid, and never ask it to do a bigger job at a time than a single function or very small class. This is hyperbolic, but the general sentiment is accurate enough, at least for now. I've noticed a bimodal distribution of quality when using these tools. The people who approach the LLM from the lens of a combo architect & PM, do all the leg work, set up the guard rails, define the acceptance criteria, these are the people who get great results. The people who walk up and say "sudo make me a sandwich" do not. Also the latter group complains that they don't see the point of the first group. Why would they put in all the work when they could just code? But what they don't see is that *someone* was always doing that work, it just wasn't them in the past. We're moving to a world where the mechanical part of grinding the code is not worth much, people who defined their existence as avoiding all the legwork will be left in the cold.

_0ffh 7mo ago
> This is hyperbolic Maybe a bit, but unfortunately sometimes not so much. I recently had an LLM write a couple of transforms on a tree in Python. The node class just had "kind" and "children" defined, nothing else. The LLM added new attributes to use in the new node kinds (Python allows to just do "foo.bar=baz" to add one). Apparently it saw a lot of code doing that during training. I corrected the code by hand and modified the Node class to raise an error when new attributes are added, with an emphatic source code comment to not add new attributes. A couple of sessions later it did it again, even adding it's own comment about circumventing the restriction! X-| Anyways, I think I mostly agree with your assessment. I might be dating myself here, but I'm not even sure what happened that made "coding" grunt work. It used to be every "coder" was an "architect" as well, and did their own legwork as needed. Maybe labor shortages changed that.
- queenkjuul 7mo ago
  I'm with you, it's constantly doing stupid shit and ignoring instructions, and I've always been responsible for determining architecture and doing the "legwork." Unless the task is so small and well defined that it's less typing to tell the LLM (and clean up its output) then i may as well just do it myself
- jghn 7mo ago
  > It used to be every "coder" was an "architect" as well, and did their own legwork as needed. I disagree. I remember in the days before "software engineer" became the rage that the standard job titles had a clear delineation between the people who thought the big thoughts with titles like "analyst" and the people who did the grunt work of coding who were "programmers". You'd also see roles in between like "programmer/analyst"
  - _0ffh 7mo ago
    Might be a big company thing then, but I'm not wholly convinced. There's a big gap between designing the outline of a big system and coding instructions that can be followed without having to make your own decisions. The question of how much of that gap is filled by the "design" vs "coding" levels is a spectrum.
    - jghn 7mo ago
      I think I see what you're saying and if so we're talking past each other a bit and I agree with what you're saying as well. The point I was raising is by the time an IC developer sees something, there's already been a process of curation that happens that frames the possible solutions & constrains branch points. This is different from saying that an IC makes 0 implementation decisions. The C-suite has set a direction. A product manager has defined the shape of the solution. A tech lead, architect, or whatever may have further limited scope. And any of these could just already be in effect at a global scale or on the specific problem at hand. Then the IC picks up the work and proceeds to make the last mile decisions. And it's turtles all the way up. At almost all levels on the career ladder, there are people above and/or upstream of you who are pre-curating your potential decision tree. As an analogy, I once had a fresh tech lead under me where they didn't understand this. Their team became a mess. They'd introduce raw tickets straight from the PM to their team without having thought about them at all and things ground to a halt due to decision paralysis. From their perspective that's how it was always done when they were an IC in that group. The team tackled the tickets together to work out how to accomplish their goals. It took a lot of effort to convince them that what they *didn't see* was their prior tech lead narrowing down the search space a bit, and then framing the problem in a way that that made it easier for the team to move forward.
satvikpendem 7mo ago
It's almost as if architecture and code quality mattered just as before and that those who don't know proper engineering principles and problem decomposition will not succeed with these new tools.
gedy 7mo ago
> The people who walk up and say "sudo make me a sandwich" do not. My personal beef is the human devs get "make me a sandwich", and the LLM superfans now suddenly know how to specify requirements. That's fine but don't look down your nose at people for not getting the same info. This is happening now at my company where leadership won't explain what they want, won't answer questions, but now type all day into Claude and ChatGPT. Like you could have Slacked me the same info last year knuckleheads...
- jghn 7mo ago
  Absolutely. Merely being a member of the business class does not magically mean one has the ability to specify business requirements much less product specifications. These are *not* the people I'm talking about now having superpowers. I am picturing people who blend high level engineering and product skills, ideally with business sense.
  - godelski 7mo ago
    > Merely being a member of the business class does not magically mean one has the ability to specify business requirements much less product specifications Is this not why COBOL failed? Common Business-Oriented Language sure does look much more like natural language than a lot of other code, but it could never solve the abstraction needed to do the complex things. I don't think LLMs will ever get rid of coders. Business people can no more tell an LLM what to build than they can a team of programmers. I've long argued that the contention between the "business monkeys" and "coding monkeys" is a good one. That the former focuses on making money and the latter focuses on making a better product. The contention is good because they need each other (though I do not think the dependence is symmetric). Maybe one day AI will get there, but I don't see how it does without achieving AGI. To break down intent. To differentiate what was asked from what was intended. To understand the depth and all the context surrounding the many little parts. To understand the needs of the culture. The needs of the users. The needs of the business. This is all quite complex and it's why the number of employees typically grows quite rapidly. How do we move forward without asking how we got here? Why we got here? How optimizing for decades (or much longer) led us to these patters. Under what conditions makes these patterns (near) optimal? I've yet to see a good answer to how LLMs actually address this. If typing was the bottleneck I think we would have optimized in very different ways.
valicord 7mo ago
I agree with your first paragraph but not the second one. In many cases it's easier for me to directly write the code that satisfies the unwritten acceptance criteria I have in my head than to write those criteria down in English, have an LLM turn them into code, and then have to carefully review that code to see if I forgot some detail that changes everything.
- Terretta 7mo ago
  > easier for me to directly write the code that satisfies the unwritten acceptance criteria I have in my head than to write those criteria down in English Yes, and for team or company code, "there's the problem". Those acceptance criteria are guardrails for the change that comes after, and getting those out of your head into English is more important over the long haul than your undocumented short-term solution to the criteria. Virtually all teams — because virtually all PgMs, PjMs, TLs, and Devs — miscalculate this. Easier for you, not better for team or firm. • • • FWIW, perpetuation of this problem isn't really a fault of culture or skill or education. It's largely thanks to "leadership" having no idea how to correctly incentivize what the outcome should holistically be, as they don't know enough to know what long-haul good looks like. FWIW, you can make that easier for them by having the LLM derive your acceptance criteria into English (based not only on code but on your entire conversation+iteration history) and write that up, which you can read and correct, after the countless little iterations you made since your head-spec wasn't as concrete as you imagined before you started iterating. Even if you refuse to do spec driven development, LLMs can do development-driven spec. You can review that, you must correct it, and then ... Change can come after more easily — thanks to that context.
  - valicord 7mo ago
    > Those acceptance criteria are guardrails for the change that comes after, and getting those out of your head into English is more important over the long haul than your undocumented short-term solution to the criteria. I have a lot of context about the system/codebase inside my head. 99.9% of it is not relevant to the specific task I need to do this week. The 0.1% that is relevant to this task is not relevant to other tasks that I or my teammates will need to do next week. You're suggesting that I write down this particular 0.1% in some markdown file so that LLM can write the code for me, instead of writing the code myself (which would have been faster). Chances are, nobody is going to touch that particular piece of code again for a long time. By the time they do, whatever I have written down is likely out of date, so the long term benefit of writing everything down disappears. > after the countless little iterations you made since your head-spec wasn't as concrete as you imagined before you started iterating. That's exactly the point. If I need to iterate on the spec anyway, why would I use an intermediary (LLM) instead of just writing the code myself?

theshrike79 7mo ago

If you're using words in English to "tell it in painstaking detail", you're doing it the hard way. You can provide it with a tool to do that. Agents run tools in a loop, give it good tools. We have linters, code analysers, fuzzers and everything else. Configure them correctly, tell the agent to use them (in painstaking detail) and it can't mess things up.