4 ms·

Respectfully, unless someone is really really bad at articulating what the quality standards are or works with a very niche stack that is definitely not the cas

Respectfully, unless someone is really really bad at articulating what the quality standards are or works with a very niche stack that is definitely not the case anymore with SOTA models

WorldMaker 5mo ago
Respectfully, the current models are all trained on everyone else's legacy code as of roughly six months ago and largely always will be. If I'm doing my job right an LLM cannot meet my personal quality bar on its own because I will always need innovation and excellence they will never see and thus cannot deliver. I also think that training these tools on my personal quality bar is more work than just writing it myself.
- dilyevsky 5mo ago
  High quality c++ code today looks exactly same as it did six months, hell, six years ago. Innovation doesn’t come from code tricks
  - WorldMaker 5mo ago
    True innovation is doing something new that has never been seen before. Even if you are doing it in a stable language that's just a stable poetry form (rhyme and meter and formatting), the real magic is the content of the poetry, and the real beauty of code is when the poetry reads well to both audiences, the compiler and the next maintainer, on multiple levels, the literal and the metaphorical. LLMs can't reach the metaphorical. LLMs don't know what true beauty is. I will grant you they have gotten great at the literal and the poetry forms. But it is the beauty that elevates things to my quality bar, and makes a difference between "legacy code" and "innovation" to me.
    - dilyevsky 5mo ago
      What you're describing is not innovation in the business sense but some kind of experimental art. Software engineering is not art as the name suggests - it's a craft, even though there's a self-expression component to it.
      - WorldMaker 5mo ago
        I think we have different definitions of what a craft is. My boundary between art and craft is a lot looser, for instance. The way I see it there is craft to the best art and there is art embedded in the best craft. I think you are right that companies seem to ignore both the art and craft of software engineering and long wish the process were a predictable industrial machine much more than an art or a craft. But I will absolutely disagree with you that just because companies desire it to be an industrial process doesn't mean it isn't my job to understand the art of software design and how that influences the crafting of software.
- thedevilslawyer 5mo ago
  LLMs are plenty innovative and generate good quality code by default, and great quality code when directed well. If you're not seeing this, at best you're probably unable to direct them or use them well. FWIW, if you don't believe the above, I challenge you to put up a quick git repo, where you are unable to get the deserved quality out, and we can quickly show you how the same quality is available via SOTA agents, within a fraction of hand-coded time.
  - larsfaye 5mo ago
    I do agree with this. I was able to get exactly what I needed even out of GPT3.5. If you put enough parameters and examples, along with a real solid system prompt and (if you can) proper temperature and topK/topP, there's no reason they can't basically function like "smart typing assistants". The issue is that its a sliding scale of ambiguity to chaos. The more ambiguity, the more the LLM fills that in. And it can be very difficult and time consuming to know when you're being ambiguous or not (you don't know what you don't know, OR, you can't track what you aren't tracking). Depending on the task, it can sometimes be just as arduous to produce enough guidance and guardrails to get the LLM to output exactly what you need that you can trust without issue or extensive review than it is to write it yourself and use the LLM just for ad-hoc generation. It's a constant balance and an endless amount of micro-decisions, honestly, but it's pretty essential to stay engaged and not YOLO with agents the way so many are. Most of my interactions with models these days are done in pseudo-code.
    - abalashov 5mo ago
      Great article! I use LLMs 99.9% in the chat box, to bounce ideas around, ask research and reference questions, and so forth. However, I have sworn off 'agents' entirely for serious code, for reasons your article captures almost totally and perfectly. I'll still use 'agents' for throwaway tasks--mostly with local models--including tasks where some sort of ad-hoc code generation is in the critical path (e.g. scraping data).
nyssos 5mo ago
You're presuming too much about what OP's quality standards are. Can SOTA models outperform the average junior engineer? Yes, obviously. Can they match the best human engineers, if those humans were given all the time and interest in the world? Equally obviously not. I use hundreds of millions of tokens a month, and LLMs have completely transformed the way I work. They're also, frankly, pretty mid programmers.
- dilyevsky 5mo ago
  most people are around mid by definition. mid is good. a lot of companies pay good money to find mid devs.
  - nyssos 5mo ago
    Yes, and? Something can be both scare and inadequate to a given task. FAANG L5s cost a pretty penny but I wouldn't trust a random one to prove a crypto library correct.
    - dilyevsky 5mo ago
      Number of people who i personally met and would entrust a crypto library to i can count on one hand. You're moving that goalpost at relativistic speeds now.

Respectfully, unless someone is really really bad at articulating what the quality standards are or works with a very niche stack that is definitely not the case anymore with SOTA models

WorldMaker 5mo ago

Respectfully, the current models are all trained on everyone else's legacy code as of roughly six months ago and largely always will be. If I'm doing my job right an LLM cannot meet my personal quality bar on its own because I will always need innovation and excellence they will never see and thus cannot deliver. I also think that training these tools on my personal quality bar is more work than just writing it myself.

dilyevsky 5mo ago
High quality c++ code today looks exactly same as it did six months, hell, six years ago. Innovation doesn’t come from code tricks
- WorldMaker 5mo ago
  True innovation is doing something new that has never been seen before. Even if you are doing it in a stable language that's just a stable poetry form (rhyme and meter and formatting), the real magic is the content of the poetry, and the real beauty of code is when the poetry reads well to both audiences, the compiler and the next maintainer, on multiple levels, the literal and the metaphorical. LLMs can't reach the metaphorical. LLMs don't know what true beauty is. I will grant you they have gotten great at the literal and the poetry forms. But it is the beauty that elevates things to my quality bar, and makes a difference between "legacy code" and "innovation" to me.
  - dilyevsky 5mo ago
    What you're describing is not innovation in the business sense but some kind of experimental art. Software engineering is not art as the name suggests - it's a craft, even though there's a self-expression component to it.
    - WorldMaker 5mo ago
      I think we have different definitions of what a craft is. My boundary between art and craft is a lot looser, for instance. The way I see it there is craft to the best art and there is art embedded in the best craft. I think you are right that companies seem to ignore both the art and craft of software engineering and long wish the process were a predictable industrial machine much more than an art or a craft. But I will absolutely disagree with you that just because companies desire it to be an industrial process doesn't mean it isn't my job to understand the art of software design and how that influences the crafting of software.
thedevilslawyer 5mo ago
LLMs are plenty innovative and generate good quality code by default, and great quality code when directed well. If you're not seeing this, at best you're probably unable to direct them or use them well. FWIW, if you don't believe the above, I challenge you to put up a quick git repo, where you are unable to get the deserved quality out, and we can quickly show you how the same quality is available via SOTA agents, within a fraction of hand-coded time.
- larsfaye 5mo ago
  I do agree with this. I was able to get exactly what I needed even out of GPT3.5. If you put enough parameters and examples, along with a real solid system prompt and (if you can) proper temperature and topK/topP, there's no reason they can't basically function like "smart typing assistants". The issue is that its a sliding scale of ambiguity to chaos. The more ambiguity, the more the LLM fills that in. And it can be very difficult and time consuming to know when you're being ambiguous or not (you don't know what you don't know, OR, you can't track what you aren't tracking). Depending on the task, it can sometimes be just as arduous to produce enough guidance and guardrails to get the LLM to output exactly what you need that you can trust without issue or extensive review than it is to write it yourself and use the LLM just for ad-hoc generation. It's a constant balance and an endless amount of micro-decisions, honestly, but it's pretty essential to stay engaged and not YOLO with agents the way so many are. Most of my interactions with models these days are done in pseudo-code.
  - abalashov 5mo ago
    Great article! I use LLMs 99.9% in the chat box, to bounce ideas around, ask research and reference questions, and so forth. However, I have sworn off 'agents' entirely for serious code, for reasons your article captures almost totally and perfectly. I'll still use 'agents' for throwaway tasks--mostly with local models--including tasks where some sort of ad-hoc code generation is in the critical path (e.g. scraping data).

nyssos 5mo ago

You're presuming too much about what OP's quality standards are. Can SOTA models outperform the average junior engineer? Yes, obviously. Can they match the best human engineers, if those humans were given all the time and interest in the world? Equally obviously not. I use hundreds of millions of tokens a month, and LLMs have completely transformed the way I work. They're also, frankly, pretty mid programmers.

dilyevsky 5mo ago
most people are around mid by definition. mid is good. a lot of companies pay good money to find mid devs.
- nyssos 5mo ago
  Yes, and? Something can be both scare and inadequate to a given task. FAANG L5s cost a pretty penny but I wouldn't trust a random one to prove a crypto library correct.
  - dilyevsky 5mo ago
    Number of people who i personally met and would entrust a crypto library to i can count on one hand. You're moving that goalpost at relativistic speeds now.