4 ms·

That doesn't quite work, and precisely for the reason I mentioned: You can definitely tell the AI to follow some strategy, but at some point the strategy will n

by pron 6mo ago

That doesn't quite work, and precisely for the reason I mentioned: You can definitely tell the AI to follow some strategy, but at some point the strategy will need to change, and the AI won't tell you that (even if you tell it to). Unless you read the code every time you won't know if the AI is following the strategy and producing good results or following it and producing bad results because the strategy has to change. This can happen even in small changes: the AI will follow the strategy even if the change proves it's wrong, and if you don't pay close attention, these mistakes pile up. So yes, you might get good results in one round, but not over time. What does work is to carefully review the AI's output, although the review needs to be more careful than review of human-written code because the agents are very good at hiding the time bombs they leave behind.

lukan 6mo ago
How do you define "bad code"? If I instruct the AI to make small modules where I can verify they work, have tests and no side effects - then it is good enough code for me. It works, is readable and can be extended - and will turn into bad code if this is not done with care.
- throwaway173738 6mo ago
  The concept of a small module is an architecture invariant. You’re making that decision, not the LLM. And you’ve made that decision because the machine is not good at certain things. You’re doing that because you can’t trust the LLM to make that decision on its own.
  - ChicagoDave 6mo ago
    I’m doing it because as a DDD adherent, I’ve been building software that way for 15 years without GenAI and now with GenAI I can do it faster. You can’t play whack-a-mole with GenAI. You have to start from well-known principles and watch everything it produces. Every module or bounded context has to have its own invariants. You can’t fully automate software engineering with GenAI. It seems the vast majority of GenAI users think they can and end up in the same place as the OP. Maybe learn Domain-Driven Design, Event Sourcing, and then try again. The results will be dramatically improved. https://devarch.ai/ https://devarch.ai/
    - dominotw 6mo ago
      [flagged]
      - gabrielhidasy 6mo ago
        That's not the point, the point is they can generate pretty good code, and do that most of the time, so ask them to generate the code, review it as you would review a more junior teammate or an opensource collaboration from an unknown source, and take advantage of their speed to test everything. You can't make a great vibe-coded thing that you couldn't make yourself, but you can get pretty much the same code you would have made in a fraction of the time.
      - ChicagoDave 6mo ago
        I have not discovered anything new. I have applied established architecture and engineering practices that I've been following for 15 years to using Claude Code. Domain-Driven Design started with Eric Evan's "blue book" in 2003 and continued with Vaughn Vernon's "red book" a few years later. Event Storming workshops and event sourcing data storage have also emerged as important tools. I am just a practitioner. Eric, Vaughn, Paul Reyner, Alberto Brandolini, and a few other architects lead. My DevArch guardrails is a toolkit to follow these practices.
        ChicagoDave 6mo ago
        I encourage skeptics to look at the Sharpee repo at https://github.com/chicagodave/sharpee https://github.com/chicagodave/sharpee and specifically docs/context. Every session summary is there going back months.
    - peterbell_nyc 6mo ago
      Love the DDD callout. I have explicit steps to review and rate delta's to the ubiquitous language and one of my architectural reviewers will often engage with me about where the bounded contexts should be and will probably the translation layers. I find the more good practices I add to my envision/scope/spec/build/test/deploy loops the happier I am with the outcomes. I will say that I am finding the actual code to be somewhat ephemeral for me - the more precise the specifications are and generally the tighter and more elegant the design is, the less the code matters as a long term artifact. I'm not at the "code is assembler" point yet - but I could see that with more, richer specs I could end up there. Of course the specs are then substantial, but declarative specs can be robust and unambigous (with sufficient read teaming review) and - like domain specific languages - reduce the accidental complexity of the syntax when compared to an implementation in a given language. There are exceptions to all of this, but it's fascinating to see how it's evolving!
- pron 6mo ago
  Sure, if you carefully review the agent's output, including tests, you can get good results. If you don't carefully review the output, you obviously have no idea if it's good enough for you. The only way to find out is that 30 changes down the line the agent won't be able to change one thing without breaking another, but by then the codebase will be too far gone to fix.
  - readitalready 6mo ago
    I let agents break things 30 changes down the line. If something breaks, I add a check to my project validator and start over, with the validator providing instructions on what was wrong and how to fix it. It's all automatic, and now I have a guard against the exact same error in the future. Some of these checks have caught thousands of the same error, even with the latest Opus 4.7 writing the original code.
    - hunterpayne 6mo ago
      You proved that testing is a good idea, not that vibe coding is a good idea.
      - lukan 6mo ago
        To be honest, I am past the point of wanting to convince people that AI is useful, if you want to refuse new tools other people find helpful, your loss. (Also I stick to the original definition of "vibe coding = not looking at generated code", "LLM assisted coding = verify generated code", I do both, depending on the task)
        LtWorf 6mo ago
        So basically your only test is "it compiles" since you have no idea what it's actually testing.
        deleted 6mo ago
        [deleted]
- WalterBright 6mo ago
  > How do you define "bad code"? The harder the code is to understand, the badder it is (and the more likely it is infested with bugs).
- pron 6mo ago
  > How do you define "bad code"? Code that will not be able to evolve for more than one-two years is terrible code. Agents write terrible code while doing a truly impressive job hiding it (including in the tests they write) unless, of course, you keep them under very close supervision.
lobocinza 6mo ago
This is reductive. What you're describing already happened in codebases without AI. LLM's just speed up thing because they are a great calculator and not a replacement for human input.
- Forgeties79 6mo ago
  > not a replacement for human input Countless individuals/companies unfortunately do believe it is a replacement still and are unlikely to change their minds.
atomicnumber3 6mo ago
I also find that every additional "constraint" you add in your context window, the dumber the agent gets, and it goes double if your constraint is unusual. To illustrate: "Do x" - for baseline, assume this generally does X fine. "Do X, don't use javascript". - even if X already didn't use javascript, this will often perform worse. It will perform even _more_ worse if X is difficult or unusual to do without javascript even if there is some perfectly serviceable way to do it. Also, despite "don't use javascript", sometimes it just still uses a little bit of javascript anyway, and usually in a spot that would actually be extremely annoying/inconvenient to them remove that js yourself (when you would've otherwise reconsidered your approach at a higher level, to either use js, or to just want something different that is easier to do without js).
- joquarky 5mo ago
  I feel like there's a limit on constraints that doesn't necessarily follow the context limits. I've assumed this is "attention heads" which I understand are an independent limitation, but I'm not smart enough to understand all the layers involved in these models so I could be wrong there. I do observe the same thing. There are a limited number of constraints you can add and once you exceed that, you'll play whack-a-mole if you insist on all of them. This is why I tend toward a more wu-wei attitude to constraints. For example: - Do I really need this constraint? - How does the agent tend to behave in this scenario it if unconstrained? Is this behavior/result an acceptable pattern for this solution? - Is the constraint implicitly followed often enough that I can trade spending tokens recovering from a deterministic test that enforces the constraint rather than preemptively state it in the prompt? If I get into the situation where I need more constraints than can fit in context/attention without the need to regularly play whack-a-mole, then I break the module down into sub-modules with fewer, more specific constraints.
aldousd666 6mo ago
you are never going to get away from reading the code every time. at least I haven't seen how you could possibly. That being said, it is considerably less work to read and check the code than it is to have to build it all, even if you know what you're doing and have done it before.
- LtWorf 6mo ago
  I find this to be false. Same way teaching your child to do something is much harder than to just do it yourself. Except the child learns.

by pron 6mo ago

lukan 6mo ago

How do you define "bad code"? If I instruct the AI to make small modules where I can verify they work, have tests and no side effects - then it is good enough code for me. It works, is readable and can be extended - and will turn into bad code if this is not done with care.

throwaway173738 6mo ago
The concept of a small module is an architecture invariant. You’re making that decision, not the LLM. And you’ve made that decision because the machine is not good at certain things. You’re doing that because you can’t trust the LLM to make that decision on its own.
- ChicagoDave 6mo ago
  I’m doing it because as a DDD adherent, I’ve been building software that way for 15 years without GenAI and now with GenAI I can do it faster. You can’t play whack-a-mole with GenAI. You have to start from well-known principles and watch everything it produces. Every module or bounded context has to have its own invariants. You can’t fully automate software engineering with GenAI. It seems the vast majority of GenAI users think they can and end up in the same place as the OP. Maybe learn Domain-Driven Design, Event Sourcing, and then try again. The results will be dramatically improved. https://devarch.ai/ https://devarch.ai/
  - dominotw 6mo ago
    [flagged]
    - gabrielhidasy 6mo ago
      That's not the point, the point is they can generate pretty good code, and do that most of the time, so ask them to generate the code, review it as you would review a more junior teammate or an opensource collaboration from an unknown source, and take advantage of their speed to test everything. You can't make a great vibe-coded thing that you couldn't make yourself, but you can get pretty much the same code you would have made in a fraction of the time.
    - ChicagoDave 6mo ago
      I have not discovered anything new. I have applied established architecture and engineering practices that I've been following for 15 years to using Claude Code. Domain-Driven Design started with Eric Evan's "blue book" in 2003 and continued with Vaughn Vernon's "red book" a few years later. Event Storming workshops and event sourcing data storage have also emerged as important tools. I am just a practitioner. Eric, Vaughn, Paul Reyner, Alberto Brandolini, and a few other architects lead. My DevArch guardrails is a toolkit to follow these practices.
      - ChicagoDave 6mo ago
        I encourage skeptics to look at the Sharpee repo at https://github.com/chicagodave/sharpee https://github.com/chicagodave/sharpee and specifically docs/context. Every session summary is there going back months.
  - peterbell_nyc 6mo ago
    Love the DDD callout. I have explicit steps to review and rate delta's to the ubiquitous language and one of my architectural reviewers will often engage with me about where the bounded contexts should be and will probably the translation layers. I find the more good practices I add to my envision/scope/spec/build/test/deploy loops the happier I am with the outcomes. I will say that I am finding the actual code to be somewhat ephemeral for me - the more precise the specifications are and generally the tighter and more elegant the design is, the less the code matters as a long term artifact. I'm not at the "code is assembler" point yet - but I could see that with more, richer specs I could end up there. Of course the specs are then substantial, but declarative specs can be robust and unambigous (with sufficient read teaming review) and - like domain specific languages - reduce the accidental complexity of the syntax when compared to an implementation in a given language. There are exceptions to all of this, but it's fascinating to see how it's evolving!
pron 6mo ago
Sure, if you carefully review the agent's output, including tests, you can get good results. If you don't carefully review the output, you obviously have no idea if it's good enough for you. The only way to find out is that 30 changes down the line the agent won't be able to change one thing without breaking another, but by then the codebase will be too far gone to fix.
- readitalready 6mo ago
  I let agents break things 30 changes down the line. If something breaks, I add a check to my project validator and start over, with the validator providing instructions on what was wrong and how to fix it. It's all automatic, and now I have a guard against the exact same error in the future. Some of these checks have caught thousands of the same error, even with the latest Opus 4.7 writing the original code.
  - hunterpayne 6mo ago
    You proved that testing is a good idea, not that vibe coding is a good idea.
    - lukan 6mo ago
      To be honest, I am past the point of wanting to convince people that AI is useful, if you want to refuse new tools other people find helpful, your loss. (Also I stick to the original definition of "vibe coding = not looking at generated code", "LLM assisted coding = verify generated code", I do both, depending on the task)
      - LtWorf 6mo ago
        So basically your only test is "it compiles" since you have no idea what it's actually testing.
        deleted 6mo ago
        [deleted]
WalterBright 6mo ago
> How do you define "bad code"? The harder the code is to understand, the badder it is (and the more likely it is infested with bugs).
pron 6mo ago
> How do you define "bad code"? Code that will not be able to evolve for more than one-two years is terrible code. Agents write terrible code while doing a truly impressive job hiding it (including in the tests they write) unless, of course, you keep them under very close supervision.

lobocinza 6mo ago

This is reductive. What you're describing already happened in codebases without AI. LLM's just speed up thing because they are a great calculator and not a replacement for human input.

Forgeties79 6mo ago
> not a replacement for human input Countless individuals/companies unfortunately do believe it is a replacement still and are unlikely to change their minds.

atomicnumber3 6mo ago

I also find that every additional "constraint" you add in your context window, the dumber the agent gets, and it goes double if your constraint is unusual. To illustrate: "Do x" - for baseline, assume this generally does X fine. "Do X, don't use javascript". - even if X already didn't use javascript, this will often perform worse. It will perform even _more_ worse if X is difficult or unusual to do without javascript even if there is some perfectly serviceable way to do it. Also, despite "don't use javascript", sometimes it just still uses a little bit of javascript anyway, and usually in a spot that would actually be extremely annoying/inconvenient to them remove that js yourself (when you would've otherwise reconsidered your approach at a higher level, to either use js, or to just want something different that is easier to do without js).

joquarky 5mo ago
I feel like there's a limit on constraints that doesn't necessarily follow the context limits. I've assumed this is "attention heads" which I understand are an independent limitation, but I'm not smart enough to understand all the layers involved in these models so I could be wrong there. I do observe the same thing. There are a limited number of constraints you can add and once you exceed that, you'll play whack-a-mole if you insist on all of them. This is why I tend toward a more wu-wei attitude to constraints. For example: - Do I really need this constraint? - How does the agent tend to behave in this scenario it if unconstrained? Is this behavior/result an acceptable pattern for this solution? - Is the constraint implicitly followed often enough that I can trade spending tokens recovering from a deterministic test that enforces the constraint rather than preemptively state it in the prompt? If I get into the situation where I need more constraints than can fit in context/attention without the need to regularly play whack-a-mole, then I break the module down into sub-modules with fewer, more specific constraints.

aldousd666 6mo ago

you are never going to get away from reading the code every time. at least I haven't seen how you could possibly. That being said, it is considerably less work to read and check the code than it is to have to build it all, even if you know what you're doing and have done it before.

LtWorf 6mo ago
I find this to be false. Same way teaching your child to do something is much harder than to just do it yourself. Except the child learns.