4 ms·

Sure, if you carefully review the agent's output, including tests, you can get good results. If you don't carefully review the output, you obviously have no ide

by pron 6mo ago

Sure, if you carefully review the agent's output, including tests, you can get good results. If you don't carefully review the output, you obviously have no idea if it's good enough for you. The only way to find out is that 30 changes down the line the agent won't be able to change one thing without breaking another, but by then the codebase will be too far gone to fix.

readitalready 6mo ago
I let agents break things 30 changes down the line. If something breaks, I add a check to my project validator and start over, with the validator providing instructions on what was wrong and how to fix it. It's all automatic, and now I have a guard against the exact same error in the future. Some of these checks have caught thousands of the same error, even with the latest Opus 4.7 writing the original code.
- hunterpayne 6mo ago
  You proved that testing is a good idea, not that vibe coding is a good idea.
  - lukan 6mo ago
    To be honest, I am past the point of wanting to convince people that AI is useful, if you want to refuse new tools other people find helpful, your loss. (Also I stick to the original definition of "vibe coding = not looking at generated code", "LLM assisted coding = verify generated code", I do both, depending on the task)
    - LtWorf 6mo ago
      So basically your only test is "it compiles" since you have no idea what it's actually testing.
      - deleted 6mo ago
        [deleted]
  - readitalready 6mo ago
    How do you think the tests were generated? You don't actually think I look at the code, do you?
- pron 6mo ago
  Down the line the agent is no longer able to fix one failure without causing another and the codebase is unsalvageable, but you may not have reached that point yet. Agents can help a lot when you carefully review everything they output and find all the time bombs they like hiding in your code and your tests. If not, then they're fine for codebases that don't need to last more than a year or two.
Phelinofist 6mo ago
I get what you mean, but that can also happen with code written by humans.
- LtWorf 6mo ago
  Sure, by inexperienced ones.
  - lukan 6mo ago
    Hm. Some have rather a lot of experience of making such mess themself. I mean for real, is the idea here, that all programmers are or were some kind of semi gods? Because this is not what I remember from the pre LLM time, rather this: https://xkcd.com/2030/ https://xkcd.com/2030/ I know I got into such developement hell myself. Fix a bug here, results in braking something there. Experience surely helps in avoiding it .. but even senior devs can make a mess. Otherwise there wouldn't be so many projects canceled. So sure, agents can multiply a mess in a amazingly short time, but .. that is up to the humans guiding them.
  - reassess_blind 6mo ago
    Experienced developers write bad code all the time.
  - deterministic 5mo ago
    30 years of experience writing bad code, with no effort to improve, doesn't make you any good. You need to right attitude and humility to become good. Some of the worst programmers I have ever worked with had 30+ years of experience. They basically spend all of their time fixing bug after bug in a never ending cycle because the software they produced was so fragile that it would crash if you just looked at it wrong or the temperature in the room wasn't perfect. While others with the same number of years of experience had massive systems in production for years with not a single bug reported by the happy users.
- dpoloncsak 6mo ago
  Nobody is going to argue that humans are capable of writing bad code
K0balt 6mo ago
This is essentially true. There are other ways to achieve this goal though, that don’t require exhaustive human review, better models are able to do that part as well if properly guided. The key is that yes, some of the design constraints will morph over time, necessarily, since coding is as often about discovering the problem as solving it. But design principles don’t drift. If you have a design principle that can not be adhered to, it is not a proper principle, it’s an opinion about the problem. The main thing that helps me in my workflow is to develop documentation around the code. If the code drifts from the docs, the model will notice and you can decide which was correct, the plan, the maintainer manual, or the code, or the comments in the code. Notice that there is 3 separate things written about the code, and the code itself…. Keeping all of that correct, coherent, and consistent (with a separate, invariant document that describes your design principles) keeps the model from going off the rails and gives ample opportunity to sense bad smells before they get set in stone. It’s a token fire and you need a minimum 250k context model… but I still get as much work done in an hour as I used to do in a day, and the code I coauthor is better documented, more maintainable, and more tested than any code I have ever written before.
- pron 6mo ago
  > There are other ways to achieve this goal though, that don’t require exhaustive human review, better models are able to do that part as well if properly guided. Not at this time. Even if you could somehow get their success rate to 90%, it's still far too low because the mistakes can be (and are occassionally) catastrophic. It's only when you review everything that you find mistakes that will bite you down the line. If you don't review everything, you just don't know, but the rate of bad mistakes introduced by the agents is too high to trust, no matter how much prompting and orchestration you do. Maybe future models will address that, but we're not there yet. > The main thing that helps me in my workflow is to develop documentation around the code. If the code drifts from the docs, the model will notice and you can decide which was correct, the plan, the maintainer manual, or the code, or the comments in the code. That's helpful but it doesn't solve the problem, which is that the agents are happy to introduce horrendous workarounds, and they don't tell you that the code they've written is a horrendous workaround. The docs are fine and reflect the code and the code reflects the strategy, but you just don't know that the strategy is wrong.
deterministic 5mo ago
That is correct. Using an AI to generate code and then not verify it yourself is IMHO unprofessional and should get you at a minimum a verbal warning. YOU are responsible for the code NOT the AI.

by pron 6mo ago

readitalready 6mo ago

I let agents break things 30 changes down the line. If something breaks, I add a check to my project validator and start over, with the validator providing instructions on what was wrong and how to fix it. It's all automatic, and now I have a guard against the exact same error in the future. Some of these checks have caught thousands of the same error, even with the latest Opus 4.7 writing the original code.

hunterpayne 6mo ago
You proved that testing is a good idea, not that vibe coding is a good idea.
- lukan 6mo ago
  To be honest, I am past the point of wanting to convince people that AI is useful, if you want to refuse new tools other people find helpful, your loss. (Also I stick to the original definition of "vibe coding = not looking at generated code", "LLM assisted coding = verify generated code", I do both, depending on the task)
  - LtWorf 6mo ago
    So basically your only test is "it compiles" since you have no idea what it's actually testing.
    - deleted 6mo ago
      [deleted]
- readitalready 6mo ago
  How do you think the tests were generated? You don't actually think I look at the code, do you?
pron 6mo ago
Down the line the agent is no longer able to fix one failure without causing another and the codebase is unsalvageable, but you may not have reached that point yet. Agents can help a lot when you carefully review everything they output and find all the time bombs they like hiding in your code and your tests. If not, then they're fine for codebases that don't need to last more than a year or two.

Phelinofist 6mo ago

I get what you mean, but that can also happen with code written by humans.

LtWorf 6mo ago
Sure, by inexperienced ones.
- lukan 6mo ago
  Hm. Some have rather a lot of experience of making such mess themself. I mean for real, is the idea here, that all programmers are or were some kind of semi gods? Because this is not what I remember from the pre LLM time, rather this: https://xkcd.com/2030/ https://xkcd.com/2030/ I know I got into such developement hell myself. Fix a bug here, results in braking something there. Experience surely helps in avoiding it .. but even senior devs can make a mess. Otherwise there wouldn't be so many projects canceled. So sure, agents can multiply a mess in a amazingly short time, but .. that is up to the humans guiding them.
- reassess_blind 6mo ago
  Experienced developers write bad code all the time.
- deterministic 5mo ago
  30 years of experience writing bad code, with no effort to improve, doesn't make you any good. You need to right attitude and humility to become good. Some of the worst programmers I have ever worked with had 30+ years of experience. They basically spend all of their time fixing bug after bug in a never ending cycle because the software they produced was so fragile that it would crash if you just looked at it wrong or the temperature in the room wasn't perfect. While others with the same number of years of experience had massive systems in production for years with not a single bug reported by the happy users.
dpoloncsak 6mo ago
Nobody is going to argue that humans are capable of writing bad code

K0balt 6mo ago

This is essentially true. There are other ways to achieve this goal though, that don’t require exhaustive human review, better models are able to do that part as well if properly guided. The key is that yes, some of the design constraints will morph over time, necessarily, since coding is as often about discovering the problem as solving it. But design principles don’t drift. If you have a design principle that can not be adhered to, it is not a proper principle, it’s an opinion about the problem. The main thing that helps me in my workflow is to develop documentation around the code. If the code drifts from the docs, the model will notice and you can decide which was correct, the plan, the maintainer manual, or the code, or the comments in the code. Notice that there is 3 separate things written about the code, and the code itself…. Keeping all of that correct, coherent, and consistent (with a separate, invariant document that describes your design principles) keeps the model from going off the rails and gives ample opportunity to sense bad smells before they get set in stone. It’s a token fire and you need a minimum 250k context model… but I still get as much work done in an hour as I used to do in a day, and the code I coauthor is better documented, more maintainable, and more tested than any code I have ever written before.

pron 6mo ago
> There are other ways to achieve this goal though, that don’t require exhaustive human review, better models are able to do that part as well if properly guided. Not at this time. Even if you could somehow get their success rate to 90%, it's still far too low because the mistakes can be (and are occassionally) catastrophic. It's only when you review everything that you find mistakes that will bite you down the line. If you don't review everything, you just don't know, but the rate of bad mistakes introduced by the agents is too high to trust, no matter how much prompting and orchestration you do. Maybe future models will address that, but we're not there yet. > The main thing that helps me in my workflow is to develop documentation around the code. If the code drifts from the docs, the model will notice and you can decide which was correct, the plan, the maintainer manual, or the code, or the comments in the code. That's helpful but it doesn't solve the problem, which is that the agents are happy to introduce horrendous workarounds, and they don't tell you that the code they've written is a horrendous workaround. The docs are fine and reflect the code and the code reflects the strategy, but you just don't know that the strategy is wrong.

deterministic 5mo ago

That is correct. Using an AI to generate code and then not verify it yourself is IMHO unprofessional and should get you at a minimum a verbal warning. YOU are responsible for the code NOT the AI.