3 ms·

I think you can keep the vibe coding totally under control with good tests. I'm baffled how such a huge regression would not be caught. Then again, the google

I think you can keep the vibe coding totally under control with good tests. I'm baffled how such a huge regression would not be caught. Then again, the google home page was broken on FF on Android for how long?

cyanydeez 9mo ago
Whose going to write those good tests, and are they going to write them or disable them cause they dont work.
AstroBen 9mo ago
I think you underestimate how impossible of a task it is to write sufficient test coverage to keep AI in line You can assert that something you want to happen is actually happening How do you assert all the things it shouldn't be doing? They're endless. And AI WILL mess up It's enough if you're actively reviewing the code in depth.. but if you're vibe coding? Good luck
- brookst 9mo ago
  How do we check humans’ work for these unknown errors?
  - gpm 9mo ago
    An expectation of professionalism, training and written material on software design, providing incentives (like promotions) to not produce crap, etc. It's not a world where everything produced is immediately verified. If a human consistently only produced the quality of work Claude Opus 4.5 is capable of I would expect them to be fired from just about any job in short order. Yes, they'd get some stuff done, but they'd do too much damage to be worth it. Of course humans are much more expensive than LLMs to manage so this doesn't mean it can't be a useful tool... just it's not that useful a tool yet.
  - ASalazarMX 9mo ago
    Humans may be prone to err, but they don't confabulate like LLMs do. Also, the unit tests are done by people who know intimately the expected behavior of the code, which surprisingly, it's frequently the same programmer. This can be abused because the programmer is both judge and jury, but people tend to handle this paradox much better than LLMs.
  - AstroBen 9mo ago
    We have many layers to prevent them: 1. Competent humans architecting and leading the system who understand the specs, business needs, have critical thinking skills and are good at their job 2. Automated tests 3. Competent human reviewers 4. QA 5. Angry users Cutting out 1 and 3 in favor of more tests isn't gunna work
    - kaydub 9mo ago
      Ugh, I just think everyone in these threads are talking past each other. I'm personally not advocating for not having humans in the loop. I don't know of anybody using llm tools or advocating for them that are saying there shouldn't be humans in the loop. "vibe coding" seems to mean different things to different people.
gpm 9mo ago
Not my experience at all when I occasionally try making something purely coded by AI for fun. It starts off fine but the pile of sub-optimal patterns slowly builds towards an unmaintainable mess with tons of duplication of code, and state that somehow needs to be kept in sync. Tests and linters can't test that the code is actually reasonable code... Doesn't mean it's not a useful tool - if you read and think about the output you can keep it in check. But the "100% of my contributions to Claude Code were written by Claude Code" claim by the creator makes me doubt this is being done.
- gpm 9mo ago
  PS. In the 5 minutes between starting and finishing writing the parent comment https://claude.ai/settings/usage https://claude.ai/settings/usage just stopped displaying my quota usage... fun. Edit: And 3 minutes later it is back...
- AstroBen 9mo ago
  Everyone has been stressing over losing their job because of AI. I'm genuinely starting to think this will end in 5x more work needing to clean up the mess caused. Who's going to maintain all this generated code?
  - inimino 9mo ago
    Nobody is going to maintain it, the spec that generated it will be given to better systems and it will be rewritten.
    - nosianu 9mo ago
      That would be possible if you had just the spec, but after sometime most of the code will not have been generated through the original spec, but through lots of back and forth for adding features and big fixing. No way to run all that again. Not that old big non-AI software doesn't have similar maintainability issues (I keep posting this example, but I don't actually want to callthat company out specifically, the problem is widespread: https://news.ycombinator.com/item?id=18442941 https://news.ycombinator.com/item?id=18442941). That's why I'm reluctant to complain about the AI code issues too much. The problem of how software is written, on the higher level, the teams, the decisions, the rotating programmers, may be bigger than that of any particular technology or person actually writing the code. I remember a company where I looked at a contractor job, they wanted me to fix a lot of code they had received from their Eastern European programmers. They complained about them a lot in our meeting. However, after hearing them out I was convinced the problem was not the people generating the code, but the ones above them who failed to provide them with accurate specs and clear guidance, and got surprised at the very end that it did not work as expected. Similar with AI. It may be hard to disentangle what is project management, what is actually the fault of the AI. I found that you can live with pockets of suboptimal but mostly working code well enough, even adding features and fixing bugs easily, if the overall architecture is solid, and components are well isolated. That is why I don't worry too much about the complaints here about bad error checks and other small stuff. Even if it is bad, you will have lots of such issues in typical large corporate projects, even with competent people. That's because programmers keep changing, management focuses on features over anything else (usually customers, internal or external, don't pay for code reorg, only for new features). The layers above the low level code are more important in deciding if the project is and remains viable. From what the commenters say, it seems to me the problem starts much higher than the Claude code, so it is hard to say how much at fault AI generated code actually is IMHO. Whether you have inexperienced juniors or an AI producing code, you need solid project lead and architecture layers above the lines of code first of all.
tyfon 9mo ago
I've been trying opencode a bit with gemini pro (and claude via those) with a rust project, and I have a push pre-hook to cargo check the code. The amount of times I have to "yell" at the llm for adding #[allow] statements to silence the linter instead of fixing the code is crazy and when I point it out they go "Oops, you caught me, let me fix it the proper way". So the tests don't necessarily make them produce proper code.
- egeozcan 9mo ago
  I added a bunch of lines telling it to never do that in CLAUDE.md and it worked flawlessly. So I have a different experience with Claude Code, but I'm not trying to say you're holding it wrong, just adding a data point, and then, maybe I got lucky.
  - ASalazarMX 9mo ago
    I'm curious how many of those directives you'll have in that file at the end of the year.
    - egeozcan 9mo ago
      I think it won't be bigger than the giant set of rules people are supposed to read through (they never do) when onboarding. At least with AGENTS/CLAUDE.md file, you know the agent will re-read those rules on every new session.
- ASalazarMX 9mo ago
  I was doing a somewhat elaborate form/graph in Google Worksheets, had to translate a bunch of cells from English to Spanish, and said "Why not use Gemini for this easy, grunt work? They tend to output good translations". I spent 20 minutes between guiding it because it was putting the translation in the wrong cells, asking it not to convert the cells to a fancy table, and finally, convincing it that it really had access to alter the document, because at some point it denied it. I wasn't being rude, but it seems I somehow made it evasive. I had to ask it to translate in the chat, and manually copy-pasted the translations in the proper cells myself. Bonus points because it only translated like ten cells at a time, and truncated the reply with a "More cells translated" message. I can't imagine how hard it would be to handhold an LLM while working in a complex code base. I guess they are a godsend for prototypes and proofs of concept, but they can't beat a competent engineer yet. It's like that joke where a student answers that 2+2=5, and when questioned, he replies that his strength is speed, not accuracy.
swalsh 9mo ago
Be careful with that confidence. Sometimes the AI changes the test when the test tells the AI it's recent changes broke something.
- egeozcan 9mo ago
  You can see if a test is changed. If you're paranoid like I am, you can simply make the tests read-only for the user you're running the agent from.
kaydub 9mo ago
Not just tests. I run multiple agents in separate sessions. It starts with one agent, building out features or working on a task/bug fix. Once it gets some progress, I spin up another session and have it just review the code. I explicitly tell it things to look out for. I tell it to let me know about things I'm not thinking of and to make me aware of any blind spots. Whatever it reviews I send back to the agent building out features (I used to also review what the review agent told me about, but now I probably only review it like 20% of the time). I'll also have an agent session started just for writing tests, I tell it to look at the code and see if it's testable, find duplicate code, stale/dead code. And so on and so forth. Between all of that + deterministic testing it's hard for shit to end up in the code base.

cyanydeez 9mo ago

Whose going to write those good tests, and are they going to write them or disable them cause they dont work.

AstroBen 9mo ago

I think you underestimate how impossible of a task it is to write sufficient test coverage to keep AI in line You can assert that something you want to happen is actually happening How do you assert all the things it shouldn't be doing? They're endless. And AI WILL mess up It's enough if you're actively reviewing the code in depth.. but if you're vibe coding? Good luck

brookst 9mo ago
How do we check humans’ work for these unknown errors?
- gpm 9mo ago
  An expectation of professionalism, training and written material on software design, providing incentives (like promotions) to not produce crap, etc. It's not a world where everything produced is immediately verified. If a human consistently only produced the quality of work Claude Opus 4.5 is capable of I would expect them to be fired from just about any job in short order. Yes, they'd get some stuff done, but they'd do too much damage to be worth it. Of course humans are much more expensive than LLMs to manage so this doesn't mean it can't be a useful tool... just it's not that useful a tool yet.
- ASalazarMX 9mo ago
  Humans may be prone to err, but they don't confabulate like LLMs do. Also, the unit tests are done by people who know intimately the expected behavior of the code, which surprisingly, it's frequently the same programmer. This can be abused because the programmer is both judge and jury, but people tend to handle this paradox much better than LLMs.
- AstroBen 9mo ago
  We have many layers to prevent them: 1. Competent humans architecting and leading the system who understand the specs, business needs, have critical thinking skills and are good at their job 2. Automated tests 3. Competent human reviewers 4. QA 5. Angry users Cutting out 1 and 3 in favor of more tests isn't gunna work
  - kaydub 9mo ago
    Ugh, I just think everyone in these threads are talking past each other. I'm personally not advocating for not having humans in the loop. I don't know of anybody using llm tools or advocating for them that are saying there shouldn't be humans in the loop. "vibe coding" seems to mean different things to different people.

gpm 9mo ago

Not my experience at all when I occasionally try making something purely coded by AI for fun. It starts off fine but the pile of sub-optimal patterns slowly builds towards an unmaintainable mess with tons of duplication of code, and state that somehow needs to be kept in sync. Tests and linters can't test that the code is actually reasonable code... Doesn't mean it's not a useful tool - if you read and think about the output you can keep it in check. But the "100% of my contributions to Claude Code were written by Claude Code" claim by the creator makes me doubt this is being done.

gpm 9mo ago
PS. In the 5 minutes between starting and finishing writing the parent comment https://claude.ai/settings/usage https://claude.ai/settings/usage just stopped displaying my quota usage... fun. Edit: And 3 minutes later it is back...
AstroBen 9mo ago
Everyone has been stressing over losing their job because of AI. I'm genuinely starting to think this will end in 5x more work needing to clean up the mess caused. Who's going to maintain all this generated code?
- inimino 9mo ago
  Nobody is going to maintain it, the spec that generated it will be given to better systems and it will be rewritten.
  - nosianu 9mo ago
    That would be possible if you had just the spec, but after sometime most of the code will not have been generated through the original spec, but through lots of back and forth for adding features and big fixing. No way to run all that again. Not that old big non-AI software doesn't have similar maintainability issues (I keep posting this example, but I don't actually want to callthat company out specifically, the problem is widespread: https://news.ycombinator.com/item?id=18442941 https://news.ycombinator.com/item?id=18442941). That's why I'm reluctant to complain about the AI code issues too much. The problem of how software is written, on the higher level, the teams, the decisions, the rotating programmers, may be bigger than that of any particular technology or person actually writing the code. I remember a company where I looked at a contractor job, they wanted me to fix a lot of code they had received from their Eastern European programmers. They complained about them a lot in our meeting. However, after hearing them out I was convinced the problem was not the people generating the code, but the ones above them who failed to provide them with accurate specs and clear guidance, and got surprised at the very end that it did not work as expected. Similar with AI. It may be hard to disentangle what is project management, what is actually the fault of the AI. I found that you can live with pockets of suboptimal but mostly working code well enough, even adding features and fixing bugs easily, if the overall architecture is solid, and components are well isolated. That is why I don't worry too much about the complaints here about bad error checks and other small stuff. Even if it is bad, you will have lots of such issues in typical large corporate projects, even with competent people. That's because programmers keep changing, management focuses on features over anything else (usually customers, internal or external, don't pay for code reorg, only for new features). The layers above the low level code are more important in deciding if the project is and remains viable. From what the commenters say, it seems to me the problem starts much higher than the Claude code, so it is hard to say how much at fault AI generated code actually is IMHO. Whether you have inexperienced juniors or an AI producing code, you need solid project lead and architecture layers above the lines of code first of all.

tyfon 9mo ago

I've been trying opencode a bit with gemini pro (and claude via those) with a rust project, and I have a push pre-hook to cargo check the code. The amount of times I have to "yell" at the llm for adding #[allow] statements to silence the linter instead of fixing the code is crazy and when I point it out they go "Oops, you caught me, let me fix it the proper way". So the tests don't necessarily make them produce proper code.

egeozcan 9mo ago
I added a bunch of lines telling it to never do that in CLAUDE.md and it worked flawlessly. So I have a different experience with Claude Code, but I'm not trying to say you're holding it wrong, just adding a data point, and then, maybe I got lucky.
- ASalazarMX 9mo ago
  I'm curious how many of those directives you'll have in that file at the end of the year.
  - egeozcan 9mo ago
    I think it won't be bigger than the giant set of rules people are supposed to read through (they never do) when onboarding. At least with AGENTS/CLAUDE.md file, you know the agent will re-read those rules on every new session.
ASalazarMX 9mo ago
I was doing a somewhat elaborate form/graph in Google Worksheets, had to translate a bunch of cells from English to Spanish, and said "Why not use Gemini for this easy, grunt work? They tend to output good translations". I spent 20 minutes between guiding it because it was putting the translation in the wrong cells, asking it not to convert the cells to a fancy table, and finally, convincing it that it really had access to alter the document, because at some point it denied it. I wasn't being rude, but it seems I somehow made it evasive. I had to ask it to translate in the chat, and manually copy-pasted the translations in the proper cells myself. Bonus points because it only translated like ten cells at a time, and truncated the reply with a "More cells translated" message. I can't imagine how hard it would be to handhold an LLM while working in a complex code base. I guess they are a godsend for prototypes and proofs of concept, but they can't beat a competent engineer yet. It's like that joke where a student answers that 2+2=5, and when questioned, he replies that his strength is speed, not accuracy.

swalsh 9mo ago

Be careful with that confidence. Sometimes the AI changes the test when the test tells the AI it's recent changes broke something.

egeozcan 9mo ago
You can see if a test is changed. If you're paranoid like I am, you can simply make the tests read-only for the user you're running the agent from.

kaydub 9mo ago

Not just tests. I run multiple agents in separate sessions. It starts with one agent, building out features or working on a task/bug fix. Once it gets some progress, I spin up another session and have it just review the code. I explicitly tell it things to look out for. I tell it to let me know about things I'm not thinking of and to make me aware of any blind spots. Whatever it reviews I send back to the agent building out features (I used to also review what the review agent told me about, but now I probably only review it like 20% of the time). I'll also have an agent session started just for writing tests, I tell it to look at the code and see if it's testable, find duplicate code, stale/dead code. And so on and so forth. Between all of that + deterministic testing it's hard for shit to end up in the code base.