6 ms·

I feel like Claude Code is starting to fall over from being entirely written by LLMs. How do you even begin to fix precise bugs in a 1M+ LOC codebase all writte

I feel like Claude Code is starting to fall over from being entirely written by LLMs. How do you even begin to fix precise bugs in a 1M+ LOC codebase all written by AI? It seems like LLMs are great for quickly adding large new features but not great for finding and fixing edge-cases.

throwjjj 9mo ago
Wishful thinking. Lol. It’s GPu related
- rigel8 9mo ago
  compaction actually reduces GPU usage?
  - throwjjj 9mo ago
    [flagged]
eunoia 9mo ago
This is real. I’ve seen some baffling bugs in prompt based stop hook behavior. When I investigated I found the docs and implementation are completely out of sync, but the implementation doesn’t work anyway. Then I went poking on GitHub and found a vibed fix diff that changed the behavior in a totally new direction (it did not update the documentation). Seems like everyone over there is vibing and no one is rationalizing the whole.
- heliumtera 9mo ago
  >Seems like everyone over there is vibing and no one is rationalizing the whole. Claude Code creator literally brags about running 10 agents in parallel 24/7. It doesn't just seems like it, they confirmed like it is the most positive thing ever.
  - TrainedMonkey 9mo ago
    It's software engineering crack. Starting a project feels amazing, features are shipping, a complex feature in the afternoon - ezpz. But AI lacks permanence, for every feature you start over from scratch, except there is more of codebase now, but the context window is still the same. So there is drift, codebase randomizes, edge cases proliferate, and the implementation velocity slows down. Full disclosure - I am a heavy codex user and I review and understand every line of code. I manually fight spurious tests it tries to add by pointing a similar one already exists and we can get coverage with +1 LOC vs +50. It's exhausting, but personal productivity is still way up. I think the future is bright because training / fine-tuning taste, dialing down agentic frameworks, introducing adversarial agents, and increasing model context windows all seem attainable and stackable.
    - tuhgdetzhh 9mo ago
      I think that the current test suite is far too small. For the Claude Code codebase, a sensible next step would be to generate thousands of tests. Without that kind of coverage, regressions are likely, and the existing checks and review process do not appear sufficient to reliably prevent them. My request is that an entirely LLM-written feature should only be eligible for merge once all of those generated tests pass, so we have objective evidence that the change preserves existing behavior.
    - kaydub 9mo ago
      I usually have multiple agents up working on a codebase. But it's typically 1 agent building out features and 1 or 2 agents code reviewing, finding code smells, bad architecture, duplicated code, stale/dead code, etc. I'm definitely faster, but there's a lot of LLM overhead to get things done right. I think if you're just using a single agent/session you're missing out on some of the speed gains. I think a lot of the gains I get using an LLM is because I can have the multiple different agent sessions work on different projects at the same time.
  - MrDarcy 9mo ago
    I know at least one of the companies behind a coding agent we all have heard of has called in human experts to clean up their vibe coded IAC mess created in the last year.
- klodolph 9mo ago
  I’m happy to throw an LLM at our projects but we also spend time refactoring and reviewing each other’s code. When I look at the AI-generated code I can visualize the direction it’s headed in—lots of copy-pasted code with tedious manual checks for specific error conditions and little thought about how somebody reading it could be confident that the code is correct. I can’t understand how people would run agents 24/7. The agent is producing mediocre code and is bottlenecked on my review & fixes. I think I’m only marginally faster than I was without LLMs.
  - gpm 9mo ago
    > with tedious manual checks for specific error conditions And specifically: Lots of checks for impossible error conditions - often then supplying an incorrect "default value" in the case of those error conditions which would result in completely wrong behavior that would be really hard to debug if a future change ever makes those branches actually reachable.
    - klodolph 9mo ago
      I always thought that the vast majority of your codebase, the right thing to do with an error is to propagate it. Either blindly, or by wrapping it with a bit of context info. I don’t know where the LLMs are picking up this paranoid tendency to handle every single error case. It’s worth knowing about the error cases, but it requires a lot more knowledge and reasoning about the current state of the program to think about how they should be handled. Not something you can figure out just by looking at a snippet.
      - stefan_ 9mo ago
        The answer (as usual) is reinforcement learning. They gave ten idiots some code snippets, and all of them went for the "belt and braces" approach. So now thats all we get, ever. It's like the previous versions that spammed emojis everywhere despite that not being a thing whatsoever in their training data. I don't think they ever fixed that, just put a "spare us the emojis" instruction in the system prompt bandaid.
      - 9mo ago
- skerit 9mo ago
  I switched to OpenCode, away from Claude-Code, because Claude-Code is _so_ buggy.
- einpoklum 9mo ago
  > When I investigated I found the docs and implementation are completely out of sync, but the implementation doesn’t work anyway. That is not an uncommon occurrence in human-written code as well :-\
  - tobyjsullivan 9mo ago
    Someone said it best after one of those AWS outages from a fat-fingered config change: > Automation doesn't just allow you to create/fix things faster. It also allows you to break things faster. https://news.ycombinator.com/item?id=13775966 https://news.ycombinator.com/item?id=13775966 Edit: found the original comment from NikolaeVarius
- data_ders 9mo ago
  omg are you me? I had this exact same problem last week
- nrds 9mo ago
  What else could they do? If they don't vibecode Claude Code it is a bad look.
agumonkey 9mo ago
structural (team) recursion and statistical output don't mesh well together ?
egeozcan 9mo ago
I think you can keep the vibe coding totally under control with good tests. I'm baffled how such a huge regression would not be caught. Then again, the google home page was broken on FF on Android for how long?
- cyanydeez 9mo ago
  Whose going to write those good tests, and are they going to write them or disable them cause they dont work.
- AstroBen 9mo ago
  I think you underestimate how impossible of a task it is to write sufficient test coverage to keep AI in line You can assert that something you want to happen is actually happening How do you assert all the things it shouldn't be doing? They're endless. And AI WILL mess up It's enough if you're actively reviewing the code in depth.. but if you're vibe coding? Good luck
  - brookst 9mo ago
    How do we check humans’ work for these unknown errors?
    - gpm 9mo ago
      An expectation of professionalism, training and written material on software design, providing incentives (like promotions) to not produce crap, etc. It's not a world where everything produced is immediately verified. If a human consistently only produced the quality of work Claude Opus 4.5 is capable of I would expect them to be fired from just about any job in short order. Yes, they'd get some stuff done, but they'd do too much damage to be worth it. Of course humans are much more expensive than LLMs to manage so this doesn't mean it can't be a useful tool... just it's not that useful a tool yet.
    - ASalazarMX 9mo ago
      Humans may be prone to err, but they don't confabulate like LLMs do. Also, the unit tests are done by people who know intimately the expected behavior of the code, which surprisingly, it's frequently the same programmer. This can be abused because the programmer is both judge and jury, but people tend to handle this paradox much better than LLMs.
OptionOfT 9mo ago
You don't. These seems to be this idea that LLMs can do it all, but the reality is that it itself has limited amounts of memory, and thus context. And this is not tied to the LLMs. It's that to EVERYTHING we do. There are limits everywhere. And for humans the context window might be smaller, but at least we have developed methods of abstracting different context windows, by making libraries. Now, as a trade-off of trying to go super-fast, changes need to be made in response to your current prompts, and there is no time validate behavior in cases you haven't considered. And regardless of whether you have abstractions in libraries, or whether you have inlined code everywhere, you're gonna have issues. With libraries changes in behavior are going to impact code in places you don't want, but also, you don't necessarily know, as you haven't tested all paths. With inlined code everywhere you're probably going to miss instances, or code goes on to live its own life and you lose track of it. They built a skyscraper while shifting out foundational pieces. And now a part of the skyscraper is on the foundation of your backyard shed.
quietsegfault 9mo ago
What differences do you see between AI written codebases and a codebase written by engineers? Both parties create buggy code, but I can imagine the types of bugs are different. Is it just that bug fixing doesn't really scale because we don't have the ability to chomp down 1M+ LOC codebases into LLM context?
dataviz1000 9mo ago
> I feel like Claude Code is starting to fall over from being entirely written by LLMs. The degradation is palpable. I have been using vscode github copilot chat with mostly the claude opus 4.5 model. The underlying code for vscode github copilot chat has turned to shit. It will continuously make mistakes no matter what for 20 minutes. This morning I was researching Claude Code and pricing thinking about switching however this post sounds like it has turned to shit also. I don't mind spending $300-$500 a month for a tool that was a month ago accomplishing in a day what would take me 3-4 days to code. However, the days since the last update have been shit. Clearly the AI companies can't afford to run these models at profit. Do I buy puts?
swalsh 9mo ago
Just like a leveraged ETF, the returns are twice as good when things are on the up and up, but when you dig a hole it takes three times the effort to dig yourself out because now going down twice as fast, and you're also paying interest (ie, you have no clue where the bodies are burried as you bury them twice as fast).
bmurphy1976 9mo ago
You do it the same way you fix every other disaster of a code-base. You add a ton of tests and start breaking it up into modules. You then rewrite each module/component/service/etc. one at a time using good practices. That's how every project gets out of the muck. That's a big, slow, and expensive process though. Will Anthropic actually do that or will they keep throwing AI at it and hope the AI figures this approach out? We shall see...
- AstroBen 9mo ago
  With the competition biting at their heels I don't think they have time to do that. They're stuck with what they have. At least until innovation settles a little
borg16 9mo ago
vibe around and find out folks have created software by "vibe coding". It is now time to "face the music" when doing so for production grade software at scale.
root_axis 9mo ago
This explanation fits my intuition, but from an outsider's perspective, I can't say the user experience with claude code is noticeably more bug-ridden than what is typical for a rapidly scaling startup rushing crap out the door. It's vibes all the way down.
mark_l_watson 9mo ago
Despite having written a few books on LLM applications I use them sparingly for coding: to design and get started, and occasionally for debugging. I have no interest criticizing other people’s practices but I enjoy mostly writing code myself.
ankit219 9mo ago
think this particular complaint is about claude ai - the website - and not claude code. I see your point though.
charcircuit 9mo ago
LLMs have been great at finding and fixing edge cases in my experience. It's a useful tool for improving software quality.

throwjjj 9mo ago

Wishful thinking. Lol. It’s GPu related

rigel8 9mo ago
compaction actually reduces GPU usage?
- throwjjj 9mo ago
  [flagged]

eunoia 9mo ago

This is real. I’ve seen some baffling bugs in prompt based stop hook behavior. When I investigated I found the docs and implementation are completely out of sync, but the implementation doesn’t work anyway. Then I went poking on GitHub and found a vibed fix diff that changed the behavior in a totally new direction (it did not update the documentation). Seems like everyone over there is vibing and no one is rationalizing the whole.

heliumtera 9mo ago
>Seems like everyone over there is vibing and no one is rationalizing the whole. Claude Code creator literally brags about running 10 agents in parallel 24/7. It doesn't just seems like it, they confirmed like it is the most positive thing ever.
- TrainedMonkey 9mo ago
  It's software engineering crack. Starting a project feels amazing, features are shipping, a complex feature in the afternoon - ezpz. But AI lacks permanence, for every feature you start over from scratch, except there is more of codebase now, but the context window is still the same. So there is drift, codebase randomizes, edge cases proliferate, and the implementation velocity slows down. Full disclosure - I am a heavy codex user and I review and understand every line of code. I manually fight spurious tests it tries to add by pointing a similar one already exists and we can get coverage with +1 LOC vs +50. It's exhausting, but personal productivity is still way up. I think the future is bright because training / fine-tuning taste, dialing down agentic frameworks, introducing adversarial agents, and increasing model context windows all seem attainable and stackable.
  - tuhgdetzhh 9mo ago
    I think that the current test suite is far too small. For the Claude Code codebase, a sensible next step would be to generate thousands of tests. Without that kind of coverage, regressions are likely, and the existing checks and review process do not appear sufficient to reliably prevent them. My request is that an entirely LLM-written feature should only be eligible for merge once all of those generated tests pass, so we have objective evidence that the change preserves existing behavior.
  - kaydub 9mo ago
    I usually have multiple agents up working on a codebase. But it's typically 1 agent building out features and 1 or 2 agents code reviewing, finding code smells, bad architecture, duplicated code, stale/dead code, etc. I'm definitely faster, but there's a lot of LLM overhead to get things done right. I think if you're just using a single agent/session you're missing out on some of the speed gains. I think a lot of the gains I get using an LLM is because I can have the multiple different agent sessions work on different projects at the same time.
- MrDarcy 9mo ago
  I know at least one of the companies behind a coding agent we all have heard of has called in human experts to clean up their vibe coded IAC mess created in the last year.
klodolph 9mo ago
I’m happy to throw an LLM at our projects but we also spend time refactoring and reviewing each other’s code. When I look at the AI-generated code I can visualize the direction it’s headed in—lots of copy-pasted code with tedious manual checks for specific error conditions and little thought about how somebody reading it could be confident that the code is correct. I can’t understand how people would run agents 24/7. The agent is producing mediocre code and is bottlenecked on my review & fixes. I think I’m only marginally faster than I was without LLMs.
- gpm 9mo ago
  > with tedious manual checks for specific error conditions And specifically: Lots of checks for impossible error conditions - often then supplying an incorrect "default value" in the case of those error conditions which would result in completely wrong behavior that would be really hard to debug if a future change ever makes those branches actually reachable.
  - klodolph 9mo ago
    I always thought that the vast majority of your codebase, the right thing to do with an error is to propagate it. Either blindly, or by wrapping it with a bit of context info. I don’t know where the LLMs are picking up this paranoid tendency to handle every single error case. It’s worth knowing about the error cases, but it requires a lot more knowledge and reasoning about the current state of the program to think about how they should be handled. Not something you can figure out just by looking at a snippet.
    - stefan_ 9mo ago
      The answer (as usual) is reinforcement learning. They gave ten idiots some code snippets, and all of them went for the "belt and braces" approach. So now thats all we get, ever. It's like the previous versions that spammed emojis everywhere despite that not being a thing whatsoever in their training data. I don't think they ever fixed that, just put a "spare us the emojis" instruction in the system prompt bandaid.
    - 9mo ago
skerit 9mo ago
I switched to OpenCode, away from Claude-Code, because Claude-Code is _so_ buggy.
einpoklum 9mo ago
> When I investigated I found the docs and implementation are completely out of sync, but the implementation doesn’t work anyway. That is not an uncommon occurrence in human-written code as well :-\
- tobyjsullivan 9mo ago
  Someone said it best after one of those AWS outages from a fat-fingered config change: > Automation doesn't just allow you to create/fix things faster. It also allows you to break things faster. https://news.ycombinator.com/item?id=13775966 https://news.ycombinator.com/item?id=13775966 Edit: found the original comment from NikolaeVarius
data_ders 9mo ago
omg are you me? I had this exact same problem last week
nrds 9mo ago
What else could they do? If they don't vibecode Claude Code it is a bad look.

agumonkey 9mo ago

structural (team) recursion and statistical output don't mesh well together ?

egeozcan 9mo ago

I think you can keep the vibe coding totally under control with good tests. I'm baffled how such a huge regression would not be caught. Then again, the google home page was broken on FF on Android for how long?

cyanydeez 9mo ago
Whose going to write those good tests, and are they going to write them or disable them cause they dont work.
AstroBen 9mo ago
I think you underestimate how impossible of a task it is to write sufficient test coverage to keep AI in line You can assert that something you want to happen is actually happening How do you assert all the things it shouldn't be doing? They're endless. And AI WILL mess up It's enough if you're actively reviewing the code in depth.. but if you're vibe coding? Good luck
- brookst 9mo ago
  How do we check humans’ work for these unknown errors?
  - gpm 9mo ago
    An expectation of professionalism, training and written material on software design, providing incentives (like promotions) to not produce crap, etc. It's not a world where everything produced is immediately verified. If a human consistently only produced the quality of work Claude Opus 4.5 is capable of I would expect them to be fired from just about any job in short order. Yes, they'd get some stuff done, but they'd do too much damage to be worth it. Of course humans are much more expensive than LLMs to manage so this doesn't mean it can't be a useful tool... just it's not that useful a tool yet.
  - ASalazarMX 9mo ago
    Humans may be prone to err, but they don't confabulate like LLMs do. Also, the unit tests are done by people who know intimately the expected behavior of the code, which surprisingly, it's frequently the same programmer. This can be abused because the programmer is both judge and jury, but people tend to handle this paradox much better than LLMs.

OptionOfT 9mo ago

You don't. These seems to be this idea that LLMs can do it all, but the reality is that it itself has limited amounts of memory, and thus context. And this is not tied to the LLMs. It's that to EVERYTHING we do. There are limits everywhere. And for humans the context window might be smaller, but at least we have developed methods of abstracting different context windows, by making libraries. Now, as a trade-off of trying to go super-fast, changes need to be made in response to your current prompts, and there is no time validate behavior in cases you haven't considered. And regardless of whether you have abstractions in libraries, or whether you have inlined code everywhere, you're gonna have issues. With libraries changes in behavior are going to impact code in places you don't want, but also, you don't necessarily know, as you haven't tested all paths. With inlined code everywhere you're probably going to miss instances, or code goes on to live its own life and you lose track of it. They built a skyscraper while shifting out foundational pieces. And now a part of the skyscraper is on the foundation of your backyard shed.

quietsegfault 9mo ago

What differences do you see between AI written codebases and a codebase written by engineers? Both parties create buggy code, but I can imagine the types of bugs are different. Is it just that bug fixing doesn't really scale because we don't have the ability to chomp down 1M+ LOC codebases into LLM context?

dataviz1000 9mo ago

> I feel like Claude Code is starting to fall over from being entirely written by LLMs. The degradation is palpable. I have been using vscode github copilot chat with mostly the claude opus 4.5 model. The underlying code for vscode github copilot chat has turned to shit. It will continuously make mistakes no matter what for 20 minutes. This morning I was researching Claude Code and pricing thinking about switching however this post sounds like it has turned to shit also. I don't mind spending $300-$500 a month for a tool that was a month ago accomplishing in a day what would take me 3-4 days to code. However, the days since the last update have been shit. Clearly the AI companies can't afford to run these models at profit. Do I buy puts?

swalsh 9mo ago

Just like a leveraged ETF, the returns are twice as good when things are on the up and up, but when you dig a hole it takes three times the effort to dig yourself out because now going down twice as fast, and you're also paying interest (ie, you have no clue where the bodies are burried as you bury them twice as fast).

bmurphy1976 9mo ago

You do it the same way you fix every other disaster of a code-base. You add a ton of tests and start breaking it up into modules. You then rewrite each module/component/service/etc. one at a time using good practices. That's how every project gets out of the muck. That's a big, slow, and expensive process though. Will Anthropic actually do that or will they keep throwing AI at it and hope the AI figures this approach out? We shall see...

AstroBen 9mo ago
With the competition biting at their heels I don't think they have time to do that. They're stuck with what they have. At least until innovation settles a little

borg16 9mo ago

vibe around and find out folks have created software by "vibe coding". It is now time to "face the music" when doing so for production grade software at scale.

root_axis 9mo ago

This explanation fits my intuition, but from an outsider's perspective, I can't say the user experience with claude code is noticeably more bug-ridden than what is typical for a rapidly scaling startup rushing crap out the door. It's vibes all the way down.

mark_l_watson 9mo ago

Despite having written a few books on LLM applications I use them sparingly for coding: to design and get started, and occasionally for debugging. I have no interest criticizing other people’s practices but I enjoy mostly writing code myself.

ankit219 9mo ago

think this particular complaint is about claude ai - the website - and not claude code. I see your point though.

charcircuit 9mo ago

LLMs have been great at finding and fixing edge cases in my experience. It's a useful tool for improving software quality.