4 ms·

The moment you point it at a real, existing codebase - even a small one - everything falls apart. Not my experience. It excels in existing codebases too. I of

The moment you point it at a real, existing codebase - even a small one - everything falls apart. Not my experience. It excels in existing codebases too. I often ask it "I have this bug. Why?" And it almost always figures it out and fixes it. Huge code base. Codex user, not Claude Code.

bsaul 9mo ago
Not my experience too and i'm on claude code. I'd be really curious to see what when wrong in OP case. Maybe too much indication ? Could it be that it used a fast model instead of the deep ones ?
- n4r9 9mo ago
  They say explicitly what model they're using.
- deleted 9mo ago
  [deleted]
- aurareturn 9mo ago
  No, OP said he used the Max Opus 4.6. Anyways, I think one area where Codex and Claude Code falls short is that they do not test the changes they made by using the app. In this case, the LLM should ideally render the page in a real browser, and actually click on the buttons to verify. Best if the LLM test it before the changes, and then after so that it is the same. Maybe it should take a screenshot of before the change, then take a screenshot after. And match. I asked why Codex and Claude don't do this here: https://news.ycombinator.com/item?id=46792066 https://news.ycombinator.com/item?id=46792066
  - threetonesun 9mo ago
    Yeah, if you have these tools in place to validate it's changes you can quickly iterate with it to the right results. But think through how it's making UI changes and it becomes obvious quickly why it can make absolutely wrong and terrible guesses about the implementation details, it can't _see_ what it's doing, or interact with it, it's just pattern matching other implementations its seen.
    - aurareturn 9mo ago
      Yea, the next breakthrough for Codex or Claude Code would be to actually use/test the app like a real human would during the development process.
      - simonw 9mo ago
        Here's a document produced by Claude Code using my Showboat testing tool this morning to help explore SeaweedFS (a local S3 clone) - it includes trying things out with curl and getting screenshots from Chrome using my Rodney tool: https://github.com/simonw/research/blob/main/seaweedfs-testing/README.md https://github.com/simonw/research/blob/main/seaweedfs-testi...
  - throwup238 9mo ago
    See the /chrome command in Claude code.
  - mwigdahl 9mo ago
    You can easily do this, at least with Claude Code. Ask it to install and use Playwright to confirm rendering and flow. You're correct that it is a failing to not do this. When you do, it definitely helps cut down on bugs. EDIT: Sorry, just noticed you said "real browser". Haven't tried this but Playwright gets you a long way down the road.
    - aurareturn 9mo ago
      Will check it out. Looks like there is also chrome-devtools-mcp for Codex.
  - lenerdenator 9mo ago
    FWIW, I've found Playwright tests to be a decent way of getting Claude to do what you're talking about.
netdevphoenix 9mo ago
> Not my experience. It excels in existing codebases too. Why don't you prove it? 1. Find an old large codebase in codeberg (avoiding the octopus for obvious reasons) 2. Video stream the session and make the LLM convo public 3. Ask your LLM to remove jQuery from the db and submit regular commits to a public remote branch Then we will be able to judge if the evidence stands
- aurareturn 9mo ago
  I don't have to prove it. I do it every single day at work in a real production codebase that my business relies on. And I don't remove jQuery every day. Maybe the OP is right that Opus 4.6 sucks at removing jQuery. I don't know. I've never asked an AI to do it. The moment you point it at a real, existing codebase - even a small one - everything falls apart. This statement is absolutely not true based on my experience. Codex has been amazing for me at existing code bases.
  - netdevphoenix 9mo ago
    Extraordinary claims require extraordinary evidence. "Works on my machine" ain't it.
    - aurareturn 9mo ago
      Is it an extraordinary claim that Opus 4.6 or GPT 5.3 works amazing on existing code bases in my experience? That's funny. I feel like it's the opposite. Claiming that Opus 4.6 or GPT 5.3 fails as soon as you point them to an existing code base, big or small, is a much more extraordinary claim.
- simonw 9mo ago
  What are the obvious reasons?
  - netdevphoenix 9mo ago
    I thought it would be obvious: OpenAI has used repos on GitHub as training data. Would be like testing someone using a past paper publicly available. Are you planning on carrying out the experiment? Regardless of the outcome, it would be of value to developers.
uludag 9mo ago
There could be a whole spectrum of types of repositories where these tools exceed and fail. I can immagine a large repository, poorly documented, with confusing inconsistent usages/patterns, in a dynamic language, with poor tests will almost always lead to failure. I honestly think that size and age alone are sufficient to lead these tools into failure cases.
- aurareturn 9mo ago
  It could be. I mainly use LLMs with Typescript and Go, both typed languages.
netdevphoenix 9mo ago
> I often ask it "I have this bug. Why?" And it almost always figures it out and fixes it. Huge code base. Is your AI PR publicly available in github?
- aurareturn 9mo ago
  No. I don't do any open source work. I work for a private company.
  - whiplash451 9mo ago
    These two things are not mutually exclusive.

bsaul 9mo ago

Not my experience too and i'm on claude code. I'd be really curious to see what when wrong in OP case. Maybe too much indication ? Could it be that it used a fast model instead of the deep ones ?

n4r9 9mo ago
They say explicitly what model they're using.
deleted 9mo ago
[deleted]
aurareturn 9mo ago
No, OP said he used the Max Opus 4.6. Anyways, I think one area where Codex and Claude Code falls short is that they do not test the changes they made by using the app. In this case, the LLM should ideally render the page in a real browser, and actually click on the buttons to verify. Best if the LLM test it before the changes, and then after so that it is the same. Maybe it should take a screenshot of before the change, then take a screenshot after. And match. I asked why Codex and Claude don't do this here: https://news.ycombinator.com/item?id=46792066 https://news.ycombinator.com/item?id=46792066
- threetonesun 9mo ago
  Yeah, if you have these tools in place to validate it's changes you can quickly iterate with it to the right results. But think through how it's making UI changes and it becomes obvious quickly why it can make absolutely wrong and terrible guesses about the implementation details, it can't _see_ what it's doing, or interact with it, it's just pattern matching other implementations its seen.
  - aurareturn 9mo ago
    Yea, the next breakthrough for Codex or Claude Code would be to actually use/test the app like a real human would during the development process.
    - simonw 9mo ago
      Here's a document produced by Claude Code using my Showboat testing tool this morning to help explore SeaweedFS (a local S3 clone) - it includes trying things out with curl and getting screenshots from Chrome using my Rodney tool: https://github.com/simonw/research/blob/main/seaweedfs-testing/README.md https://github.com/simonw/research/blob/main/seaweedfs-testi...
- throwup238 9mo ago
  See the /chrome command in Claude code.
- mwigdahl 9mo ago
  You can easily do this, at least with Claude Code. Ask it to install and use Playwright to confirm rendering and flow. You're correct that it is a failing to not do this. When you do, it definitely helps cut down on bugs. EDIT: Sorry, just noticed you said "real browser". Haven't tried this but Playwright gets you a long way down the road.
  - aurareturn 9mo ago
    Will check it out. Looks like there is also chrome-devtools-mcp for Codex.
- lenerdenator 9mo ago
  FWIW, I've found Playwright tests to be a decent way of getting Claude to do what you're talking about.

netdevphoenix 9mo ago

> Not my experience. It excels in existing codebases too. Why don't you prove it? 1. Find an old large codebase in codeberg (avoiding the octopus for obvious reasons) 2. Video stream the session and make the LLM convo public 3. Ask your LLM to remove jQuery from the db and submit regular commits to a public remote branch Then we will be able to judge if the evidence stands

aurareturn 9mo ago
I don't have to prove it. I do it every single day at work in a real production codebase that my business relies on. And I don't remove jQuery every day. Maybe the OP is right that Opus 4.6 sucks at removing jQuery. I don't know. I've never asked an AI to do it. The moment you point it at a real, existing codebase - even a small one - everything falls apart. This statement is absolutely not true based on my experience. Codex has been amazing for me at existing code bases.
- netdevphoenix 9mo ago
  Extraordinary claims require extraordinary evidence. "Works on my machine" ain't it.
  - aurareturn 9mo ago
    Is it an extraordinary claim that Opus 4.6 or GPT 5.3 works amazing on existing code bases in my experience? That's funny. I feel like it's the opposite. Claiming that Opus 4.6 or GPT 5.3 fails as soon as you point them to an existing code base, big or small, is a much more extraordinary claim.
simonw 9mo ago
What are the obvious reasons?
- netdevphoenix 9mo ago
  I thought it would be obvious: OpenAI has used repos on GitHub as training data. Would be like testing someone using a past paper publicly available. Are you planning on carrying out the experiment? Regardless of the outcome, it would be of value to developers.

uludag 9mo ago

There could be a whole spectrum of types of repositories where these tools exceed and fail. I can immagine a large repository, poorly documented, with confusing inconsistent usages/patterns, in a dynamic language, with poor tests will almost always lead to failure. I honestly think that size and age alone are sufficient to lead these tools into failure cases.

aurareturn 9mo ago
It could be. I mainly use LLMs with Typescript and Go, both typed languages.

netdevphoenix 9mo ago

> I often ask it "I have this bug. Why?" And it almost always figures it out and fixes it. Huge code base. Is your AI PR publicly available in github?

aurareturn 9mo ago
No. I don't do any open source work. I work for a private company.
- whiplash451 9mo ago
  These two things are not mutually exclusive.