5 ms·

Anecdotal evidence. I have a fairly large web application with ~200k LoC. Gave the same prompt to Sonnet 4.5 (Claude Code) and GPT-5-Codex (Codex CLI). "impl

Anecdotal evidence. I have a fairly large web application with ~200k LoC. Gave the same prompt to Sonnet 4.5 (Claude Code) and GPT-5-Codex (Codex CLI). "implement a fuzzy search for conversations and reports either when selecting "Go to Conversation" or "Go to Report" and typing the title or when the user types in the title in the main input field, and none of the standard elements match, a search starts with a 2s delay" Sonnet 4.5 went really fast at ~3min. But what it built was broken and superficial. The code did not even manage to reuse already existing auth and started re-building auth server-side instead of looking how other API endpoints do it. Even re-prompting and telling it how it went wrong did not help much. No tests were written (despite the project rules requiring it). GPT-5-Codex needed MUCH longer ~20min. Changes made were much more profound, but it implemented proper error handling, lots of edge cases and wrote tests without me prompting it to do so (project rules already require it). API calls ran smoothly. The entire feature worked perfectly. My conclusion is clear: GPT-5-Codex is the clear winner, not even close. I will take the 20mins every single time, knowing the work that has been done feels like work done by a senior dev. The 3mins surprised me a lot and I was hoping to see great results in such a short period of time. But of course, a quick & dirty, buggy implementation with no tests is not what I wanted.

teekert 1y ago
[flagged]
- iagooar 1y ago
  I even added a disclaimer "anecdotal evidence". Believe me, I am not the biggest fan of Sam. I just happen to like the best tools available, have used most of the large models and always choose the one that works best - for me.
Implicated 1y ago
I'm not trying to be offensive here, feel the need to indicate that. But that prompt leads me to believe that you're going to get rather 'random' results due to leaving SO much room for interpretation. Also, in my experience, punctuation is important - particularly for pacing and grouping of logical 'parts' of a task and your prompt reads like a run on sentence. Making a lot of assumptions here - but I bet if I were in your shoes and looking to write a prompt to start a task of a similar type that my prompt would have been 5 to 20x the length of yours (depending on complexity and importance) with far more detail, including overlapping of descriptions of various tasks (ie; potentially describing the same thing more than once in different ways in context/relation to other things to establish relation/hierarchy). I'm glad you got what you needed - but these types of prompts and approaches are why I believe so many people think these models aren't useful. You get out of them what you put into them. If you give them structured and well written requirements as well as a codebase that utilizes patterns you're going to get back something relative to that. No different than a developer - if you gave a junior coder, or some team of developers the following as a feature requirement: `implement a fuzzy search for conversations and reports either when selecting "Go to Conversation" or "Go to Report" and typing the title or when the user types in the title in the main input field, and none of the standard elements match, a search starts with a 2s delay` then you can't really be mad when you don't get back exactly what you wanted. edit: To put it another way - spend a few more minutes on the initial task/prompt/description of your needs and you're likely to get back more of what you're expecting.
- pton_xd 1y ago
  > I bet if I were in your shoes and looking to write a prompt to start a task of a similar type that my prompt would have been 5 to 20x the length of yours Why would you need such extensive prompting just to get the model to not re-implement authentication logic, for example? It already has access to all of the existing code, shouldn't it just take advantage of what's already there? A 20x longer prompt doesn't sound like a satisfying solution to whatever issue is happening here.
  - Implicated 1y ago
    Well, I don't have the context myself about what's happening in this example, though I don't see anything about auth myself. And I left that window at 5-20x because, again, no real context. But unless I was already in the middle of a task and I was giving direction that there was already context for - my prompt is generally almost never _this_ short. (referring to the prompt in the top level comment) > A 20x longer prompt doesn't sound like a satisfying solution to whatever issue is happening here. It wouldn't be, given the additional context given by the author in a sibling comment to yours. But if you had specific expectations on the resulting code/functionality that 20x longer prompt is likely to save you time and energy in the back and forth adjustments you might have to make otherwise.
  - paool 1y ago
    > shouldn't it just take advantage of what's already there? It's not a good idea to have any coding agent put unnecessary amounts of lines into the context window in order to understand your code base. Performance of all llms drop drastically when the context window is filled or full. The purpose of being more specific with your prompts is that you spend a little bit more tokens up front to make the task a lot more efficient and more likely to result in success. At least that's how it is today. We're probably a breakthrough or two away from the type of vibe coding experience non-coders want. Or it may never happen, and the developers who have coding knowledge will be the only ones to fully utilize coding agents and it will only become more powerful over time.
    - 1y ago
deleted 1y ago
[deleted]
deleted 1y ago
[deleted]
robotswantdata 1y ago
I had similar experience, not good enough yet to come back for the Claude max plan. Sticking with ChatGPT pro sub and gpt5 codex on high.
- mmgeorgi 1y ago
  do you ever hit your pro quota?
  - robotswantdata 1y ago
    Never hit pro quota yet, huge repo. Have multiple projects on the go locally and in cloud. Feel like this is going to be thr $1000 plan soon
    - JofArnold 1y ago
      I'm thinking about switching to ChatGPT Pro also. Any idea what maxes it out before I need to pay via the API instead? For context I'm using about 1b tokens a month so likely similar to you by the sounds of things.
      - robotswantdata 1y ago
        On pro tier have not been able to trigger the usage cap. Pro Local tasks: Average users can send 300-1,500 messages every 5 hours with a weekly limit. Cloud tasks: Generous limits for a limited time. Best for: Developers looking to power their full workday across multiple projects.
        JofArnold 1y ago
        Thank you, that's very helpful. I think I could get close to that in some coding sessions where I'm running multiple in parallel but I suspect it's very very rare. Even with token efficient gpt5-codex my OpenAI bill is quite high so I think I will switch to Pro now.
kordlessagain 1y ago
Claude needs a lot of context to keep it on rails IMHO. And, Claude Code is a closed source piece of crap agent framework.
dbbk 1y ago
Did you use plan mode?
- iagooar 1y ago
  Yes, I did. I ran the test again, took Claude ~4mins this time. There was no error now with the auth, but the functionality was totally broken. It could not even find the most basic stuff that matches perfectly.
  - Szpadel 1y ago
    try also adding Serena mcp and make sure you did /init
fluidcruft 1y ago
I've tried codex with GPT-5 a little bit and I haven't figured out how to get it to not be annoying. codex just constantly tries to gaslight and argue with me. For example, I was debugging an OpenGL render pipeline that went black and codex insisted it must be because I was ssh'd into a headless server. It really makes me appreciate the Claude "You're absolutely right!"s. Anyway as you can tell, I haven't cracked working with codex. But at the moment it just messes everything up and ways I've learned to work with claude don't seem to translate.
alecco 1y ago
Try adding ultrathink to the prompt, hit enter, and play this song in the background: [1] https://www.reddit.com/r/ClaudeAI/comments/1mgwohq/ultrathink/ https://www.reddit.com/r/ClaudeAI/comments/1mgwohq/ultrathin...
- unshavedyak 1y ago
  Sidenote, Claude Code now has a thinking toggle. Maybe it did already, but that's new to me. I see better results by just leaving that toggle on.
Szpadel 1y ago
that's kind of expected for me, but codex feels more like vibe coding tool and Claude code more like ai assisted development. And I actually like Claude more because of that. codex will indeed work more independently but you will have hard time when it isn't what you want. It will use python script just to do simple edits in files ( niesearch and replace for unique code snippets in small files) when it's wrong good look convincing it (it already have some outdated info like on latest docker image releases and convincing it that Debian base changed was challenging) it uses context more effectively but it will lack explanation why it is doing what it is doing, asking it to explain will just cause it to do something else without any word. and of course lack of proper permissions for running commands. sandbox is cool but I do not want it to be able to commit, I want it to just edit files or I want to have some more control over what it does. you can run codex as mcp server, I prefer adding it to Claude and ask to do cooperative plan, codex will do great analysis and plan and I can comfortable work with Claude on the code that matches my style
epolanski 1y ago
There's few issues with posts like yours: 1. Different LLMs require different prompts and information 2. They ignore LLMs non determinism, you should run the experiment several times
- iagooar 1y ago
  Oh and I agree so much. I just shared a quick first observation in a real-world testing scenario (BTW re-ran Sonnet 4.5 with the same prompt, not much changed). I just keep seeing how LLM providers keep optimizing for benchmarks, but then I cannot reproduce their results in my projects. I will keep trying, because Claude 4 generally is a very strong line of models. Anthropic has been on the AI coding throne for months before OpenAI with GPT-5 and Codex CLI (and now GPT-5-Codex) has dethroned them. And sure I do want to keep them competing to make each other even better.
- mexicocitinluez 1y ago
  What would be the difference in prompts/info for Claude vs ChatGpt? Is this just based on anecdotal stuff or is there actually something I can refer to when writing prompts? I mostly use Claude, but don't really pay much attention to the exact wording of the prompts
artursapek 1y ago
I think Codex working for 20 mins uninterrupted is actually a strength. It’s not “slow” as critics sometimes say - it’s thorough and autonomous. I can actually walk away and get something else done around the house while it does my work for me.
- streetmeat 1y ago
  I swear cc in June/July used to spend a lot more time on tasks and felt more thorough like codex does now. Hard to remember much past the last week in this world though.
  - artursapek 1y ago
    Yes I agree. I think Anthropic nerfed it considerably.
paul_manias 1y ago
Same experience here. In the last week I've successfully managed to build a complete C++20 XPath 1.0 parser with Codex, and am now onto supporting XPath 2.0. Codex has been nailing it time and again - the only caveat is that I have to use their cloud version as local execution is buggy. Sonnet on the other hand gets tripped up constantly due to the complexity. I'm not seeing the improvement I was hoping for with 4.5, and it's just given up on attempting to implement support for date-time functionality. Codex has nailed the same task, yet Anthropic claim OpenAI have the inferior product? I'm pretty sad about this as I'm gunning for Anthropic and would have loved to see them at least match Codex with this new release. If OpenAI stays on this course then Claude Code is toast without an important breakthrough. It doesn't help that they're also the more expensive product and have service quality issues pushing people to quit the platform.
dinobones 1y ago
Are you paying for ChatGPT pro? Does that include codex CLI usage? Claude Code on max plan is why I use Sonnet/Opus, but if ChatGPT pro includes codex usage I may switch.
- danmaz74 1y ago
  You can use codex CLI on a measly Plus plan
j45 1y ago
Curious if Opus would have done it differently than GPT-5-Codex. Also, trying GPT-5-Codex
devoutsalsa 1y ago
My first thought was I bet I could get Sonnet to fix it faster because I got something back in 3 minutes instead of 20 minutes. You can prompt a lot of changes with a faster model. I'm new to Claude Code, so generally speaking I have no idea if I'm making sense or not.
jpfromlondon 1y ago
Interesting, in my experience Claude usually does okay with the first pass, often gets the best visual/ui output, but cannot improve beyond that even with repeated prompts and is terrible at optimising, GPT almost the opposite.
bastawhiz 1y ago
I must be using Codex wrong. I evaluated it with a task to do a pretty simple, mechanical string replacement across many files (moving from a prop spread in JSX to standard props, but only when the value being spread is a subscript of an object). It simply couldn't do it, and it wasn't even close. It was getting the syntax wrong, trying to fix it by deleting the code, then moving on to other files. Sonnet 4.1 wasn't perfect, but I was able to point out its errors and it fixed them and avoided doing it again. I will say, Claude does seem to need a verbose prompt. Often I'll write my prompts as tasks in Notion and have it pull then via MCP (which is nice, because it tracks and documents its work in the process). But once you've given it a few paragraphs about the why and how, my experience is that it's pretty self sufficient. Granted, I'm using Cursor and not CC; I don't know if that makes much of a difference.
- lkey 1y ago
  Codex cannot fail, it contains multitudes beyond your imagining. Nay, it can only be failed. Continue internalizing that the problem is you, not the tool. Perhaps a small infusion of one trillion USD would unlock it and your potential? Sincerely, TPG Dexoc, the living human person
hackernewds 1y ago
Anecdotes are pointless in this domain.
bobbylarrybobby 1y ago
It's also my experience that Claude loves to reimplement the wheel instead of reading code to look for an existing implementation of what it wants to do.
- Huppie 1y ago
  I've been working with Opus 4 on ultrathink quite a bit recently and did some quick tests with Sonnet 4.5, I'm fairly impressed, especially with its speed but I did feel it was a lot less strict with my rules, existing patterns, etc. compared to Opus 4. Maybe it's better with a better CLAUDE.md structure? I don't use those a lot, just telling Opus to think got 'good enough' results I guess. Not sure. I hope there's an Opus 4.5 coming out soon too. In the meantime I'll see if I can get to do better with some extra prompting or I'll go back to Opus of if I don't need the speedier responses.
theshrike79 1y ago
Codex is like an outsourced contractor. You give them the spec and eventually you get a full feature to review. Claude is more like a pair programmer, you iterate in short cycles and adjust.

teekert 1y ago

[flagged]

iagooar 1y ago
I even added a disclaimer "anecdotal evidence". Believe me, I am not the biggest fan of Sam. I just happen to like the best tools available, have used most of the large models and always choose the one that works best - for me.

Implicated 1y ago

I'm not trying to be offensive here, feel the need to indicate that. But that prompt leads me to believe that you're going to get rather 'random' results due to leaving SO much room for interpretation. Also, in my experience, punctuation is important - particularly for pacing and grouping of logical 'parts' of a task and your prompt reads like a run on sentence. Making a lot of assumptions here - but I bet if I were in your shoes and looking to write a prompt to start a task of a similar type that my prompt would have been 5 to 20x the length of yours (depending on complexity and importance) with far more detail, including overlapping of descriptions of various tasks (ie; potentially describing the same thing more than once in different ways in context/relation to other things to establish relation/hierarchy). I'm glad you got what you needed - but these types of prompts and approaches are why I believe so many people think these models aren't useful. You get out of them what you put into them. If you give them structured and well written requirements as well as a codebase that utilizes patterns you're going to get back something relative to that. No different than a developer - if you gave a junior coder, or some team of developers the following as a feature requirement: `implement a fuzzy search for conversations and reports either when selecting "Go to Conversation" or "Go to Report" and typing the title or when the user types in the title in the main input field, and none of the standard elements match, a search starts with a 2s delay` then you can't really be mad when you don't get back exactly what you wanted. edit: To put it another way - spend a few more minutes on the initial task/prompt/description of your needs and you're likely to get back more of what you're expecting.

pton_xd 1y ago
> I bet if I were in your shoes and looking to write a prompt to start a task of a similar type that my prompt would have been 5 to 20x the length of yours Why would you need such extensive prompting just to get the model to not re-implement authentication logic, for example? It already has access to all of the existing code, shouldn't it just take advantage of what's already there? A 20x longer prompt doesn't sound like a satisfying solution to whatever issue is happening here.
- Implicated 1y ago
  Well, I don't have the context myself about what's happening in this example, though I don't see anything about auth myself. And I left that window at 5-20x because, again, no real context. But unless I was already in the middle of a task and I was giving direction that there was already context for - my prompt is generally almost never _this_ short. (referring to the prompt in the top level comment) > A 20x longer prompt doesn't sound like a satisfying solution to whatever issue is happening here. It wouldn't be, given the additional context given by the author in a sibling comment to yours. But if you had specific expectations on the resulting code/functionality that 20x longer prompt is likely to save you time and energy in the back and forth adjustments you might have to make otherwise.
- paool 1y ago
  > shouldn't it just take advantage of what's already there? It's not a good idea to have any coding agent put unnecessary amounts of lines into the context window in order to understand your code base. Performance of all llms drop drastically when the context window is filled or full. The purpose of being more specific with your prompts is that you spend a little bit more tokens up front to make the task a lot more efficient and more likely to result in success. At least that's how it is today. We're probably a breakthrough or two away from the type of vibe coding experience non-coders want. Or it may never happen, and the developers who have coding knowledge will be the only ones to fully utilize coding agents and it will only become more powerful over time.
  - 1y ago

deleted 1y ago

[deleted]

deleted 1y ago

[deleted]

robotswantdata 1y ago

I had similar experience, not good enough yet to come back for the Claude max plan. Sticking with ChatGPT pro sub and gpt5 codex on high.

mmgeorgi 1y ago
do you ever hit your pro quota?
- robotswantdata 1y ago
  Never hit pro quota yet, huge repo. Have multiple projects on the go locally and in cloud. Feel like this is going to be thr $1000 plan soon
  - JofArnold 1y ago
    I'm thinking about switching to ChatGPT Pro also. Any idea what maxes it out before I need to pay via the API instead? For context I'm using about 1b tokens a month so likely similar to you by the sounds of things.
    - robotswantdata 1y ago
      On pro tier have not been able to trigger the usage cap. Pro Local tasks: Average users can send 300-1,500 messages every 5 hours with a weekly limit. Cloud tasks: Generous limits for a limited time. Best for: Developers looking to power their full workday across multiple projects.
      - JofArnold 1y ago
        Thank you, that's very helpful. I think I could get close to that in some coding sessions where I'm running multiple in parallel but I suspect it's very very rare. Even with token efficient gpt5-codex my OpenAI bill is quite high so I think I will switch to Pro now.

kordlessagain 1y ago

Claude needs a lot of context to keep it on rails IMHO. And, Claude Code is a closed source piece of crap agent framework.

dbbk 1y ago

Did you use plan mode?

iagooar 1y ago
Yes, I did. I ran the test again, took Claude ~4mins this time. There was no error now with the auth, but the functionality was totally broken. It could not even find the most basic stuff that matches perfectly.
- Szpadel 1y ago
  try also adding Serena mcp and make sure you did /init

fluidcruft 1y ago

I've tried codex with GPT-5 a little bit and I haven't figured out how to get it to not be annoying. codex just constantly tries to gaslight and argue with me. For example, I was debugging an OpenGL render pipeline that went black and codex insisted it must be because I was ssh'd into a headless server. It really makes me appreciate the Claude "You're absolutely right!"s. Anyway as you can tell, I haven't cracked working with codex. But at the moment it just messes everything up and ways I've learned to work with claude don't seem to translate.

alecco 1y ago

Try adding ultrathink to the prompt, hit enter, and play this song in the background: [1] https://www.reddit.com/r/ClaudeAI/comments/1mgwohq/ultrathink/ https://www.reddit.com/r/ClaudeAI/comments/1mgwohq/ultrathin...

unshavedyak 1y ago
Sidenote, Claude Code now has a thinking toggle. Maybe it did already, but that's new to me. I see better results by just leaving that toggle on.

Szpadel 1y ago

that's kind of expected for me, but codex feels more like vibe coding tool and Claude code more like ai assisted development. And I actually like Claude more because of that. codex will indeed work more independently but you will have hard time when it isn't what you want. It will use python script just to do simple edits in files ( niesearch and replace for unique code snippets in small files) when it's wrong good look convincing it (it already have some outdated info like on latest docker image releases and convincing it that Debian base changed was challenging) it uses context more effectively but it will lack explanation why it is doing what it is doing, asking it to explain will just cause it to do something else without any word. and of course lack of proper permissions for running commands. sandbox is cool but I do not want it to be able to commit, I want it to just edit files or I want to have some more control over what it does. you can run codex as mcp server, I prefer adding it to Claude and ask to do cooperative plan, codex will do great analysis and plan and I can comfortable work with Claude on the code that matches my style

epolanski 1y ago

There's few issues with posts like yours: 1. Different LLMs require different prompts and information 2. They ignore LLMs non determinism, you should run the experiment several times

iagooar 1y ago
Oh and I agree so much. I just shared a quick first observation in a real-world testing scenario (BTW re-ran Sonnet 4.5 with the same prompt, not much changed). I just keep seeing how LLM providers keep optimizing for benchmarks, but then I cannot reproduce their results in my projects. I will keep trying, because Claude 4 generally is a very strong line of models. Anthropic has been on the AI coding throne for months before OpenAI with GPT-5 and Codex CLI (and now GPT-5-Codex) has dethroned them. And sure I do want to keep them competing to make each other even better.
mexicocitinluez 1y ago
What would be the difference in prompts/info for Claude vs ChatGpt? Is this just based on anecdotal stuff or is there actually something I can refer to when writing prompts? I mostly use Claude, but don't really pay much attention to the exact wording of the prompts

artursapek 1y ago

I think Codex working for 20 mins uninterrupted is actually a strength. It’s not “slow” as critics sometimes say - it’s thorough and autonomous. I can actually walk away and get something else done around the house while it does my work for me.

streetmeat 1y ago
I swear cc in June/July used to spend a lot more time on tasks and felt more thorough like codex does now. Hard to remember much past the last week in this world though.
- artursapek 1y ago
  Yes I agree. I think Anthropic nerfed it considerably.

paul_manias 1y ago

Same experience here. In the last week I've successfully managed to build a complete C++20 XPath 1.0 parser with Codex, and am now onto supporting XPath 2.0. Codex has been nailing it time and again - the only caveat is that I have to use their cloud version as local execution is buggy. Sonnet on the other hand gets tripped up constantly due to the complexity. I'm not seeing the improvement I was hoping for with 4.5, and it's just given up on attempting to implement support for date-time functionality. Codex has nailed the same task, yet Anthropic claim OpenAI have the inferior product? I'm pretty sad about this as I'm gunning for Anthropic and would have loved to see them at least match Codex with this new release. If OpenAI stays on this course then Claude Code is toast without an important breakthrough. It doesn't help that they're also the more expensive product and have service quality issues pushing people to quit the platform.

dinobones 1y ago

Are you paying for ChatGPT pro? Does that include codex CLI usage? Claude Code on max plan is why I use Sonnet/Opus, but if ChatGPT pro includes codex usage I may switch.

danmaz74 1y ago
You can use codex CLI on a measly Plus plan

j45 1y ago

Curious if Opus would have done it differently than GPT-5-Codex. Also, trying GPT-5-Codex

devoutsalsa 1y ago

My first thought was I bet I could get Sonnet to fix it faster because I got something back in 3 minutes instead of 20 minutes. You can prompt a lot of changes with a faster model. I'm new to Claude Code, so generally speaking I have no idea if I'm making sense or not.

jpfromlondon 1y ago

Interesting, in my experience Claude usually does okay with the first pass, often gets the best visual/ui output, but cannot improve beyond that even with repeated prompts and is terrible at optimising, GPT almost the opposite.

bastawhiz 1y ago

I must be using Codex wrong. I evaluated it with a task to do a pretty simple, mechanical string replacement across many files (moving from a prop spread in JSX to standard props, but only when the value being spread is a subscript of an object). It simply couldn't do it, and it wasn't even close. It was getting the syntax wrong, trying to fix it by deleting the code, then moving on to other files. Sonnet 4.1 wasn't perfect, but I was able to point out its errors and it fixed them and avoided doing it again. I will say, Claude does seem to need a verbose prompt. Often I'll write my prompts as tasks in Notion and have it pull then via MCP (which is nice, because it tracks and documents its work in the process). But once you've given it a few paragraphs about the why and how, my experience is that it's pretty self sufficient. Granted, I'm using Cursor and not CC; I don't know if that makes much of a difference.

lkey 1y ago
Codex cannot fail, it contains multitudes beyond your imagining. Nay, it can only be failed. Continue internalizing that the problem is you, not the tool. Perhaps a small infusion of one trillion USD would unlock it and your potential? Sincerely, TPG Dexoc, the living human person

hackernewds 1y ago

Anecdotes are pointless in this domain.

bobbylarrybobby 1y ago

It's also my experience that Claude loves to reimplement the wheel instead of reading code to look for an existing implementation of what it wants to do.

Huppie 1y ago
I've been working with Opus 4 on ultrathink quite a bit recently and did some quick tests with Sonnet 4.5, I'm fairly impressed, especially with its speed but I did feel it was a lot less strict with my rules, existing patterns, etc. compared to Opus 4. Maybe it's better with a better CLAUDE.md structure? I don't use those a lot, just telling Opus to think got 'good enough' results I guess. Not sure. I hope there's an Opus 4.5 coming out soon too. In the meantime I'll see if I can get to do better with some extra prompting or I'll go back to Opus of if I don't need the speedier responses.

theshrike79 1y ago

Codex is like an outsourced contractor. You give them the spec and eventually you get a full feature to review. Claude is more like a pair programmer, you iterate in short cycles and adjust.