3 ms·

Not sure how Claude and CC has become the defacto best model given gpt 5.3 codex and 5.4 exist. This space moves so fast you should be testing your workflows on

by baq 7mo ago

Not sure how Claude and CC has become the defacto best model given gpt 5.3 codex and 5.4 exist. This space moves so fast you should be testing your workflows on different models at least once every quarter, prudently once a month.

m-schuetz 7mo ago
Checking different models once every quarter is exactly what made me move to claude code.
- skippyboxedhero 7mo ago
  Anthropic models haven't been far ahead for a while. Quite a few months at least. Chinese models are roughly equal at 1/6th the cost. Minimax is roughly equal to Opus. Chinese providers also haven't had the issues with uptime and variable model quality. The gap with OpenAI also isn't huge and GLM is a noticeably more compliant model (unsurprisingly given the hubristic internal culture at Anthropic around safety). CC is a better implementation and seems to be fairly economic with token usage. That is the really the only defining point and, I suspect, Anthropic are going to have a lot of trouble staying relevant with all the product issues. They were far ahead for a brief period in November/December which is driving the hype cycle that now appears to be collapsing the company. You have to test at least every month, things are moving quickly. Stepfun is releasing soon and seems to have an Opus-level model with more efficient architecture.
  - nwienert 7mo ago
    Minimax is nowhere near Opus in my tests, though for me at least oddly 4.6 felt worse than 4.5. I haven't use Minimax extensively, but I have an API driven test suite for a product and even Sonnet 4.6 outperforms it in my testing unless something changed in the last month. One example is I have a multi-stage distillation/knowledge extraction script for taking a Discord channel and answering questions. I have a hardcoded 5k message test set where I set up 20 questions myself based on analyzing it. In my harness Minimax wasn't even getting half of them right, whereas Sonnet was 100%. Granted this isn't code, but my usage on pi felt about the same.
  - epistasis 7mo ago
    > CC is a better implementation and seems to be fairly economic with token usage. That is the really the only defining point and, I suspect, Anthropic are going to have a lot of trouble staying relevant with all the product issues. What are you using to drive the Chinese models in order to evaluate this? OpenCode? Some of Claude Code's features, like remote sessions, are far more important than the underlying model for my productivity.
    - skippyboxedhero 7mo ago
      Yes, 100% agree. OpenHands has self-hosted, KiloCode and RooCode both have a cloud option. I don't think you are able to pass a session around with any of them. Codex seems to have comparable features afaik. CC tool usage is also significantly ahead imo (doesn't negate the price but it is something). I have seen issues with heavy thinking models (like Minimax) and client implementations with poor tool usage (like Cline). CC has had a period over the last six months of delivering significant value...but, of course, you can just use CC with OpenRouter.
  - SkyPuncher 7mo ago
    Claude is exceptionally better at long running agentic sessions. I keep coming back to it because I can run it as a manager for the smaller tasks.
    - baq 7mo ago
      ...and codex is at least 10x better than Claude. I don't even bother starting a new session when working on a feature, a single compaction is basically unnoticeable. You have to compact several times to start needing to remind the model about a rule or two.
    - skippyboxedhero 7mo ago
      I haven't noticed a huge difference with other models but I agree that is definitely a strength (and CC has better tooling for this). However, I do think there are practical limitations to agentic workflows because of the relatively poor output vs humans. You can generate lots of code, but most of it will be shit. Agentic workflows do have a place in well-defined, structured tasks...but I don't think that is what most people are trying to do with it.
Quothling 7mo ago
We've got access to opus 4-6, gpt 5.4, gemini pro and a few others through corprate. I have customized agents on claude, gpt and gemini since we tend to run out of tokens for x model by the end of a month. Out of all of them I've consistently been using sonnet for most tasks. Opus functions mainly as hand-off agents and reviewer". In my anecdotal experience Claude is miles ahead of the other models and has been for a long while... when it comes to writing code the way we want it. Which eksplicit, no-abstraction, no-external packages, fail fast defensive programming. I imagine you'd get different milage with different models and different coding styles. The rest of the organisation, which is not software development or IT related, mainly uses GPT models. I just wish I hadn't taught risk management about claude code so they weren't wasting MY tokens.
fakwandi_priv 7mo ago
I've been an avid fan of codex for the last few month's but finally hit the weekly limit so I've wanted to try out claude code before biting the bullet and going for the 200 dollar codex sub. Obviously in hindsight it would be unfair to Anthropic to judge them on an unstable day so I'l leave those complaints aside but I hit the session limit way too fast. I planned out 3 tasks and it couldn't finish the first plan completely, for that implementation task it has seen a grand total of 1 build log and hasn't even run any tests which already caused it to enter in the red territory of the context circle. It was even asking me during planning which endpoints the new feature should use to hook into the existing system, codex would never ask this and just simply look these up during planning and whenever it encounters ambiguity it would either ask straight away or put it as an open question. I have to wonder if they're limiting this behavior due trying to keep the context as small as possible and preventing even earlier session limits. Maybe codex's limits are not sustainable in the long run and I'm very spoiled by the limits but at this point CC(sonnet) and Codex(5.4) are simply not in the same league when comparing both 20 dollar subscriptions. I will also clearly state that the value both these tools provide at these price points are absolutely worth it, it's just that codex's value/money ratio is much better.

by baq 7mo ago

m-schuetz 7mo ago

Checking different models once every quarter is exactly what made me move to claude code.

skippyboxedhero 7mo ago
Anthropic models haven't been far ahead for a while. Quite a few months at least. Chinese models are roughly equal at 1/6th the cost. Minimax is roughly equal to Opus. Chinese providers also haven't had the issues with uptime and variable model quality. The gap with OpenAI also isn't huge and GLM is a noticeably more compliant model (unsurprisingly given the hubristic internal culture at Anthropic around safety). CC is a better implementation and seems to be fairly economic with token usage. That is the really the only defining point and, I suspect, Anthropic are going to have a lot of trouble staying relevant with all the product issues. They were far ahead for a brief period in November/December which is driving the hype cycle that now appears to be collapsing the company. You have to test at least every month, things are moving quickly. Stepfun is releasing soon and seems to have an Opus-level model with more efficient architecture.
- nwienert 7mo ago
  Minimax is nowhere near Opus in my tests, though for me at least oddly 4.6 felt worse than 4.5. I haven't use Minimax extensively, but I have an API driven test suite for a product and even Sonnet 4.6 outperforms it in my testing unless something changed in the last month. One example is I have a multi-stage distillation/knowledge extraction script for taking a Discord channel and answering questions. I have a hardcoded 5k message test set where I set up 20 questions myself based on analyzing it. In my harness Minimax wasn't even getting half of them right, whereas Sonnet was 100%. Granted this isn't code, but my usage on pi felt about the same.
- epistasis 7mo ago
  > CC is a better implementation and seems to be fairly economic with token usage. That is the really the only defining point and, I suspect, Anthropic are going to have a lot of trouble staying relevant with all the product issues. What are you using to drive the Chinese models in order to evaluate this? OpenCode? Some of Claude Code's features, like remote sessions, are far more important than the underlying model for my productivity.
  - skippyboxedhero 7mo ago
    Yes, 100% agree. OpenHands has self-hosted, KiloCode and RooCode both have a cloud option. I don't think you are able to pass a session around with any of them. Codex seems to have comparable features afaik. CC tool usage is also significantly ahead imo (doesn't negate the price but it is something). I have seen issues with heavy thinking models (like Minimax) and client implementations with poor tool usage (like Cline). CC has had a period over the last six months of delivering significant value...but, of course, you can just use CC with OpenRouter.
- SkyPuncher 7mo ago
  Claude is exceptionally better at long running agentic sessions. I keep coming back to it because I can run it as a manager for the smaller tasks.
  - baq 7mo ago
    ...and codex is at least 10x better than Claude. I don't even bother starting a new session when working on a feature, a single compaction is basically unnoticeable. You have to compact several times to start needing to remind the model about a rule or two.
  - skippyboxedhero 7mo ago
    I haven't noticed a huge difference with other models but I agree that is definitely a strength (and CC has better tooling for this). However, I do think there are practical limitations to agentic workflows because of the relatively poor output vs humans. You can generate lots of code, but most of it will be shit. Agentic workflows do have a place in well-defined, structured tasks...but I don't think that is what most people are trying to do with it.

Quothling 7mo ago

We've got access to opus 4-6, gpt 5.4, gemini pro and a few others through corprate. I have customized agents on claude, gpt and gemini since we tend to run out of tokens for x model by the end of a month. Out of all of them I've consistently been using sonnet for most tasks. Opus functions mainly as hand-off agents and reviewer". In my anecdotal experience Claude is miles ahead of the other models and has been for a long while... when it comes to writing code the way we want it. Which eksplicit, no-abstraction, no-external packages, fail fast defensive programming. I imagine you'd get different milage with different models and different coding styles. The rest of the organisation, which is not software development or IT related, mainly uses GPT models. I just wish I hadn't taught risk management about claude code so they weren't wasting MY tokens.

fakwandi_priv 7mo ago

I've been an avid fan of codex for the last few month's but finally hit the weekly limit so I've wanted to try out claude code before biting the bullet and going for the 200 dollar codex sub. Obviously in hindsight it would be unfair to Anthropic to judge them on an unstable day so I'l leave those complaints aside but I hit the session limit way too fast. I planned out 3 tasks and it couldn't finish the first plan completely, for that implementation task it has seen a grand total of 1 build log and hasn't even run any tests which already caused it to enter in the red territory of the context circle. It was even asking me during planning which endpoints the new feature should use to hook into the existing system, codex would never ask this and just simply look these up during planning and whenever it encounters ambiguity it would either ask straight away or put it as an open question. I have to wonder if they're limiting this behavior due trying to keep the context as small as possible and preventing even earlier session limits. Maybe codex's limits are not sustainable in the long run and I'm very spoiled by the limits but at this point CC(sonnet) and Codex(5.4) are simply not in the same league when comparing both 20 dollar subscriptions. I will also clearly state that the value both these tools provide at these price points are absolutely worth it, it's just that codex's value/money ratio is much better.