3 ms·

Claude wins by a large margin * Claude Opus 4.6 : 0.71 * Claude Opus 4.5 : 0.51 * KIMI-K2.5 : 0.37 * GLM-5 : 0.36 * GPT-5.2 : 0.23 Note: later GPT version

Claude wins by a large margin * Claude Opus 4.6 : 0.71 * Claude Opus 4.5 : 0.51 * KIMI-K2.5 : 0.37 * GLM-5 : 0.36 * GPT-5.2 : 0.23 Note: later GPT versions seem to be only available within openAi's proprietary codex cli, so can't be tested - and if tested via the codex cli "harness" it wouldn't be a pure model-to-model comparison any more. --- Of course, the interesting follow-up question is: How well perform these models with added agent tooling ("harness") ? Maybe someone has tokens to burn and can run a matrix of agent tools over the top models and provide the results?

mike_hearn 7mo ago
It's the other way around - Claude Code is the proprietary one. Codex CLI is open source: https://github.com/openai/codex https://github.com/openai/codex You can definitely access the latest models via the API. That's how Codex CLI works.
- pizlonator 7mo ago
  gpt-5.3 was not accessible via API, at least for me But it was in codex
  - mike_hearn 7mo ago
    Ah ha, I know what you're hitting. 5.3-codex is only available via the Responses API, not the Completions API. Two different APIs for model access. If you were using Completions you have to port to Responses. It's not that hard. I did this for my own agent the other week. I think it might be like that for all their new models from now on. Responses is a much more powerful API. It's more like a front to ChatGPT than the underlying models.
    - pizlonator 7mo ago
      I am using responses API (these days I only use responses API when talking to OpenAI) I'll try again, maybe I was doing something else wrong, or something changed
      - mike_hearn 7mo ago
        OK, in that case I'm not sure. But Codex can't hide anything. I don't think there are special codes compiled into the binaries or anything like that.
jsemrau 7mo ago
I reached the same conclusion. I tried using both for my personal investment ambient using agent-pair programming to build and agentic intelligence layer for stocks and the difference between the 2 models if astounding.
andai 7mo ago
>if tested via the codex cli "harness" it wouldn't be a pure model-to-model comparison any more. Well that's already not a very fair comparison, we've known for years (one of the early-ish LLM papers, maybe someone knows which one) that prompting makes an enormous difference on agent performance, and most strikingly, the same prompt that massively boosts performance on one model, can massively reduce performance on another. So you already need to fine-tune the prompts for the model, if you want anything approaching best results. Now what's really amusing is that if you run models without their official harness, they can actually do way better on some benchmarks! [0] e.g. On Terminal Bench 2, Claude Opus 4.6 goes from #33 (Claude Code) to #5 (custom harness). Similar results for Codex. Now, this is "for this one very specific benchmark", but I still thought it was funny, since you'd expect "the harness made by the same company" to be the best for all tasks, but that's clearly not the case. (For specific tasks, it's actually quite trivial to outperform a general purpose harness.) [0] https://www.tbench.ai/leaderboard/terminal-bench/2.0 https://www.tbench.ai/leaderboard/terminal-bench/2.0
climike 7mo ago
We are working on supporting agent harnesses @ www.cliwatch.com, so both 1. LLM model as well 2. LLM model + harness performance can be evaluated against your software/CLI. We also support building evals against your doc suite. End result is that you’ll feel more comfortable shipping CLIs that work for your agentic users!:)
pizlonator 7mo ago
> and if tested via the codex cli "harness" it wouldn't be a pure model-to-model comparison any more. But the interesting comparison when evaluating coding agent capabilities is to evaluate the offerings given to users. So this means comparing Claude Code to Codex to whatever CLI tools Kimi, GLM, and others give you. And it might mean throwing Cursor, OpenCode, Amp, Pi, mini-swe-agent, etc into the mix

mike_hearn 7mo ago

It's the other way around - Claude Code is the proprietary one. Codex CLI is open source: https://github.com/openai/codex https://github.com/openai/codex You can definitely access the latest models via the API. That's how Codex CLI works.

pizlonator 7mo ago
gpt-5.3 was not accessible via API, at least for me But it was in codex
- mike_hearn 7mo ago
  Ah ha, I know what you're hitting. 5.3-codex is only available via the Responses API, not the Completions API. Two different APIs for model access. If you were using Completions you have to port to Responses. It's not that hard. I did this for my own agent the other week. I think it might be like that for all their new models from now on. Responses is a much more powerful API. It's more like a front to ChatGPT than the underlying models.
  - pizlonator 7mo ago
    I am using responses API (these days I only use responses API when talking to OpenAI) I'll try again, maybe I was doing something else wrong, or something changed
    - mike_hearn 7mo ago
      OK, in that case I'm not sure. But Codex can't hide anything. I don't think there are special codes compiled into the binaries or anything like that.

jsemrau 7mo ago

I reached the same conclusion. I tried using both for my personal investment ambient using agent-pair programming to build and agentic intelligence layer for stocks and the difference between the 2 models if astounding.

andai 7mo ago

>if tested via the codex cli "harness" it wouldn't be a pure model-to-model comparison any more. Well that's already not a very fair comparison, we've known for years (one of the early-ish LLM papers, maybe someone knows which one) that prompting makes an enormous difference on agent performance, and most strikingly, the same prompt that massively boosts performance on one model, can massively reduce performance on another. So you already need to fine-tune the prompts for the model, if you want anything approaching best results. Now what's really amusing is that if you run models without their official harness, they can actually do way better on some benchmarks! [0] e.g. On Terminal Bench 2, Claude Opus 4.6 goes from #33 (Claude Code) to #5 (custom harness). Similar results for Codex. Now, this is "for this one very specific benchmark", but I still thought it was funny, since you'd expect "the harness made by the same company" to be the best for all tasks, but that's clearly not the case. (For specific tasks, it's actually quite trivial to outperform a general purpose harness.) [0] https://www.tbench.ai/leaderboard/terminal-bench/2.0 https://www.tbench.ai/leaderboard/terminal-bench/2.0

climike 7mo ago

We are working on supporting agent harnesses @ www.cliwatch.com, so both 1. LLM model as well 2. LLM model + harness performance can be evaluated against your software/CLI. We also support building evals against your doc suite. End result is that you’ll feel more comfortable shipping CLIs that work for your agentic users!:)

pizlonator 7mo ago

> and if tested via the codex cli "harness" it wouldn't be a pure model-to-model comparison any more. But the interesting comparison when evaluating coding agent capabilities is to evaluate the offerings given to users. So this means comparing Claude Code to Codex to whatever CLI tools Kimi, GLM, and others give you. And it might mean throwing Cursor, OpenCode, Amp, Pi, mini-swe-agent, etc into the mix