4 ms·

5.3 codex https://openai.com/index/introducing-gpt-5-3-codex/ https://openai.com/index/introducing-gpt-5-3-codex/ crushes with a 77.3% in Terminal Bench. The sh

5.3 codex https://openai.com/index/introducing-gpt-5-3-codex/ https://openai.com/index/introducing-gpt-5-3-codex/ crushes with a 77.3% in Terminal Bench. The shortest lived lead in less than 35 minutes. What a time to be alive!

nharada 8mo ago
That's a massive jump, I'm curious if there's a materially different feeling in how it works or if we're starting to reach the point of benchmark saturation. If the benchmark is good then 10 points should be a big improvement in capability...
jkelleyrtp 8mo ago
claude swe-bench is 80.8 and codex is 56.8 Seems like 4.6 is still all-around better?
- gizmodo59 8mo ago
  Its SWE bench pro not swe bench verified. The verified benchmark has stagnated
  - joshuahedlund 8mo ago
    Any ideas why verified has stagnated? It was increasing rapidly and then basically stopped.
    - Snuggly73 8mo ago
      it has been pretty much a benchmark for memorization for a while. there is a paper on the subject somewhere. swe bench pro public is newer, but its not live, so it will get slowly memorized as well. the private dataset is more interesting, as are the results there: https://scale.com/leaderboard/swe_bench_pro_private https://scale.com/leaderboard/swe_bench_pro_private
- Rudybega 8mo ago
  You're comparing two different benchmarks. Pro vs Verified.
purplerabbit 8mo ago
The lack of broad benchmark reports in this makes me curious: Has OpenAI reverted to benchmaxxing? Looking forward to hearing opinions once we all try both of these out
- MallocVoidstar 8mo ago
  The -codex models are only for 'agentic coding', nothing else.
  - dingnuts 8mo ago
    [dead]
wasmainiac 8mo ago
Dumb question. Can these benchmarks be trusted when the model performance tends to vary depending on the hours and load on OpenAI’s servers? How do I know I’m not getting a severe penalty for chatting at the wrong time. Or even, are the models best after launch then slowly eroded away at to more economical settings after the hype wears off?
- aaaalone 8mo ago
  At the end of the day you test it for your use cases anyway but it makes it a great initial hint if it's worth it to test out.
- Corence 8mo ago
  It is a fair question. I'd expect the numbers are all real. Competitors are going to rerun the benchmark with these models to see how the model is responding and succeeding on the tasks and use that information to figure out how to improve their own models. If the benchmark numbers aren't real their competitors will call out that it's not reproducible. However it's possible that consumers without a sufficiently tiered plan aren't getting optimal performance, or that the benchmark is overfit and the results won't generalize well to the real tasks you're trying to do.
  - mrandish 8mo ago
    > I'd expect the numbers are all real. I think a lot of people are concerned due to 1) significant variance in performance being reported by a large number of users, and 2) We have specific examples of OpenAI and other labs benchmaxxing in the recent past (https://grok.com/share/c2hhcmQtMw_66c34055-740f-43a3-a63c-4b03d336c214 https://grok.com/share/c2hhcmQtMw_66c34055-740f-43a3-a63c-4b...). It's tricky because there are so many subtle ways in which "the numbers are all real" could be technically true in some sense, yet still not reflect what a customer will experience (eg harnesses, etc). And any of those ways can benefit the cost structures of companies currently subsidizing models well below their actual costs with limited investor capital. All with billions of dollars in potential personal wealth at stake for company employees and dozens of hidden cost/performance levers at their disposal. And it doesn't even require overt deception on anyone's part. For example, the teams doing benchmark testing of unreleased new models aren't the same people as the ops teams managing global deployment/load balancing at scale day-to-day. If there aren't significant ongoing resources devoted to specifically validating those two things remain in sync - they'll almost certainly drift apart. And it won't be anyone's job to even know it's happening until a meaningful number of important customers complain or sales start to fall. Of course, if an unplanned deviation causes costs to rise over budget, it's a high-priority bug to be addressed. But if the deviation goes the other way and costs are little lower than expected, no one's getting a late night incident alert. This isn't even a dig at OpenAI in particular, it's just the default state of how large orgs work.
callamdelaney 8mo ago
Anthropic models generally are right first time for me. Chatgpt and Gemini are often way, way out with some fundamental misunderstanding of the task at hand.
kys_gizmodo59 8mo ago
> "What a time to be alive!" fuck off.
deleted 8mo ago
[deleted]

nharada 8mo ago

That's a massive jump, I'm curious if there's a materially different feeling in how it works or if we're starting to reach the point of benchmark saturation. If the benchmark is good then 10 points should be a big improvement in capability...

jkelleyrtp 8mo ago

claude swe-bench is 80.8 and codex is 56.8 Seems like 4.6 is still all-around better?

gizmodo59 8mo ago
Its SWE bench pro not swe bench verified. The verified benchmark has stagnated
- joshuahedlund 8mo ago
  Any ideas why verified has stagnated? It was increasing rapidly and then basically stopped.
  - Snuggly73 8mo ago
    it has been pretty much a benchmark for memorization for a while. there is a paper on the subject somewhere. swe bench pro public is newer, but its not live, so it will get slowly memorized as well. the private dataset is more interesting, as are the results there: https://scale.com/leaderboard/swe_bench_pro_private https://scale.com/leaderboard/swe_bench_pro_private
Rudybega 8mo ago
You're comparing two different benchmarks. Pro vs Verified.

purplerabbit 8mo ago

The lack of broad benchmark reports in this makes me curious: Has OpenAI reverted to benchmaxxing? Looking forward to hearing opinions once we all try both of these out

MallocVoidstar 8mo ago
The -codex models are only for 'agentic coding', nothing else.
- dingnuts 8mo ago
  [dead]

wasmainiac 8mo ago

Dumb question. Can these benchmarks be trusted when the model performance tends to vary depending on the hours and load on OpenAI’s servers? How do I know I’m not getting a severe penalty for chatting at the wrong time. Or even, are the models best after launch then slowly eroded away at to more economical settings after the hype wears off?

aaaalone 8mo ago
At the end of the day you test it for your use cases anyway but it makes it a great initial hint if it's worth it to test out.
Corence 8mo ago
It is a fair question. I'd expect the numbers are all real. Competitors are going to rerun the benchmark with these models to see how the model is responding and succeeding on the tasks and use that information to figure out how to improve their own models. If the benchmark numbers aren't real their competitors will call out that it's not reproducible. However it's possible that consumers without a sufficiently tiered plan aren't getting optimal performance, or that the benchmark is overfit and the results won't generalize well to the real tasks you're trying to do.
- mrandish 8mo ago
  > I'd expect the numbers are all real. I think a lot of people are concerned due to 1) significant variance in performance being reported by a large number of users, and 2) We have specific examples of OpenAI and other labs benchmaxxing in the recent past (https://grok.com/share/c2hhcmQtMw_66c34055-740f-43a3-a63c-4b03d336c214 https://grok.com/share/c2hhcmQtMw_66c34055-740f-43a3-a63c-4b...). It's tricky because there are so many subtle ways in which "the numbers are all real" could be technically true in some sense, yet still not reflect what a customer will experience (eg harnesses, etc). And any of those ways can benefit the cost structures of companies currently subsidizing models well below their actual costs with limited investor capital. All with billions of dollars in potential personal wealth at stake for company employees and dozens of hidden cost/performance levers at their disposal. And it doesn't even require overt deception on anyone's part. For example, the teams doing benchmark testing of unreleased new models aren't the same people as the ops teams managing global deployment/load balancing at scale day-to-day. If there aren't significant ongoing resources devoted to specifically validating those two things remain in sync - they'll almost certainly drift apart. And it won't be anyone's job to even know it's happening until a meaningful number of important customers complain or sales start to fall. Of course, if an unplanned deviation causes costs to rise over budget, it's a high-priority bug to be addressed. But if the deviation goes the other way and costs are little lower than expected, no one's getting a late night incident alert. This isn't even a dig at OpenAI in particular, it's just the default state of how large orgs work.

callamdelaney 8mo ago

Anthropic models generally are right first time for me. Chatgpt and Gemini are often way, way out with some fundamental misunderstanding of the task at hand.

kys_gizmodo59 8mo ago

> "What a time to be alive!" fuck off.

deleted 8mo ago

[deleted]