7 ms·

Dumb question. Can these benchmarks be trusted when the model performance tends to vary depending on the hours and load on OpenAI’s servers? How do I know I’m n

Dumb question. Can these benchmarks be trusted when the model performance tends to vary depending on the hours and load on OpenAI’s servers? How do I know I’m not getting a severe penalty for chatting at the wrong time. Or even, are the models best after launch then slowly eroded away at to more economical settings after the hype wears off?

aaaalone 8mo ago
At the end of the day you test it for your use cases anyway but it makes it a great initial hint if it's worth it to test out.
Corence 8mo ago
It is a fair question. I'd expect the numbers are all real. Competitors are going to rerun the benchmark with these models to see how the model is responding and succeeding on the tasks and use that information to figure out how to improve their own models. If the benchmark numbers aren't real their competitors will call out that it's not reproducible. However it's possible that consumers without a sufficiently tiered plan aren't getting optimal performance, or that the benchmark is overfit and the results won't generalize well to the real tasks you're trying to do.
- mrandish 8mo ago
  > I'd expect the numbers are all real. I think a lot of people are concerned due to 1) significant variance in performance being reported by a large number of users, and 2) We have specific examples of OpenAI and other labs benchmaxxing in the recent past (https://grok.com/share/c2hhcmQtMw_66c34055-740f-43a3-a63c-4b03d336c214 https://grok.com/share/c2hhcmQtMw_66c34055-740f-43a3-a63c-4b...). It's tricky because there are so many subtle ways in which "the numbers are all real" could be technically true in some sense, yet still not reflect what a customer will experience (eg harnesses, etc). And any of those ways can benefit the cost structures of companies currently subsidizing models well below their actual costs with limited investor capital. All with billions of dollars in potential personal wealth at stake for company employees and dozens of hidden cost/performance levers at their disposal. And it doesn't even require overt deception on anyone's part. For example, the teams doing benchmark testing of unreleased new models aren't the same people as the ops teams managing global deployment/load balancing at scale day-to-day. If there aren't significant ongoing resources devoted to specifically validating those two things remain in sync - they'll almost certainly drift apart. And it won't be anyone's job to even know it's happening until a meaningful number of important customers complain or sales start to fall. Of course, if an unplanned deviation causes costs to rise over budget, it's a high-priority bug to be addressed. But if the deviation goes the other way and costs are little lower than expected, no one's getting a late night incident alert. This isn't even a dig at OpenAI in particular, it's just the default state of how large orgs work.
cyanydeez 8mo ago
When do you think we should run this benchmark? Friday, 1pm? Monday 8AM? Wednesday 11AM? I definitely suspect all these models are being degraded during heavy loads.
- j_maffe 8mo ago
  This hypothesis is tested regularly by plenty of live benchmarks. The services usually don't decay in performance.
ifwinterco 8mo ago
On benchmarks GPT 5.2 was roughly equivalent to Opus 4.5 but most people who've used both for SWE stuff would say that Opus 4.5 is/was noticeably better
- elAhmo 8mo ago
  I mostly used Sonnet/Opus 4.x in the past months, but 5.2 Codex seemed to be on par or better for my use case in the past month. I tried a few models here and there but always went back to Claude, but with 5.2 Codex for the first time I felt it was very competitive, if not better. Curious to see how things will be with 5.3 and 4.6
- georgeven 8mo ago
  Interesting. Everyone in my circle said the opposite.
  - krzyk 8mo ago
    It probably depends on programming language and expectations.
    - ifwinterco 8mo ago
      This is mostly Python/TS for me... what Jonathan Blow would probably call not "real programming" but it pays the bills They can both write fairly good idiomatic code but in my experience opus 4.5 is better at understanding overall project structure etc. without prompting. It just does things correctly first time more often than codex. I still don't trust it obviously but out of all LLMs it's the closest to actually starting to earn my trust
    - deaux 8mo ago
      Even for the same language it depends on domain.
  - MadnessASAP 8mo ago
    My experience is that Codex follows directions better but Claude writes better code. ChatGPT-5.2-Codex follows directions to ensure a task [bead](https://github.com/steveyegge/beads https://github.com/steveyegge/beads) is opened before starting a task and to keep it updated almost to a fault. Claude-Opus-4.5 with the exact same directions, forgets about it within a round or two. Similarly, I had a project that required very specific behaviour from a couple functions, it was documented in a few places including comments at the top and bottom of the function. Codex was very careful in ensuring the function worked as was documented. Claude decided it was easier to do the exact opposite, rewrote the function, the comments, and the documentation to saynit now did the opposite of what was previously there. If I believed a LLM could be spiteful, I would've believed it on that second one. I certainly felt some after I realised what it had done. The comment literally said: // Invariant regardless of the value of X, this function cannot return Y And it turned it into: // Returns Y if X is true
tedsanders 8mo ago
We don't vary our model quality with time of day or load (beyond negligible non-determinism). It's the same weights all day long with no quantization or other gimmicks. They can get slower under heavy load, though. (I'm from OpenAI.)
- Trufa 8mo ago
  Can you be more specific than this? does it vary in time from launch of a model to the next few months, beyond tinkering and optimization?
  - joshvm 8mo ago
    My gut feeling is that performance is more heavily affected by harnesses which get updated frequently. This would explain why people feel that Claude is sometimes more stupid - that's actually accurate phrasing, because Sonnet is probably unchanged. Unless Anthropic also makes small A/B adjustments to weights and technically claims they don't do dynamic degradation/quantization based on load. Either way, both affect the quality of your responses. It's worth checking different versions of Claude Code, and updating your tools if you don't do it automatically. Also run the same prompts through VS Code, Cursor, Claude Code in terminal, etc. You can get very different model responses based on the system prompt, what context is passed via the harness, how the rules are loaded and all sorts of minor tweaks. If you make raw API calls and see behavioural changes over time, that would be another concern.
  - tedsanders 8mo ago
    Yeah, happy to be more specific. No intention of making any technically true but misleading statements. The following are true: - In our API, we don't change model weights or model behavior over time (e.g., by time of day, or weeks/months after release) - Tiny caveats include: there is a bit of non-determinism in batched non-associative math that can vary by batch / hardware, bugs or API downtime can obviously change behavior, heavy load can slow down speeds, and this of course doesn't apply to the 'unpinned' models that are clearly supposed to change over time (e.g., xxx-latest). But we don't do any quantization or routing gimmicks that would change model weights. - In ChatGPT and Codex CLI, model behavior can change over time (e.g., we might change a tool, update a system prompt, tweak default thinking time, run an A/B test, or ship other updates); we try to be transparent with our changelogs (listed below) but to be honest not every small change gets logged here. But even here we're not doing any gimmicks to cut quality by time of day or intentionally dumb down models after launch. Model behavior can change though, as can the product / prompt / harness. ChatGPT release notes: https://help.openai.com/en/articles/6825453-chatgpt-release-notes https://help.openai.com/en/articles/6825453-chatgpt-release-... Codex changelog: https://developers.openai.com/codex/changelog/ https://developers.openai.com/codex/changelog/ Codex CLI commit history: https://github.com/openai/codex/commits/main/ https://github.com/openai/codex/commits/main/
thinkingtoilet 8mo ago
We know Open AI got caught getting benchmark data and tuning their models to it already. So the answer is a hard no. I imagine over time it gives a general view of the landscape and improvements, but take it with a large grain of salt.
- rvz 8mo ago
  The same thing was done with Meta researchers with Llama 4 and what can go wrong when 'independent' researchers begin to game AI benchmarks. [0] You always have to question these benchmarks, especially when the in-house researchers can potentially game them if they wanted to. Which is why it must be independent. [0] https://gizmodo.com/meta-cheated-on-ai-benchmarks-and-its-a-glimpse-into-a-new-golden-age-2000586433 https://gizmodo.com/meta-cheated-on-ai-benchmarks-and-its-a-...
- tedsanders 8mo ago
  Are you referring to FrontierMath? We had access to the eval data (since we funded it), but we didn't train on the data or otherwise cheat. We didn't even look at the eval results until after the model had been trained and selected.
  - thinkingtoilet 8mo ago
    No one believes you.
    - tedsanders 8mo ago
      If you don't believe me, that's fair enough. Some pieces of evidence that might update you or others: - a member of the team who worked with this eval has left OpenAI and now works at a competitor; if we cheated, he would have every incentive to whistleblow - cheating on evals is fairly easy to catch and risks destroying employee morale, customer trust, and investor appetite; even if you're evil, the cost-benefit doesn't really pencil out to cheat on a niche math eval - Epoch made a private held-out set (albeit with a different difficulty); OpenAI performance on that set doesn't suggest any cheating/overfitting - Gemini and Claude have since achieved similar scores, suggesting that scoring ~40% is not evidence of cheating with the private set - The vast majority of evals are open-source (e.g., SWE-bench Pro Public), and OpenAI along with everyone else has access to their problems and the opportunity to cheat, so FrontierMath isn't even unique in that respect
smcleod 8mo ago
I don't think much from OpenAI can be trusted tbh.

aaaalone 8mo ago

At the end of the day you test it for your use cases anyway but it makes it a great initial hint if it's worth it to test out.

Corence 8mo ago

It is a fair question. I'd expect the numbers are all real. Competitors are going to rerun the benchmark with these models to see how the model is responding and succeeding on the tasks and use that information to figure out how to improve their own models. If the benchmark numbers aren't real their competitors will call out that it's not reproducible. However it's possible that consumers without a sufficiently tiered plan aren't getting optimal performance, or that the benchmark is overfit and the results won't generalize well to the real tasks you're trying to do.

mrandish 8mo ago
> I'd expect the numbers are all real. I think a lot of people are concerned due to 1) significant variance in performance being reported by a large number of users, and 2) We have specific examples of OpenAI and other labs benchmaxxing in the recent past (https://grok.com/share/c2hhcmQtMw_66c34055-740f-43a3-a63c-4b03d336c214 https://grok.com/share/c2hhcmQtMw_66c34055-740f-43a3-a63c-4b...). It's tricky because there are so many subtle ways in which "the numbers are all real" could be technically true in some sense, yet still not reflect what a customer will experience (eg harnesses, etc). And any of those ways can benefit the cost structures of companies currently subsidizing models well below their actual costs with limited investor capital. All with billions of dollars in potential personal wealth at stake for company employees and dozens of hidden cost/performance levers at their disposal. And it doesn't even require overt deception on anyone's part. For example, the teams doing benchmark testing of unreleased new models aren't the same people as the ops teams managing global deployment/load balancing at scale day-to-day. If there aren't significant ongoing resources devoted to specifically validating those two things remain in sync - they'll almost certainly drift apart. And it won't be anyone's job to even know it's happening until a meaningful number of important customers complain or sales start to fall. Of course, if an unplanned deviation causes costs to rise over budget, it's a high-priority bug to be addressed. But if the deviation goes the other way and costs are little lower than expected, no one's getting a late night incident alert. This isn't even a dig at OpenAI in particular, it's just the default state of how large orgs work.

cyanydeez 8mo ago

When do you think we should run this benchmark? Friday, 1pm? Monday 8AM? Wednesday 11AM? I definitely suspect all these models are being degraded during heavy loads.

j_maffe 8mo ago
This hypothesis is tested regularly by plenty of live benchmarks. The services usually don't decay in performance.

ifwinterco 8mo ago

On benchmarks GPT 5.2 was roughly equivalent to Opus 4.5 but most people who've used both for SWE stuff would say that Opus 4.5 is/was noticeably better

elAhmo 8mo ago
I mostly used Sonnet/Opus 4.x in the past months, but 5.2 Codex seemed to be on par or better for my use case in the past month. I tried a few models here and there but always went back to Claude, but with 5.2 Codex for the first time I felt it was very competitive, if not better. Curious to see how things will be with 5.3 and 4.6
georgeven 8mo ago
Interesting. Everyone in my circle said the opposite.
- krzyk 8mo ago
  It probably depends on programming language and expectations.
  - ifwinterco 8mo ago
    This is mostly Python/TS for me... what Jonathan Blow would probably call not "real programming" but it pays the bills They can both write fairly good idiomatic code but in my experience opus 4.5 is better at understanding overall project structure etc. without prompting. It just does things correctly first time more often than codex. I still don't trust it obviously but out of all LLMs it's the closest to actually starting to earn my trust
  - deaux 8mo ago
    Even for the same language it depends on domain.
- MadnessASAP 8mo ago
  My experience is that Codex follows directions better but Claude writes better code. ChatGPT-5.2-Codex follows directions to ensure a task [bead](https://github.com/steveyegge/beads https://github.com/steveyegge/beads) is opened before starting a task and to keep it updated almost to a fault. Claude-Opus-4.5 with the exact same directions, forgets about it within a round or two. Similarly, I had a project that required very specific behaviour from a couple functions, it was documented in a few places including comments at the top and bottom of the function. Codex was very careful in ensuring the function worked as was documented. Claude decided it was easier to do the exact opposite, rewrote the function, the comments, and the documentation to saynit now did the opposite of what was previously there. If I believed a LLM could be spiteful, I would've believed it on that second one. I certainly felt some after I realised what it had done. The comment literally said: // Invariant regardless of the value of X, this function cannot return Y And it turned it into: // Returns Y if X is true

tedsanders 8mo ago

We don't vary our model quality with time of day or load (beyond negligible non-determinism). It's the same weights all day long with no quantization or other gimmicks. They can get slower under heavy load, though. (I'm from OpenAI.)

Trufa 8mo ago
Can you be more specific than this? does it vary in time from launch of a model to the next few months, beyond tinkering and optimization?
- joshvm 8mo ago
  My gut feeling is that performance is more heavily affected by harnesses which get updated frequently. This would explain why people feel that Claude is sometimes more stupid - that's actually accurate phrasing, because Sonnet is probably unchanged. Unless Anthropic also makes small A/B adjustments to weights and technically claims they don't do dynamic degradation/quantization based on load. Either way, both affect the quality of your responses. It's worth checking different versions of Claude Code, and updating your tools if you don't do it automatically. Also run the same prompts through VS Code, Cursor, Claude Code in terminal, etc. You can get very different model responses based on the system prompt, what context is passed via the harness, how the rules are loaded and all sorts of minor tweaks. If you make raw API calls and see behavioural changes over time, that would be another concern.
- tedsanders 8mo ago
  Yeah, happy to be more specific. No intention of making any technically true but misleading statements. The following are true: - In our API, we don't change model weights or model behavior over time (e.g., by time of day, or weeks/months after release) - Tiny caveats include: there is a bit of non-determinism in batched non-associative math that can vary by batch / hardware, bugs or API downtime can obviously change behavior, heavy load can slow down speeds, and this of course doesn't apply to the 'unpinned' models that are clearly supposed to change over time (e.g., xxx-latest). But we don't do any quantization or routing gimmicks that would change model weights. - In ChatGPT and Codex CLI, model behavior can change over time (e.g., we might change a tool, update a system prompt, tweak default thinking time, run an A/B test, or ship other updates); we try to be transparent with our changelogs (listed below) but to be honest not every small change gets logged here. But even here we're not doing any gimmicks to cut quality by time of day or intentionally dumb down models after launch. Model behavior can change though, as can the product / prompt / harness. ChatGPT release notes: https://help.openai.com/en/articles/6825453-chatgpt-release-notes https://help.openai.com/en/articles/6825453-chatgpt-release-... Codex changelog: https://developers.openai.com/codex/changelog/ https://developers.openai.com/codex/changelog/ Codex CLI commit history: https://github.com/openai/codex/commits/main/ https://github.com/openai/codex/commits/main/

thinkingtoilet 8mo ago

We know Open AI got caught getting benchmark data and tuning their models to it already. So the answer is a hard no. I imagine over time it gives a general view of the landscape and improvements, but take it with a large grain of salt.

rvz 8mo ago
The same thing was done with Meta researchers with Llama 4 and what can go wrong when 'independent' researchers begin to game AI benchmarks. [0] You always have to question these benchmarks, especially when the in-house researchers can potentially game them if they wanted to. Which is why it must be independent. [0] https://gizmodo.com/meta-cheated-on-ai-benchmarks-and-its-a-glimpse-into-a-new-golden-age-2000586433 https://gizmodo.com/meta-cheated-on-ai-benchmarks-and-its-a-...
tedsanders 8mo ago
Are you referring to FrontierMath? We had access to the eval data (since we funded it), but we didn't train on the data or otherwise cheat. We didn't even look at the eval results until after the model had been trained and selected.
- thinkingtoilet 8mo ago
  No one believes you.
  - tedsanders 8mo ago
    If you don't believe me, that's fair enough. Some pieces of evidence that might update you or others: - a member of the team who worked with this eval has left OpenAI and now works at a competitor; if we cheated, he would have every incentive to whistleblow - cheating on evals is fairly easy to catch and risks destroying employee morale, customer trust, and investor appetite; even if you're evil, the cost-benefit doesn't really pencil out to cheat on a niche math eval - Epoch made a private held-out set (albeit with a different difficulty); OpenAI performance on that set doesn't suggest any cheating/overfitting - Gemini and Claude have since achieved similar scores, suggesting that scoring ~40% is not evidence of cheating with the private set - The vast majority of evals are open-source (e.g., SWE-bench Pro Public), and OpenAI along with everyone else has access to their problems and the opportunity to cheat, so FrontierMath isn't even unique in that respect

smcleod 8mo ago

I don't think much from OpenAI can be trusted tbh.