4 ms·
GPT 6 Astra benchmarks https://cdn.thenewstack.io/media/2026/09/358eb84a-screenshot-2026-09-03-at-10.51.35-am.png https://cdn.thenewstack.io/media/2026/09/358eb
by tristanj 2mo ago
GPT 6 Astra benchmarks https://cdn.thenewstack.io/media/2026/09/358eb84a-screenshot-2026-09-03-at-10.51.35-am.png https://cdn.thenewstack.io/media/2026/09/358eb84a-screenshot...
Performance is significantly higher than Fable 5.1
Source: https://thenewstack.io/openai-gpt6-astra-benchmarks/ https://thenewstack.io/openai-gpt6-astra-benchmarks/
- opus5_hater 2mo agoany benchmark where opus 5 achieves higher scores than fable 5 in any way is not a benchmark worth trusting.
- ActionHank 2mo agousername checks out
- machomaster 2mo agoWhy would Anthropic trust and use these tests in their official comparisons?
- r_lee 2mo agogreat username lol
- merlindru 1mo agodo you feel you're free of bias and predisposition in saying this
- andxor 2mo ago> Performance is significantly higher than Fable 5.1 That's not clear. Need to see independent benchmarks first.
- andxor 2mo agoArtificial Analysis just published their aggregate score (61). Still below Fable 5, let alone Fable 5.1. EDIT: This is suspiciously low. Calls the relevance of existing benchmarks into question.
- timpera 2mo agoI agree, Opus 5 scoring higher than Fable 5 on Artificial Analysis really makes me question the relevance of these scores.
- CamperBob2 2mo agoThere is a very simple explanation for why weaker models appear to kick sand in Fable's face: Fable cannot be benchmarked because of its batshit out-of-control refusal policy. If it actually tackled all of the problems it was assigned, it would presumably kick Opus into the weeds.
- natsucks 2mo agoI saw this too and I'm really confused.
- forgot-my-pw 2mo agoWe need them pelicans on bikes.
- bwat49 2mo agoIts time to move on to the flamingo on a unicycle bench
- forgot-my-pw 2mo agoAA benchmark: https://artificialanalysis.ai/articles/benchmarking-gpt-6-astra https://artificialanalysis.ai/articles/benchmarking-gpt-6-as... TLDR: it's about the same intelligence level as Opus/Fable, but it's suppose to be 70% more token efficient than GPT 5.6 Sol. So it's currently the new leader for cost efficiency frontier.
- tintor 2mo ago66 vs 61 is not 'about the same'. GPT 5.6 is also 61 like Astra.
- forgot-my-pw 1mo agoLate reply: I'm not sure which chart you're referring to. The top 2 charts shows Astra has the same score as Fable 5.1. Even the title says "GPT-6 Astra ties leadership with Claude Fable 5.1 in both of our flagship Indices, at lower cost. Astra equals Fable 5.1 in the Intelligence Index at ~40% of the cost, and in the Coding Agent Index at ~60% of the cost."
- scrlk 2mo agoIs the ARC-AGI-3 score with their custom harness? I'm guessing that is what the footnote is for? (per https://openai.com/index/how-two-settings-tripled-our-arc-agi-3-scores/ https://openai.com/index/how-two-settings-tripled-our-arc-ag...)
- kasperni 2mo agoyes it is.
- woah 2mo agoHaven't people demonstrated all kinds of weak LLMs getting good ARC-AGI-3 scores with special harnesses?
- tintor 2mo agoThose people haven't verified their results against the private set: https://arcprize.org/leaderboard https://arcprize.org/leaderboard
- andriy_koval 2mo agoAstra also not verified using private set, but on "semi-private" set
- andrewchambers 2mo agoif that is true then why is astra on the official ARC leaderboard now ?
- andriy_koval 2mo agoARC leaderboard has results from semi-private data for frontier models, they have another competition for private data. It is described in their methodology: https://arcprize.org/policy https://arcprize.org/policy It makes sense, since once OpenAI API receive task, it is not private anymore but leaked to OpenAI.
- leumon 2mo agoThe annotation on arc-agi-3 is this: > OpenAI's own evaluation notes say Astra uses the company's Responses API harness, while comparison models can operate under different configurations. With this configuration gpt-5.6-sol was able to reach 38,3%. So this is misleading.
- tedsanders 2mo agoJust to clarify, the 38.3% is on the public set, which is easier. On the private set it’s probably more like 30ish. (This hasn’t been run by ARC, so we can only estimate at the moment.)
- jjice 2mo ago100% on ExploitBench seems fitting given recent events.
- malshe 2mo agoI think we need a few writing related benchmarks.
- boutell 2mo agoThat looks more than slight.