4 ms·
For what it's worth, the Qwen team misreported an ARC-AGI score benchmark on the non-thinking model by a factor of 4, which has not been explained yet. They cla
by christianqchung 1y ago
For what it's worth, the Qwen team misreported an ARC-AGI score benchmark on the non-thinking model by a factor of 4, which has not been explained yet. They claimed a score of 41.8% on ARC-AGI 1 [0] which is much higher than what non-chain of thought models have been able to achieve (GPT 4.5 got 10%). The ARC team later benchmarked it at 11%[1], which is still a high score, but not the same as 41.8%. It's still probably a significant update on the model though.
[0] https://huggingface.co/Qwen/Qwen3-235B-A22B-Instruct-2507 https://huggingface.co/Qwen/Qwen3-235B-A22B-Instruct-2507
[1] https://x.com/arcprize/status/1948453132184494471 https://x.com/arcprize/status/1948453132184494471
- ducviet00 1y agoMaybe 41.8% is the score of Qwen3-235B-A22B-Thinking-2507, lol. 11% for the non-thinking model is pretty high
- jug 1y agoMakes sense, it's in line with Gemini 2.5 Pro in that case. It aligns with their other results in the post.
- christianqchung 1y agoThey made it very clear that they were reporting that score for the non-thinking model[0]. I still don't have any guesses as to what happened here, maybe something format related. I can't see a motivation to blatantly lie on a benchmark which would very obviously be publicly corrected. [0] https://x.com/JustinLin610/status/1947836526853034403 https://x.com/JustinLin610/status/1947836526853034403
- mattnewton 1y agoCould it be the public eval set vs the private eval the ARC team has? The public eval set is slightly easier and may have had a some unintentional data leakage since it was released before their training data cutoff.
- coolspot 1y agoThey have provided repro for the 41.8% result here: https://github.com/QwenLM/Qwen3/tree/main/eval https://github.com/QwenLM/Qwen3/tree/main/eval
- deleted 1y ago[deleted]