> These comments are getting ridiculous. Not really. Francois (co-creator of the ARC Prize) has this to say: The v1 version of the benchmark is starting to

> These comments are getting ridiculous. Not really. Francois (co-creator of the ARC Prize) has this to say: The v1 version of the benchmark is starting to saturate. There were already signs of this in the Kaggle competition this year: an ensemble of all submissions would score 81% Early indications are that ARC-AGI-v2 will represent a complete reset of the state-of-the-art, and it will remain extremely difficult for o3. Meanwhile, a smart human or a small panel of average humans would still be able to score >95% ... This shows that it's still feasible to create unsaturated, interesting benchmarks that are easy for humans, yet impossible for AI, without involving specialist knowledge. We will have AGI when creating such evals becomes outright impossible. For me, the main open question is where the scaling bottlenecks for the techniques behind o3 are going to be. If human-annotated CoT data is a major bottleneck, for instance, capabilities would start to plateau quickly like they did for LLMs (until the next architecture). If the only bottleneck is test-time search, we will see continued scaling in the future. https://x.com/fchollet/status/1870169764762710376 https://x.com/fchollet/status/1870169764762710376 / https://ghostarchive.org/archive/Sqjbf https://ghostarchive.org/archive/Sqjbf