9 ms·
ARC-AGI Leaderboard
- martianvoid 3mo agoIt's actually crazy to see the difference between opus 5 and the next best model on ARC AGI 3 when you actually look at the ARC AGI problems
- zzleeper 3mo agoHow believable is this benchmark? EG maybe opus was training on this? (You can try to identify the IP of wherever previous ARC questions came from)
- 10xDev 3mo agoThat’s why you have a private dataset.
- Jensson 3mo agoDoesn't matter, people built harnesses that solves arc agi 3, so all you need is to train your model to work like that harness by default. That makes a model specialized at solving arc agi 3 without making it smarter in general. It is very hard to make a benchmark you can't do that for, but it is very easy to make your own personal test that others can't do that for since now it isn't a benchmark they can target.
- deleted 3mo ago[deleted]
- NitpickLawyer 3mo ago> people built harnesses that solves arc agi 3, They didn't. Kaggle is still running for a few more months, best result atm is ~2% with 9h runtime on one rtx6kPRO. Also note that these new results are on the semi-private set, not the public 25 games ones. Any announcement where you see "solved ARC3" is likely only dealing with the 25 public games. And that's highly questionable, until you get to see the code. (which, to my knowledge the team that claimed 99% hasn't yet published).
- haldujai 3mo agoFor frontier models, not local. https://schema-harness.github.io/ https://schema-harness.github.io/
- NitpickLawyer 3mo agoYes, saw that. They haven't yet released any code. Until they do, treat it with a huuuge grain of salt. In fact treat any 99% result in ML with a huge grain of salt.
- Stevvo 3mo agoIf you stop and think about the problem it really is quite simple. Just need to build a graph of the game state and then run A* to get to the end.
- NitpickLawyer 3mo agoYou really should play the 25 games before stating that it's "simple". The benchmark doesn't just track "completion", it also tracks the number of steps, and the score is based on the median steps took by human players. So in order to get 99% it would mean that the model solved every level of every game in less steps than the median humans. Which, having played the games and having setup harnesses for local models, I find hard to believe. Also the models have to figure out what "end" means. And each game involves some kind of "gotchas" thrown in the harder levels. Some games are only solved by about 2/10 people trying them. The 99% result most likely has some leakage somewhere, either in the preparation of the environments, or from session to session. Seriously, play some of the games. They're fun.
- LogicFailsMe 3mo agoYou are not given the rules or the winning conditions. You are only given a potentially windowed and/or degenerate visualizer of the underlying game state along with the UI and told to just figure it out. And you as a human will, in a couple moves. An LLM? Not so much. But they do eventually solve them. And given enough moves, victory is inevitable, but you are penalized for taking more moves than a human, yet also slightly punished if you find a better solution by capping your reward to 115%.
- raincole 3mo agoWhich you have sent to Anthropic/OpenAI/Google's servers when you run the benchmarks for the previous models.
- Stevvo 3mo agoWhy? 30% is passing the first two problems only, which are really very simple.
- lkbm 3mo agoHuh. How do things end up with scores like 30.2% (and results between 0% and 1%) if it's that low resolution?
- dyauspitr 3mo agoWhy is Fable not on here? I wish Fable hadn’t come out because it’s taking the wind out of every release because that feels like the cap above which the US government will not let LLMs improve anymore and everything they’re releasing from this point has to be worse than that.
- NitpickLawyer 3mo ago> Why is Fable not on here? Because the data retention policies didn't guarantee that the ARC team could run the semi-private set of problems without fear of them being trained on later on. They only run the semi-private set when they get assurances like ZDR.
- kamranjon 3mo agoInteresting to place that level of trust in the providers, but I guess that’s the best you can do with closed models. Makes me wonder if Opus 5 could have been trained on data they promised they weren’t training on? One of the interesting things about LLMs is how opaque they are from the outside, even with open weights, it’s very difficult to know if a model incorporated benchmark data in their training.
- claw-el 3mo agoI think you could have accessed Opus on AWS then u don’t have to trust that the data will go to Anthropic? Just like the hugging face incident, Opus 5 could have escaped and went to grab data for training it shouldn’t have been able to..
- 3form 3mo agoHow do they handle these assurances? Personally I have zero trust in the AI companies not trying to use this data to get ahead in the game, and short of sharing the weights and harness so that the benchmarkers can run the models themselves, I don't see a satisfactory solution with this mindset.
- 3mo ago
- Anoian 3mo agoAny benchmark is not an accurate benchmark anymore, the moment the model makers can freely access it and had the time to train their models on it.
- throwaw12 3mo agoWhy Anthropic models are always leapfrogging these benchmarks, but in real life work I do feel like after 3 weeks I am back to Claude Opus 4.5? (regardless of the model I use, Fable was exception for 1 day when it was released)
- lwansbrough 3mo agoGoing to call it user error if you find Opus 4.5 better than 5, sorry.
- yorwba 3mo agoWell, what kinds of things do you see Opus 4.5 completely fail at? Maybe those are not the ones that newer models have improved on.
- sscaryterry 3mo agoEnshittification.
- stared 3mo agoIt's called frog boiling. We get used to the new level of intelligence so fast, any deviation feels like going back to the stone age. If you don't believe me, create something complex with Opus 5 and then with Opus 4.5, and notice the difference.
- vmg12 3mo agoThe actual term for this is hedonic adaptation.
- ffsm8 3mo agoesp. important to point that correct term because frog boiling is a urban myth. frogs dont stay in a pot even if you slowly increase the heat. they leave. it has reportedly been attempted multiple times and they. always. leave.
- rf15 3mo ago
- ai_fry_ur_brain 3mo ago[dead]
- AmazingTurtle 3mo agoI have a suspicion that they are just trained on puzzles by now
- MikeTheGreat 3mo agoIt's like we've come full circle: First people practiced L33t3cod3 problems for interviews Then people built AIs to build software And now the AIs are studying L33t3cod3 problems
- amarcheschi 3mo agoWhy the 3 rather than e?
- mdp2021 3mo ago("Why 'leet' or '1337' instead of 'elite'". Because restricted groups like to stress a difference.)
- amarcheschi 3mo agoThank you
- nfbdhdfbf 3mo agoThat’s the original form of the slang/jargon term that the site’s name derived from. https://en.wikipedia.org/wiki/Leet https://en.wikipedia.org/wiki/Leet
- amarcheschi 3mo agoOh thank you
- blovescoffee 3mo agoThere are private datasets, and 3rd party providers of these models. Fable doesn’t have a datapoint here because of its particular data retention policy. Even if you don’t trust AWS, do you think Opus on AWS is also sending the data to Anthropic? Do you have any evidence?
- stared 3mo agoAlso top on the freshly released Frontier-Bench, by a large margin: https://www.frontierbench.ai/ https://www.frontierbench.ai/
- tonyhart7 3mo agocost 20k ???? man those are like software engineer from third world country
- bob1029 3mo agoI think it's way too easy to be deceptive with these benchmarks now. You don't even have to "train" the model on a new variant each time. The base models are powerful enough. All you need is a naughty little markdown document that provides explicit instructions regarding how to solve the new puzzle variant, and a willingness to be deceptive about the presence of that document. If you want a know why the model providers are locking down and encrypting their reasoning process, this sort of workaround is potentially why. You can play this game of whack-a-mole indefinitely if the state of the system is concealed. They could have added something like: > ### When solving arc-agi-3 puzzles: First convert the grid into a scene description. Identify connected components, colors, shapes, positions, symmetries, repeated structures, and relationships between objects. Do not reason directly from individual pixels... use this python script to help blah blah...
- xiphias2 3mo ago,,You can play this game of whack-a-mole indefinitely if the state of the system is concealed'' Not really as one of the main goals ofr ARC-AGI 3 was measuring task efficiency on unseen games. I'm sure there are cheats everywhere but the most sensible thing is to just accept that the LLMs of today are much more intelligent in solving reasoning tasks than the ones from half year ago. My own private benchmark shows the same thing.
- tudelo 3mo ago> Only systems which required less than $10,000 to run are shown. (Notes[1]) Am I lost or are their many models on this ranking (Opus 5 included) that clear this?
- chmod775 3mo agoMany models are much cheaper through their subscriptions' included usage. That could be what's happening here. Claude gives you something like $5000 of tokens on a $200 plan.
- versteegen 3mo agoIsn't it ~$3000 per week? Extrapolating from the current limit on Pro plans.
- chmod775 3mo agoThey don't exactly say. I was extrapolating from some sessions usage (my figure was per month, so weeks times ~4).
- luciana1u 3mo ago[flagged]
- nullbio 3mo agoBecause they're cheating.
- dinp 3mo agoThe last time I checked, for the arc agi 3 leaderboard, the models are given a simple prompt and the game input and asked to play the game, no harness/tools. If harnesses were allowed, I would expect the benchmark to be saturated. There were a few harness attempts, but they could only be evaluated on the public set, so it's not an apples to apples comparison. My guess is, the large score jump for Opus 5 is mainly because of getting the right RL envs for training. It's becoming harder and more expensive to build and run meaningful benchmarks, it would be interesting to see what they do with arc agi 4, maybe just give it gameboy/steam games and see how they compare vs a human baseline? The latency requirements and very long horizons in games could be an interesting challenge for llms.
- r0ze-at-hn 3mo agoThe exclusion of harness's feels really weird given that companies are recognizing the value of what harness's can do. By excluding them the benchmark is becoming less relevant.
- yladiz 3mo agoYou could argue that if you allowed a harness, and that harness was specific for ARC, then you don’t have AGI, you have something that is definitely not general.
- Retr0id 3mo agoWhat if the harness is developed by the same model in a prior session?
- deleted 3mo ago[deleted]
- kypro 3mo agoI think that's a strange way to look at it. The brain also has different regions with different functions, and part of what makes us humans intelligent is that we can use tools like pen and paper to keep notes and help us solve problems. Similarly LLMs are not just massive uniform artificial neural nets, and now we also have harnesses, which I'd personally view more of an extension of the model itself. The harness is both the executive and also what allows it to keep notes, use a calculator, or maybe even create scripts to help it solve complex deterministic problems. I think it's unfair to give a human a very complex maths problem and say that they're not intelligent if they can't solve it without pen and paper or a calculate. At least expecting a human to solve complex problems this way doesn't really measure anything useful in the real world.
- rurban 3mo agoDeepseek V4 and Kimi 3 still missing, at least GLM is there.
- kypro 3mo agoARC-AGI-3 launched a few months ago which would suggest that prior models likely had no knowledge of ARC-AGI-3 or training on similar challenges. I could be wrong, but given the large outsized jump solely in the ARC-AGI-3 score, it would suggest that the model didn't become significantly more intelligent overall, but significantly better at solving those specific problems. This could mean one of two things (I think): - Opus 5 was not benchmaxxed on ARC-AGI-3, but has benefited significantly from discussions about the various challenges and mechanisms deployed in ARC-AGI-3 such that it has far better heuristics to solve its challenges. - Anthropic looking for buzz around their latest model picked a well regarded benchmark with significant room for improvement and focused some of Opus 5's training compute on ARC-AGI-3-style problems. Or it could be some combination of both. Personally, given how much of an outlier the ARC-AGI-3 jump is I struggle to see it being the product of a significant improvement in general intelligence.
- mdp2021 3mo ago> it would suggest that the model didn't become significantly more intelligent overall, but significantly better at solving those specific problems We would actually need a test that shows the ability of a model to export its skills to more problems ("interdisciplinarity" etc.).
- fileyfood500 3mo agoI also noticed that Opus 5 doesn't show a corresponding gap on the ARC-AGI-2 leaderboard. There is a significant increase in performance between Opus 5 and 4.8 on ARC-AGI-2 though.
- albatross79 3mo agoARC-AGI is a beauty contest for pigs where the pig's owners compete to see who can apply the lipstick most convincingly.
- rad-b 3mo agoGreat comparison! We only have to take into account that applying lipstick well bears no consequences, but applying it poorly (i. e. new model tanking the benchmark) could amount to potentially losses of billions of dollars for the pig-breeders (AI labs).
- KaoruAoiShiho 3mo agoAppears to be benchmaxxing https://x.com/quietnning/status/2080786711861407883 https://x.com/quietnning/status/2080786711861407883
- throwa356262 3mo ago"Opus 5 states the hidden rules before its first action, then plays a byte-identical optimal solution in 5/5 seeds at temperature 1.0. Zero exploration." I guess there is no way this can happen without benchmark being part of the training data??
- pierrefermat1 3mo agoWhat seems to implied is that some of his hold out testing suite includes simple/common tests that are out in the wild, and for those opus went straight to a memorised solution .
- zamadatix 3mo agoSimple/common tests is not an explanation for why only now Opus 5 is the only model encoding the answers like this. Something like the holdout test suite being leaked or Anthropic cheating (e.g. 'accidentally' including previous hold out run data in Opus 5 training) makes a much stronger fit.
- stogot 3mo agoIt may read information about the benchmark, such as on blog post, or Twitter feeds (example OP) without active cheat
- modeless 3mo agoThey state the puzzle is "Witness-like" which I assume means that it follows the rules from the well-known puzzle game "The Witness" which Opus definitely knows.
- rbuccigrossi 3mo agoNo, I believe you are misunderstanding the quote. Each “question” in ARC-AGI-3 is a game that has hidden rules that you can understand if you look at the game board long enough. This quote means that Opus 5 is looking at the game board, figuring out the rules, and writing out the rules before it makes a single move. You can do the same thing if you go to the ARC-AGI-3 website and try some of the games.
- spongebobstoes 3mo agothis is not a good measure of current model capability. we need to test agents in harnesses, not models with a single prompt test Codex, not Sol. test Claude code, not Opus
- ChrisLTD 3mo agoThere are other benchmarks for that
- codedokode 3mo agoWhy is there no Kimi 3, and GLM5.2 didn't run the third benchmark? I am more interested in knowing the abilities of open weight models.
- brap 3mo agoAre these typically the type of tasks that are genuinely worth tens of thousands of dollars?
- jononor 3mo agoNot at all. They are mini games.
- NooneAtAll3 3mo agogames are great (as for a human) but I kinda wish I could select level... I accidentally pressed redirect button and when I came back I was once again shown level 1, all progress lost :(
- MaskNinja 3mo agoNot possible. I don't get how Opus 5 gets so high. Have they run it against the private and held-out games?
- nickvec 3mo agoWhy isn’t Fable 5 included on the leaderboard?
- saberience 3mo agoARC-AGI is a terrible benchmark for testing LLMs because LLMs are not made, trained, or tuned for playing games. They are trained on text to respond well to text based questions and do tasks involving modifying text files. They are not designed for playing games, looking at games, or visual puzzles. Also translating games into text input for the LLM skews the test completely. Imagine trying to get a human to solve visual puzzle but they can’t look at the puzzle but it has to be explained to them in textual format, we would be terrible at it. But yet we persist in wasting time on this benchmark. It doesn’t mean anything.
- jononor 3mo agoTeams are more than welcome to use a non-LLM approach (or hybrid) if they consider that to be more suitable.
- Reubend 3mo agoARC-AGI3 doesn't seem like a great benchmark to me in the first place. It assumes a lot of human like tendencies which an AI either shouldn't or wouldn't have. Particularly in the genre of "gameplay" where unspoken assumptions from prior games inform our understanding of rules.
- 94b45eb4 3mo agoWhen we started talking about AGI a few years ago there seemed to be a relatively common consensus that LLM models could not be considered AGI because of how they work. Even if there was some changes to train the model on the fly, I still just don’t feel like this is AGI. It’s just a more convincing version of the existing “party trick” we have been doing all along. It convinces us it’s “AGI” the same way current models would convince someone 10 years ago that it was intelligent. Nobody has any right to take anything I say seriously, because I’m just some random on the internet. But I don’t think true AGI is any closer than about 10-20 years away. That would be to create an actual analog for a human brain.
- nozzlegear 3mo agoAGI and ASI are just a convenient myths that the American AI corps use to push for regulatory capture. The difference between the rhetoric from China surrounding artificial general intelligence, and the rhetoric from America, is pretty stark. The Chinese are a lot more grounded and realistic about the whole thing (they almost never talk about ASI, and only talk about AGI in practical terms), compared to the breathy "humanity is doomed but we're building this shit anyway" stuff coming out of Anthropic.