10 ms·
Congratulations to Francois Chollet on making the most interesting and challenging LLM benchmark so far. A lot of people have criticized ARC as not being relev
by modeless 2y ago
Congratulations to Francois Chollet on making the most interesting and challenging LLM benchmark so far.
A lot of people have criticized ARC as not being relevant or indicative of true reasoning, but I think it was exactly the right thing. The fact that scaled reasoning models are finally showing progress on ARC proves that what it measures really is relevant and important for reasoning.
It's obvious to everyone that these models can't perform as well as humans on everyday tasks despite blowout scores on the hardest tests we give to humans. Yet nobody could quantify exactly the ways the models were deficient. ARC is the best effort in that direction so far.
We don't need more "hard" benchmarks. What we need right now are "easy" benchmarks that these models nevertheless fail. I hope Francois has something good cooked up for ARC 2!
- dtquad 2y agoAre there any single-step non-reasoner models that do well on this benchmark? I wonder how well the latest Claude 3.5 Sonnet does on this benchmark and if it's near o1.
- throwaway71271 2y ago| Name | Semi-private eval | Public eval | |--------------------------------------|-------------------|-------------| | Jeremy Berman | 53.6% | 58.5% | | Akyürek et al. | 47.5% | 62.8% | | Ryan Greenblatt | 43% | 42% | | OpenAI o1-preview (pass@1) | 18% | 21% | | Anthropic Claude 3.5 Sonnet (pass@1) | 14% | 21% | | OpenAI GPT-4o (pass@1) | 5% | 9% | | Google Gemini 1.5 (pass@1) | 4.5% | 8% | https://arxiv.org/pdf/2412.04604 https://arxiv.org/pdf/2412.04604
- kandesbunzler 2y agowhy is this missing the o1 release / o1 pro models? Would love to know how much better they are
- Freebytes 2y agoThis might be because they are referencing single step, and I do not think o1 is single step.
- aimanbenbaha 2y agoAkyürek et al uses test-time compute.
- YetAnotherNick 2y agoHere are the results for base models[1]: o3 (coming soon) 75.7% 82.8% o1-preview 18% 21% Claude 3.5 Sonnet 14% 21% GPT-4o 5% 9% Gemini 1.5 4.5% 8% Score (semi-private eval) / Score (public eval) [1]: https://arcprize.org/2024-results https://arcprize.org/2024-results
- simonw 2y agoI'd love to know how Claude 3.5 Sonnet does so well despite (presumably) not having the same tricks as the o-series models.
- Bjorkbat 2y agoIt's easy to miss, but if you look closely at the first sentence of the announcement they mention that they used a version of o3 trained on a public dataset of ARC-AGI, so technically it doesn't belong on this list.
- dot1x 2y agoIt's all scam. ClosedAI trained on the data they were tested on, so no, nothing here is impressive.
- Tanjreeve 2y agoThey definitely did or they probably did? Is there any source for that just so I can point It out to people?
- catalypso 2y agoJust a clarification, they tuned on the public training dataset, not the semi-private one. The 87.5% score was on the semi-private eval, which means the model was still able to generalize well. That being said, the fact that this is not a "raw" base model, but one tuned on the ARC-AGI tests distribution takes away from the impressiveness of the result — How much ? — I'm not sure, we'd need the un-tuned base o3 model score for that. In the meantime, comparing this tuned o3 model to other un-tuned base models is unfair (apples-to-oranges kind of comparison).
- refulgentis 2y agoThis emphasizes persons and a self-conceived victory narrative over the ground truth. Models have regularly made progress on it, this is not new with the o-series. Doing astoundingly well on it, and having a mutually shared PR interest with OpenAI in this instance, doesn't mean a pile of visual puzzles is actually AGI or some well thought out and designed benchmark of True Intelligence(tm). It's one type of visual puzzle. I don't mean to be negative, but to inject a memento mori. Real story is some guys get together and ride off Chollet's name with some visual puzzles from ye olde IQ test, and the deal was Chollet then gets to show up and say it proves program synthesis is required for True Intelligence. Getting this score is extremely impressive but I don't assign more signal to it than any other benchmark with some thought to it.
- modeless 2y agoSolving ARC doesn't mean we have AGI. Also o3 presumably isn't doing program synthesis, seemingly proving Francois wrong on that front. (Not sure I believe the speculation about o3's internals in the link.) What I'm saying is the fact that as models are getting better at reasoning they are also scoring better on ARC proves that it is measuring something relating to reasoning. And nobody else has come up with a comparable benchmark that is so easy for humans and so hard for LLMs. Even today, let alone five years ago when ARC was released. ARC was visionary.
- hdjjhhvvhga 2y agoYour argumentation seems convincing but I'd like to offer a competitive narrative: any benchmark that is public becomes completely useless because companies optimize for it - especially AI that depends on piles of money and they need some proof they are developing. That's why I have some private benchmarks and I'm sorry to say that the transition from GTP4 to o1 wasn't unambiguously a step forward (in some tasks yes, in some not). On the other hand, private benchmarks are even less useful to the general public than the public ones, so we have to deal with what we have - but many of us just treat it as noise and don't give it much significance. Ultimately, the models should defend themselves by performing the tasks individual users want them to do.
- lossolo 2y ago> making the most interesting and challenging LLM benchmark so far. This[1] is currently the most challenging benchmark. I would like to see how O3 handles it, as O1 solved only 1%. 1. https://epoch.ai/frontiermath/the-benchmark https://epoch.ai/frontiermath/the-benchmark
- pynappo 2y agoApparently o3 scored about 25% https://youtu.be/SKBG1sqdyIU?t=4m40s https://youtu.be/SKBG1sqdyIU?t=4m40s
- FiberBundle 2y agoThis is actually the result that I find way more impressive. Elite mathematicians think these problems are challenging and thought they were years away from being solvable by AI.
- modeless 2y agoYou're right, I was wrong to say "most challenging" as there have been harder ones coming out recently. I think the correct statement would be "most challenging long-standing benchmark" as I don't believe any other test designed in 2019 has resisted progress for so long. FrontierMath is only a month old. And of course the real key feature of ARC is that it is easy for humans. FrontierMath is (intentionally) not.
- skywhopper 2y ago"The fact that scaled reasoning models are finally showing progress on ARC proves that what it measures really is relevant and important for reasoning." Not sure I understand how this follows. The fact that a certain type of model does well on a certain benchmark means that the benchmark is relevant for a real-world reasoning? That doesn't make sense.
- bagels 2y agoIt doesn't follow, faulty logic. The two are probably correlated though.
- munchler 2y agoIt shows objectively that the models are getting better at some form of reasoning, which is at least worth noting. Whether that improved reasoning is relevant for the real world is a different question.
- moffkalast 2y agoIt shows objectively that one model got better at this specific kind of weird puzzle that doesn't translate to anything because it is just a pointless pattern matching puzzle that can be trained for, just like anything else. In fact they specifically trained for it, they say so upfront. It's like the modern equivalent of saying "oh when AI solves chess it'll be as smart as a person, so it's a good benchmark" and we all know how that nonsense went.
- munchler 2y agoHmm, you could be right, but you could also be very wrong. Jury's still out, so the next few years will be interesting. Regarding the value of "pointless pattern matching" in particular, I would refer you to Douglas Hofstadter's discussion of Bongard problems starting on page 652 of _Godel, Escher, Bach_. Money quote: "I believe that the skill of solving Bongard [pattern recognition] problems lies very close to the core of 'pure' intelligence, if there is such a thing."
- jug 2y agoI liked the SimpleQA benchmark that measures hallucinations. OpenAI models did surprisingly poorly, even o1. In fact, it looks like OpenAI often does well on benchmarks by taking the shortcut to be more risk prone than both Anthropic and Google.
- zone411 2y agoIt's the least interesting benchmark for language models among all they've released, especially now that we already had a large jump in its best scores this year. It might be more useful as a multimodal reasoning task since it clearly involves visual elements, but with o3 already performing so well, this has proven unnecessary. ARC-AGI served a very specific purpose well: showcasing tasks where humans easily outperformed language models, so these simple puzzles had their uses. But tasks like proving math theorems or programming are far more impactful.
- versteegen 2y agoARC wasn't designed as a benchmark for LLMs, and it doesn't make much sense to compare them on it since it's the wrong modality. Even a MLM with image inputs can't be expected to do well, since they're nothing like 99.999% of the training data. The fact that even a text-only LLM can solve ARC problems with the proper framework is important, however.
- danielmarkbruce 2y agoHighly challenging for LLMs because it has nothing to do with language. LLMs and their training processes have all kinds of optimizations for language and how it's presented. This benchmark has done a wonderful job with marketing by picking a great name. It's largely irrelevant for LLMs despite the fact it's difficult. Consider how much of the model is just noise for a task like this given the low amount of information in each token and the high embedding dimensions used in LLMs.
- computerex 2y agoThe benchmark is designed to test for AGI and intelligence, specifically the ability to solve novel problems. If the hypothesis is that LLMs are the “computer” that drives the AGI then of course the benchmark is relevant in testing for AGI. I don’t think you understand the benchmark and its motivation. ARC AGI benchmark problems are extremely easy and simple for humans. But LLMs fail spectacularly at them. Why they fail is irrelevant, the fact they fail though means that we don’t have AGI.
- danielmarkbruce 2y ago> The benchmark is designed to test for AGI and intelligence, specifically the ability to solve novel problems. It's a bunch of visual puzzles. They aren't a test for AGI because it's not general. If models (or any other system for that matter) could solve it, we'd be saying "this is a stupid puzzle, it has no practical significance". It's a test of some sort of specific intelligence. On top of that, the vast majority of blind people would fail - are they not generally intelligent? The name is marketing hype. The benchmark could be called "random puzzles LLMs are not good at because they haven't been optimized for it because it's not valuable benchmark". Sure, it wasn't designed for LLMs, but throwing LLMs at it and saying "see?" is dumb. We can throw in benchmarks for tennis playing, chess playing, video game playing, car driving and a bajillion other things while we are at it.
- NateEag 2y agoAnd all that is kind of irrelevant, because if LLMs were human-level general intelligence, they would solve all these questions correctly without blinking. But they don't. Not even the best ones.
- adamgordonbell 2y agoThere is a benchmark, NovelQA, that LLMs don't dominate when it feels like they should. The benchmark is to read a novel and answer questions about it. LLMs are below human evaluation, as I last looked, but it doesn't get much attention. Once it is passed, I'd like to see one that is solving the mystery in a mystery book right before it's revealed. We'd need unpublished mystery novels to use for that benchmark, but I think it gets at what I think of as reasoning. https://novelqa.github.io/ https://novelqa.github.io/
- CamperBob2 2y agoDoes it work on short stories, but not novels? If so, then that's just a minor question of context length that should self-resolve over time.
- adamgordonbell 2y agoThe books fit in the current long context models, so it's not merely the context size constraint but the length is part of the issue, for sure.
- meta_x_ai 2y agoLooks like it's not updated for nearly a year and I'm guessing Gemini 2.0 Flash with 2m context will simply crush it
- adamgordonbell 2y agoThat's true. They don't have Claude 3.5 on there either. So maybe it's not relevant anymore, but I'm not sure. If so, let's move on to the murder mysteries or more complex literary analysis.
- rowanG077 2y agoBenchmark how? Is it good if the LLM can or can't solve it?
- loxias 2y ago
- deleted 2y ago[deleted]
- aimanbenbaha 2y agoBecause LLMs are on an off-ramp path towards AGI. A generally intelligent system can brute force its way with just memory. Once a model recognizes a weakness through reasoning with CoT when posed to a certain problem and gets the agency to adapt to solve that problem that's a precursor towards real AGI capability!
- justanotherjoe 2y agoi am confused cause this dataset is visual-based, and yet being used to measure 'LLM'. I feel like the visual nature of it was really the biggest hurdle to solving it.
- internet_points 2y ago> The fact that scaled reasoning models are finally showing progress on ARC proves that what it measures really is relevant and important for reasoning. One might also interpret that as "the fact that models which are studying to the test are getting better at the test" (Goodhart's law), not that they're actually reasoning.