18 ms·
Getting 50% (SoTA) on Arc-AGI with GPT-4o
- deleted 2y ago[deleted]
- deleted 2y ago[deleted]
- traject_ 2y agoWe don't actually know if it is SOTA, the previous SOTA solution also got around the same on the evaluation set.
- cma 2y agoYeah and GPT4o was potentially trained on this test set and if the tried to hold it out it was still likely trained on discussions of the problems.
- extr 2y agoVery cool. When GPT-4 first came out I tried some very naive approaches using JSON representations on the puzzles [0], [1]. GPT-4 did "okay", but in some cases it felt like it was falling for the classic LLM issue of saying all the right things but then then failing to grasp some critical bit of logic and missing the solution entirely. At the time I noticed that many of the ARC problems rely on visual-spatial priors that are "obvious" when viewing the grids, but become less so when transmuted to some other representation. Many of them rely on some kind of symmetry, counting, or the very human bias to assume a velocity or continued movement when seeing particular patterns. I had always thought maybe multimodality was key: the model needs to have similar priors around grounded physical spaces and movement to be able to do well. I'm not sure the OP really fleshes this line of thinking out, brute forcing python solutions is a very "non human" approach. [0] https://x.com/eatpraydiehard/status/1632671307254099968 https://x.com/eatpraydiehard/status/1632671307254099968 [1] https://x.com/eatpraydiehard/status/1632683214329479169 https://x.com/eatpraydiehard/status/1632683214329479169
- refulgentis 2y ago> brute forcing python solutions is a very "non human" approach. ARC-AGI has odd features that leave me flummoxed by the naming and the attendant prize money and hype. It is one singular task and frankly I strongly suspect someone could beat it within 30 days[1], in an unsatisfying way, as you note. There's so much alpha that can be pieced together from here, ex. the last couple Google papers use the 1M context to do *500-shot*, i.e. 500 question answer examples. IIRC most recent showed raising travelling-salesman problem solve rate from 3 to 35%. [1] I pre-registered this via a Twitter post, about 48 hours ago, i.e. before this result was announced.
- elicksaur 2y agoThe private test set has been available to crack for almost four years now. There was also a monetary prize competition run last year. In your opinion, what has changed that would accelerate a solution to the next 30 days?
- refulgentis 2y agoPrize money meant people would more cleverly strain the rule that "the private test set stays private, no GPT4o, Claude etc.", as shown by the TFA. This sort of idea would then be shared openly on new sites, creating more attempts. Fallout I did not anticipate was getting widespread attentional on general tech news sites, and then getting public comment from a prize co-founder confirming it was acceptable.
- elicksaur 2y agoIt seems like you don’t understand the rules of the competition. Entries don’t have access to the internet. The OP acknowledges in their post that this is not eligible for the prize. The HN comment from the prize co-founder specifically says the OP’s claims haven’t been scrutinized. (implicit: they won’t be for the prize set unless the OP submits with an open LLM implementation) There is a plan for a “public” leaderboard, but it currently has no entries, so we don’t actually know what the SOTA for the unrestrained version is. [1] The general idea - test time augmentation - is what the current private set SOTA uses. [2] Generating more examples via transforming the samples is not a new idea. Really, it seems like all the publicity has just gotten a bunch of armchair software architects coming up with 1-4 year-old ideas thinking they are geniuses. [1] https://arcprize.org/leaderboard https://arcprize.org/leaderboard [2] https://lab42.global/community-interview-jack-cole/ https://lab42.global/community-interview-jack-cole/
- rgbrgb 2y ago> 50% accuracy on the public test set for ARC-AGI by having GPT-4o Isn't the public test set public on github and therefore GPT-4o trained on it?
- bongodongobob 2y agoI keep seeing this comment all over the place. Just because something exists 1 time in the training data doesn't mean it can just regurgitate that. That's not how training works. An LLM is not a knowledge database.
- adroniser 2y agoAnd yet it doesn't rule out that it can't. See new york times lawsuit
- bongodongobob 2y agoFrom old pieces of articles that are quoted all over the internet? That's not surprising.
- ben_w 2y agoThat's still sufficient for both The Times and for it to be a potential problem in this case.
- spencerchubb 2y agoIt could exist many times. People can fork and clone the repo. People are likely to copy the examples and share them online.
- daemonologist 2y agoIn this case I don't think having seen the Arc set would help much in writing and selecting python scripts to solve the test cases. (Unless someone else has tried this approach before and _their_ results are in the training data.) It will be good to see the private set results though.
- bashfulpup 2y agoI looked at the website and have no idea how Arc is supposed to be AGI. Can someone explain?
- TheDudeMan 2y agoIt is necessary but not sufficient. If you can't do ARC, you aren't general enough. But even if you can do ARC, you still might not be general enough.
- Workaccount2 2y agoIt's also possible that you are an AGI and simply cannot pass ARC.
- TheDudeMan 2y agoHow so? If there is a task that humans can do but the AI cannot, I would not call it AGI. But that's just my definition.
- awwaiid 2y agoYeah but if my brother can't pass it, that doesn't mean he is NOT human.
- infgeoax 2y agoIsn't chatGPT already proven to be smarter than many of us in many ways?
- Jensson 2y agoCould he pass it if he was educated to do the task from birth? Human level intelligence includes being able to be educated, the ML models we have done so far can't be educated so have to match the level of educated humans to compare. General intelligence as we know it requires ability to receive education.
- 2y ago
- eigenvalue 2y agoThe Arc stuff just felt intuitively wrong as soon as I heard it. I don't find any of Chollet's critiques of LLMs to be convincing. It's almost as if he's being overly negative about them to make a point or something to push back against all the unbridled optimism. The problem is, the optimism really seems to be justified, and the rate of improvement of LLMs in the past 12 months has been nothing short of astonishing. So it's not at all surprising to me to see Arc already being mostly solved using existing models, just with different prompting techniques and some tool usage. At some point, the naysayers about LLMs are going to have to confront the problem that, if they are right about LLMs not really thinking/understanding/being sentient, then a very large percentage of people living today are also not thinking/understanding/sentient!
- HarHarVeryFunny 2y agoActually the solution being discussed here is the one that Chollet mentioned in his interview with Dwarkesh, and only bolsters his case. The LLM isn't doing the reasoning here, it's just pattern matching the before/after diff and generating thousands of Python programs. The actual reasoning is done by an agentic like loop wrapped around the LLM, as described in the linked blog.
- awwaiid 2y agoWhen you peer into the soul of the machine it delicately resolves to `while(1){...}`. All Hail The REPL.
- Smaug123 2y ago> a very large percentage of people living today are also not thinking/understanding/sentient This isn't that big a bullet to bite (https://www.lesswrong.com/posts/4AHXDwcGab5PhKhHT/humans-who-are-not-concentrating-are-not-general https://www.lesswrong.com/posts/4AHXDwcGab5PhKhHT/humans-who... comes from well before ChatGPT's launch), and I myself am inclined to bite it. System 1 alone does not a general intelligence make, although the article is extremely interesting in asking the question "is System 1 plus Python enough for a general intelligence?". But it's not a very relevant philosophical point, because Chollet's position is consistent with humans being obsoleted and/or driven extinct whether or not the LLMs are "general intelligences". His position is that training LLMs results in an ever-larger number of learned algorithms and no ability to construct new algorithms. This is consistent with the possibility that, after some threshold of size and training, the LLM has learned every algorithm it needs to supplant humans in (say) 99.9% of cases. (It would definitely be going out with a whimper rather than a bang, on that hypothesis, to be out-competed by something that _really is_ just a gigantic lookup table!)
- greatpostman 2y agoYou know you’re approaching AGI when creating benchmarks gets difficult. This is only just beginning
- elicksaur 2y agoAlternatively, society has no common understanding of what AGI means.
- Workaccount2 2y agoWhich is why the first "artificial" sentient beings will likely go through the wringer that humanity historically put other "sub-human" beings through.
- Slyfox33 2y agoBenchmarks being difficult to create has no connection to something being agi.
- mikeknoop 2y ago(ARC Prize co-founder here). Ryan's work is legitimately interesting and novel "LLM reasoning" research! The core idea: > get GPT-4o to generate around 8,000 python programs which attempt to implement the transformation, select a program which is right on all the examples (usually there are 3 examples), and then submit the output this function produces when applied to the additional test input(s) Roughly, he's implemented an outer loop and using 4o to sample reasoning traces/programs from training data and test. Hybrid DL + program synthesis approaches are solutions we'd love to see more of. A couple important notes: 1. this result is on the public eval set vs private set (ARC Prize $). 2. the current private set SOTA ~35% solution also performed ~50% on the public set. so this new result might be SOTA but hasn't been validated or scrutinized yet. All said, I do expect verified public set results to flow down to the private set over time. We'll be publishing all the SOTA scores and open source reproductions here once available: https://arcprize.org/leaderboard https://arcprize.org/leaderboard EDIT: also, congrats and kudos to Ryan for achieving this and putting the effort in to document and share his approach. we hope to inspire more frontier AI research sharing like this
- refibrillator 2y agoDo you have any perspectives to share on Ryan's observation of a potential scaling law for these tasks and his comment that "ARC-AGI will be one benchmark among many that just gets solved by scale"?
- mikeknoop 2y agoARC isn't perfect and I hope ARC is not the last AGI benchmark. I've spoken with a few other benchmark creators looking to emulate ARC's novelty in other domains, so I think we'll see more. The evolution of AGI benchmarks likely needs to evolve alongside the tech -- humans have to design these tasks today to ensure novelty but should expect that to shift. One core idea we've been advocating with ARC is that pure LLM scaling (parameters...) is insufficient to achieve AGI. Something new is needed. And OPs approach using a novel outer loop is one cool demonstration of this.
- Nimitz14 2y ago
- asperous 2y agoHaving tons of people employ human ingenuity to manipulate existing LLMs into passing this one benchmark kind of defeats the purpose of testing for "AGI". The author points this out as it's more of a pattern matching test. Though on the other hand figuring out which manipulations are effective does teach us something. And I think most problems boil down to pattern matching, creating a true, easily testable AGI test may be tough.
- janalsncm 2y agoPerhaps if we don’t know how to create an evaluation that can’t be “gamed” it tells us something about how special our intelligence really is?
- lucianbr 2y agoI don't know how to create a liver, or test one, so what does that say about my liver? Pretty much nothing.
- opdahl 2y agoWouldn’t the real AGI test be that an AI would be able to do what the author did here and write this blog post?
- atroche 2y agoYep, but a float is more useful than a bool for tracking progress, especially if you want to answer questions like "how soon can we expect (drivers/customer support staff/programmers) to lose their jobs?" Hard to find the right float but worth trying I think.
- opdahl 2y agoI agree, but it does seem a bit strange that you are allowed to "custom-fit" an AI program to solve a specific benchmark. Shouldn't there be some sort of rule that for something to be AGI it should work as "off-the-shelf" as possible?
- trott 2y agoFrançois Chollet says LLMs do not learn in-context. But Geoff Hinton says LLMs' few-shot learning compares quite favorably with people! https://www.youtube.com/watch?v=QWWgr2rN45o&t=46m20s https://www.youtube.com/watch?v=QWWgr2rN45o&t=46m20s The truth is in the middle, I think. They learn in-context, but not as well as humans. The approach in the article hides the unreliability of current LLMs by generating thousands of programs, and still the results aren't human-level. (This is impressive work though -- I'm not criticizing it.)
- hackpert 2y agoI'm not sure how to quantify how quickly or well humans learn in-context (if you know of any work on this I'd love to read it!) In general, there is too much fluff and confusion floating around about what these models are and are not capable of (regardless of the training mechanism.) I think more people need to read Song Mei's lovely slides[1] and related work by others. These slides are the best exposition I've found of neat ideas around ICL that researchers have been aware of for a while. [1] https://www.stat.berkeley.edu/~songmei/Presentation/Algorithm_approx_BSJC.pdf https://www.stat.berkeley.edu/~songmei/Presentation/Algorith...
- TheDudeMan 2y ago"Vision is an especially large weakness." But you can have GPT write code to reliably convert the image grid into a textual representation, right? And code to convert back to image and auto-verify.
- p1esk 2y agoGPT-4o might not have been trained on a sufficiently large amount of visual data to develop advanced spatial intelligence. Perhaps it needs to see a lot more images, or perhaps it needs to be trained differently (e.g. predict the next frame in a video). I suspect SORA has more spatial intelligence internally than 4o.
- deleted 2y ago[deleted]
- machiaweliczny 2y agoThe task provides json grid of numbers
- gibsonf1 2y agoIsn't 50% kind of a failing grade?
- p1esk 2y agoIt's only been a week since the million dollar prize was offered. Let's see what the SOTA is in a month.
- badrunaway 2y agoWhen we talk about system 2; is it possible that [generating large number of programs; evaluating them of the task; choosing top K outcomes; feeding it back to Neural net] can act as system 2 for a AGI? Isn't that how we think intelligently as well- by making lot of hypothesis internally and evaluating them - and updating our model?
- awwaiid 2y agoI think it's more like humans are a chaotic choir of subsystems all doing their thing and tossing up their directives until some sort of "win" happens or the volume is loud enough in some direction that it then gets reverse engineered into a "thought". But yes.
- badrunaway 2y agolike darwin selection between the subsystem approaches? Put in a lot of different kind of LLMs and let them play the same game inside with each other.. whosoever wins the simulation is allowed to externally present the approach... something like that?
- spencerchubb 2y agoPossibly I think we need those pieces, and also a piece for determining hypotheses in an efficient manner. Monte Carlo Tree Search could be that piece. Probabilistically choose a node to search, and then backpropagate the probabilities back to the root node.
- badrunaway 2y agoIntuitively I feel efficiency is the outcome of existing world model.. approach can look like yours - I don't see why there has not been efforts on scaling monte carlo tree search for extending the existing world model via tree search. My guess is that it would diverge to hallucinations too fast because it doesn't have a strong logical building block already
- bshanks 2y ago
- Imnimo 2y agoTo me the big take-aways here are: 1) Most of the heavy lifting is being done by search. We're talking about having the LLM generate thousands of candidate solutions, and they're mostly bad enough that "just pick the ones that get kinda close on the examples" is a meaningful operation. 2) More samples improves performance despite the fact that GPT-4o's vision is not capable of parsing the inputs. I'm curious how much performance would degrade if you shuffled the images passed to the model (but used the correct images when evaluating which candidates to keep). 3) It's definitely true that the LLM has to be giving you something more than random programs. At the very least, the LLM knows how to craft parsimonious programs that are more likely to be the solution. It may be that it's providing more than that, but it's not clear to me exactly how much information on the correct search space is coming from the hand-crafted examples in the prompt. Overall, the work to get this far is very impressive, but it doesn't really move the needle for me on whether GPT-4 can do ARC puzzles. It does, however, show me that search is surprisingly powerful on this task.
- bearjaws 2y agoSeems that Arc-AGI is more flawed rather than GPT-4o is more AGI. Maybe a AI version of Hanlons Razor. Never attribute to AGI what could be easily explained by being in the training set.
- bjornsing 2y agoCan we be sure GPT-4o hasn’t been trained on the public test set?
- atleastoptimal 2y agoI'll say what a lot of people seem to be denying. GPT-4 is an AGI, just a very bad one. Even GPT-1 was an AGI. There isn't a hard boundary between non AGI and AGI. A lot of people wish there was so they imagine absolutes regarding LLM's like "they cannot create anything new" or something like that. Just think: we consider humans a general intelligence, but obviously wouldn't consider an embryo or infant a general intelligence. So at what point does a human go from not generally intelligent to generally intelligent? And I don't mean an age or brain size, I mean suite of testable abilities. Intelligence is an ability that is naturally gradual and emerges over many domains. It is a collection of tools via which general abstractive principles can be applied, not a singular universally applicable ability to think in abstractions. GPT-4, compared to a human, is a very very small brain trained for the single purpose of textual thinking with some image capabilities. Claiming that ARC is the absolute market of general intelligence fails to account for the big picture of what intelligence is.
- theptip 2y agoThis seems to be so broad a definition as to no longer mean anything useful. People in general are interested in capabilities or economic impact, and GPT-2 cleared no notable thresholds in those regards. I prefer the exact opposite approach: let’s use a strict definition, and have levels to make it really explicit what we are talking about. Here is a good one: “Levels of AGI for Operationalizing Progress on the Path to AGI” https://arxiv.org/abs/2311.02462 https://arxiv.org/abs/2311.02462
- sigmoid10 2y agoPeople will never agree on this. We've known about the concept of intelligence for much, much longer than computers have been around and we still don't have a common definition or set of rules to check. That also makes things like consciousness and death pretty hard to define in medicine, leading to inconsistent rules across jurisdictions. For AGI in particular, I guarantee you that no matter which test gets beaten, the majority of humanity will always just move the goalpost and claim it's not "real" AGI because "reasons." Because the opposite would mean they have to admit that they are now the lesser intelligence on the planet.
- whiplash451 2y agoThe article jumps to the conclusion that "Given that current LLMs can perform decently well on ARC-AGI" after having used multiple hand-crafted tricks to get to these results, including "I also did a small amount of iteration on a 100 problem subset of the public test set" which is hidden in the middle of the article and not mentioned in the bullet list at the top. Adding the close-to ad-hominem attack on Francois Chollet with the comics at the beginning (Francois never claimed to be a neuro-symbolic believer), this work does a significant disservice to the community.
- z7 2y ago>Francois never claimed to be a neuro-symbolic believer His response: "This has been the most promising branch of approaches so far -- leveraging a LLM to help with discrete program search, by using the LLM as a way to sample programs or branching decisions. This is exactly what neurosymbolic AI is, for the record..." "Deep learning-guided discrete search over program space is the approach I've been advocating, yes... there are many different flavors it could take though. This is one of them (perhaps the simplest one)." https://x.com/fchollet/status/1802773156341641480 https://x.com/fchollet/status/1802773156341641480
- YeGoblynQueenne 2y agoThat kind of neuro-symbolic AI is a bit like British cuisine: place two different things next to each other in the same plate, like bangers and mash, and call it "a dish". Nope. This is neurosymbolic AI: Abductive Knowledge Induction From Raw Data https://www.doc.ic.ac.uk/~shm/Papers/abdmetarawIJCAI.pdf https://www.doc.ic.ac.uk/~shm/Papers/abdmetarawIJCAI.pdf That's a symbolic learning engine trained in tandem with a neural net. The symbolic engine is learning to label examples for the neural net that learns to label examples for the symbolic engine. I call that cooking! (Full disclosure: the authors of the paper are my thesis advisor and a dear colleague).
- deleted 2y ago[deleted]
- whiplash451 2y ago
- comfortabledoug 2y agoI'm glad someone else finally said it, those born blind cannot possibly have AGI! /sarcasm :D
- nadam 2y agoAmazing work, prompt engineering at its finest. One future direction for Arc AGI could be to use not Python, but a much more concise programming language that is more suited for brute-force methods like genetic mutations. The problem would be of course to train an LLM that is proficient enough in such a language. I am thinking about stack based languages. For this competition I would develop a careful bit-level encoding of a variant of the 'Joy' programming language. (https://en.wikipedia.org/wiki/Joy_(programming_language) https://en.wikipedia.org/wiki/Joy_(programming_language)) It would be a considerable effort though which I don't have time for, hence I post this idea publicly. A promising direction is a mix of things in my opinion: Special stack-based concise language, consulting LLMs like the OP did, and genetic algorithms combined.
- fire_lake 2y agoS expressions are great for making valid syntax more likely - an old trick from Genetic Programming.
- YeGoblynQueenne 2y ago>> Claim 1 seems likely true to me for a reasonable notion of “learning”. I think François Chollet agrees here. Most of my doubts about this claim are concerns that you can basically brute force ARC-AGI without interestingly doing learning (e.g. brute-force search over some sort of DSL or training on a huge array of very similar problems). These concerns apply much less to the kind of approach I used The approach described in the article is exactly "brute-force search over some sort of DSL". The "DSL" is a model of Python syntax that GPT-4o has learned after training on the entire internet. This "DSL" is locked up in the black box of GPT-4o's weights, but just because no-one can see it, it doesn't mean it's not there; and we can see GPT-4o generating Python programs, so we know it is there, even if we don't know what it looks like. That DSL may not be "domain specific" in the sense of being specifically tailored to solve ARC-AGI tasks, or any other particular task, but it is "domain specific" in the sense of generating Python programs for some subset of all possible Python programs that includes programs that can solve some ARC-AGI tasks. That's a very broad category, but that's why it over-generates so much: it needs to draw 8k samples total until one works for just 50% of the public eval set.
- llama_drama 2y ago[dead]
- htrp 2y agoThe expectation is that you'll have to have dynamically generated benchmarks with better eval at some point given the potential for brute forcing the private validation set.
- sparsely 2y agoYou can have a go at the problems here: https://arcprize.org/play?task=00576224 https://arcprize.org/play?task=00576224 None of them are terribly hard but some aren't trivial either, a couple took me a bit of thinking to work out. By far the most tedious part is inputting the result (I didn't bother after the first) which is definitely something AI is better at!
- killerstorm 2y agoFWIW GPT-4 is able to generate a plan very similar to one in the article: also involves feature extraction, program synthesis, iterative refinement. https://chatgpt.com/share/2fde1db5-00cf-404d-9ae5-192aa5ac909e https://chatgpt.com/share/2fde1db5-00cf-404d-9ae5-192aa5ac90... So it's pretty close to being able to plan solution completely on its own. It's just rather bad at coding and visual inputs, so it doesn't know what it doesn't know.
- uptownfunk 2y agoArc agi is a small stepping stone to agi but is not agi. Program search mimics what humans do to a certain extent but not in entirety. A more general world model and reference will be required for agi.
- cchance 2y agoLOL i looked at that first complex test sample and closed the page, it made my brain hurt.
- machiaweliczny 2y agoThis challenge looks quite solvable but it's relies on physics understanding and it's has a lot of human/world priors in sense of space understanding and object boundaries. Seems like it relies on identification of objects and then mapping them somehow. Most of the cases so far that I've seen are based on some transformation or relation between the objects. So far it seems like some search among common transformatiosn and relations could solve it. Plus some heuristics/computation for counting order, wholeness(boundary) or pattern. IMO it can be solved by search of programs that combine these + some LLM to guide heuristics most likely. The only hard one was applied noise or one testing understanding of "gravity". Did anyone test human baseline for this?