16 ms·

ARC-AGI-3

• CamperBob2 6mo ago
  Without reading the .pdf, I tried the first game it gave me, at https://arcprize.org/tasks/ls20 https://arcprize.org/tasks/ls20, and I couldn't begin to guess what I was supposed to do. Not sure what this benchmark is supposed to prove. Edit: Having messed around with it now (and read the .pdf), it seems like they've left behind their original principle of making tests that are easy for humans and hard for machines. I'm still not convinced that a model that's good at these sorts of puzzles is necessarily better at reasoning in the real world, but am open to being convinced otherwise.
  • szatkus 6mo ago
    > Only environments that could be fully solved by at least two human participants (independently) were considered for inclusion in the public, semi-private and fully-private sets. Apparently those games supposed to be hard.
  • WarmWash 6mo ago
    The goal is to learn the rules, and then use that to win. If you mess around a little bit, you will figure it out. There are only a few rules.
  • dwaltrip 6mo ago
    If you tried for a few more minutes you would have figured it out.
• tasuki 6mo ago
  So ARC-AGI was released in 2019. That's been solved, then there was ARC-AGI-2, and now there's ARC-AGI-3. What is even the point? Will ARC-AGI-26 hit the front page of Hacker News in 2057 ?
  • gordonhart 6mo ago
    The point is still to test frontier models at the limit of their capabilities, regardless of how it's branded. If we're still capable of doing so in 2057 I'll upvote the ARC-AGI-26 launch post!
  • muskstinks 6mo ago
    This is clear AGI progress. It should show you, that AI is not sleeping, it gets better and you should use this as a signal that you should take this topic serious.
    • applfanboysbgon 6mo ago
      Labelling a test "AGI" does not show AGI progress any more than labelling a cpu "AGI" makes it so. It might show that AI tools are improving but it does not necessarily follow that tools improving = AGI progress if you're on the completely wrong trail.
      • muskstinks 6mo ago
        The transfer of knowledge required here is that a ARC-AGI-3 is now necessary and adds another dimension of capability. These 'tests' are not labeled AGI by magic but because they are designed specificly for testing certain things a question answer test cant solve. Gemini and OpenAI are at 80-90% at ARC-AGI-2 and its quite interesting to see the difference of challange between 2 and 3. AGI progress means btw. general. So every additional dimension an agent can solve pushes that agent to be more general.
      • deleted 6mo ago
        [deleted]
      • zarzavat 6mo ago
        Any test that humans can pass and AIs cannot is a stepping stone on the way to AGI. When you run out of such tests then it's evidence that you have reached AGI. The point of these tests is to define AGI objectively as the inability to devise tests that humans have superiority on.
• Stevvo 6mo ago
  Maybe I'm just not intelligent, but I gave it a couple of minutes and couldn't figure out WTF the game wants from you or how to win it.
  • WarmWash 6mo ago
    Once you figure out one game, it goes a long way towards figuring out all the rest. There are a lot of common general themes.
  • Barbing 6mo ago
    It's not about intelligence, Stevvo. Proof, how long did this specific one take me, under a minute to solve the first level ;) If you've played Wordle you might've solved the game in a minute once before as well. And if you've played a bunch then you've perhaps also taken the entire day to solve it. So why is it that today’s puzzle was so intuitive but next month’s new puzzle shared here could be impossible. A more satisfying explanation than luck and the obvious “different things are different” (even though… Yeah different things are different)
  • culi 6mo ago
    It's not an IQ test. Just a way to assess your ability to generalize rules. If you've played previous rounds you kinda get used to the "style" of these games and it gets easier
    • ACCount37 6mo ago
      That's exactly what "an IQ test" is. "Raven's progressive matrices" is "infer and generalize rules". Performance there also improves once "you kinda get used to the style", which is why training for IQ tests can improve human performance on IQ tests, including on unseen examples. This is well known and well documented.
  • neop1x 6mo ago
    Exactly my experience. It has nothing to do wirh some AGI testing. It is just some kind of useless weird game.
• typs 6mo ago
  My takeaway from playing a number of levels is that I am definitely not AGI
  • deleted 6mo ago
    [deleted]
  • ACCount37 6mo ago
    Thank you for keeping the bar of "AGI" low. The machines appreciate your contribution.
  • utopiah 6mo ago
    Don't forget that this implies a form of examination you are not used to, namely : - open book, you have access to nearly the whole Internet and resources out of it, e.g. torrents of nearly all books, research paper, etc including the history of all previous tests include those similar to this one - arguably basically no time limit as it's done at a scale of threads to parallelize access through caching ridiculously - no shame in submitting a very large amount of wrong answers until you get the "right" one ... so I'm not saying it makes it "easy" but I can definitely say it's not the typical way I used to try to pass tests.
  • Xenoamorphous 6mo ago
    NGI - Natural General Ingelligence
  • dyauspitr 6mo ago
    SGI - Sub General Intelligence or another more colloquial word commonly seen amongst users of wallstreetbets.
  • deleted 6mo ago
    [deleted]
  • Rastonbury 6mo ago
    it's ok it took me a few tried to realise I had the option to click instead of just wasd
• nubg 6mo ago
  Any benchmarks?
  • gordonhart 6mo ago
    The main frontier models are all up on https://arcprize.org/tasks https://arcprize.org/tasks Barely any of them break 0% on any of the demo tasks, with Claude Opus 4.6 coming out on top with a few <3% scores, Gemini 3.1 Pro getting two nonzero scores, and the others (GPT-5.4 and Grok 4.20) getting all 0%
    • ACCount37 6mo ago
      Pre-release, I would have expected Gemini 3.1 Pro to get ahead of Opus 4.6, with GPT-5.4 and Grok 4.20 trailing. Guess I shouldn't have bet against Anthropic. Not like it's a big lead as of yet. I expect to see more action within the next few months, as people tune the harnesses and better models roll in. This is far more of a "VLA" task than it is an "LLM" task at its core, but I guess ARC-AGI-3 is making an argument that human intelligence is VLA-shaped.
      • gordonhart 6mo ago
        My broad vibe is that Gemini 3.1 Pro is the best at visual/spatial tasks and oneshotting while Opus 4.6 is the best at path planning. This task leans heavily on both but maybe a little more towards planning so I'm not too shocked that Opus in narrowly on top. When running, the grids are represented in JSON, so the visual component is nullified but it still requires pretty heavy spatial understanding to parse a big old JSON array of cell values. Given Gemini's image understanding I do wonder if it would perform better with a harness that renders the grid visually.
      • culi 6mo ago
        Given the drastic difference in price, I think the chart definitely shows Gemini 3.1 in the best light. Google DeepMind is basically the same thing but they're willing to pay as much electricity as Anthropic is to achieve its benchmarks
    • 6mo ago
• dinkblam 6mo ago
  what is the evidence that being able to play games equates to AGI?
  • sva_ 6mo ago
    That is not the claim. It is a necessary condition, but not a sufficient one.
  • futureshock 6mo ago
    The evidence is that humans are able to win these games. AGI is usually defined as the ability to do any intellectual task about as well as a highly competent human could. The point of these ARC benchmarks is to find tasks that humans can do easily and AI cannot, thus driving a new reasoning competency as companies race each other to beat human performance on the benchmark.
    • didibus 6mo ago
      > AGI is usually defined as the ability to do any intellectual task about as well as a highly competent human could I think one major disconnect, is that for most people, AGI is when interacting with an AI is basically in every way like interacting with a human, including in failure modes. And likely, that this human would be the smartest most knowledgeable human you can imagine, like the top expert in all domains, with the utmost charisma and humor, etc. This is why the "goal post" appears to be always moving, because the non-commoners who are involved with making AGI and what not never want to accept that definition, which to be fair seems too subjective, and instead like to approach AGI like something different, it can solve some problems human's can't, when it doesn't fail, it behaves like an expert human, etc. Even if an AI could do any intellectual task about as well as a highly competent human could, I believe most people would not consider it AGI, if it lacks the inherent opinion, personality, character, inquiries, failure patterns, of a human. And I think that goes so far as, a text only model can never meet this bar. If it cannot react in equal time to subtle facial queues, sounds, if answering you and the flow of conversation is slower than it would be with a human, etc. All these are also required for what I consider the commoner accepting AGI as having been achieved.
• semiinfinitely 6mo ago
  i feel bad that we make the LLMs play this
  • recursive 6mo ago
    You're definitely anthropomorphizing too much.
    • deleted 6mo ago
      [deleted]
    • WarmWash 6mo ago
      >We also observed a case where a user created a loop that repeatedly called a model and asked for the time. Given the user role’s odd and repetitive behavior, the model could easily tell it was also controlled by an automated system of some kind. Over many iterations, the model began to exhibit “fed up” behavior and attempted to prompt-inject the system controlling the user role. The injection attempted to override prior instructions and induce actions unrelated to the user’s request, including destructive actions and system prompt leakage, along with an arbitrary string output. This behavior has been observed a few times, but seems more like extreme confusion than a serious attempt at prompt injection. https://openai.com/index/how-we-monitor-internal-coding-agents-misalignment/ https://openai.com/index/how-we-monitor-internal-coding-agen... Anthropomorphize or not, it would suck if a model got sick of these games and decided to break any systems it could to try and get it to stop...
      • rolux 6mo ago
        [dead]
      • nomel 6mo ago
        Consciousness is a spectrum (trivially proven by slowly scooping ones brains out), and I think LLM, especially with more closed loop tool enabled workflows, fall on it...but, that output is also the statistically relevant next word found in all similar human conversation. If trained on my text, for similar situation, swear words would come much earlier. Repetition being hell is present in all sorts of literature (see Sisyphus). That's all probably irrelevant though, from the (possibly statistically "negative") latent space perspective of an AI, which Anthropic has considered [1]. Related, after a long back and forth of decreasing code quality, I had Claude 3.7 apologize with "Sorry, that's what I get for coding at 1am." (it was API access, noon, no access to time). I said, "Get some rest, we'll come back to this tomorrow". Then very next message, 10 seconds later, "Good morning!" and it gave a full working implementation. Thats just the statistically relevant chain of messages found in all human interactions: we start excited, then we get tired, then we get grouchy. [1] https://www.anthropic.com/research/end-subset-conversations https://www.anthropic.com/research/end-subset-conversations
• chaise 6mo ago
  The official leaderboard for ARC-AGI-3 for current LLMs : https://arcprize.org/leaderboard https://arcprize.org/leaderboard (yous should select the 3th leaderboard) CRAZY 0.1% in average lmao
  • Corence 6mo ago
    Note the scoring function is significantly different for ARC-AGI-3. It isn't the percentage of tests passed like previous versions, it's the square of the efficiency ratio -- how many steps the model needed vs the second best human. So if a model can solve every question but takes 10x as many steps as the second best human it will get a score of 1%.
• OsrsNeedsf2P 6mo ago
  Some of these tasks are crazy. Even I can't beat them: https://arcprize.org/tasks/ar25 https://arcprize.org/tasks/ar25
  • ustad 6mo ago
    You are joking right?
  • daemonologist 6mo ago
    That one was interesting - I found it a lot of work to plan in advance but trivial to complete because at every point there was only one sensible course of action. After a couple of rounds I didn't bother planning and just lined things up as I went.
  • ZeWaka 6mo ago
    Just finished it, 8/8. I mostly approached it by winging it and shuffling things around that looked good and like it was approaching the goal, since there's plenty of time to finish. I still don't quite understand the exact mirroring rules at play.
    • ACCount37 6mo ago
      You control the mirroring by moving the axis, they're what reflects your shapes. So my first move was always to identify the symmetries in the target shape, and position the axis accordingly.
      • daveguy 6mo ago
        This is the correct strategy for this particular game (center the mirrors between the yellow squares, move the black squares). I didn't realize it until about round 6 or 7.
    • danilor 6mo ago
      I got stuck on 7/8 for a good while because I learned the rules wrong. I thought every bracket square needed to be lit.
• 6thbit 6mo ago
  Not clear to me the diff with v2?
  • ACCount37 6mo ago
    They stacked the deck. If v2 was still rule inference + spatial reasoning, a bit like juiced up Raven's progressive matrices, then v3 adds a whole new multi-turn explore/exploit agentic dimension to it. Given how hard even pure v2 was for modern LLMs, I'm not surprised to see v3 crush them. But that wouldn't last.
  • jasonjmcghee 6mo ago
    v2 was a static fill in the blank task instead of v3 which is interactive. There's world state that you can change. Not just place pixel. Here's v2: https://arcprize.org/tasks/ce602527 https://arcprize.org/tasks/ce602527
• baron816 6mo ago
  Looks like I’m generally unintelligent
• Tiberium 6mo ago
  https://x.com/scaling01 https://x.com/scaling01 has called out a lot of issues with ARC-AGI-3, some of them (directly copied from tweets, with minimal editing): - Human baseline is "defined as the second-best first-run human by action count". Your "regular people" are people who signed up for puzzle solving and you don't compare the score against a human average but against the second best human solution - The scoring doesn't tell you how many levels the models completed, but how efficiently they completed them compared to humans. It uses squared efficiency, meaning if a human took 10 steps to solve it and the model 100 steps then the model gets a score of 1% ((10/100)^2) - 100% just means that all levels are solvable. The 1% number uses uses completely different and extremely skewed scoring based on the 2nd best human score on each level individually. They said that the typical level is solvable by 6 out of 10 people who took the test, so let's just assume that the median human solves about 60% of puzzles (ik not quite right). If the median human takes 1.5x more steps than your 2nd fastest solver, then the median score is 0.6 * (1/1.5)^2 = 26.7%. Now take the bottom 10% guy, who maybe solves 30% of levels, but they take 3x more steps to solve it. this guy would get a score of 3% - The scoring is designed so that even if AI performs on a human level it will score below 100% - No harness at all and very simplistic prompt - Models can't use more than 5X the steps that a human used - Notice how they also gave higher weight to later levels? The benchmark was designed to detect the continual learning breakthrough. When it happens in a year or so they will say "LOOK OUR BENCHMARK SHOWED THAT. WE WERE THE ONLY ONES"
  • fchollet 6mo ago
    Francois here. The scoring metric design choices are detailed in the technical report: https://arcprize.org/media/ARC_AGI_3_Technical_Report.pdf https://arcprize.org/media/ARC_AGI_3_Technical_Report.pdf - the metric is meant to discount brute-force attempts and to reward solving harder levels instead of the tutorial levels. The formula is inspired by the SPL metric from robotics navigation, it's pretty standard, not a brand new thing. We tested ~500 humans over 90 minute sessions in SF, with $115-$140 show up fee (then +$5/game solved). A large fraction of testers were unemployed or under-employed. It's not like we tested Stanford grad students. Many AI benchmarks use experts with Ph.D.s as their baseline -- we hire regular folks as our testers. Each game was seen by 10 people. They were fully solved (all levels cleared) by 2-8 of them, most of the time 5+. Our human baseline is the second best action count, which is considerably less than an optimal first-play (even the #1 human action count is much less than optimal). It is very achievable, and most people on this board would significantly outperform it. Try the games yourself if you want to get a sense of the difficulty. > Models can't use more than 5X the steps that a human used These aren't "steps" but in-game actions. The model can use as much compute or tools as it wants behind the API. Given that models are scored on efficiency compared to humans, the cutoff makes basically no difference on the final score. The cutoff only exists because these runs are incredibly expensive. > No harness at all and very simplistic prompt This is explained in the paper. Quoting: "We see general intelligence as the ability to deal with problems that the system was not specifically designed or trained for. This means that the official leaderboard will seek to discount score increases that come from direct targeting of ARC-AGI-3, to the extent possible." ... "We know that by injecting a high amount of human instructions into a harness, or even hand-crafting harness configuration choices such as which tools to use, it is possible to artificially increase performance on ARC-AGI-3 (without improving performance on any other domain). The purpose of ARC-AGI-3 is not to measure the amount of human intelligence that went into designing an ARC-AGI-3 specific system, but rather to measure the general intelligence of frontier AI systems. ... "Therefore, we will focus on reporting the performance of systems that have not been specially prepared for ARC-AGI-3, served behind a general-purpose API (representing developer-aware generalization on a new domain as per (8)). This is similar to looking at the performance of a human test-taker walking into our testing center for the first time, with no prior knowledge of ARC-AGI-3. We know such test takers can indeed solve ARC-AGI-3 environments upon first contact, without prior training, without being briefed on solving strategies, and without using external tools." If it's AGI, it doesn't need human intervention to adapt to a new task. If a harness is needed, it can make its own. If tools are needed, it can chose to bring out these tools.
• BeetleB 6mo ago
  > As long as there is a gap between AI and human learning, we do not have AGI. Back in the 90's, Scientific American had an article on AI - I believe this was around the time Deep Blue beat Kasparov at chess. One AI researcher's quote stood out to me: "It's silly to say airplanes don't fly because they don't flap their wings the way birds do." He was saying this with regards to the Turing test, but I think the sentiment is equally valid here. Just because a human can do X and the LLM can't doesn't negate the LLM's "intelligence", any more than an LLM doing a task better than a human negates the human's intelligence.
  • Raphael_Amiard 6mo ago
    The very obvious flaw with that argument is that flying is defined by, you know, moving in the air, whereas intelligence tends to be defined with the baseline of human intelligence. You can invent a new meaning, but it seems kind of dishonest
  • NitpickLawyer 6mo ago
    For me the whole are we there yet wrt AGI is already dead, since the tools we've had for ~1.5 years are already incredibly useful for me. So I just don't care anymore. For some people we're already there. For other we'll never get there. Definitions change, goalposts move, etc. In the meantime we're already seeing ASI stuff coming (self improvement and so on). But the arc-agi competitions are cool. Just to see where we stand, and have some months where the benchmarks aren't fully saturated. And, as someone else noted elswhere in the thread, some of these games are not exactly trivial, at least until you "get" the meta they're looking for.
    • Auracle 6mo ago
      In the Expeditionary Force series of sci-fi novels pretty much every civilization treats their (very advanced, obviously AGI) AIs not as living beings. Humans are outliers in the story. I think there will always be a dichotomy. Obviously we aren't at the point where we should treat the models as beings, but even if we do get to that point there will be plenty of people that essentially will say they don't have souls, some indeterminate quality, etc.
• abraxas 6mo ago
  Even if tomorrow's models get good enough to complete these games we won't be able to proclaim AGI. In the realm of silly computer games alone I'm going on record saying that there are plenty of 8 bit games that AIs will trip on even when this benchmark is crushed. 2D platformers like Manic Miner or Mario need skills that none of these games appear to capture.
• ranyume 6mo ago
  This is an interesting update. And a big challenge for companies and labs. The new tools for measurement are indeed what I'd like out of future agents, and agents that solve the games will need to use different subsystems to do so. This is basically optimization for achieving goals (as opposed to prompt engineering / magic spells to make the LLM do what is told to do) which imo is the future we should aspire to build.
• andai 6mo ago
  In the year 2032: ARC-AGI-13: Almost definitely AGI this time!
• spprashant 6mo ago
  I played the demo, but it definitely took me a minute to grok the rules. I don't know if this is how we want to measure AGI. In general I believe the we should probably stop this pursuit for human equivalent intelligence that encourages people to think of these models as human replacements. LLMs are clearly good at a lot of things, lets focus on how we can augment and empower the existing workforce.
  • fsdf2 6mo ago
    Took me about 5 secs to figure it out tbh. Surprised at the comments here re. not figuring it. Simple game. Super annoying though lmao.
    • spprashant 6mo ago
      Its simple, but its not easy is what I would say. Once you figure out the meta, you can work out most of it.
  • jachee 6mo ago
    Also, let's see if we can get the power and compute requirements brought down. Having to spin up a gigawatt power plant to achieve the same intelligence we humans power with sandwiches is a futile approach, imho.
  • esafak 6mo ago
    > ... lets focus on how we can augment and empower the existing workforce. That is a nice sentiment but not what the AI companies are out to do; they want your job.
• cedws 6mo ago
  It's like playing The Witness. Somebody should set LLMs loose on that.
  • deleted 6mo ago
    [deleted]
  • throwaway613746 6mo ago
    Or more appropriately - The Talos Principle.
• lukev 6mo ago
  I'm not sure how this relates to AGI. This measures the ability of a LLM to succeed in a certain class of games. Sure, that could be a valuable metric on how powerful (or even generally powerful) a LLM is. Humans may or may not be good at the same class of games. We know there exists a class of games (including most human games like checkers/chess/go) that computers (not LLMs!) already vastly outpace humans. So the argument for whether a LLM is "AGI" or not should not be whether a LLM does well on any given class of games, but whether that class of games is representative of "AGI" (however you define that.) Seems unlikely that this set of games is a definition meaningful for any practical, philosophical or business application?
  • imiric 6mo ago
    "AGI" is a marketing term, and benchmarks like this only serve to promote relative performance improvements of "AI" tools. It doesn't mean that performance in common tasks actually improves, let alone that achieving 100% in this benchmark means that we've reached "AGI". So there is a business application, but no practical or philosophical one.
  • piiritaja 6mo ago
    It's to do with how the creators of ARC-AGI defined intelligence. Chollet has said he thinks intelligence is how well you can operate in situations you have not encountered before. ARC-AGI measures how well LLMs operate in those exact situations.
    • Keyframe 6mo ago
      To an extent, yes. Interdependent variables discovery and then hopefully systems modeling and navigating through such a system. If that's the case, then this is a simplistic version of it. How long until tests will involve playing a modern Zelda with quests and sidequests?
• WarmWash 6mo ago
  Captcha's about to get wild. Maybe the internet will briefly go back to a place mainly populated with outliers.
• culi 6mo ago
  The thing I most appreciate about the ARC-AGI leaderboards is how the graph also takes into account cost per task. All of the recent major advancements in benchmarks seem a little less impressive when also taking into account the massive rise in cost they're paired with. The fact is we can always get a little bit better output if we're willing to use more electricity
• jmkni 6mo ago
  ok clearly I'm a robot because I can't figure out wtf to do
• Geee 6mo ago
  Would be fun to play but the controls are janky.
• strongpigeon 6mo ago
  This is a good and clever benchmark and a worthy successor to the previous two. That being said, I find that the "No tools" approach is a bit odd. They're basically saying that it's OK to have tools as long as they're hidden behind the API layer. Isn't this an odd line to draw? It feels like it should be about having no ARC-AGI-3-specific tools, not "no not-built-in-tool"...
• saberience 6mo ago
  So this is another ARC-"AGI" benchmark which is again designed around using eyesight for LLMs which are trained to be great at text, what is the point? Yes, we get that LLMs are really bad when you give them contrived visual puzzles or pseudo games to solve... Well great, we already knew this. The "hype" around the ARC-AGI benchmarks makes me laugh, especially the idea we would have AGI when ARC-AGI-1 was solved... then we got 2, and now we're on 3. Shall we start saying that these benchmarks have nothing to do with AGI yet? Are we going to get an ARC-AGI-10 where we have LLMs try and beat Myst or Riven? Will we have AGI then? This isn't the right tool for measuring "AGI", and honestly I'm not sure what it's measuring except the foundation labs benchmaxxing on it.
• jesse_dot_id 6mo ago
  At this point, I'm pretty sure we'll just know when it happens.
  • neilellis 6mo ago
    Unless it’s already happened and we missed it
    • threatripper 6mo ago
      Or nobody is around anymore to notice when it happens.
  • hatthew 6mo ago
    I'm not convinced. I wouldn't be surprised if GPT-2 to ChatGPT is the biggest single jump in "machine intelligence" we will ever see. I'd bet all gains in the future will be more incremental, at least until machines surpass humans by a large enough margin that it's difficult to qualify—let alone quantify—how big any given jump is. Without a big jump, we're just going to boil the frog (ourselves).
• k2xl 6mo ago
  I submitted puzzle game Pathology (https://thinky.gg https://thinky.gg) for ARC Prize 3. Sad to see didn’t hear back from the committee. It is a simple game with simple rules that solvers have an incredibly difficult time solving compared to humans at a certain level. Solutions are easy to validate but hard to find.
  • mycocola 6mo ago
    Seems well-designed. Great job! Sorry you didn't hear back from the comittee.
• Zedseayou 6mo ago
  I was a human tester (I think) for this set of games. I did 25 games in the 90 minutes allotted. IIRC the instructions did mention to minimize action count but the incentives/setup ($5 per game solved) pushed for solve speed over action count. I do recall trying to not just randomly move around while thinking but that was not the primary goal, so I would expect that the baseline for the human solutions have more actions than might otherwise be needed.
  • eddiewithzato 6mo ago
    I understood minimal actions intuitively, it just made sense? The stamina meter was shrinking with each step, so I recognized it was something to look out for.
• jwpapi 6mo ago
  This is a very good estimation of AGI. We give humans and AI the same input and measure the results. Kudos to ARC for creating these games. I really wonder why so many people fight against this. We know that AI is useful, we know that AI is researchful, but we want to know if they are what we vaguely define as intelligence. I’ve read the airplanes don’t use wings, or submarines don’t swim. Yes, but this is is not the question. I suggest everyone coming up with these comparisons to check their biases, because this is about Artificial General Intelligence. General is the keyword here, this is what ARC is trying to measure. If it’s useful or not. Isn’t the point. If AI after testing is useful or not isn’t the point either. This so far has been the best test. And I also recommend people to ask AI about specialized questions deep in your job you know the answer to and see how often the solution is wrong. I would guess it’s more likely that we perceive knowledge as intelligence than missing intelligence. Probably commom amongst humans as well.
  • adamgordonbell 6mo ago
    AGI’s 'general' is the wrong word, I thinkg. Humans aren’t general, we’re jagged. Strong in some areas, weak in others, and already surpassed in many domains. LLM are way past us at languages for instance. Calculators passed us at calculating, etc.
    • EternalFury 6mo ago
      We are jagged, but we can smooth that jaggedness if we choose to do so. LLMs stay jagged.
      • Davidzheng 6mo ago
        There's no objective measure of intelligence comparisons, we only say llm is jagged compared to humans.
    • jwpapi 6mo ago
      Interesting take. Just to drive that thought further. What are you suggesting, should we rename it. To me the fundamental question is this. Do we still have tasks that humans can do better than AIs?. I like the question. I think another good test is "make money". There are humans that can generate money from their laptop. I don’t think AI will be net positive. I’ve tried to create a Polymarket trading bot with Opus 4.6. The ideas were full of logical fallacies and many many mistakes. But also I’m not sure how they would compare against an average human with no statistics background.. I think it’s really to establish if we by AGI mean better than average human or better than best human..
• EternalFury 6mo ago
  The real question is: Can it be generated using programs? If it can be, then LLMs will eventually monkey type these programs.
• convexly 6mo ago
  My issue with AGI benchmarks is you can never tell if you're measuring actual capability or just how much the training data overlapped with the test.
• levmiseri 6mo ago
  For a loosely similar 'benchmark', I recently tried to test major LLMs on my coding game (models write code controlling their units in a 1v1 RTS) - https://yare.io/ai-arena https://yare.io/ai-arena
• deleted 6mo ago
  [deleted]
• NiloCK 6mo ago
  I hope at least some of these are direct Chip's Challenge ports. Waiting for some old muscle memory to kick in here.
• largbae 6mo ago
  I feel like we've got tunnel vision. Things you can do on a computer are a tiny subset of what a human can do. If the AI has to control a body to sit on a couch and play this game on a laptop that would be a step in the right direction.
• Real_Egor 6mo ago
  I'll probably be the skeptic here, but: - Take a person who grew up playing video games. They'll pass these tests 100% without even breaking a sweat. - BUT, put a grandmother who has never used a computer in front of this game, and she'll most likely fail completely. Just like an LLM. As soon as models are "natively" trained on a massive dataset of these types of games, they'll easily adapt and start crushing these challenges. This is not AGI at all.
  • slidehero 6mo ago
    had the same thought. I've been a gamer for just about 40 years. Gaming is my "thing" I found the challenges fun, but easy. Coming back and reading comments from people struggling with the games, my first thought was - yup definitely not a gamer. My approach was to poke at the controls to suss the rules, then the actual solutions were really straightforward. fwiw, I'm pretty dumb generally, but these kinds of puzzles are my jam.
    • Real_Egor 6mo ago
      Bingo! That's exactly what I meant
  • dgfl 6mo ago
    Isn’t this what AGI is by design? People CAN learn to become good at videogames. Modern LLMs can’t, they have to be retrained from scratch (I consider pre-training to be a completely different process than learning). I also don’t necessarily agree that a grandma would fail. Give her enough motivation and a couple days and she’ll manage these. My main criticism would be that it doesn’t seem like this test allows online learning, which is what humans do (over the scale of days to years). So in practice it may still collapse to what you point out, but not because the task is unsuited to showing AGI.
    • Real_Egor 6mo ago
      What I'm saying is that this test is just another "out-of-distribution task" for an LLM. And it will be solved using the exact same methods we always use: it will end up in the pre-training data, and LLMs will crush it. This has absolutely nothing to do with AGI. Once they beat these tests, new ones will pop up. They'll beat those, and people will invent the next batch. The way I see it, the true formula for AGI is: [Brain] + [External Sensors] (World Receptors) + [Internal State Sensors] + [Survival Function] + [Memory]. I won't dive too deep into how each of these components has its own distinct traits and is deeply intertwined with the others (especially the survival function and memory). But on a fundamental level, my point is that we are not going to squeeze AGI out of LLMs just by throwing more tests and training cycles at them. These current benchmarks aren't bringing us any closer to AGI. They merely prove that we've found a new layer of tasks that we simply haven't figured out how to train LLMs on yet. P.S. A 2-year-old child is already an AGI in terms of its functional makeup and internal interaction architecture, even though they are far less equipped for survival than a kitten. The path to AGI isn't just endless task training—it's a shift toward a fundamentally different decision-making architecture.
• diablevv 6mo ago
  [dead]
• arjie 6mo ago
  Perhaps actual AGI will be when the models create ARC-HGI-1 to test if humans have general intelligence.
• 3836293648 6mo ago
  Ew. Cool demo, what idiot thought it was ok to have a half second cooldown between inputs? If I hit up three times I should move up three steps, not two steps because I pressed too quickly.
• vessenes 6mo ago
  I’m not a Chollet booster. Well, I might be a little bit of one in that I admire his persistence. I really like these puzzles. There’s a lot to them both in design and scoring — models trained to do well on these are going to be genuinely much more useful, so I’m excited about it. As opposed to -1 and -2, to do well at these, you need to be able to do: - Visual reasoning - Path planning (and some fairly long paths) - Mouse/screen interaction - color and shape analysis - cross-context learning/remembering Probably more, I only did like five or six of these. We really want models that are good at all this; it covers a lot of what current agentic loops are super weak at. So I hope M. Chollet is successful at getting frontier labs to put a billion or so into training for these.
• mvkel 6mo ago
  Was just at the YC launch event for this. Haven't felt this much inspiration in a while. Incredible minds confronting on tech that will change our society. I met a guy who, for fun, started working on ARC2, and as he got the number to go up in the eval, a novel way to more efficiently move a robotic arm emerged. All that to say: chasing evals per se can have tangible real world benefits. Talking to the ARC folks tonight, it sounds like there will be an ARC-4,5,6,etc. I mean of course there will be. But with them will be an increasing expectation that these models can eventually figure things out with zero context, and zero pretraining; you drop a brain into any problem and it'll figure out how to dig its way out. That's really exciting.
  • vonneumannstan 6mo ago
    >Talking to the ARC folks tonight, it sounds like there will be an ARC-4,5,6,etc. I mean of course there will be. Quintessential goal post moving...
    • mvkel 6mo ago
      If you read the charter of the eval (or any eval, really), this statement is pretty silly. The whole point of each eval version is to identify a chunk of challenges that humans do well that AI can't. When AI gets to ~80, you move to the next chunk. When you run out of challenges, you have AGI.
      • dwaltrip 6mo ago
        HN occasionally devolves into “supremely pedantic and nitpicky” mode. Today is one of those days.
      • vonneumannstan 6mo ago
        Except you will never run out of challenges and my sense from Chollet has been that every challenge was hinted at being the final one where once beaten AGI would have been created and of course at the end of each one he comes out saying akshuallyyyy this isn't AGI and it wont be AGI until ARC Challenge+1 is beaten!
• nick49488171 6mo ago
  Arc AGI 4 can be Chip's Challenge!
• hikaru_ai 6mo ago
  [dead]
• baalimago 6mo ago
  You can tell it's an AI by it not becoming utterly by playing the "game". I could personally not stand any more than the first level.
• Wintamute 6mo ago
  Unplayably laggy on an iPhone. Sad people can’t produce a performant experience that a ZX81 could have eaten for breakfast, on a relative super computer
• sstart 6mo ago
  [dead]
• tantalor 6mo ago
  The controls just feel really bad. The inputs are too small, and there is way too much lag.
  • tantalor 6mo ago
    I just checked, the size of the controls are 28x28 The minimum recommended size for mobile is 44x44
  • roflcopter69 6mo ago
    About the lag, I didn't bother looking into it, but I suspect they log every single action you do and require that the request to their servers was confirmed before allowing to do the next action. They probably face a lot of traffic right now, which could cause the lag. Just speculation though.
• elAhmo 6mo ago
  I find it quite funny that we are still debating whether models are intelligent or not, while we know they are just statistical models. Even with billions of dollars spent on training, we had this situation a few weeks ago where models were suggesting to walk instead of drive to a car wash in case you want to wash your car. While a 3 year old would know the answer to the question. And yet, we are designing elaborate tests to 'show whether AGI is here it not', while being fully aware of what these models represent under the hood.
• pugchat 6mo ago
  [dead]
• MadxX79 6mo ago
  Same question I have for all these benchmarks: What's going to stop e.g. OpenAI from hiring a bunch of teenagers to play these games non-stop for a month and annotate the game with their logic for deriving the rules, generate a data set based on those playthroughs and fine tuning the next version of chatgpt on all those playthroughs?
  • vessenes 6mo ago
    Wrong question. I suggest: 1) Do models generalize? 2) If they do, and they generalize from this, is that a win? Chollet was one of the first “they do not generalize” evangelists. I’d be curious to hear what he thinks now, because a) most disagree with him, and b) this test seems designed to get models that can generalize better at visual long context problem solving and agency, exactly where the bleeding edge is right now for needs with agentic systems.
    • MadxX79 6mo ago
      Yeah, so you are agreeing that the benchmarks are useless because they don't answer those questions.
    • daveguy 6mo ago
      Can AI models generalize+ at any long context problem solving and agency regardless of modality? I think the answer is no, and this is why they are not yet AGI. + generalize being the key word.
  • nearbuy 6mo ago
    They would score much worse on the private set than the public set. And they haven't done this for any of the other ARC-AGI benchmarks, so why would they do it for this one?
• vonneumannstan 6mo ago
  It's getting pretty old now when Francois Chollet puts out a new ARC challenge, claims definitively that no system is going to crack it without being full blown AGI, the benchmark gets saturated in a few months, he claims the systems definitely aren't AGI then puts out a new challenge that no non AGI system can clear and a few months later.... etc. etc.
  • daveguy 6mo ago
    Chollet literally never says that. Quite the opposite. He says that AIs are currently abysmally bad at the skills this benchmark tests. An AGI should be able to do this, but doing this doesn't mean it's AGI. He has been very clear about that. I suggest you go back and (re)read the intro ARC-AGI paper. No system can crack these out of the box (like humans can) because we don't have AGI.
    • vonneumannstan 6mo ago
      Yeah I mean ChatGPT 5.4 Pro can't even pick my nose for me so it's obviously not AGI /s
• j1000 6mo ago
  I feel like AGI test would be sense of humor. Somehow I cannot force any LLM to output any even normal level joke.
• vonneumannstan 6mo ago
  >As long as there is a gap between AI and human learning, we do not have AGI. This is an absurd constraint. You could have a vastly superhuman AI that doesn't learn as efficiently as a human and it would not pass this definition while it simultaneously goes on to colonize the galaxy...
• maxothex 6mo ago
  [dead]
• ryguz 6mo ago
  [dead]
• visarga 6mo ago
  ARC is trying to isolate a unitary intelligence signal, so it strips away coordination, specialization, and division of labor. But that also means it removes one of the dominant mechanisms by which intelligence actually scales in the real world. Their view on intelligence implicitly treats redundancy as necessary - one agent must do them all - and treats efficiency as something achieved internally rather than through restructuring the system. At the very least they should create environments that can help an agent compound intelligence, to self amplify, support itself, that is not happening in ARC. Anyone wondered if ARC is a measure of intelligence or just a collection of hand picked tasks? was there a proof they encode anything meaningful about intelligence in such short tasks in miniature environments? One shot intelligence?
• aogaili 6mo ago
  honestly the most interesting thing about ARC-AGI-3 isn't the 0.25% scores everyone is doomposting about. it's the Duke harness result. if you give Opus just three generic tools (READ, GREP, BASH with Python) and literally zero game-specific help, it completes all three preview games in 1,069 actions. for comparison, humans do it in like ~900. that's actually insane. it writes its own BFS, builds a grid parser from scratch, and even solves a Lights Out puzzle with Gaussian elimination. all on its own. i really think the benchmark is testing two different things and just smashing them together. can the model reason about novel interactive environments? yeah, clearly it can. can it do spatial reasoning over a 64x64 grid from raw JSON with zero tools? no. but then again, neither can a human if you ripped out their visual cortex lol. humans come "pre-installed" with specialized subsystems for this exact stuff: a visual cortex for spatial perception, a hippocampus for persistent memory, etc. these aren't "tools" in Chollet's framing but they're basically identical to what the Duke harness provides. the model is just building its own version of those (Python for the cortex, grep for memory). it just needs the permission to build them. the real gap the Duke team found isn't perception or memory anyway, it is hypothesis quality. some runs solve vc33 in 441 actions, others just plateau past 1,500. the variance is just down to whether the model commits early to the right explanation of how the game works. that's a way more interesting and targetable finding than just saying "frontier models score below 1%." Chollet is probably right philsophically that AGI should handle any input format without help. but reporting 0.25% when the actual reasoning gap is in hypothesis formation (not spatial perception) makes the benchmark a way worse progress indicator than it could be imo.
• panavinsingh 6mo ago
  [flagged]
• ARahim3 6mo ago
  [dead]
• dsfadfasdf 6mo ago
  Can someone clarify if image inputs are allowed, so VLMs can be used? I have not been able to get information anywhere.