9 ms·

I got the highest score on ARC-AGI again swapping Python for English

178 points by freediver 1y ago

pilooch 1y ago
Congrats, this solution resembles AlphaEvolve. Text serves as the high-level search space, and genetic mixing (map-elites in AE) merges attemps at lower levels.
doctorpangloss 1y ago
you would be interested in dSPY
Davidzheng 1y ago
Actually really promising stuff. I think a lot of the recent advances in the last 6mo - 1yr is in the other loop (for ex. the google deepthink model which got IMO gold and the OAI IMO gold all use substantive other loop search strategies [though it's unclear what these are] to maybe parallelize some generation/verification process). So there's no reason why we can't have huge advances in this area even outside of the industry labs in my view (I'm uninformed in general so take this comment with a large grain of salt).
modeless 1y ago
I've been testing LLMs on Sokoban-like puzzles (in the style of ARC-AGI-3) and they are completely awful at them. It really highlights how poor their memory is. They can't remember abstract concepts or rules between steps, even if they discover them themselves. They can only be presented with lossy text descriptions of such things which they have to re-read and re-interpret at every step. LLMs are completely helpless on agentic tasks without a ton of scaffolding. But the scaffolding is inflexible and brittle, unlike the models themselves. Whoever figures out how to reproduce the functions of this type of scaffolding within the models, with some kind of internal test-time-learned memory mechanism, is going to win.
- M4v3R 1y ago
  I wonder scaffolding synthesis is the way to go. Namely the LLM itself first reasons about the problem and creates scaffolding for a second agent that will do the actual solving. All inside a feedback loop to adjust the scaffolding based on results.
  - modeless 1y ago
    In general I think the more of the scaffolding that can be folded into the model, the better. The model should learn problem solving strategies like this and be able to manage them internally.
  - sixo 1y ago
    I toyed around with the idea of using an LLM to "compile" user instructions into a kind of AST of scaffolding, which can then be run by another LLM. It worked fairly wellbfor the kind of semi-structured tasks LLMs choke on like "for each of 100 things, do...", but I haven't taken it beyond a minimal impl.
    - harshitaneja 1y ago
      I am working on something similar but with an AST for legal documents. So far, it seems promising but still rudimentary.
  - plantain 1y ago
    If you've ever used Claude Code + Plan mode - you know that exactly this is true.
- sunrunner 1y ago
  I'm not sure how similar this is but I tried the same quite a while back with a simple 5x5 nonogram (Picross) and had similar difficulties. I found not only incorrect 'reasoning' but also even after being explicit about why a certain deduction was not correct the same incorrect deduction would then appear later, and this happened over and over. Also, there's already a complete database of valid answers at [1], so I'm not sure why the correct answer couldn't just come from that, and the 'reasoning' can be 'We solved this here, look...' ;) [1] The wonderful https://pixelogic.app/every-5x5-nonogram https://pixelogic.app/every-5x5-nonogram
  - Akronymus 1y ago
    > I found not only incorrect 'reasoning' but also even after being explicit about why a certain deduction was not correct the same incorrect deduction would then appear later, and this happened over and over. Because its in the context window and a lot of training material refers to earlier stuff for later stuff it is trained to bring up that stuff again and again. Even if it is in the window as a negative.
- low_tech_love 1y ago
  Try to get your LLM of choice to find its way out of a labyrinth that you describe in text form. It's absolutely awful even with the simplest mazes. I'm not sure the problem here is memory, though? I think it has to do with spatial reasoning. I'd be willing to bet every company right now is working on spatial reasoning (at least up to 3D) and as soon as that is working, a huge amount of pieces will fall into place.
  - modeless 1y ago
    Spatial reasoning is weak, but still I frequently see models come up with the right answer in reasoning steps, only to make the wrong move in the following turn because they forget what they just learned. For models with hidden reasoning it's often not even possible to retain the reasoning tokens in context through multiple steps, but even if you could the context windows are big but not big enough to contain all the past reasoning for every step for hundreds of steps. And then even if they were the retrieval from context for abstract concepts (vs verbatim copying) is terrible. Text is too lossy and inefficient. The models need to be able to internally store and retrieve a more compact, abstract, non-verbal representation of facts and procedures.
- Workaccount2 1y ago
  I really think that the problem is with tokenizing vision. Any kind of visually based reasoning and they become dumb as rocks. It feels similar to having a person play sokoban but blindfolded and only with text prompts. The same issue cropped up with playing pokemon. Like the image gets translated to text, and then the model works on that. I'm no expert on transformers, but it just feels like there is some kind of limit that prevents the models from "thinking" visually.
  - modeless 1y ago
    Yes, vision is a problem, but I don't think it's the biggest problem for the specific task I'm testing. The memory problem is bigger. The models frequently do come up with the right answer, but they promptly forget it between turns. Sometimes they forget because the full reasoning trace is not preserved in context (either due to API limitations or simply because the context isn't big enough to hold dozens or hundreds of steps of full reasoning traces). Sometimes it's because retrieval from context is bad for abstract concepts and rules vs. keyword matching, and to me the reason for that is that text is lossy and inefficient. The models need to be able to internally store and retrieve a more compact, abstract, non-verbal representation of facts and procedures.
    - Workaccount2 1y ago
      I think the problem is though that they need to store it in text context. When I am solving a sokoban style game, it's entirely visual. I don't need to remember a lot because the visual holds so much information. It's like the average person trying to play a game of chess with just text. It's nightmarishly hard compared to having a board in front of you. The LLMs seem stuck having to play everything through just text.
      - modeless 1y ago
        It's not just visual. You also need a representation of the rules of the game and the strategies that make sense. The puzzles I'm solving are not straight Sokoban, they have per-game varying rules that need to be discovered (again, ARC-AGI-3 style) that affect the strategies that you need to use. For example, in classic Sokoban you can't push two crates at once, but in some of the puzzles I'm using you can, and this is taught by forcing you to do it in the first level, and you need to remember it through the rest of the levels. This is not a purely visual concept and models still struggle with it.
jokoon 1y ago
Those are bold claims
albertzeyer 1y ago
This sounds interesting. I would really like to read a full research paper made out of this, which describes the method in more detail, gives some more examples, does more analysis on it, etc. Btw, this uses LLMs on pure text-level? Why not images? Most of these patterns are easy to detect on image-level, but I assume when presented as text, it's much harder. > LLMs are PhD-level reasoners in math and science, yet they fail at children's puzzles. How is this possible? I think this argument is a bit flawed. Yes, you can define AGI as being better than (average) humans in every possible task. But isn't this very arbitrary? Isn't it more reasonable to expect that different intelligent systems (including animals, humans) can have different strengths, and it is unreasonable to expect that one system is really better in everything? Maybe it's more reasonable to define ASI that way, but even for ASI, if a system is already better in a majority of tasks (but not necessarily in every task), I think this should already count as ASI. Maybe really being better in every possible task is just not possible. You could design a task that is very specifically tailored for human intelligence.
- bubblyworld 1y ago
  I suspect (to use the language of the author) current LLMs have a bit of a "reasoning dead zone" when it comes to images. In my limited experience they struggle with anything more complex than "transcribe the text" or similarly basic tasks. Like I tried to create an automated QA agent with Claude Sonnet 3.5 to catch regressions in my frontend, and it will look at an obviously broken frontend component (using puppeteer to drive and screenshot a headless browser) and confidently proclaim it's working correctly, often making up a supporting argument too. I've had much more success passing the code for the component and any console logs directly to the agent in text form. My memory is a bit fuzzy, but I've seen another QA agent that takes a similar approach of structured text extraction rather than using images. So I suspect I'm not the only one finding image-based reasoning an issue. Could also be for cost reasons though, so take that with a pinch of salt.
  - ACCount37 1y ago
    LLM image frontends suck, and a lot of them suck big time. The naive approach of "use a pretrained encoder to massage the input pixels into a bag of soft tokens and paste those tokens into the context window" is good enough to get you a third of the way to humanlike vision performance - but struggles to go much further. Claude's current vision implementation is also notoriously awful. Like, "a goddamn 4B Gemma 3 beats it" level of awful. For a lot of vision-heavy tasks, you'd be better off using literally anything else.
0x20cowboy 1y ago
> LLMs are PhD-level reasoners in math and science, yet they fail at children's puzzles. How is this possible? Because they are not. Pattern matching questions on a contrived test is not the same thing as understanding or reasoning. It’s the same reason why most of the people who pass your leetcode tests don’t actually know how to build anything real. They are taught to the test not taught to reality.
- wiseowise 1y ago
  [flagged]
  - bloqs 1y ago
    please consider a less emotive, flaming/personal tone in the future, hacker news is much more readable without it! I would broadly agree that it's a bit far, but the OPs point does have some validity, its often the same formulaic methodology
- ACCount37 1y ago
  "Not understanding or reasoning" is anthropocentric cope. There is very little practical difference between "understanding" and "reasoning" implemented in human mind and that implemented in LLMs. One notable difference, however, is that LLMs disproportionately suck at spatial reasoning. Which shouldn't be surprising, considering that their training datasets are almost entirely text. The ultimate wordcel makes for a poor shape rotator. All ARC-AGI tasks are "spatial reasoning" tasks. They aren't in any way special. They just force LLMs to perform in an area they're spectacularly weak at. And LLMs aren't good enough yet to be able to brute force through this innate deficiency with raw intelligence.
  - HighGoldstein 1y ago
    > There is very little practical difference between "understanding" and "reasoning" implemented in human mind and that implemented in LLMs. Source?
justatdotin 1y ago
> LLMs have "dead reasoning zones" — areas in their weights where logic doesn't work. Humans have dead knowledge zones (things we don't know), but not dead reasoning zones. blank stare
- mjburgess 1y ago
  We have dead-zones in adductive reasoning, not in induction or deduction. Almost all failures of reasoning in people are in abducing what model describes the situation at hand. eg., we can apply the rule, "-A cannot follow from A", etc. regardless of the A eg., we always know that if the number of apples is 2, then it cannot be any of "all numbers without 2" -- which quantifies over all numbers You will not find a "gap" for a given number, whereas with LLMs, gaps of this kind are common
  - rel_ic 1y ago
    > we can apply the rule, "-A cannot follow from A", etc. regardless of the A You can't think of any domains where we are unable to apply this rule? I feel like I'm surrounded by people claiming "A, therefore -A!!" And if I'm one of them, and this were a reasoning dead-zone for me, I wouldn't be able to tell!
    - mjburgess 1y ago
      That's an abductive failure to recognise that something is A, and something else is not-A I dont see cases where people recognise the contradiction and then perform it.
      - virgilp 1y ago
        How can you know? One could argue that the entire phenomenon of cognitive dissonance is "people (internally) recognize the contradiction and then perform it"
      - rel_ic 1y ago
        People who know alcohol is bad for them and don't want to keep being drunks but keep drinking, people who believe phones are bad for their kids but still buy them, people who understand AI will significantly degrade the environment if it becomes ubiquitous but still work to help it become ubiquitous... Mathematicians who publish proofs that are later proven inconsistent! I suspect we have fundamentally different views of how humans work. I see our behavior and beliefs as _mostly_ irrational, with only a few "reasoning live-zones" where, with great effort, we can achieve logical thought.
didroe 1y ago
>With RL, models no longer just learn what sounds correct based on patterns they've seen. They learn what words to output to be correct. RL is the process of forcing the pre-trained weights to be logically consistent. How does Reinforcement Learning force the weights to be logically consistent? Isn't it just about training using a coarser/more-fuzzy granularity of fitness? More generally, is it really solving the task if it's given a large number of attempts and an oracle to say whether it's correct? Humans can answer the questions in one shot and self-check the answer, whereas this is like trial and error with an external expert who tells you to try again.
wiz21c 1y ago
isn't the author actually overfitting a solution ? He'll sure beat ARC AGI, but that will be all.
- deyiao 1y ago
  I don't think so. The author isn't training an LLM, but rather using an LLM to solve a specific problem. This method could also be applied to solve other problems.
- falcor84 1y ago
  Seeing how ARC-AGI is pretty much the only non-embodied short-duration type of challenge where humans are still an order of magnitude better than AIs, beating it would possibly bring us a lot closer to actual AGI.
  - wiz21c 1y ago
    https://arxiv.org/html/2505.07859v1 https://arxiv.org/html/2505.07859v1
amelius 1y ago
This sounds like it is just slightly smarter than brute forcing your way to a solution. Oh well, more support for my prediction: nobody will win a Nobel prize for reaching AGI.
imiric 1y ago
Congrats, you made LLMs perform slightly better at a contrived puzzle. This finally proves that we've cracked intelligence and are well on our way towards AGI.
Garlef 1y ago
That's a super neat approach. But the core issue seems to be: How do you come up with the fitness function that drives the evolutionary process without human intervention in the first place? (I've tried something similar with a coding agent where I let the agent modify parts of its system prompt... But it got stuck very fast since there was no clear fitness function)
FergusArgyll 1y ago
The biggest issue I have with ARC-AGI is it's a visual problem. LLMs (even the newfangled multi-modal ones) are still far worse at vision than at purely text based problems. I don't think it's possible to build a test of purely text-based questions that would be easy for humans and hard for SOTA models. Yes, there's a few gotchas you can throw at them but not 500.
cahaya 1y ago
Are there any existing scripts/ tools to use these evolutionary algorithms also at home with e.g. Codex/GPT-5 / Claude Code?
- tiagopavan 1y ago
  dspy approach seems rather similar to that: https://dspy.ai/tutorials/gepa_ai_program/ https://dspy.ai/tutorials/gepa_ai_program/
d_burfoot 1y ago
To me the reason ARC-AGI puzzles are difficult for LLMs and possible for humans is that they are expressed in a format for which humans have powerful preprocessing capabilities. Imagine the puzzle layouts were expressed in JSON instead of as a pattern of visual blocks. How many humans could solve them in that case?
- pessimizer 1y ago
  Every one who had access to a computer that could convert json into something more readable for humans, and would know that was the first thing they needed to do? You might as well have asked how many English speakers could solve the questions if they were in Chinese. All of them. They would call up someone who spoke Chinese, pay them to translate the questions, then solve them. Or failing that, they would go to the bookstore, buy books on learning Chinese, and solve them three years from now.
- kenjackson 1y ago
  Bingo. We simply made a test for which we are well trained. We are constantly making real time decisions with our eyes. Interestingly certain monkeys are much better at certain visual pattern recognition than we are. They might laugh and think humans haven’t reached AGI yet.
- jononor 1y ago
  We have powerful preprocessing blocks for images: Strong computer vision capabilities predates LLMs by several years. Image classification, segmentation, object detection, etc. All differential and trainable in same way as LLMs, including jointly. To the best of my knowledge, no team has shown really high scores by adding in a image preprocessing block?
- timonofathens 1y ago
  [dead]
frozenseven 1y ago
Code: https://github.com/jerber/arc-lang-public https://github.com/jerber/arc-lang-public Kaggle: https://www.kaggle.com/code/jerber/jeremy-arc2 https://www.kaggle.com/code/jerber/jeremy-arc2
jvanderbot 1y ago
Can someone explain to me why a new LLMs ability to solve highly publicized puzzles is not "just" (sorry) it having access to the blog posts talking about those puzzles? It's fine, that's what I would do to solve them, but it doesn't obviously and immediately make me confident in new reasoning capability w that suspicion floating around.
- wasabi991011 1y ago
  Should be easy to test by picking two similar models with different publishing dates (before and after ARC v2), and also comparing with/without the new reasoning technique from the article.
- Legend2440 1y ago
  Because people already tried to get LLMs to solve ARC-AGI puzzles by training on millions of similar puzzles, and it doesn’t work. Some problems fundamentally require many serial steps to solve. Reasoning LLMs can work through those steps, base LLMs can’t.
yujzgzc 1y ago
> LLMs have "dead reasoning zones" — areas in their weights where logic doesn't work. Humans have dead knowledge zones (things we don't know), but not dead reasoning zones. Religion often is, as "the Lord's ways are inscrutable"
- unfitted2545 1y ago
  And people have started seeing LLM's as a quasi-religion.
causal 1y ago
I love this sort of self-starter experimenting. Curious what models have been tried, I saw Grok4 mentioned, curious how well it transfers to other models.
waffletower 1y ago
Transformer models, typically architected for and trained on 1d text streams, are not going to perform well on ARC-AGI. I like that the test corpus exists as I believe it suggests that other model architectures (perhaps co-existing with LLMs in a MoE fashion) are needed to generalize AI performance further. For example, if we constructed a 3d version of ARC-AGI (rather than relying on grids) humans would probably still outperform reasoning LLMs handily. However, expand ARC-AGI to 4d and I think human performance might start to become more comparable to LLM performance. 4d is as alien to us as 2d is to LLMs, in this narrow test corpus.

178 points by freediver 1y ago

Congrats, this solution resembles AlphaEvolve. Text serves as the high-level search space, and genetic mixing (map-elites in AE) merges attemps at lower levels.

doctorpangloss 1y ago

you would be interested in dSPY

Davidzheng 1y ago

Actually really promising stuff. I think a lot of the recent advances in the last 6mo - 1yr is in the other loop (for ex. the google deepthink model which got IMO gold and the OAI IMO gold all use substantive other loop search strategies [though it's unclear what these are] to maybe parallelize some generation/verification process). So there's no reason why we can't have huge advances in this area even outside of the industry labs in my view (I'm uninformed in general so take this comment with a large grain of salt).

modeless 1y ago

I've been testing LLMs on Sokoban-like puzzles (in the style of ARC-AGI-3) and they are completely awful at them. It really highlights how poor their memory is. They can't remember abstract concepts or rules between steps, even if they discover them themselves. They can only be presented with lossy text descriptions of such things which they have to re-read and re-interpret at every step. LLMs are completely helpless on agentic tasks without a ton of scaffolding. But the scaffolding is inflexible and brittle, unlike the models themselves. Whoever figures out how to reproduce the functions of this type of scaffolding within the models, with some kind of internal test-time-learned memory mechanism, is going to win.

M4v3R 1y ago
I wonder scaffolding synthesis is the way to go. Namely the LLM itself first reasons about the problem and creates scaffolding for a second agent that will do the actual solving. All inside a feedback loop to adjust the scaffolding based on results.
- modeless 1y ago
  In general I think the more of the scaffolding that can be folded into the model, the better. The model should learn problem solving strategies like this and be able to manage them internally.
- sixo 1y ago
  I toyed around with the idea of using an LLM to "compile" user instructions into a kind of AST of scaffolding, which can then be run by another LLM. It worked fairly wellbfor the kind of semi-structured tasks LLMs choke on like "for each of 100 things, do...", but I haven't taken it beyond a minimal impl.
  - harshitaneja 1y ago
    I am working on something similar but with an AST for legal documents. So far, it seems promising but still rudimentary.
- plantain 1y ago
  If you've ever used Claude Code + Plan mode - you know that exactly this is true.
sunrunner 1y ago
I'm not sure how similar this is but I tried the same quite a while back with a simple 5x5 nonogram (Picross) and had similar difficulties. I found not only incorrect 'reasoning' but also even after being explicit about why a certain deduction was not correct the same incorrect deduction would then appear later, and this happened over and over. Also, there's already a complete database of valid answers at [1], so I'm not sure why the correct answer couldn't just come from that, and the 'reasoning' can be 'We solved this here, look...' ;) [1] The wonderful https://pixelogic.app/every-5x5-nonogram https://pixelogic.app/every-5x5-nonogram
- Akronymus 1y ago
  > I found not only incorrect 'reasoning' but also even after being explicit about why a certain deduction was not correct the same incorrect deduction would then appear later, and this happened over and over. Because its in the context window and a lot of training material refers to earlier stuff for later stuff it is trained to bring up that stuff again and again. Even if it is in the window as a negative.
low_tech_love 1y ago
Try to get your LLM of choice to find its way out of a labyrinth that you describe in text form. It's absolutely awful even with the simplest mazes. I'm not sure the problem here is memory, though? I think it has to do with spatial reasoning. I'd be willing to bet every company right now is working on spatial reasoning (at least up to 3D) and as soon as that is working, a huge amount of pieces will fall into place.
- modeless 1y ago
  Spatial reasoning is weak, but still I frequently see models come up with the right answer in reasoning steps, only to make the wrong move in the following turn because they forget what they just learned. For models with hidden reasoning it's often not even possible to retain the reasoning tokens in context through multiple steps, but even if you could the context windows are big but not big enough to contain all the past reasoning for every step for hundreds of steps. And then even if they were the retrieval from context for abstract concepts (vs verbatim copying) is terrible. Text is too lossy and inefficient. The models need to be able to internally store and retrieve a more compact, abstract, non-verbal representation of facts and procedures.
Workaccount2 1y ago
I really think that the problem is with tokenizing vision. Any kind of visually based reasoning and they become dumb as rocks. It feels similar to having a person play sokoban but blindfolded and only with text prompts. The same issue cropped up with playing pokemon. Like the image gets translated to text, and then the model works on that. I'm no expert on transformers, but it just feels like there is some kind of limit that prevents the models from "thinking" visually.
- modeless 1y ago
  Yes, vision is a problem, but I don't think it's the biggest problem for the specific task I'm testing. The memory problem is bigger. The models frequently do come up with the right answer, but they promptly forget it between turns. Sometimes they forget because the full reasoning trace is not preserved in context (either due to API limitations or simply because the context isn't big enough to hold dozens or hundreds of steps of full reasoning traces). Sometimes it's because retrieval from context is bad for abstract concepts and rules vs. keyword matching, and to me the reason for that is that text is lossy and inefficient. The models need to be able to internally store and retrieve a more compact, abstract, non-verbal representation of facts and procedures.
  - Workaccount2 1y ago
    I think the problem is though that they need to store it in text context. When I am solving a sokoban style game, it's entirely visual. I don't need to remember a lot because the visual holds so much information. It's like the average person trying to play a game of chess with just text. It's nightmarishly hard compared to having a board in front of you. The LLMs seem stuck having to play everything through just text.
    - modeless 1y ago
      It's not just visual. You also need a representation of the rules of the game and the strategies that make sense. The puzzles I'm solving are not straight Sokoban, they have per-game varying rules that need to be discovered (again, ARC-AGI-3 style) that affect the strategies that you need to use. For example, in classic Sokoban you can't push two crates at once, but in some of the puzzles I'm using you can, and this is taught by forcing you to do it in the first level, and you need to remember it through the rest of the levels. This is not a purely visual concept and models still struggle with it.

jokoon 1y ago

Those are bold claims

albertzeyer 1y ago

This sounds interesting. I would really like to read a full research paper made out of this, which describes the method in more detail, gives some more examples, does more analysis on it, etc. Btw, this uses LLMs on pure text-level? Why not images? Most of these patterns are easy to detect on image-level, but I assume when presented as text, it's much harder. > LLMs are PhD-level reasoners in math and science, yet they fail at children's puzzles. How is this possible? I think this argument is a bit flawed. Yes, you can define AGI as being better than (average) humans in every possible task. But isn't this very arbitrary? Isn't it more reasonable to expect that different intelligent systems (including animals, humans) can have different strengths, and it is unreasonable to expect that one system is really better in everything? Maybe it's more reasonable to define ASI that way, but even for ASI, if a system is already better in a majority of tasks (but not necessarily in every task), I think this should already count as ASI. Maybe really being better in every possible task is just not possible. You could design a task that is very specifically tailored for human intelligence.

bubblyworld 1y ago
I suspect (to use the language of the author) current LLMs have a bit of a "reasoning dead zone" when it comes to images. In my limited experience they struggle with anything more complex than "transcribe the text" or similarly basic tasks. Like I tried to create an automated QA agent with Claude Sonnet 3.5 to catch regressions in my frontend, and it will look at an obviously broken frontend component (using puppeteer to drive and screenshot a headless browser) and confidently proclaim it's working correctly, often making up a supporting argument too. I've had much more success passing the code for the component and any console logs directly to the agent in text form. My memory is a bit fuzzy, but I've seen another QA agent that takes a similar approach of structured text extraction rather than using images. So I suspect I'm not the only one finding image-based reasoning an issue. Could also be for cost reasons though, so take that with a pinch of salt.
- ACCount37 1y ago
  LLM image frontends suck, and a lot of them suck big time. The naive approach of "use a pretrained encoder to massage the input pixels into a bag of soft tokens and paste those tokens into the context window" is good enough to get you a third of the way to humanlike vision performance - but struggles to go much further. Claude's current vision implementation is also notoriously awful. Like, "a goddamn 4B Gemma 3 beats it" level of awful. For a lot of vision-heavy tasks, you'd be better off using literally anything else.

0x20cowboy 1y ago

> LLMs are PhD-level reasoners in math and science, yet they fail at children's puzzles. How is this possible? Because they are not. Pattern matching questions on a contrived test is not the same thing as understanding or reasoning. It’s the same reason why most of the people who pass your leetcode tests don’t actually know how to build anything real. They are taught to the test not taught to reality.

wiseowise 1y ago
[flagged]
- bloqs 1y ago
  please consider a less emotive, flaming/personal tone in the future, hacker news is much more readable without it! I would broadly agree that it's a bit far, but the OPs point does have some validity, its often the same formulaic methodology
ACCount37 1y ago
"Not understanding or reasoning" is anthropocentric cope. There is very little practical difference between "understanding" and "reasoning" implemented in human mind and that implemented in LLMs. One notable difference, however, is that LLMs disproportionately suck at spatial reasoning. Which shouldn't be surprising, considering that their training datasets are almost entirely text. The ultimate wordcel makes for a poor shape rotator. All ARC-AGI tasks are "spatial reasoning" tasks. They aren't in any way special. They just force LLMs to perform in an area they're spectacularly weak at. And LLMs aren't good enough yet to be able to brute force through this innate deficiency with raw intelligence.
- HighGoldstein 1y ago
  > There is very little practical difference between "understanding" and "reasoning" implemented in human mind and that implemented in LLMs. Source?

justatdotin 1y ago

> LLMs have "dead reasoning zones" — areas in their weights where logic doesn't work. Humans have dead knowledge zones (things we don't know), but not dead reasoning zones. blank stare

mjburgess 1y ago
We have dead-zones in adductive reasoning, not in induction or deduction. Almost all failures of reasoning in people are in abducing what model describes the situation at hand. eg., we can apply the rule, "-A cannot follow from A", etc. regardless of the A eg., we always know that if the number of apples is 2, then it cannot be any of "all numbers without 2" -- which quantifies over all numbers You will not find a "gap" for a given number, whereas with LLMs, gaps of this kind are common
- rel_ic 1y ago
  > we can apply the rule, "-A cannot follow from A", etc. regardless of the A You can't think of any domains where we are unable to apply this rule? I feel like I'm surrounded by people claiming "A, therefore -A!!" And if I'm one of them, and this were a reasoning dead-zone for me, I wouldn't be able to tell!
  - mjburgess 1y ago
    That's an abductive failure to recognise that something is A, and something else is not-A I dont see cases where people recognise the contradiction and then perform it.
    - virgilp 1y ago
      How can you know? One could argue that the entire phenomenon of cognitive dissonance is "people (internally) recognize the contradiction and then perform it"
    - rel_ic 1y ago
      People who know alcohol is bad for them and don't want to keep being drunks but keep drinking, people who believe phones are bad for their kids but still buy them, people who understand AI will significantly degrade the environment if it becomes ubiquitous but still work to help it become ubiquitous... Mathematicians who publish proofs that are later proven inconsistent! I suspect we have fundamentally different views of how humans work. I see our behavior and beliefs as _mostly_ irrational, with only a few "reasoning live-zones" where, with great effort, we can achieve logical thought.

didroe 1y ago

>With RL, models no longer just learn what sounds correct based on patterns they've seen. They learn what words to output to be correct. RL is the process of forcing the pre-trained weights to be logically consistent. How does Reinforcement Learning force the weights to be logically consistent? Isn't it just about training using a coarser/more-fuzzy granularity of fitness? More generally, is it really solving the task if it's given a large number of attempts and an oracle to say whether it's correct? Humans can answer the questions in one shot and self-check the answer, whereas this is like trial and error with an external expert who tells you to try again.

wiz21c 1y ago

isn't the author actually overfitting a solution ? He'll sure beat ARC AGI, but that will be all.

deyiao 1y ago
I don't think so. The author isn't training an LLM, but rather using an LLM to solve a specific problem. This method could also be applied to solve other problems.
falcor84 1y ago
Seeing how ARC-AGI is pretty much the only non-embodied short-duration type of challenge where humans are still an order of magnitude better than AIs, beating it would possibly bring us a lot closer to actual AGI.
- wiz21c 1y ago
  https://arxiv.org/html/2505.07859v1 https://arxiv.org/html/2505.07859v1

amelius 1y ago

This sounds like it is just slightly smarter than brute forcing your way to a solution. Oh well, more support for my prediction: nobody will win a Nobel prize for reaching AGI.

imiric 1y ago

Congrats, you made LLMs perform slightly better at a contrived puzzle. This finally proves that we've cracked intelligence and are well on our way towards AGI.

Garlef 1y ago

That's a super neat approach. But the core issue seems to be: How do you come up with the fitness function that drives the evolutionary process without human intervention in the first place? (I've tried something similar with a coding agent where I let the agent modify parts of its system prompt... But it got stuck very fast since there was no clear fitness function)

FergusArgyll 1y ago

The biggest issue I have with ARC-AGI is it's a visual problem. LLMs (even the newfangled multi-modal ones) are still far worse at vision than at purely text based problems. I don't think it's possible to build a test of purely text-based questions that would be easy for humans and hard for SOTA models. Yes, there's a few gotchas you can throw at them but not 500.

cahaya 1y ago

Are there any existing scripts/ tools to use these evolutionary algorithms also at home with e.g. Codex/GPT-5 / Claude Code?

tiagopavan 1y ago
dspy approach seems rather similar to that: https://dspy.ai/tutorials/gepa_ai_program/ https://dspy.ai/tutorials/gepa_ai_program/

d_burfoot 1y ago

To me the reason ARC-AGI puzzles are difficult for LLMs and possible for humans is that they are expressed in a format for which humans have powerful preprocessing capabilities. Imagine the puzzle layouts were expressed in JSON instead of as a pattern of visual blocks. How many humans could solve them in that case?

pessimizer 1y ago
Every one who had access to a computer that could convert json into something more readable for humans, and would know that was the first thing they needed to do? You might as well have asked how many English speakers could solve the questions if they were in Chinese. All of them. They would call up someone who spoke Chinese, pay them to translate the questions, then solve them. Or failing that, they would go to the bookstore, buy books on learning Chinese, and solve them three years from now.
kenjackson 1y ago
Bingo. We simply made a test for which we are well trained. We are constantly making real time decisions with our eyes. Interestingly certain monkeys are much better at certain visual pattern recognition than we are. They might laugh and think humans haven’t reached AGI yet.
jononor 1y ago
We have powerful preprocessing blocks for images: Strong computer vision capabilities predates LLMs by several years. Image classification, segmentation, object detection, etc. All differential and trainable in same way as LLMs, including jointly. To the best of my knowledge, no team has shown really high scores by adding in a image preprocessing block?
timonofathens 1y ago
[dead]

frozenseven 1y ago

Code: https://github.com/jerber/arc-lang-public https://github.com/jerber/arc-lang-public Kaggle: https://www.kaggle.com/code/jerber/jeremy-arc2 https://www.kaggle.com/code/jerber/jeremy-arc2

jvanderbot 1y ago

Can someone explain to me why a new LLMs ability to solve highly publicized puzzles is not "just" (sorry) it having access to the blog posts talking about those puzzles? It's fine, that's what I would do to solve them, but it doesn't obviously and immediately make me confident in new reasoning capability w that suspicion floating around.

wasabi991011 1y ago
Should be easy to test by picking two similar models with different publishing dates (before and after ARC v2), and also comparing with/without the new reasoning technique from the article.
Legend2440 1y ago
Because people already tried to get LLMs to solve ARC-AGI puzzles by training on millions of similar puzzles, and it doesn’t work. Some problems fundamentally require many serial steps to solve. Reasoning LLMs can work through those steps, base LLMs can’t.

yujzgzc 1y ago

> LLMs have "dead reasoning zones" — areas in their weights where logic doesn't work. Humans have dead knowledge zones (things we don't know), but not dead reasoning zones. Religion often is, as "the Lord's ways are inscrutable"

unfitted2545 1y ago
And people have started seeing LLM's as a quasi-religion.

causal 1y ago

I love this sort of self-starter experimenting. Curious what models have been tried, I saw Grok4 mentioned, curious how well it transfers to other models.

waffletower 1y ago

Transformer models, typically architected for and trained on 1d text streams, are not going to perform well on ARC-AGI. I like that the test corpus exists as I believe it suggests that other model architectures (perhaps co-existing with LLMs in a MoE fashion) are needed to generalize AI performance further. For example, if we constructed a 3d version of ARC-AGI (rather than relying on grids) humans would probably still outperform reasoning LLMs handily. However, expand ARC-AGI to 4d and I think human performance might start to become more comparable to LLM performance. 4d is as alien to us as 2d is to LLMs, in this narrow test corpus.