9 ms·

François Chollet, creator of ARC-AGI, has consistently said that solving the benchmark does not mean we have AGI. It has always been meant as a stepping stone t

François Chollet, creator of ARC-AGI, has consistently said that solving the benchmark does not mean we have AGI. It has always been meant as a stepping stone to encourage progress in the correct direction rather than as an indicator of reaching the destination. That's why he is working on ARC-AGI-3 (to be released in a few weeks) and ARC-AGI-4. His definition of reaching AGI, as I understand it, is when it becomes impossible to construct the next version of ARC-AGI because we can no longer find tasks that are feasible for normal humans but unsolved by AI.

hmmmmmmmmmmmmmm 8mo ago
I don't think the creator believes ARC3 can't be solved but rather that it can't be solved "efficiently" and >$13 per task for ARC2 is certainly not efficient. But at this rate, the people who talk about the goal posts shifting even once we achieve AGI may end up correct, though I don't think this benchmark is particularly great either.
beklein 8mo ago
https://x.com/fchollet/status/2022036543582638517 https://x.com/fchollet/status/2022036543582638517
- joelthelion 8mo ago
  Do opus 4.6 or gemini deep think really use test time adaptation ? How does it work in practice?
mapontosevenths 8mo ago
> His definition of reaching AGI, as I understand it, is when it becomes impossible to construct the next version of ARC-AGI because we can no longer find tasks that are feasible for normal humans but unsolved by AI. That is the best definition I've yet to read. If something claims to be conscious and we can't prove it's not, we have no choice but to believe it. Thats said, I'm reminded of the impossible voting tests they used to give black people to prevent them from voting. We dont ask nearly so much proof from a human, we take their word for it. On the few occasions we did ask for proof it inevitably led to horrific abuse. Edit: The average human tested scores 60%. So the machines are already smarter on an individual basis than the average human.
- woah 8mo ago
  > If something claims to be conscious and we can't prove it's not, we have no choice but to believe it. Can you "prove" that GPT2 isn't concious?
  - mapontosevenths 8mo ago
    If we equate self awareness with consciousness then yes. Several papers have now shown that SOTA models have self awareness of at least a limited sort. [0][1] As far as I'm aware no one has ever proven that for GPT 2, but the methodology for testing it is available if you're interested. [0]https://arxiv.org/pdf/2501.11120 https://arxiv.org/pdf/2501.11120 [1]https://transformer-circuits.pub/2025/introspection/index.html https://transformer-circuits.pub/2025/introspection/index.ht...
    - pixl97 8mo ago
      Honestly our ideas of consciousness and sentience really don't fit well with machine intelligence and capabilities. There is the idea of self as in 'i am this execution' or maybe I am this compressed memory stream that is now the concept of me. But what does consciousness mean if you can be endlessly copied? If embodiment doesn't mean much because the end of your body doesnt mean the end of you? A lot of people are chasing AI and how much it's like us, but it could be very easy to miss the ways it's not like us but still very intelligent or adaptable.
      - idiotsecant 8mo ago
        I'm not sure what consciousness has to do with whether or not you can be copied. If I make a brain scanner tomorrow capable of perfectly capturing your brain state do you stop being conscious?
    - olyjohn 8mo ago
      We don't equate self awareness with consciousness. Dogs are conscious, but still bark at themselves in a mirror.
UltraSane 8mo ago
ARC-AGI-3 uses dynamic games that LLMs must determine the rules and is MUCH harder. LLMs can also be ranked on how many steps they required.
vessenes 8mo ago
Please let’s hold M Chollet to account, at least a little. He launched ARC claiming transformer architectures could never do it and that he thought solving it would be AGI. And he was smug about it. ARC 2 had a very similar launch. Both have been crushed in far less time without significantly different architectures than he predicted. It’s a hard test! And novel, and worth continuing to iterate on. But it was not launched with the humility your last sentence describes.
- modeless 8mo ago
  Here is what the original paper for ARC-AGI-1 said in 2019: > Our definition, formal framework, and evaluation guidelines, which do not capture all facets of intelligence, were developed to be actionable, explanatory, and quantifiable, rather than being descriptive, exhaustive, or consensual. They are not meant to invalidate other perspectives on intelligence, rather, they are meant to serve as a useful objective function to guide research on broad AI and general AI [...] > Importantly, ARC is still a work in progress, with known weaknesses listed in [Section III.2]. We plan on further refining the dataset in the future, both as a playground for research and as a joint benchmark for machine intelligence and human intelligence. > The measure of the success of our message will be its ability to divert the attention of some part of the community interested in general AI, away from surpassing humans at tests of skill, towards investigating the development of human-like broad cognitive abilities, through the lens of program synthesis, Core Knowledge priors, curriculum optimization, information efficiency, and achieving extreme generalization through strong abstraction.
  - vessenes 8mo ago
    https://www.dwarkesh.com/p/francois-chollet https://www.dwarkesh.com/p/francois-chollet (June 2024, about ARC-AGI-1. Note the AGI right in the name) > I’m pretty skeptical that we’re going to see an LLM do 80% in a year. That said, if we do see it, you would also have to look at how this was achieved. If you just train the model on millions or billions of puzzles similar to ARC, you’re relying on the ability to have some overlap between the tasks that you train on and the tasks that you’re going to see at test time. You’re still using memorization. > Maybe it can work. Hopefully, ARC is going to be good enough that it’s going to be resistant to this sort of brute force attempt but you never know. Maybe it could happen. I’m not saying it’s not going to happen. ARC is not a perfect benchmark. Maybe it has flaws. Maybe it could be hacked in that way. e.g. If ARC is solved not through memorization, then it does what it says on the tin. [Dwarkesh suggests that larger models get more generalization capabilities and will therefore continue to become more intelligent] > If you were right, LLMs would do really well on ARC puzzles because ARC puzzles are not complex. Each one of them requires very little knowledge. Each one of them is very low on complexity. You don't need to think very hard about it. They're actually extremely obvious for human > Even children can do them but LLMs cannot. Even LLMs that have 100,000x more knowledge than you do still cannot. If you listen to the podcast, he was super confident, and super wrong. Which, like I said, NBD. I'm glad we have the ARC series of tests. But they have "AGI" right in the name of the test.
grantcas 8mo ago
[dead]
peheje 8mo ago
Hello Gemini, please fix: Biological Aging: Find the cellular "reset switch" so humans can live indefinitely in peak physical health. Global Hunger: Engineer a food system where nutritious meals are a universal right and never a scarcity. Cancer: Develop a precision "search and destroy" therapy that eliminates every malignant cell without side effects. War: Solve the systemic triggers of conflict to transition humanity into an era of permanent global peace. Chronic Pain: Map the nervous system to shut off persistent physical suffering for every person on Earth. Infectious Disease: Create a universal shield that detects and neutralizes any pathogen before it can spread. Clean Energy: Perfect nuclear fusion to provide the world with limitless, carbon-free power forever. Mental Health: Unlock the brain's biology to fully cure depression, anxiety, and all neurological disorders. Clean Water: Scale low-energy desalination so that safe, fresh water is available in every corner of the globe. Ecological Collapse: Restore the Earth’s biodiversity and stabilize the climate to ensure a thriving, permanent biosphere.

hmmmmmmmmmmmmmm 8mo ago

I don't think the creator believes ARC3 can't be solved but rather that it can't be solved "efficiently" and >$13 per task for ARC2 is certainly not efficient. But at this rate, the people who talk about the goal posts shifting even once we achieve AGI may end up correct, though I don't think this benchmark is particularly great either.

beklein 8mo ago

https://x.com/fchollet/status/2022036543582638517 https://x.com/fchollet/status/2022036543582638517

joelthelion 8mo ago
Do opus 4.6 or gemini deep think really use test time adaptation ? How does it work in practice?

mapontosevenths 8mo ago

> His definition of reaching AGI, as I understand it, is when it becomes impossible to construct the next version of ARC-AGI because we can no longer find tasks that are feasible for normal humans but unsolved by AI. That is the best definition I've yet to read. If something claims to be conscious and we can't prove it's not, we have no choice but to believe it. Thats said, I'm reminded of the impossible voting tests they used to give black people to prevent them from voting. We dont ask nearly so much proof from a human, we take their word for it. On the few occasions we did ask for proof it inevitably led to horrific abuse. Edit: The average human tested scores 60%. So the machines are already smarter on an individual basis than the average human.

woah 8mo ago
> If something claims to be conscious and we can't prove it's not, we have no choice but to believe it. Can you "prove" that GPT2 isn't concious?
- mapontosevenths 8mo ago
  If we equate self awareness with consciousness then yes. Several papers have now shown that SOTA models have self awareness of at least a limited sort. [0][1] As far as I'm aware no one has ever proven that for GPT 2, but the methodology for testing it is available if you're interested. [0]https://arxiv.org/pdf/2501.11120 https://arxiv.org/pdf/2501.11120 [1]https://transformer-circuits.pub/2025/introspection/index.html https://transformer-circuits.pub/2025/introspection/index.ht...
  - pixl97 8mo ago
    Honestly our ideas of consciousness and sentience really don't fit well with machine intelligence and capabilities. There is the idea of self as in 'i am this execution' or maybe I am this compressed memory stream that is now the concept of me. But what does consciousness mean if you can be endlessly copied? If embodiment doesn't mean much because the end of your body doesnt mean the end of you? A lot of people are chasing AI and how much it's like us, but it could be very easy to miss the ways it's not like us but still very intelligent or adaptable.
    - idiotsecant 8mo ago
      I'm not sure what consciousness has to do with whether or not you can be copied. If I make a brain scanner tomorrow capable of perfectly capturing your brain state do you stop being conscious?
  - olyjohn 8mo ago
    We don't equate self awareness with consciousness. Dogs are conscious, but still bark at themselves in a mirror.

UltraSane 8mo ago

ARC-AGI-3 uses dynamic games that LLMs must determine the rules and is MUCH harder. LLMs can also be ranked on how many steps they required.

vessenes 8mo ago

Please let’s hold M Chollet to account, at least a little. He launched ARC claiming transformer architectures could never do it and that he thought solving it would be AGI. And he was smug about it. ARC 2 had a very similar launch. Both have been crushed in far less time without significantly different architectures than he predicted. It’s a hard test! And novel, and worth continuing to iterate on. But it was not launched with the humility your last sentence describes.

modeless 8mo ago
Here is what the original paper for ARC-AGI-1 said in 2019: > Our definition, formal framework, and evaluation guidelines, which do not capture all facets of intelligence, were developed to be actionable, explanatory, and quantifiable, rather than being descriptive, exhaustive, or consensual. They are not meant to invalidate other perspectives on intelligence, rather, they are meant to serve as a useful objective function to guide research on broad AI and general AI [...] > Importantly, ARC is still a work in progress, with known weaknesses listed in [Section III.2]. We plan on further refining the dataset in the future, both as a playground for research and as a joint benchmark for machine intelligence and human intelligence. > The measure of the success of our message will be its ability to divert the attention of some part of the community interested in general AI, away from surpassing humans at tests of skill, towards investigating the development of human-like broad cognitive abilities, through the lens of program synthesis, Core Knowledge priors, curriculum optimization, information efficiency, and achieving extreme generalization through strong abstraction.
- vessenes 8mo ago
  https://www.dwarkesh.com/p/francois-chollet https://www.dwarkesh.com/p/francois-chollet (June 2024, about ARC-AGI-1. Note the AGI right in the name) > I’m pretty skeptical that we’re going to see an LLM do 80% in a year. That said, if we do see it, you would also have to look at how this was achieved. If you just train the model on millions or billions of puzzles similar to ARC, you’re relying on the ability to have some overlap between the tasks that you train on and the tasks that you’re going to see at test time. You’re still using memorization. > Maybe it can work. Hopefully, ARC is going to be good enough that it’s going to be resistant to this sort of brute force attempt but you never know. Maybe it could happen. I’m not saying it’s not going to happen. ARC is not a perfect benchmark. Maybe it has flaws. Maybe it could be hacked in that way. e.g. If ARC is solved not through memorization, then it does what it says on the tin. [Dwarkesh suggests that larger models get more generalization capabilities and will therefore continue to become more intelligent] > If you were right, LLMs would do really well on ARC puzzles because ARC puzzles are not complex. Each one of them requires very little knowledge. Each one of them is very low on complexity. You don't need to think very hard about it. They're actually extremely obvious for human > Even children can do them but LLMs cannot. Even LLMs that have 100,000x more knowledge than you do still cannot. If you listen to the podcast, he was super confident, and super wrong. Which, like I said, NBD. I'm glad we have the ARC series of tests. But they have "AGI" right in the name of the test.

grantcas 8mo ago

[dead]

peheje 8mo ago

Hello Gemini, please fix: Biological Aging: Find the cellular "reset switch" so humans can live indefinitely in peak physical health. Global Hunger: Engineer a food system where nutritious meals are a universal right and never a scarcity. Cancer: Develop a precision "search and destroy" therapy that eliminates every malignant cell without side effects. War: Solve the systemic triggers of conflict to transition humanity into an era of permanent global peace. Chronic Pain: Map the nervous system to shut off persistent physical suffering for every person on Earth. Infectious Disease: Create a universal shield that detects and neutralizes any pathogen before it can spread. Clean Energy: Perfect nuclear fusion to provide the world with limitless, carbon-free power forever. Mental Health: Unlock the brain's biology to fully cure depression, anxiety, and all neurological disorders. Clean Water: Scale low-energy desalination so that safe, fresh water is available in every corner of the globe. Ecological Collapse: Restore the Earth’s biodiversity and stabilize the climate to ensure a thriving, permanent biosphere.