5 ms·

Why is this considered such a great AGI test? It seems possible to extensively train a model on the algorithms used to solve these cases, and some cases feel be

Why is this considered such a great AGI test? It seems possible to extensively train a model on the algorithms used to solve these cases, and some cases feel beyond what a human could straightforwardly figure out.

riku_iki 2y ago
I think huge advantage is that they keep eval tests private, so corps can't finetune them to model and claim breakthrough, which possibly happened with many other benchmarks.
visarga 2y ago
There is a hidden test set with new puzzle types not seen in the open part. It's designed so that humans do well and AI models have a hard time.
- YeGoblynQueenne 2y ago
  "Designed" is not right. What gives "AI models" (i.e. deep neural nets) a hard time is that there are very few examples in the public training and evaluation set: each task has three examples. So basically it's not a test of intelligence but a test of sample efficiency. Besides which, it is unfair because it excludes an entire category of systems, not to mention a dominant one. If F. Chollet really believes ARC is a test of intelligence, then why not provide enough examples for deep nets or some other big data approach to be trained effectively? The answer is: because a big data approach would then easily beat the test. But if the test can be beaten without intelligence, just with data, then it's not a test of intelligence. My guess for a long time has been that ARC will fall just like the Winograd Schema challenge (WSC) [1] fell: someone will do the work to generate enough (tens of thousands) examples of ARC-like tasks, then train a deep neural net and go to town. That's what happened with the WSC. A large dataset of Winograd schema sentences was crowd-sourced and a big BERT-era Transformer got around 90% accuracy on the WSC [2]. Bye bye WSC, and any wishful thinking about Winograd schemas requiring human intuition and other undefined stuff. Or, ARC might go the way of the Bongard Problems [3]: the original 100 problems by Bongard still stand unsolved, but the machine learning community has effectively sidestepped them. Someone made a generator of Bongard-like problems [4], and while this was not enough to solve the original problems, everyone simply switched to training CNNs and reporting results on the new dataset [5]. We basically have no idea how to create a test for intelligence that computers cannot beat by brute force or big data approaches so we have no effective way to test computers for (artificial) intelligence. The only thing we know humans can do that computers can't is identify undecidable problems (like Barber Paradoxes i.e. statements of the form "this sentence is false", as in Gödel's second incompleteness theorem). Unfortunately we already know there is no computer that can ever do that, and even if we observe say ChatGPT returning the right answer we can be sure it has only memorised, not calculated it, so we're a bit stuck. ARC won't get us unstuck in any way shape or form and so it's just a distraction. _____________________ [1] https://en.wikipedia.org/wiki/Winograd_schema_challenge https://en.wikipedia.org/wiki/Winograd_schema_challenge [2] WinoGrande: An Adversarial Winograd Schema Challenge at Scale https://arxiv.org/abs/1907.10641 https://arxiv.org/abs/1907.10641 Although note the results are interpreted to mean LLMs are more or less memorising answers, which is right of course. [3] Index of Bongard Problems https://www.foundalis.com/res/bps/bpidx.htm https://www.foundalis.com/res/bps/bpidx.htm [4] Comparing machines and humans on a visual categorization test https://www.pnas.org/doi/abs/10.1073/pnas.1109168108 https://www.pnas.org/doi/abs/10.1073/pnas.1109168108 [5] 25 years of CNNs: Can we compare to human abstraction capabilities? https://arxiv.org/abs/1607.08366 https://arxiv.org/abs/1607.08366
  - falcor84 2y ago
    > My guess for a long time has been that ARC will fall just like the Winograd Schema challenge (WSC) fell: someone will do the work to generate enough (tens of thousands) examples of ARC-like tasks, then train a deep neural net and go to town. I think that this would be the real AGI (or even superintelligence hurdle) - having essentially a metacognitive AI understand that something given to it is a novel problem, for which it would use the given examples to automatically generate synthetic data sets and then train itself (or a subordinate model) based on these examples to gain the skill of solving this general type of problem. > The only thing we know humans can do that computers can't is identify undecidable problems (like Barber Paradoxes i.e. statements of the form "this sentence is false", as in Gödel's second incompleteness theorem). Unfortunately we already know there is no computer that can ever do that Where did the "ever" come from? Why wouldn't future computers be able to do this at (least at) a human level?
    - YeGoblynQueenne 2y ago
      The "ever" comes from the Church-Turing thesis. Maybe in the future computers will not be Turing machines, but that we can't know yet.
      - falcor84 2y ago
        That refers to the unsolvability of the general case, which is of course also unsolvable by humans.
        deleted 2y ago
        [deleted]
        YeGoblynQueenne 2y ago
        I'm sorry, I'm not sure what "unsolvability" means. What I'm saying above is that humans can identify undecidable statements, i.e. we can recognise them as undecidable. If we couldn't, Gödel, Church, and Turing would not have a proof. But we can. We just don't do it algorithmically, obviously- because there is no algorithm that can do that, and so no computer that can, either.
isotypic 2y ago
Do you have some examples of ones you found beyond what a human could straightforwardly figure out? I tried a bunch and they all seemed reasonable, so I would be interested in seeing - I didn't try all 400, for obvious reasons, so I don't doubt there are difficult ones. I think regardless one of the reasons people are interested in it is that is a fairly simple logic puzzle - given some examples, extrapolate a pattern, execute the pattern - that humans achieve high accuracy on (a study linked on the website has ~84% accuracy for humans, some more recent study seems to put it closer to 75%). Yet ML approaches have yet to reach that level, in contrast to other problems ML has been applied to. Given there is a large prize pool for the challenge, I would imagine actually training a model in the way you describe would already have been tried and is more difficult that it seems.
- deleted 2y ago
  [deleted]
- ec109685 2y ago
  I realize I didn’t scroll to other examples for one I found very hard. I guess the question is whether someone who solves this will have cracked AGI as a necessary precondition or like other Turing tests that have been solved, someone will find a technique that isn’t broadly applicable to general intelligence.

riku_iki 2y ago

I think huge advantage is that they keep eval tests private, so corps can't finetune them to model and claim breakthrough, which possibly happened with many other benchmarks.

visarga 2y ago

There is a hidden test set with new puzzle types not seen in the open part. It's designed so that humans do well and AI models have a hard time.

YeGoblynQueenne 2y ago
"Designed" is not right. What gives "AI models" (i.e. deep neural nets) a hard time is that there are very few examples in the public training and evaluation set: each task has three examples. So basically it's not a test of intelligence but a test of sample efficiency. Besides which, it is unfair because it excludes an entire category of systems, not to mention a dominant one. If F. Chollet really believes ARC is a test of intelligence, then why not provide enough examples for deep nets or some other big data approach to be trained effectively? The answer is: because a big data approach would then easily beat the test. But if the test can be beaten without intelligence, just with data, then it's not a test of intelligence. My guess for a long time has been that ARC will fall just like the Winograd Schema challenge (WSC) [1] fell: someone will do the work to generate enough (tens of thousands) examples of ARC-like tasks, then train a deep neural net and go to town. That's what happened with the WSC. A large dataset of Winograd schema sentences was crowd-sourced and a big BERT-era Transformer got around 90% accuracy on the WSC [2]. Bye bye WSC, and any wishful thinking about Winograd schemas requiring human intuition and other undefined stuff. Or, ARC might go the way of the Bongard Problems [3]: the original 100 problems by Bongard still stand unsolved, but the machine learning community has effectively sidestepped them. Someone made a generator of Bongard-like problems [4], and while this was not enough to solve the original problems, everyone simply switched to training CNNs and reporting results on the new dataset [5]. We basically have no idea how to create a test for intelligence that computers cannot beat by brute force or big data approaches so we have no effective way to test computers for (artificial) intelligence. The only thing we know humans can do that computers can't is identify undecidable problems (like Barber Paradoxes i.e. statements of the form "this sentence is false", as in Gödel's second incompleteness theorem). Unfortunately we already know there is no computer that can ever do that, and even if we observe say ChatGPT returning the right answer we can be sure it has only memorised, not calculated it, so we're a bit stuck. ARC won't get us unstuck in any way shape or form and so it's just a distraction. _____________________ [1] https://en.wikipedia.org/wiki/Winograd_schema_challenge https://en.wikipedia.org/wiki/Winograd_schema_challenge [2] WinoGrande: An Adversarial Winograd Schema Challenge at Scale https://arxiv.org/abs/1907.10641 https://arxiv.org/abs/1907.10641 Although note the results are interpreted to mean LLMs are more or less memorising answers, which is right of course. [3] Index of Bongard Problems https://www.foundalis.com/res/bps/bpidx.htm https://www.foundalis.com/res/bps/bpidx.htm [4] Comparing machines and humans on a visual categorization test https://www.pnas.org/doi/abs/10.1073/pnas.1109168108 https://www.pnas.org/doi/abs/10.1073/pnas.1109168108 [5] 25 years of CNNs: Can we compare to human abstraction capabilities? https://arxiv.org/abs/1607.08366 https://arxiv.org/abs/1607.08366
- falcor84 2y ago
  > My guess for a long time has been that ARC will fall just like the Winograd Schema challenge (WSC) fell: someone will do the work to generate enough (tens of thousands) examples of ARC-like tasks, then train a deep neural net and go to town. I think that this would be the real AGI (or even superintelligence hurdle) - having essentially a metacognitive AI understand that something given to it is a novel problem, for which it would use the given examples to automatically generate synthetic data sets and then train itself (or a subordinate model) based on these examples to gain the skill of solving this general type of problem. > The only thing we know humans can do that computers can't is identify undecidable problems (like Barber Paradoxes i.e. statements of the form "this sentence is false", as in Gödel's second incompleteness theorem). Unfortunately we already know there is no computer that can ever do that Where did the "ever" come from? Why wouldn't future computers be able to do this at (least at) a human level?
  - YeGoblynQueenne 2y ago
    The "ever" comes from the Church-Turing thesis. Maybe in the future computers will not be Turing machines, but that we can't know yet.
    - falcor84 2y ago
      That refers to the unsolvability of the general case, which is of course also unsolvable by humans.
      - deleted 2y ago
        [deleted]
      - YeGoblynQueenne 2y ago
        I'm sorry, I'm not sure what "unsolvability" means. What I'm saying above is that humans can identify undecidable statements, i.e. we can recognise them as undecidable. If we couldn't, Gödel, Church, and Turing would not have a proof. But we can. We just don't do it algorithmically, obviously- because there is no algorithm that can do that, and so no computer that can, either.

isotypic 2y ago

Do you have some examples of ones you found beyond what a human could straightforwardly figure out? I tried a bunch and they all seemed reasonable, so I would be interested in seeing - I didn't try all 400, for obvious reasons, so I don't doubt there are difficult ones. I think regardless one of the reasons people are interested in it is that is a fairly simple logic puzzle - given some examples, extrapolate a pattern, execute the pattern - that humans achieve high accuracy on (a study linked on the website has ~84% accuracy for humans, some more recent study seems to put it closer to 75%). Yet ML approaches have yet to reach that level, in contrast to other problems ML has been applied to. Given there is a large prize pool for the challenge, I would imagine actually training a model in the way you describe would already have been tried and is more difficult that it seems.

deleted 2y ago
[deleted]
ec109685 2y ago
I realize I didn’t scroll to other examples for one I found very hard. I guess the question is whether someone who solves this will have cracked AGI as a necessary precondition or like other Turing tests that have been solved, someone will find a technique that isn’t broadly applicable to general intelligence.