6 ms·

This claim that these tests are easy for humans seems dubious, and so I went looking a bit. Melanie Mitchell chimed in on Chollet's thread and posted their rel

This claim that these tests are easy for humans seems dubious, and so I went looking a bit. Melanie Mitchell chimed in on Chollet's thread and posted their related test [ConceptARC]. In it they question the ease of Chollet's tests: "One limitation on ARC’s usefulness for AI research is that it might be too challenging. Many of the tasks in Chollet’s corpus are difficult even for humans, and the corpus as a whole might be sufficiently difficult for machines that it does not reveal real progress on machine acquisition of core knowledge." ConceptARC is designed to be easier, but then also has to filter ~15% of its own test takers for "[failing] at solving two or more minimal tasks... or they provided empty or nonsensical explanations for their solutions" After this filtering, ConceptARC finds another 10-15% failure rate amongst humans on the main corpus questions, so they're seeing maybe 25-30% unable to solve these simpler questions meant to test for "AGI". ConceptARC's main results show CG4 scoring well below the filtered humans, which would agree with a [Mensa] test result that its IQ=85. Chollet and Mitchell could instead stratify their human groups to estimate IQ then compare with the Mensa measures and see if e.g. Claude3@IQ=100 compares with their ARC scores for their average human [ConceptArc]https://arxiv.org/pdf/2305.07141 https://arxiv.org/pdf/2305.07141 [Mensa]https://www.maximumtruth.org/p/ais-ranked-by-iq-ai-passes-100-iq https://www.maximumtruth.org/p/ais-ranked-by-iq-ai-passes-10...

mikeknoop 2y ago
Here is some published research on the human difficulty of ARC-AGI: https://cims.nyu.edu/~brenden/papers/JohnsonEtAl2021CogSci.pdf https://cims.nyu.edu/~brenden/papers/JohnsonEtAl2021CogSci.p... > We found that humans were able to infer the underlying program and generate the correct test output for a novel test input example, with an average of 84% of tasks solved per participant
mark_l_watson 2y ago
I saw Melanie’s post and I am intrigued by an easier AGI suite. I would like some experimenting done by individuals like myself snd smaller organizations.
- bbor 2y ago
  Are you working on (a book detailing) AGI also? It’s a lonely field but I have no doubt there are a sea of malcontent engineers across the world who saw the truth early on and are pushing solo for AGI. It’s going well for me, but I’m not sure whether to take that as “you’re great” or “it’s really that easy”, so was interested to see such a fellow brazen American on HN of all places. Game on for the million, if so :). If not, apologies for distracting from the good fight for OSS/noncorp devs! E: it occurred to me on the drive home how easily we (engineers) can fall into competitiveness, even when we’ve all read the thinkpieces about why an AI Race would/will be/is incredibly dangerous. Maybe not “game on”, perhaps… “god I hope it’s impossible but best of luck anyway to both of us”?
- PaulDavisThe1st 2y ago
  You actually think that has not been going for 30, 40 or 50 years?
- neoneye2 2y ago
  Melanie is coauthor/supervisor of ConceptARC, that can be tried here: https://neoneye.github.io/arc/?dataset=ConceptARC https://neoneye.github.io/arc/?dataset=ConceptARC
salamo 2y ago
They claim that the average score for humans is between 85% and 100%, so I think there's a disagreement on whether the test is actually too hard. Taking them at their word, if no existing model can score even half what the average human can, the test is certainly measuring some kind of significant difference. I guess there might be a disagreement of whether the problems in ARC are a representative sample of all of the possible abstract programs which could be synthesized, but then again most LLMs are also trained on human data.
- gkbrk 2y ago
  The tasks are very easy for humans. Out of the 6 tasks assigned when I opened the web page, I got all of them correct on the first try. Maybe if you run into some exceptionally difficult tasks it might not be 100%, but there's no way the challenge can be called unfair because it's too difficult for humans too.
kenjackson 2y ago
I just tried the first puzzle and I can't get it right. I think my solution makes logical sense and I explain why the patterns are consistent with the input, but it says its wrong. I'm either a lot dumber than I thought or they need to do a better job of vetting their tests.
- mikeknoop 2y ago
  (You can direct link to a task like this: https://arcprize.org/play?task=009d5c81 https://arcprize.org/play?task=009d5c81 in case you want to share!)
- saati 2y ago
  It's pretty easy, just follow the second example with the colors from the test input. (if it's the same puzzle 00576224 for you too)
  - kenjackson 2y ago
    https://arcprize.org/play?task=00576224 https://arcprize.org/play?task=00576224 Yes the same puzzle. And I followed the second example. This was my solution: GRG OBO RGR B is the cyan like blue color. My solution looks right, but it says it’s wrong.
    - halter73 2y ago
      You need to resize the output grid to 6x6.
      - kenjackson 2y ago
        How come? The pattern should work for any size grid.
        bigyikes 2y ago
        You might be technically correct, but if you extend that logic, why not just make the grid 1x1 and select a single color? The grid size is part of the pattern in the same way that the colors are part of the pattern. It’s not just a color pattern, it’s a generalized mapping of input to output. In short: you need to resize the grid because that’s what the examples do.

mikeknoop 2y ago

Here is some published research on the human difficulty of ARC-AGI: https://cims.nyu.edu/~brenden/papers/JohnsonEtAl2021CogSci.pdf https://cims.nyu.edu/~brenden/papers/JohnsonEtAl2021CogSci.p... > We found that humans were able to infer the underlying program and generate the correct test output for a novel test input example, with an average of 84% of tasks solved per participant

mark_l_watson 2y ago

I saw Melanie’s post and I am intrigued by an easier AGI suite. I would like some experimenting done by individuals like myself snd smaller organizations.

bbor 2y ago
Are you working on (a book detailing) AGI also? It’s a lonely field but I have no doubt there are a sea of malcontent engineers across the world who saw the truth early on and are pushing solo for AGI. It’s going well for me, but I’m not sure whether to take that as “you’re great” or “it’s really that easy”, so was interested to see such a fellow brazen American on HN of all places. Game on for the million, if so :). If not, apologies for distracting from the good fight for OSS/noncorp devs! E: it occurred to me on the drive home how easily we (engineers) can fall into competitiveness, even when we’ve all read the thinkpieces about why an AI Race would/will be/is incredibly dangerous. Maybe not “game on”, perhaps… “god I hope it’s impossible but best of luck anyway to both of us”?
PaulDavisThe1st 2y ago
You actually think that has not been going for 30, 40 or 50 years?
neoneye2 2y ago
Melanie is coauthor/supervisor of ConceptARC, that can be tried here: https://neoneye.github.io/arc/?dataset=ConceptARC https://neoneye.github.io/arc/?dataset=ConceptARC

salamo 2y ago

They claim that the average score for humans is between 85% and 100%, so I think there's a disagreement on whether the test is actually too hard. Taking them at their word, if no existing model can score even half what the average human can, the test is certainly measuring some kind of significant difference. I guess there might be a disagreement of whether the problems in ARC are a representative sample of all of the possible abstract programs which could be synthesized, but then again most LLMs are also trained on human data.

gkbrk 2y ago
The tasks are very easy for humans. Out of the 6 tasks assigned when I opened the web page, I got all of them correct on the first try. Maybe if you run into some exceptionally difficult tasks it might not be 100%, but there's no way the challenge can be called unfair because it's too difficult for humans too.

kenjackson 2y ago

I just tried the first puzzle and I can't get it right. I think my solution makes logical sense and I explain why the patterns are consistent with the input, but it says its wrong. I'm either a lot dumber than I thought or they need to do a better job of vetting their tests.

mikeknoop 2y ago
(You can direct link to a task like this: https://arcprize.org/play?task=009d5c81 https://arcprize.org/play?task=009d5c81 in case you want to share!)
saati 2y ago
It's pretty easy, just follow the second example with the colors from the test input. (if it's the same puzzle 00576224 for you too)
- kenjackson 2y ago
  https://arcprize.org/play?task=00576224 https://arcprize.org/play?task=00576224 Yes the same puzzle. And I followed the second example. This was my solution: GRG OBO RGR B is the cyan like blue color. My solution looks right, but it says it’s wrong.
  - halter73 2y ago
    You need to resize the output grid to 6x6.
    - kenjackson 2y ago
      How come? The pattern should work for any size grid.
      - bigyikes 2y ago
        You might be technically correct, but if you extend that logic, why not just make the grid 1x1 and select a single color? The grid size is part of the pattern in the same way that the colors are part of the pattern. It’s not just a color pattern, it’s a generalized mapping of input to output. In short: you need to resize the grid because that’s what the examples do.