4 ms·

Take a look at the ARC Prize, which is a test for achieving "AGI" created in 2019 by François Chollet. Scroll down halfway on the home page and ponder the steep

Take a look at the ARC Prize, which is a test for achieving "AGI" created in 2019 by François Chollet. Scroll down halfway on the home page and ponder the steep yellow line on the graph. That's what OpenAI o3 recently achieved. [0] https://arcprize.org/ https://arcprize.org/ [1] https://arcprize.org/blog/oai-o3-pub-breakthrough https://arcprize.org/blog/oai-o3-pub-breakthrough

mrshadowgoose 2y ago
Reviewing the actual problems is highly recommended: https://kts.github.io/arc-viewer/ https://kts.github.io/arc-viewer/ They're not particularly difficult, but clearly require reasoning to solve.
- UnlockedSecrets 2y ago
  unless you train directly against solving those problems... in which case how could you theoretically design a test that could stand against training directly against the answer sheet?
  - munchler 2y ago
    That's why they keep the evaluation set private: "Submit a solution which scores 85% on the ARC-AGI private evaluation set and win $600K." [0] https://arcprize.org/guide https://arcprize.org/guide
EdwardDiego 2y ago
So we're only 12% from AGI? I'm dubious tbh. Given we still can't simulate a nematode.
- munchler 2y ago
  That's why I put "AGI" in quotes. The point is that six months ago, no one expected an LLM to score this well.
  - EdwardDiego 2y ago
    Fair enough.
- simonw 2y ago
  ARC creator François Chollet says: https://bsky.app/profile/fchollet.bsky.social/post/3les3izgdj22j https://bsky.app/profile/fchollet.bsky.social/post/3les3izgd... I don't think people really appreciate how simple ARC-AGI-1 was, and what solving it really means. It was designed as the simplest, most basic assessment of fluid intelligence possible. Failure to pass signifies a near-total inability to adapt or problem-solve in unfamiliar situations. Passing it means your system exhibits non-zero fluid intelligence -- you're finally looking at something that isn't pure memorized skill. But it says rather little about how intelligent your system is, or how close to human intelligence it is.
  - sitkack 2y ago
    > designed as the simplest, most basic assessment of fluid intelligence possible. This was the goal, but that doesn't say what the test itself is. Try to get a human to solve this problem without their visual cortex, they couldn't do it. Stating your goal for a thing, doesn't make the thing that goal. AI researchers designing intelligence tests are like programmers designing their own cryptography. How about we have people skilled in neuropsychology, psychometrics and cognitive psychology do what they are good at.
    - baq 2y ago
      > How about we have people skilled in neuropsychology, psychometrics and cognitive psychology do what they are good at. Disagree. The thing that we will eventually call AGI will not be human. No need to have human-specific evaluations unless you’re aiming for an artificial human and not just an artificial intelligence.
      - sitkack 2y ago
        But why ignore a huge body of research in how to write scientific tests of intelligence and cognition? Smells like linear algebra exceptionalism. Is ARC AGI really the, "simplest, most basic assessment of fluid intelligence possible" ?
        2y ago
- wslh 2y ago
  Yes, the 12% impact is significant, especially in a societal context, because it represents a shift in how people access and process information. Even without AGI, the comparison between LLMs and search engines is crucial. LLMs provide synthesized, conversational responses rather than just indexing and ranking web pages. This shift reduces the need for users to evaluate multiple sources manually, which has far-reaching implications.
- deleted 2y ago
  [deleted]
0x008 2y ago
> OpenAI's new o3 system - trained on the ARC-AGI-1 Public Training set… Sounds fishy to me
- MacsHeadroom 2y ago
  That's the purpose of the "public training set." You don't take an exam before reviewing the instructions on how to fill out the answer sheet.
  - guitarlimeo 2y ago
    You would think that the training set for the models already included enough of Mensa etc iq tests so that the model knows how to do these kinds of tests. It takes humans 2 or at most 3 examples to "get" what the test is asking for, and then they can start filling the answers to the actual questions. Meanwhile it takes hundreds of answers at least (in the public set) to train o3 to do this test. The need for a huge training set to solve simple questions has never stopped bewildering me. I think to get a human-like intelligent model we need to figure out why humans learn from 2 examples and the models don't. But I don't mean to say that the current models aren't intelligent in their own way or aren't useful already.
    - orbital-decay 2y ago
      Human intelligence is bootstrapped by biological evolution and the society, neither of which is fast or efficient. Truly individual part of the intelligence is tiny, it's vastly overrated and relies on these two. Similarly, LLMs perform in-context learning, which is much more efficient because it relies on the pre-baked knowledge. Yes, the generalization ability is still incomparable to humans, but it's entirely possible that much better ability is achievable by slowly bootstrapping it.
  - 0x008 2y ago
    If you train a model on tasks in a similar format it's even less indicative of "AGI".

mrshadowgoose 2y ago

Reviewing the actual problems is highly recommended: https://kts.github.io/arc-viewer/ https://kts.github.io/arc-viewer/ They're not particularly difficult, but clearly require reasoning to solve.

UnlockedSecrets 2y ago
unless you train directly against solving those problems... in which case how could you theoretically design a test that could stand against training directly against the answer sheet?
- munchler 2y ago
  That's why they keep the evaluation set private: "Submit a solution which scores 85% on the ARC-AGI private evaluation set and win $600K." [0] https://arcprize.org/guide https://arcprize.org/guide

EdwardDiego 2y ago

So we're only 12% from AGI? I'm dubious tbh. Given we still can't simulate a nematode.

munchler 2y ago
That's why I put "AGI" in quotes. The point is that six months ago, no one expected an LLM to score this well.
- EdwardDiego 2y ago
  Fair enough.
simonw 2y ago
ARC creator François Chollet says: https://bsky.app/profile/fchollet.bsky.social/post/3les3izgdj22j https://bsky.app/profile/fchollet.bsky.social/post/3les3izgd... I don't think people really appreciate how simple ARC-AGI-1 was, and what solving it really means. It was designed as the simplest, most basic assessment of fluid intelligence possible. Failure to pass signifies a near-total inability to adapt or problem-solve in unfamiliar situations. Passing it means your system exhibits non-zero fluid intelligence -- you're finally looking at something that isn't pure memorized skill. But it says rather little about how intelligent your system is, or how close to human intelligence it is.
- sitkack 2y ago
  > designed as the simplest, most basic assessment of fluid intelligence possible. This was the goal, but that doesn't say what the test itself is. Try to get a human to solve this problem without their visual cortex, they couldn't do it. Stating your goal for a thing, doesn't make the thing that goal. AI researchers designing intelligence tests are like programmers designing their own cryptography. How about we have people skilled in neuropsychology, psychometrics and cognitive psychology do what they are good at.
  - baq 2y ago
    > How about we have people skilled in neuropsychology, psychometrics and cognitive psychology do what they are good at. Disagree. The thing that we will eventually call AGI will not be human. No need to have human-specific evaluations unless you’re aiming for an artificial human and not just an artificial intelligence.
    - sitkack 2y ago
      But why ignore a huge body of research in how to write scientific tests of intelligence and cognition? Smells like linear algebra exceptionalism. Is ARC AGI really the, "simplest, most basic assessment of fluid intelligence possible" ?
      - 2y ago
wslh 2y ago
Yes, the 12% impact is significant, especially in a societal context, because it represents a shift in how people access and process information. Even without AGI, the comparison between LLMs and search engines is crucial. LLMs provide synthesized, conversational responses rather than just indexing and ranking web pages. This shift reduces the need for users to evaluate multiple sources manually, which has far-reaching implications.
deleted 2y ago
[deleted]

0x008 2y ago

> OpenAI's new o3 system - trained on the ARC-AGI-1 Public Training set… Sounds fishy to me

MacsHeadroom 2y ago
That's the purpose of the "public training set." You don't take an exam before reviewing the instructions on how to fill out the answer sheet.
- guitarlimeo 2y ago
  You would think that the training set for the models already included enough of Mensa etc iq tests so that the model knows how to do these kinds of tests. It takes humans 2 or at most 3 examples to "get" what the test is asking for, and then they can start filling the answers to the actual questions. Meanwhile it takes hundreds of answers at least (in the public set) to train o3 to do this test. The need for a huge training set to solve simple questions has never stopped bewildering me. I think to get a human-like intelligent model we need to figure out why humans learn from 2 examples and the models don't. But I don't mean to say that the current models aren't intelligent in their own way or aren't useful already.
  - orbital-decay 2y ago
    Human intelligence is bootstrapped by biological evolution and the society, neither of which is fast or efficient. Truly individual part of the intelligence is tiny, it's vastly overrated and relies on these two. Similarly, LLMs perform in-context learning, which is much more efficient because it relies on the pre-baked knowledge. Yes, the generalization ability is still incomparable to humans, but it's entirely possible that much better ability is achievable by slowly bootstrapping it.
- 0x008 2y ago
  If you train a model on tasks in a similar format it's even less indicative of "AGI".