7 ms·

I just googled arc agi questions, and it looks like it is similar to an iq test with raven matrix. Similar as in you have some examples of images before and aft

I just googled arc agi questions, and it looks like it is similar to an iq test with raven matrix. Similar as in you have some examples of images before and after, then an image before and you have to guess the after. Could anyone confirm if this is the only kind of questions in the benchmark? If yes, how come there is such a direct connection to "oh this performs better than humans" when llm can be quite better than us in understanding and forecasting patterns? I'm just curious, not trying to stir up controversies

zebomon 2y ago
It's a test on which (apparently until now) the vast majority of humans have far outperformed all machine systems.
- patrickhogan1 2y ago
  But it’s not a test that directly shows general intelligence. I am excited no less! This is huge improvement. How does this do on SWE Bench?
  - famouswaffles 2y ago
    >How does this do on SWE Bench? 71.7%
    - throwaway0123_5 2y ago
      I've seen this figure on a few tech news websites and reddit but can't find an official source. If it was in the video I must have missed it, where is this coming from?
      - famouswaffles 2y ago
        It was in the video. I don't know if Open ai have a page up yet
ALittleLight 2y ago
Yes, it's pretty similar to Raven's. The reason it is an interesting benchmark is because humans, even very young humans, "get" the test in the sense of understanding what it's asking and being able to do pretty well on it - but LLMs have really struggled with the benchmark in the past. Chollett (one of the creators of the ARC benchmark) has been saying it proves LLMs can't reason. The test questions are supposed to be unique and not in the model's training set. The fact that LLMs struggled with the ARC challenge suggested (to Chollett and others) that models weren't "Truly reasoning" but rather just completing based on things they'd seen before - when the models were confronted with things they hadn't seen before, the novel visual patterns, they really struggled.
Eridrus 2y ago
ML is quite good at understanding and forecasting patterns when you train on the data you want to forecast. LLMs manage to do so much because we just decided to train on everything on the internet and hope that it included everything we ever wanted to know. This tries to create patterns that are intentionally not in the data and see if a system can generalize to them, which o3 super impressively does!
- yunwal 2y ago
  ARC is in the dataset though? I mean I'm aware that there are new puzzles every day, but there's still a very specific format and set of skills required to solve it. I'd bet a decent amount of money that humans get better at ARC with practice, so it seems strange to suggest that AI wouldn't.

zebomon 2y ago

It's a test on which (apparently until now) the vast majority of humans have far outperformed all machine systems.

patrickhogan1 2y ago
But it’s not a test that directly shows general intelligence. I am excited no less! This is huge improvement. How does this do on SWE Bench?
- famouswaffles 2y ago
  >How does this do on SWE Bench? 71.7%
  - throwaway0123_5 2y ago
    I've seen this figure on a few tech news websites and reddit but can't find an official source. If it was in the video I must have missed it, where is this coming from?
    - famouswaffles 2y ago
      It was in the video. I don't know if Open ai have a page up yet

ALittleLight 2y ago

Yes, it's pretty similar to Raven's. The reason it is an interesting benchmark is because humans, even very young humans, "get" the test in the sense of understanding what it's asking and being able to do pretty well on it - but LLMs have really struggled with the benchmark in the past. Chollett (one of the creators of the ARC benchmark) has been saying it proves LLMs can't reason. The test questions are supposed to be unique and not in the model's training set. The fact that LLMs struggled with the ARC challenge suggested (to Chollett and others) that models weren't "Truly reasoning" but rather just completing based on things they'd seen before - when the models were confronted with things they hadn't seen before, the novel visual patterns, they really struggled.

Eridrus 2y ago

ML is quite good at understanding and forecasting patterns when you train on the data you want to forecast. LLMs manage to do so much because we just decided to train on everything on the internet and hope that it included everything we ever wanted to know. This tries to create patterns that are intentionally not in the data and see if a system can generalize to them, which o3 super impressively does!

yunwal 2y ago
ARC is in the dataset though? I mean I'm aware that there are new puzzles every day, but there's still a very specific format and set of skills required to solve it. I'd bet a decent amount of money that humans get better at ARC with practice, so it seems strange to suggest that AI wouldn't.