5 ms·

Weren't we barely scraping 1-10% on this with state of the art models a year ago and it was considered that this is the final boss, ie solve this and its almost

by nubg 8mo ago

Weren't we barely scraping 1-10% on this with state of the art models a year ago and it was considered that this is the final boss, ie solve this and its almost AGI-like? I ask because I cannot distinguish all the benchmarks by heart.

verdverm 8mo ago
Here's a good thread over 1+ month, as each model comes out https://bsky.app/profile/pekka.bsky.social/post/3meokmizvt22v https://bsky.app/profile/pekka.bsky.social/post/3meokmizvt22... tl;dr - Pekka says Arc-AGI-2 is now toast as a benchmark
- Aperocky 8mo ago
  If you look at the problem space it is easy to see why it's toast, maybe there's intelligence in there, but hardly general.
  - verdverm 8mo ago
    the best way I've seen this describes is "spikey" intelligence, really good at some points, those make the spikes humans are the same way, we all have a unique spike pattern, interests and talents ai are effectively the same spikes across instances, if simplified. I could argue self driving vs chatbots vs world models vs game playing might constitute enough variation. I would not say the same of Gemini vs Claude vs ... (instances), that's where I see "spikey clones"
    - Aperocky 8mo ago
      You can get more spiky with AIs, whereas with human brain we are more hard wired. So maybe we are forced to be more balanced and general whereas AI don't have to.
      - verdverm 8mo ago
        I suspect the non-spikey part is the more interesting comparison Why is it so easy for me to open the car door, get in, close the door, buckle up. You can do this in the dark and without looking. There are an infinite number of little things like this you think zero about, take near zero energy, yet which are extremely hard for Ai
        gowld 8mo ago
        You are asking a robotics question, not an AI question. Robotics is more and less than AI. Boston Dynamics robots are getting quite near your benchmark.
        idiotsecant 8mo ago
        Boston dynamics is missing just about all the degrees of freedom involved in the scenario op mentions.
        pixl97 8mo ago
        >Why is it so easy for me to open the car door Because this part of your brain has been optimized for hundreds of millions of years. It's been around a long ass time and takes an amazingly low amount of energy to do these things. On the other hand the 'thinking' part of your brain, that is your higher intelligence is very new to evolution. It's expensive to run. It's problematic when giving birth. It's really slow with things like numbers, heck a tiny calculator and whip your butt in adding. There's a term for this, but I can't think of it at the moment.
        te7447 8mo ago
        > There's a term for this, but I can't think of it at the moment. Moravec's paradox: https://epoch.ai/gradient-updates/moravec-s-paradox https://epoch.ai/gradient-updates/moravec-s-paradox
        pixl97 8mo ago
  - tasuki 8mo ago
    > maybe there's intelligence in there, but hardly general. Of course. Just as our human intelligence isn't general.
fishpham 8mo ago
Yes, but benchmarks like this are often flawed because leading model labs frequently participate in 'benchmarkmaxxing' - ie improvements on ARC-AGI2 don't necessarily indicate similar improvements in other areas (though it does seem like this is a step function increase in intelligence for the Gemini line of models)
- jstummbillig 8mo ago
  Could it also be that the models are just a lot better than a year ago?
  - bigbadfeline 8mo ago
    > Could it also be that the models are just a lot better than a year ago? No, the proof is in the pudding. After AI we're having higher prices, higher deficits and lower standard of living. Electricity, computers and everything else costs more. "Doing better" can only be justified by that real benchmark. If Gemini 3 DT was better we would have falling prices of electricity and everything else at least until they get to pre-2019 levels.
    - ctoth 8mo ago
      > If Gemini 3 DT was better we would have falling prices of electricity and everything else at least Man, I've seen some maintenance folks down on the field before working on them goalposts but I'm pretty sure this is the first time I saw aliens from another Universe literally teleport in, grab the goalposts, and teleport out.
    - WarmWash 8mo ago
      You might call me crazy, but at least in 2024, consumers spent ~1% less of their income on expenses than 2019[2], which suggests that 2024 is more affordable than 2019. This is from the BLS consumer survey report released in dec[1] [1]https://www.bls.gov/news.release/cesan.nr0.htm https://www.bls.gov/news.release/cesan.nr0.htm [2]https://www.bls.gov/opub/reports/consumer-expenditures/2019/ https://www.bls.gov/opub/reports/consumer-expenditures/2019/ Prices are never going back to 2019 numbers though
modeless 8mo ago
François Chollet, creator of ARC-AGI, has consistently said that solving the benchmark does not mean we have AGI. It has always been meant as a stepping stone to encourage progress in the correct direction rather than as an indicator of reaching the destination. That's why he is working on ARC-AGI-3 (to be released in a few weeks) and ARC-AGI-4. His definition of reaching AGI, as I understand it, is when it becomes impossible to construct the next version of ARC-AGI because we can no longer find tasks that are feasible for normal humans but unsolved by AI.
- hmmmmmmmmmmmmmm 8mo ago
  I don't think the creator believes ARC3 can't be solved but rather that it can't be solved "efficiently" and >$13 per task for ARC2 is certainly not efficient. But at this rate, the people who talk about the goal posts shifting even once we achieve AGI may end up correct, though I don't think this benchmark is particularly great either.
- beklein 8mo ago
  https://x.com/fchollet/status/2022036543582638517 https://x.com/fchollet/status/2022036543582638517
  - joelthelion 8mo ago
    Do opus 4.6 or gemini deep think really use test time adaptation ? How does it work in practice?
- mapontosevenths 8mo ago
  > His definition of reaching AGI, as I understand it, is when it becomes impossible to construct the next version of ARC-AGI because we can no longer find tasks that are feasible for normal humans but unsolved by AI. That is the best definition I've yet to read. If something claims to be conscious and we can't prove it's not, we have no choice but to believe it. Thats said, I'm reminded of the impossible voting tests they used to give black people to prevent them from voting. We dont ask nearly so much proof from a human, we take their word for it. On the few occasions we did ask for proof it inevitably led to horrific abuse. Edit: The average human tested scores 60%. So the machines are already smarter on an individual basis than the average human.

by nubg 8mo ago

verdverm 8mo ago

Here's a good thread over 1+ month, as each model comes out https://bsky.app/profile/pekka.bsky.social/post/3meokmizvt22v https://bsky.app/profile/pekka.bsky.social/post/3meokmizvt22... tl;dr - Pekka says Arc-AGI-2 is now toast as a benchmark

Aperocky 8mo ago
If you look at the problem space it is easy to see why it's toast, maybe there's intelligence in there, but hardly general.
- verdverm 8mo ago
  the best way I've seen this describes is "spikey" intelligence, really good at some points, those make the spikes humans are the same way, we all have a unique spike pattern, interests and talents ai are effectively the same spikes across instances, if simplified. I could argue self driving vs chatbots vs world models vs game playing might constitute enough variation. I would not say the same of Gemini vs Claude vs ... (instances), that's where I see "spikey clones"
  - Aperocky 8mo ago
    You can get more spiky with AIs, whereas with human brain we are more hard wired. So maybe we are forced to be more balanced and general whereas AI don't have to.
    - verdverm 8mo ago
      I suspect the non-spikey part is the more interesting comparison Why is it so easy for me to open the car door, get in, close the door, buckle up. You can do this in the dark and without looking. There are an infinite number of little things like this you think zero about, take near zero energy, yet which are extremely hard for Ai
      - gowld 8mo ago
        You are asking a robotics question, not an AI question. Robotics is more and less than AI. Boston Dynamics robots are getting quite near your benchmark.
        idiotsecant 8mo ago
        Boston dynamics is missing just about all the degrees of freedom involved in the scenario op mentions.
      - pixl97 8mo ago
        >Why is it so easy for me to open the car door Because this part of your brain has been optimized for hundreds of millions of years. It's been around a long ass time and takes an amazingly low amount of energy to do these things. On the other hand the 'thinking' part of your brain, that is your higher intelligence is very new to evolution. It's expensive to run. It's problematic when giving birth. It's really slow with things like numbers, heck a tiny calculator and whip your butt in adding. There's a term for this, but I can't think of it at the moment.
        te7447 8mo ago
        > There's a term for this, but I can't think of it at the moment. Moravec's paradox: https://epoch.ai/gradient-updates/moravec-s-paradox https://epoch.ai/gradient-updates/moravec-s-paradox
        pixl97 8mo ago
- tasuki 8mo ago
  > maybe there's intelligence in there, but hardly general. Of course. Just as our human intelligence isn't general.

fishpham 8mo ago

Yes, but benchmarks like this are often flawed because leading model labs frequently participate in 'benchmarkmaxxing' - ie improvements on ARC-AGI2 don't necessarily indicate similar improvements in other areas (though it does seem like this is a step function increase in intelligence for the Gemini line of models)

jstummbillig 8mo ago
Could it also be that the models are just a lot better than a year ago?
- bigbadfeline 8mo ago
  > Could it also be that the models are just a lot better than a year ago? No, the proof is in the pudding. After AI we're having higher prices, higher deficits and lower standard of living. Electricity, computers and everything else costs more. "Doing better" can only be justified by that real benchmark. If Gemini 3 DT was better we would have falling prices of electricity and everything else at least until they get to pre-2019 levels.
  - ctoth 8mo ago
    > If Gemini 3 DT was better we would have falling prices of electricity and everything else at least Man, I've seen some maintenance folks down on the field before working on them goalposts but I'm pretty sure this is the first time I saw aliens from another Universe literally teleport in, grab the goalposts, and teleport out.
  - WarmWash 8mo ago
    You might call me crazy, but at least in 2024, consumers spent ~1% less of their income on expenses than 2019[2], which suggests that 2024 is more affordable than 2019. This is from the BLS consumer survey report released in dec[1] [1]https://www.bls.gov/news.release/cesan.nr0.htm https://www.bls.gov/news.release/cesan.nr0.htm [2]https://www.bls.gov/opub/reports/consumer-expenditures/2019/ https://www.bls.gov/opub/reports/consumer-expenditures/2019/ Prices are never going back to 2019 numbers though

modeless 8mo ago

François Chollet, creator of ARC-AGI, has consistently said that solving the benchmark does not mean we have AGI. It has always been meant as a stepping stone to encourage progress in the correct direction rather than as an indicator of reaching the destination. That's why he is working on ARC-AGI-3 (to be released in a few weeks) and ARC-AGI-4. His definition of reaching AGI, as I understand it, is when it becomes impossible to construct the next version of ARC-AGI because we can no longer find tasks that are feasible for normal humans but unsolved by AI.

hmmmmmmmmmmmmmm 8mo ago
I don't think the creator believes ARC3 can't be solved but rather that it can't be solved "efficiently" and >$13 per task for ARC2 is certainly not efficient. But at this rate, the people who talk about the goal posts shifting even once we achieve AGI may end up correct, though I don't think this benchmark is particularly great either.
beklein 8mo ago
https://x.com/fchollet/status/2022036543582638517 https://x.com/fchollet/status/2022036543582638517
- joelthelion 8mo ago
  Do opus 4.6 or gemini deep think really use test time adaptation ? How does it work in practice?
mapontosevenths 8mo ago
> His definition of reaching AGI, as I understand it, is when it becomes impossible to construct the next version of ARC-AGI because we can no longer find tasks that are feasible for normal humans but unsolved by AI. That is the best definition I've yet to read. If something claims to be conscious and we can't prove it's not, we have no choice but to believe it. Thats said, I'm reminded of the impossible voting tests they used to give black people to prevent them from voting. We dont ask nearly so much proof from a human, we take their word for it. On the few occasions we did ask for proof it inevitably led to horrific abuse. Edit: The average human tested scores 60%. So the machines are already smarter on an individual basis than the average human.