4 ms·

> Maybe he figured out a model that beats ARC-AGI by 85%? People have, I think. One of the published approaches (BARC) uses GPT-4o to generate a lot more trai

by trott 2y ago

> Maybe he figured out a model that beats ARC-AGI by 85%? People have, I think. One of the published approaches (BARC) uses GPT-4o to generate a lot more training data. The approach is scaling really well so far [1], and whether you expect linear scaling or exponential one [2], the 85% threshold can be reached, using the "transduction" model alone, after generating under 2 million tasks ($20K in OpenAI credits). Perhaps for 2025, the organizers will redesign ARC-AGI to be more resistant to this sort of approach, somehow. --- [1] https://www.kaggle.com/competitions/arc-prize-2024/discussion/543953#3036186 https://www.kaggle.com/competitions/arc-prize-2024/discussio... [2] If you are "throwing darts at a board", you get exponential scaling (the probability of not hitting bullseye at least once reduces exponentially with the number of throws). If you deliberately design your synthetic dataset to be non-redundant, you might get something akin to linear scaling (until you hit perfect accuracy, of course).

thrw42A8N 2y ago
> If you are "throwing darts at a board", you get exponential scaling (the probability of not hitting bullseye reduces exponentially with the number of throws). Honest question - is that so, and why? I thought you have to calculate the probability of each throw individually as nothing fundamentally connects the throws together, only that long term there will be a normal distribution of randomness.
- trott 2y ago
  > The probability of not hitting bullseye at least once ... I added a clarification.
fastball 2y ago
I like the idea of ARC-AGI and think it was worth a shot. But if someone has already hit the human-level threshold, I think the entire idea can be thrown out. If the ARC-AGI challenge did not actually follow their expected graph[1], I see no reason to believe that any benchmark can be designed in a way where it cannot be gamed. Rather, it seems that the existing SOTA models just weren't well-optimized for that one task. The only way to measure "AGI" is in however you define the "G". If your model can only do one thing, it is not AGI and doesn't really indicate you are closer, even if you very carefully designed your challenge. [1] https://static.supernotes.app/ai-benchmarks-2.png https://static.supernotes.app/ai-benchmarks-2.png
- TheDudeMan 2y ago
  What you're calling "gamed" could actually be research and progress in general problem solving.
  - fastball 2y ago
    Almost by definition it is not. If you are "gaming" a specific benchmark, what you have is not progress in general intelligence. The entire premise of the ARC-AGI challenge was that general problem solving would be required. As noted by the GP, one of the top contenders is BARC which performs well by generating a huge amount of training data for this particular problem. That's not general intelligence, that's gaming. There is no reason to believe that technique would not work for any particular problem. After all, this problem was the best attempt the (very intelligent) challenge designers could come up with, as evidenced by putting $1m on the line.
    - trott 2y ago
      > That's not general intelligence, that's gaming. In fairness, their approach is non-trivial. Simply asking GPT-4o to fantasize more examples wouldn't have worked very well. Instead, they have it fantasize inputs and programs, and then run the programs on the inputs to compute the outputs. I think it's a great contribution (although I'm surprised they didn't try making an even bigger dataset -- perhaps they ran out of time or funding)
- nl 2y ago
  > The only way to measure "AGI" is in however you define the "G" "I" isn't usefully defined either. At least most people agree on "Artificial"
  - echelon 2y ago
    That's the problem with intelligence vs the other things we're doing with deep learning. Vision models, image models, video models, audio models? Solved. We've understood the physics of optics and audio for over half a century. We've had ray tracers for forever. It's all well understood, and now we're teaching models to understand it. Intelligence? We can't even describe our own.
mxwsn 2y ago
My interest was piqued, but the extrapolation in [1] is uh... not the most convincing. If there were more data points then sure, maybe
- trott 2y ago
  The plot was just showing where the solid lines were trending (see prior messages), and that happened to predict the performance at 400k samples (red dot) very well. An exponential scaling curve would steer a bit more to the right, but it would still cross the 85% mark before 2000k.
TechDebtDevin 2y ago
I personally think ARC-AGI will be a forgotten, unimportant benchmark that doesn't indicate anything more than a models ability reason, which honestly is just a very small step in the path towards AGI

by trott 2y ago

thrw42A8N 2y ago

> If you are "throwing darts at a board", you get exponential scaling (the probability of not hitting bullseye reduces exponentially with the number of throws). Honest question - is that so, and why? I thought you have to calculate the probability of each throw individually as nothing fundamentally connects the throws together, only that long term there will be a normal distribution of randomness.

trott 2y ago
> The probability of not hitting bullseye at least once ... I added a clarification.

fastball 2y ago

I like the idea of ARC-AGI and think it was worth a shot. But if someone has already hit the human-level threshold, I think the entire idea can be thrown out. If the ARC-AGI challenge did not actually follow their expected graph[1], I see no reason to believe that any benchmark can be designed in a way where it cannot be gamed. Rather, it seems that the existing SOTA models just weren't well-optimized for that one task. The only way to measure "AGI" is in however you define the "G". If your model can only do one thing, it is not AGI and doesn't really indicate you are closer, even if you very carefully designed your challenge. [1] https://static.supernotes.app/ai-benchmarks-2.png https://static.supernotes.app/ai-benchmarks-2.png

TheDudeMan 2y ago
What you're calling "gamed" could actually be research and progress in general problem solving.
- fastball 2y ago
  Almost by definition it is not. If you are "gaming" a specific benchmark, what you have is not progress in general intelligence. The entire premise of the ARC-AGI challenge was that general problem solving would be required. As noted by the GP, one of the top contenders is BARC which performs well by generating a huge amount of training data for this particular problem. That's not general intelligence, that's gaming. There is no reason to believe that technique would not work for any particular problem. After all, this problem was the best attempt the (very intelligent) challenge designers could come up with, as evidenced by putting $1m on the line.
  - trott 2y ago
    > That's not general intelligence, that's gaming. In fairness, their approach is non-trivial. Simply asking GPT-4o to fantasize more examples wouldn't have worked very well. Instead, they have it fantasize inputs and programs, and then run the programs on the inputs to compute the outputs. I think it's a great contribution (although I'm surprised they didn't try making an even bigger dataset -- perhaps they ran out of time or funding)
nl 2y ago
> The only way to measure "AGI" is in however you define the "G" "I" isn't usefully defined either. At least most people agree on "Artificial"
- echelon 2y ago
  That's the problem with intelligence vs the other things we're doing with deep learning. Vision models, image models, video models, audio models? Solved. We've understood the physics of optics and audio for over half a century. We've had ray tracers for forever. It's all well understood, and now we're teaching models to understand it. Intelligence? We can't even describe our own.

mxwsn 2y ago

My interest was piqued, but the extrapolation in [1] is uh... not the most convincing. If there were more data points then sure, maybe

trott 2y ago
The plot was just showing where the solid lines were trending (see prior messages), and that happened to predict the performance at 400k samples (red dot) very well. An exponential scaling curve would steer a bit more to the right, but it would still cross the 85% mark before 2000k.

TechDebtDevin 2y ago

I personally think ARC-AGI will be a forgotten, unimportant benchmark that doesn't indicate anything more than a models ability reason, which honestly is just a very small step in the path towards AGI