3 ms·

ARC-AGI is not yet killed by LLM. O3 achieved a breakthrough only on ARC-AGI-PUB, which is semi-private. Nothing guarantees that the test data wasn't leaked to

ARC-AGI is not yet killed by LLM. O3 achieved a breakthrough only on ARC-AGI-PUB, which is semi-private. Nothing guarantees that the test data wasn't leaked to OpenAI in previous testing rounds, because the model is not running offline. See: https://news.ycombinator.com/item?id=42478098 https://news.ycombinator.com/item?id=42478098

anon373839 2y ago
I think this should be discussed more. Models that can only be accessed via API cannot be tested without giving their owners access to the test data. You just have to trust that they’ll do the right thing.
- Tepix 2y ago
  In particular, in cases where the model gets 16 hours to solve a task that a human can solve in a few minutes, cheating is trivial!
Tepix 2y ago
See https://arcprize.org/blog/oai-o3-pub-breakthrough https://arcprize.org/blog/oai-o3-pub-breakthrough ARC-AGI-1 will be replaced by ARC-AGI-2 So yes, ARC-AGI-1 was killed.
- yamrzou 2y ago
  ARC-AGI-2 was planned long before those results came out. Also from the link: ARC-AGI-2 (same format - verified easy for humans, harder for AI) will launch alongside ARC Prize 2025. So, no, it will not replace it.
  - Tepix 2y ago
    Was it? I think reaching >70% made it a necessity.