4 ms·
The submissions can't use the internet. And I imagine can't be too huge - so you can't use "newest public LLMs" on this task.
by ks2048 2y ago
The submissions can't use the internet. And I imagine can't be too huge - so you can't use "newest public LLMs" on this task.
- mikeknoop 2y agoThat is correct for ARC Prize: limited Kaggle compute (to target efficiency) and no internet (to reduce cheating). We are also trialing a secondary leaderboard called ARC-AGI-Pub that imposes no limits or constraints. Not part of the prize today but could be in the future: https://arcprize.org/leaderboard https://arcprize.org/leaderboard
- cma 2y agoUsing the internet would leak the test data, a big problem with ML benchmarks, and also allow communication with humans during the test.