4 ms·
> Ban offline training/pretraining. Models must train from scratch after submission Previously this was considered impossible so rule. My model shows this is po
by foota 3mo ago
> Ban offline training/pretraining. Models must train from scratch after submission
Previously this was considered impossible so rule. My model shows this is possible
Guarantees no synthetic data can be used
It makes the comparison fair across differet models. Otherwise some models like LLMs can benchmaxx ARC by using ungodly amounts of offline training. (Since the benchmark has been around a long time, many ARC-like datasets have been created)
I'm not an ML researcher, so YMMV, but... how could a model learn to answer these ARC-AGI questions without training beforehand?
- nicman23 3mo agohow would you as a human know the answer to the arc-agi questions?
- foota 3mo agoI spent years learning logic and doing puzzles. I don't think a baby or even average kid could solve these.
- anon291 3mo agoPredicate logic is trivially realized by linear transformations (I.e., matrices), and these matrices are easily discovered via gradient descent with appropriate reward functions. > I don't think a baby or even average kid could solve these The reward functions of a typical baby or kid is not 'get a huge dopamine boost when you solve a logic puzzle' (or whatever neurotransmitter, I don't know).
- evilmathkid 3mo agotrain from scratch only during the 12 hours allowed on Kaggle Other competitions have implemented things like this before. Eg: OpenAI's Parameter Golf and Keller Jordan's Modded NanoGPT Speedrun
- foota 3mo ago[dead]