5 ms·
> 50% accuracy on the public test set for ARC-AGI by having GPT-4o Isn't the public test set public on github and therefore GPT-4o trained on it?
by rgbrgb 2y ago
> 50% accuracy on the public test set for ARC-AGI by having GPT-4o
Isn't the public test set public on github and therefore GPT-4o trained on it?
- bongodongobob 2y agoI keep seeing this comment all over the place. Just because something exists 1 time in the training data doesn't mean it can just regurgitate that. That's not how training works. An LLM is not a knowledge database.
- adroniser 2y agoAnd yet it doesn't rule out that it can't. See new york times lawsuit
- bongodongobob 2y agoFrom old pieces of articles that are quoted all over the internet? That's not surprising.
- ben_w 2y agoThat's still sufficient for both The Times and for it to be a potential problem in this case.
- spencerchubb 2y agoIt could exist many times. People can fork and clone the repo. People are likely to copy the examples and share them online.
- daemonologist 2y agoIn this case I don't think having seen the Arc set would help much in writing and selecting python scripts to solve the test cases. (Unless someone else has tried this approach before and _their_ results are in the training data.) It will be good to see the private set results though.
- cma 2y agoPublic discussions of solutions to the public test set will presumably have somewhat similar analogies and/or embeddings to aspects of the python programs that solve them.
- Truth_In_Lies 2y agoYes, someone has tried https://iprc-dip.github.io/DARC/ https://iprc-dip.github.io/DARC/