2 ms·
Reminds me of that old quote about “the difference between average and state of the art is forgetting to turn it off over summer break” or similar. I wonder if
by taneq 2y ago
Reminds me of that old quote about “the difference between average and state of the art is forgetting to turn it off over summer break” or similar.
I wonder if this is why smaller LLMs seem to punch above their weight, are they further along in the process of distilling the data down into understanding?
- ionwake 2y agoI love this! Can you tell me more about the quote please
- scarmig 2y agoNot about that quote in particular, but the original grokking paper from Powers et al came about because they accidentally left a training job to run too long (at least as related by gwern).
- gwern 2y agoI'm not sure they've written that anywhere else (which is a cautionary lesson for anyone trying to understand how research happens, BTW), but as further evidence besides just 'Ethan told me so on EAI Discord [IIRC]', you can see that in the original Reddit discussion where I mention Caballero's poster conversation, the lead author comments several times and doesn't contradict that anecdote: https://www.reddit.com/r/mlscaling/comments/n78584/grokking_generalization_beyond_overfitting_on/ https://www.reddit.com/r/mlscaling/comments/n78584/grokking_...
- gwern 2y agohttp://karpathy.github.io/2019/04/25/recipe/ http://karpathy.github.io/2019/04/25/recipe/ > leave it training. I’ve often seen people tempted to stop the model training when the validation loss seems to be leveling off. In my experience networks keep training for unintuitively long time. One time I accidentally left a model training during the winter break and when I got back in January it was SOTA (“state of the art”). (This is not the oldest version, and doesn't mention the NN in question, but I believe this was about Neuraltalk, his image captioner.)