3 ms·

I love this! Can you tell me more about the quote please

• scarmig 2y ago
  Not about that quote in particular, but the original grokking paper from Powers et al came about because they accidentally left a training job to run too long (at least as related by gwern).
  • gwern 2y ago
    I'm not sure they've written that anywhere else (which is a cautionary lesson for anyone trying to understand how research happens, BTW), but as further evidence besides just 'Ethan told me so on EAI Discord [IIRC]', you can see that in the original Reddit discussion where I mention Caballero's poster conversation, the lead author comments several times and doesn't contradict that anecdote: https://www.reddit.com/r/mlscaling/comments/n78584/grokking_generalization_beyond_overfitting_on/ https://www.reddit.com/r/mlscaling/comments/n78584/grokking_...
• gwern 2y ago
  http://karpathy.github.io/2019/04/25/recipe/ http://karpathy.github.io/2019/04/25/recipe/ > leave it training. I’ve often seen people tempted to stop the model training when the validation loss seems to be leveling off. In my experience networks keep training for unintuitively long time. One time I accidentally left a model training during the winter break and when I got back in January it was SOTA (“state of the art”). (This is not the oldest version, and doesn't mention the NN in question, but I believe this was about Neuraltalk, his image captioner.)