3 ms·
i think y'all are both right. grokking is a phenomenon that by definition applies to severely overfit neural networks, which is a very different regime than mod
by whimsicalism 2y ago
i think y'all are both right. grokking is a phenomenon that by definition applies to severely overfit neural networks, which is a very different regime than modern ML - but we might learn something from this that we can use to improve regularization
- barfbagginus 2y agoLooks like grokking could give better reasoning and generalization to LLMs, but I'm not sure how practical it would be to overfit a larger LLM See: https://arxiv.org/abs/2405.15071 https://arxiv.org/abs/2405.15071