4 ms·
DeepMind to OpenAI and everyone else[1]: > your hyperparameters are bad and you should feel bad It's amazing to me that such a big goof was missed by so many
by _hark 4y ago
DeepMind to OpenAI and everyone else[1]:
> your hyperparameters are bad and you should feel bad
It's amazing to me that such a big goof was missed by so many for so long. All these multimillion dollar language models and people just took the scaling laws at face value.
[1]: https://arxiv.org/abs/2203.15556 https://arxiv.org/abs/2203.15556
- gwern 4y agoIsn't it amazing? One of those reminders that we don't understand DL as much as we like to pretend we do. Cyclic learning rates work well elsewhere, but I can't think of any other case where switching made such a difference. I was completely shocked to read Chinchilla, and I'm still a little baffled - a cosine schedule? Really? That's it? You guys didn't make any other changes?