2 ms·
SGD doesn't work on large Transformers, no. You need something like AdamW.
by gwern 6y ago
SGD doesn't work on large Transformers, no. You need something like AdamW.
- The_rationalist 6y agoMish is generally superior to RadamW https://lessw.medium.com/meet-mish-new-state-of-the-art-ai-activation-function-the-successor-to-relu-846a6d93471f https://lessw.medium.com/meet-mish-new-state-of-the-art-ai-a...
- dron57 6y agoThat's cool, but Mish is an activation function while SGD and AdamW are optimizers. Apples and oranges.