4 ms·
Alternatively, one could get rid of the memory used by optimizers entirely by switching to vanilla SGD. I haven’t tried this on transformers and maybe that’s w
by vladf 6y ago
Alternatively, one could get rid of the memory used by optimizers entirely by switching to vanilla SGD.
I haven’t tried this on transformers and maybe that’s what breaks down here but in “classic” supervised settings I’ve found SGD with schedule tuning just as fast as Adam.
- gwern 6y agoSGD doesn't work on large Transformers, no. You need something like AdamW.
- The_rationalist 6y agoMish is generally superior to RadamW https://lessw.medium.com/meet-mish-new-state-of-the-art-ai-activation-function-the-successor-to-relu-846a6d93471f https://lessw.medium.com/meet-mish-new-state-of-the-art-ai-a...
- dron57 6y agoThat's cool, but Mish is an activation function while SGD and AdamW are optimizers. Apples and oranges.