3 ms·

Article is from 2016. It only mentions AdamW at the very end in passing. These days I rarely see much besides AdamW in production. Messing with optimizers is o

Article is from 2016. It only mentions AdamW at the very end in passing. These days I rarely see much besides AdamW in production. Messing with optimizers is one of the ways to enter hyperparameter hell: it’s like legacy code but on steroids because changing it only breaks your training code stochastically. Much better to stop worrying and love AdamW.

nkurz 2y ago
The mention of AdamW is brief, but in his defense he includes a link that gives a gloss of it: "An updated overview of recent gradient descent algorithms" [https://johnchenresearch.github.io/demon/ https://johnchenresearch.github.io/demon/].
pizza 2y ago
Luckily we have Shampoo, SOAP, Modula, Schedule-free variants, and many more these days being researched! I am very very excited by the heavyball library in particular
- 3abiton 2y ago
  Been out of the loop for while, anything exciting?
  - pizza 2y ago
    Read the Modular Norms in Deep Learning paper, and follow the author of the heavyball library on twitter with notifications enabled
ImageXav 2y ago
Something that stuck out to me in the updated blog [0] is that Demon Adam performed much better than even AdamW, with very interesting learning curves. I'm wondering now why it didn't become the standard. Anyone here have insights into this? [0] https://johnchenresearch.github.io/demon/ https://johnchenresearch.github.io/demon/
- gzer0 2y ago
  Demon Adam didn’t become standard largely for the same reason many “better” optimizers never see wide adoption: it’s a newer tweak, not clearly superior on every problem, is less familiar to most engineers, and isn’t always bundled in major frameworks. By contrast, AdamW is now the “safe default” that nearly everyone supports and knows how to tune, so teams stick with it unless they have a strong reason not to. Edit: Demon involves decaying the momentum parameter over time, which introduces a new schedule or formula for how momentum should be reduced during training. That can feel like additional complexity or a potential hyperparameter rabbit hole. Teams trying to ship products quickly often avoid adding new hyperparameters unless the gains are decisive.

nkurz 2y ago

The mention of AdamW is brief, but in his defense he includes a link that gives a gloss of it: "An updated overview of recent gradient descent algorithms" [https://johnchenresearch.github.io/demon/ https://johnchenresearch.github.io/demon/].

pizza 2y ago

Luckily we have Shampoo, SOAP, Modula, Schedule-free variants, and many more these days being researched! I am very very excited by the heavyball library in particular

3abiton 2y ago
Been out of the loop for while, anything exciting?
- pizza 2y ago
  Read the Modular Norms in Deep Learning paper, and follow the author of the heavyball library on twitter with notifications enabled

ImageXav 2y ago

Something that stuck out to me in the updated blog [0] is that Demon Adam performed much better than even AdamW, with very interesting learning curves. I'm wondering now why it didn't become the standard. Anyone here have insights into this? [0] https://johnchenresearch.github.io/demon/ https://johnchenresearch.github.io/demon/

gzer0 2y ago
Demon Adam didn’t become standard largely for the same reason many “better” optimizers never see wide adoption: it’s a newer tweak, not clearly superior on every problem, is less familiar to most engineers, and isn’t always bundled in major frameworks. By contrast, AdamW is now the “safe default” that nearly everyone supports and knows how to tune, so teams stick with it unless they have a strong reason not to. Edit: Demon involves decaying the momentum parameter over time, which introduces a new schedule or formula for how momentum should be reduced during training. That can feel like additional complexity or a potential hyperparameter rabbit hole. Teams trying to ship products quickly often avoid adding new hyperparameters unless the gains are decisive.