3 ms·
Interesting take but: After a reread it's nice to see the optimizer is faster but how long is spent in the optimizer and can adamw be tuned for a low memory en
by rob_c 2y ago
Interesting take but:
After a reread it's nice to see the optimizer is faster but how long is spent in the optimizer and can adamw be tuned for a low memory environment given its greedy to try and reduce the impact of statistical noise on gradient calculations.
Note that when training on image1k it only becomes comparable to adamw after many epoch and infact performs measurably worse for most of the training session.
(How significant that is, is up to debate and model/task/data)
Why not incorporate 2nd order changes into adamw directly?
The lower memory footprint is nice but it's not immediately why this is the case. Is the batch size reduced? Model changed? I'll be read this after a 2nd coffee and see if it is more obvious...
Still promising if true.
- yobbo 2y agoI haven't read more than the abstract of this particular paper, but it is expected that training behaves differently with/without adam. The problem with adam is that it keeps one more statistic (the same size as the model parameters) in memory. It also adds a little computation. The way to deal with it otherwise is to tune the momentum-parameters and clip/limit the gradient in various ways.