3 ms·
This paper is about convergence speed, not flops, so I'm not sure why this observation would be relevant. And at any rate, Adam being fastest assumes that seco
by civilized 3y ago
This paper is about convergence speed, not flops, so I'm not sure why this observation would be relevant.
And at any rate, Adam being fastest assumes that second-order methods require much more computation per iteration than first-order ones, which isn't true for a lot of practical problems. Classical optimization theory favored second-order methods because their faster convergence on many problems outweighed the extra computation per iteration.
The idea that Adam is best isn't a general principle, it's an empirical observation that, as far as I know, is limited to the context of training large neural networks.
- AbrahamParangi 3y agoAsking about convergence speed in terms of "steps" or anything that isn't FLOPS or wall clock training speed is sometimes academically interesting but encourages "hiding the ball" so to speak. You can in general hide quantity in the number of steps by increasing the complexity of each step (for the trivial case, imagine one step of 5-ADAM which performs the ADAM update and evaluation 5 times - 5-ADAM will converge much faster than ADAM alone when measured in steps-to-convergence).