3 ms·

I would take a complementary side to this bet, in that I think it's relevant that it's extremely, extremely extremely hard to improve upon ADAM (Adaptive Moment

I would take a complementary side to this bet, in that I think it's relevant that it's extremely, extremely extremely hard to improve upon ADAM (Adaptive Moment Estimation) in terms of FLOPS-till-convergence. That this is so hard feels notable to me, like we're specifically misunderstanding something about the optimization process in higher dimensions.

civilized 3y ago
This paper is about convergence speed, not flops, so I'm not sure why this observation would be relevant. And at any rate, Adam being fastest assumes that second-order methods require much more computation per iteration than first-order ones, which isn't true for a lot of practical problems. Classical optimization theory favored second-order methods because their faster convergence on many problems outweighed the extra computation per iteration. The idea that Adam is best isn't a general principle, it's an empirical observation that, as far as I know, is limited to the context of training large neural networks.
- AbrahamParangi 3y ago
  Asking about convergence speed in terms of "steps" or anything that isn't FLOPS or wall clock training speed is sometimes academically interesting but encourages "hiding the ball" so to speak. You can in general hide quantity in the number of steps by increasing the complexity of each step (for the trivial case, imagine one step of 5-ADAM which performs the ADAM update and evaluation 5 times - 5-ADAM will converge much faster than ADAM alone when measured in steps-to-convergence).
josephg 3y ago
What are your thoughts on Sophia, an algorithmic replacement for ADAM which promises to converge in fewer steps? https://arxiv.org/abs/2305.14342 https://arxiv.org/abs/2305.14342
- tbalsam 3y ago
  There are dozens of dozens of optimizers. Almost every single one of them claims superiority. Some of them, like MADGRAD, are great. But sticking with the basics is pretty good in my experience.
- lucidrains 3y ago
  https://arxiv.org/abs/2307.06440 https://arxiv.org/abs/2307.06440

civilized 3y ago

This paper is about convergence speed, not flops, so I'm not sure why this observation would be relevant. And at any rate, Adam being fastest assumes that second-order methods require much more computation per iteration than first-order ones, which isn't true for a lot of practical problems. Classical optimization theory favored second-order methods because their faster convergence on many problems outweighed the extra computation per iteration. The idea that Adam is best isn't a general principle, it's an empirical observation that, as far as I know, is limited to the context of training large neural networks.

AbrahamParangi 3y ago
Asking about convergence speed in terms of "steps" or anything that isn't FLOPS or wall clock training speed is sometimes academically interesting but encourages "hiding the ball" so to speak. You can in general hide quantity in the number of steps by increasing the complexity of each step (for the trivial case, imagine one step of 5-ADAM which performs the ADAM update and evaluation 5 times - 5-ADAM will converge much faster than ADAM alone when measured in steps-to-convergence).

josephg 3y ago

What are your thoughts on Sophia, an algorithmic replacement for ADAM which promises to converge in fewer steps? https://arxiv.org/abs/2305.14342 https://arxiv.org/abs/2305.14342

tbalsam 3y ago
There are dozens of dozens of optimizers. Almost every single one of them claims superiority. Some of them, like MADGRAD, are great. But sticking with the basics is pretty good in my experience.
lucidrains 3y ago
https://arxiv.org/abs/2307.06440 https://arxiv.org/abs/2307.06440