Your formula is almost right but there is a slight difference. Namely the third line of Program 1 in their paper is absent from your statement, which will chang

Your formula is almost right but there is a slight difference. Namely the third line of Program 1 in their paper is absent from your statement, which will change the value of the momentum variable for subsequent steps. To be fair, I don't think that this operation will fundamentally change the behavior of the algorithm. In fact, I don't think their algorithm has a fundamentally different behavior than SignSGD with momentum, whose convergence behavior is well-understood [1, 2, 3]. Actually, the algorithm you described is equivalent to SignSGD with momentum. In the algorithm you described, m is a convex combination of m and g (with coefficient alpha), then c is a convex combination of updated m and g (with coefficient beta). This is exactly the same as setting c as a convex combination of m and g with coefficient alpha*beta. And that is exactly SignSGD with momentum. The only difference between the algorithm you described (equivalent to SignSGD) and LION is that they further modify the value of the momentum after computing the update with an additional convex combination operation. Again, I am skeptical that this operation will fundamentally change the behavior of the algorithm. But that's just my guess. [1] https://arxiv.org/abs/1802.04434 https://arxiv.org/abs/1802.04434 [2] http://proceedings.mlr.press/v97/karimireddy19a.html http://proceedings.mlr.press/v97/karimireddy19a.html [3] https://arxiv.org/abs/2208.11195 https://arxiv.org/abs/2208.11195