mtcrawshaw

3 ms·

1. 1.▲
   by mtcrawshaw 4y ago

   Of course interesting things happen outside of mainstream deep learning. The problem is that sorting by citation count is a terrible way to look broadly at research, because mainstream deep learning is so hyped up that there are just way mo
2. 2.▲
   by mtcrawshaw 4y ago

   Moving the momentum update before the parameter update changes the algorithm from LION to SignSGD. The only novel part of their algorithm is the fact that the momentum changes after the update is computed. They are not equivalent even if yo
3. 3.▲
   by mtcrawshaw 4y ago

   The second line appears unnecessary because the formula in that comment is missing one line from the definition of LION. See my above comment.
4. 4.▲
   by mtcrawshaw 4y ago

   Your formula is almost right but there is a slight difference. Namely the third line of Program 1 in their paper is absent from your statement, which will change the value of the momentum variable for subsequent steps. To be fair, I don&#x2