10 ms·
Symbolic Discovery of Optimization Algorithms
- cs702 4y agoThis looks great! The authors are reputable and claim to have discovered a simple lr optimizer than trains models up to 5x faster and converges to better local optima than Adam/AdamW. As usual, YMMV. You can find a PyTorch implementation here: https://github.com/lucidrains/lion-pytorch https://github.com/lucidrains/lion-pytorch Preliminary tests seem to confirm the paper's claims.
- eastWestMath 4y ago> Our method discovers a simple and effective optimization algorithm, Lion (EvoLved Sign Momentum). Come on now
- Torkel 4y agoYeah, the acronym is very far fetched… But at least the algorithm and how they got to it seems dope.
- snovv_crash 4y agoThey could have called it ELESIUM with a lot less effort...
- roenxi 4y agoAlthough your approach to it much better, they may as well have called is XERXES by combining letters that are and aren't in the phrase with some permutation. There is no link between the name "Evolved Sign Momentum" and "Lion". They can call the thing what they want, but they're making up the link.
- adammarples 4y agoCould have called it Eve
- dgacmu 4y agoOr SignUp. Even makes a good paper title: SignUp for faster training by evolving the learning algorithm.
- bhouston 4y agoMaybe they need an algorithm that makes proper algorithm acronyms…
- Nevermark 4y agoYour "maybe" is grossly unnecessary! Get an editor, sir!
- saiojd 4y agoHahaha gotta love your terrible acronyms :)
- sbayona573 4y agoWhat does it even mean? And can it be applied to AI?
- noduerme 4y agoHow is this symbolic? At a quick look, it seems NN-based.
- wizzard0 4y agoThey discover a NN optimization algorithm via search in a symbolic space. Most of the article is about using it for NNs and not discovery itself though :(
- noduerme 4y agolol. That's literally the basis of https://gwern.net/fiction/clippy https://gwern.net/fiction/clippy [edit] When people say that the road to hell is paved with good intentions, or that a little knowledge is a dangerous thing, or... you know, pick your standard folk wisdom saying... what's so goddamn disappointing to me is how a generation that should know much better than this is eager to take shortcuts with the convenience of enormous and unprecedented processing power, rather than using their own brains to puzzle out clever ways of doing the same thing.
- michaelmior 4y agoPeople have spent enormous amounts of time and effort to find clever optimizers. This is another example of that. I don't really see this as any less clever than if someone had written the algorithm by hand.
- noduerme 4y agoWell, it's exactly less clever because once you've written an optimizer to find optimizers, you've cut yourself out of the loop and you're just a manager of things you don't understand. Go use my public tool https://doxyjs.com https://doxyjs.com and find yourself a compression algo; you probably can. If you understand why it works, that's interesting.
- wizzard0 4y ago
- inductive_magic 4y agoVery cool stuff. For one, symbolic discovery is beautiful. Second, LION (what a sin of an acronym tho) seems like a pretty big deal. Really nice paper to start the saturday with.
- im3w1l 4y agoOptimization algorithm they arrived at was (quoting from the paper) def train(weight, gradient, momentum, lr): update = interp(gradient, momentum, β1) update = sign(update) momentum = interp(gradient, momentum, β2) weight_decay = weight * λ update = update + weight_decay update = update * lr return update, momentum The key thing they do, sending the update through sign, means all weight updates turn into either 1 or -1.
- nestorD 4y ago1 or -1 times the learning rate.
- yobbo 4y agoThe formula (from their pseudocode); m ← αm + (1-α)g // momentum c ← βm + (1-β)g // new part θ ← θ + η sign(c) // update of parameters The parameter update is via the sign of the weighted average of the momentum and the gradient. For comparison, the gradient of abs(x) is sign(x), which is the update with L1-norm.
- WithinReason 4y agoIs the 2nd line really necessary? Since β is 0.9, the second line becomes: c ← 0.9m + (0.1)g There doesn't seem to be a significant difference between their code: θ ← θ + η sign(0.9m + (0.1)g) and just not having the 2nd line at all: θ ← θ + η sign(m)
- yobbo 4y agoTrue, those are equivalent. Might even become identical once it is compiled.
- mtcrawshaw 4y agoThe second line appears unnecessary because the formula in that comment is missing one line from the definition of LION. See my above comment.
- mtcrawshaw 4y agoYour formula is almost right but there is a slight difference. Namely the third line of Program 1 in their paper is absent from your statement, which will change the value of the momentum variable for subsequent steps. To be fair, I don't think that this operation will fundamentally change the behavior of the algorithm. In fact, I don't think their algorithm has a fundamentally different behavior than SignSGD with momentum, whose convergence behavior is well-understood [1, 2, 3]. Actually, the algorithm you described is equivalent to SignSGD with momentum. In the algorithm you described, m is a convex combination of m and g (with coefficient alpha), then c is a convex combination of updated m and g (with coefficient beta). This is exactly the same as setting c as a convex combination of m and g with coefficient alpha*beta. And that is exactly SignSGD with momentum. The only difference between the algorithm you described (equivalent to SignSGD) and LION is that they further modify the value of the momentum after computing the update with an additional convex combination operation. Again, I am skeptical that this operation will fundamentally change the behavior of the algorithm. But that's just my guess. [1] https://arxiv.org/abs/1802.04434 https://arxiv.org/abs/1802.04434 [2] http://proceedings.mlr.press/v97/karimireddy19a.html http://proceedings.mlr.press/v97/karimireddy19a.html [3] https://arxiv.org/abs/2208.11195 https://arxiv.org/abs/2208.11195