3 ms·

> We can use a similar idea to take an existing optimizer such as Adam and convert it to a hyperparameter-free optimizer that is guaranteed to monotonically red

> We can use a similar idea to take an existing optimizer such as Adam and convert it to a hyperparameter-free optimizer that is guaranteed to monotonically reduce the loss (in the full-batch setting). I'm not a huge fan of analyses like this that make one huge assumption that never holds in practice. Nobody uses full-batch gradient descent. It's not possible in practice or even desirable in theory. The stochasticity of SGD is important for performance! There is no reason to expect results with unrealistic assumptions like these to generalize to the cases people actually care about. > at the cost of a single additional forward pass that increases the wall time for each step by a small factor (about 2x in the example above). So it doubles the training time and FLOPS. Then the graph X axis should be wall time or FLOPS when comparing the methods, instead of steps.

GistNoesis 4y ago
In the paper https://arxiv.org/pdf/2212.11429.pdf https://arxiv.org/pdf/2212.11429.pdf In point 5.5 (page 57) Future work : " Promising areas of future work include: • Mini-batch optimization. Both the theory and experiments in this chapter have been limited to full- batch optimization. However, the MM paradigm can be extended to mini-batch optimization (e.g., [54]). A universal mini-batch MM optimization algorithm may outperform Adam and AdaGrad on certain large-scale machine learning problems. " The current limits of this algorithm (point 2 page 56) is that the reverse mode bounding is not implemented yet (and related work didn't obtain tight bounds) so you can't yet apply it directly for function with huge number of input parameters, so for the time being the main usage is limited to few hyper-parameter optimizations.
fn-mote 4y ago
> So it doubles the training time and FLOPS. Then the graph X axis should be wall time or FLOPS when comparing the methods, instead of steps. Thanks for pointing this out. Inflating your results by tricky measurement methods has been around forever, but it can be hard to spot if you're not close enough to the field. Also, unfortunately, the blog post does not contain anything about the key idea used. (Well, I couldn't find it.) Trust regions have been around forever; that's not the new idea. It would be nice if somewhere there was a piece about "this is what we do better". Decoding a scientific article to figure out how much fluff it is seems like it has an uncertain return.
6gvONxR4sf7o 4y ago
So what? This isn't some kind of gotcha. They're being explicit about the limitation. The built an interesting thing and showed that it does interesting things in a toy setting. This gets me excited about future work down this path.
- modeless 4y ago
  I disagree that the blog post was explicit enough. Tacking parenthesized disclaimers onto the end of your sentences is not really very explicit. What they're comparing against isn't even Adam; calling it Adam at all is misleading. The subtitle of the Adam paper is "A Method for Stochastic Optimization". It's not designed or intended for full-batch, non-stochastic training. What they should do is lead that paragraph with the fact that this is full-batch, note that this is not normal, and explain why they chose to do it anyway. And they shouldn't call it plain "Adam". The paper is probably good, and it's fine to test things in toy settings, but you need to be upfront with your assumptions and limitations.
- godelski 4y ago
  With the hype many people have a difficult time interpreting research. Thinking it is all results oriented. For readers, you need to remember that there's still a lot of theory research that's important. Research is far more than getting benchmark results or flashy images. Benchmarks are only proxies for what we're actually trying to learn and as such need to be evaluated with care (I wrote about this recently). This is especially true for metics that use parametric models to evaluate, such as FID. It's problematic if we can't do research in a research field. Research isn't production, though the two can overlap.
popinman322 4y ago
I also liked the graph that showed only training loss vs Adam. Would have been nice to see validation loss as well; are we just helping Adam overfit the data?

GistNoesis 4y ago

In the paper https://arxiv.org/pdf/2212.11429.pdf https://arxiv.org/pdf/2212.11429.pdf In point 5.5 (page 57) Future work : " Promising areas of future work include: • Mini-batch optimization. Both the theory and experiments in this chapter have been limited to full- batch optimization. However, the MM paradigm can be extended to mini-batch optimization (e.g., [54]). A universal mini-batch MM optimization algorithm may outperform Adam and AdaGrad on certain large-scale machine learning problems. " The current limits of this algorithm (point 2 page 56) is that the reverse mode bounding is not implemented yet (and related work didn't obtain tight bounds) so you can't yet apply it directly for function with huge number of input parameters, so for the time being the main usage is limited to few hyper-parameter optimizations.

fn-mote 4y ago

> So it doubles the training time and FLOPS. Then the graph X axis should be wall time or FLOPS when comparing the methods, instead of steps. Thanks for pointing this out. Inflating your results by tricky measurement methods has been around forever, but it can be hard to spot if you're not close enough to the field. Also, unfortunately, the blog post does not contain anything about the key idea used. (Well, I couldn't find it.) Trust regions have been around forever; that's not the new idea. It would be nice if somewhere there was a piece about "this is what we do better". Decoding a scientific article to figure out how much fluff it is seems like it has an uncertain return.

6gvONxR4sf7o 4y ago

So what? This isn't some kind of gotcha. They're being explicit about the limitation. The built an interesting thing and showed that it does interesting things in a toy setting. This gets me excited about future work down this path.

modeless 4y ago
I disagree that the blog post was explicit enough. Tacking parenthesized disclaimers onto the end of your sentences is not really very explicit. What they're comparing against isn't even Adam; calling it Adam at all is misleading. The subtitle of the Adam paper is "A Method for Stochastic Optimization". It's not designed or intended for full-batch, non-stochastic training. What they should do is lead that paragraph with the fact that this is full-batch, note that this is not normal, and explain why they chose to do it anyway. And they shouldn't call it plain "Adam". The paper is probably good, and it's fine to test things in toy settings, but you need to be upfront with your assumptions and limitations.
godelski 4y ago
With the hype many people have a difficult time interpreting research. Thinking it is all results oriented. For readers, you need to remember that there's still a lot of theory research that's important. Research is far more than getting benchmark results or flashy images. Benchmarks are only proxies for what we're actually trying to learn and as such need to be evaluated with care (I wrote about this recently). This is especially true for metics that use parametric models to evaluate, such as FID. It's problematic if we can't do research in a research field. Research isn't production, though the two can overlap.

popinman322 4y ago

I also liked the graph that showed only training loss vs Adam. Would have been nice to see validation loss as well; are we just helping Adam overfit the data?