3 ms·
I suspect that's because higher LR leads to better exploration of the opt. surface, i.e. it works as an implicit regularizer. The ideal solution would be to dev
by 0-_-0 7y ago
I suspect that's because higher LR leads to better exploration of the opt. surface, i.e. it works as an implicit regularizer. The ideal solution would be to develop better regularizers to go with the better optimizers, instead of relying on the noise in the worse optimizer for implicit regularization.
- Straw 7y agoI completely agree, we shouldn't be depending on our optimizers do some approximate Bayesian inference- an optimizer should optimize only. However, I think it's a different effect- even purely in terms of optimizing the training loss, on a quadratic (with noisy gradients), the short-horizon bias effect exists.