3 ms·
Interesting, but I have questions. The OP shows how SafeRate improves plain-vanilla Adam, but almost no one uses that anymore. The most commonly used optimizer
by cs702 4y ago
Interesting, but I have questions. The OP shows how SafeRate improves plain-vanilla Adam, but almost no one uses that anymore. The most commonly used optimizer nowadays is probably AdamW. Does SafeRate improve AdamW to the same extent, or at all? How does using SafeRate compare with Bayesian hyperparameter search? How does it compare with rule-of-thumb hyperparameter selection methods? Many people seem to get a lot of mileage with AdamW and sensible defaults (e.g., a "one cycle" schedule with a max lr of 3e-4, momentum peaking at 0.95, and some weight decay)?
- ipunchghosts 4y agoLots of people still use Adam and SGD!
- cs702 4y agoHaven't seen anyone use it for large-scale (or largish-scale) learning in a long while...
- GaggiX 4y agoIt improves Adam in the full-batch setting, which is not usually done or desired since minibatch should regularize and improve the loss.
- cs702 4y agoAh yes, that too. I forgot to point it out. Thanks!
- DougBTX 4y agoBatch size can be used for regularisation, but using it for that will limit training performance. From the Google Research Tuning Playbook: > The batch size governs the training speed and shouldn't be used to directly tune the validation set performance. Often, the ideal batch size will be the largest batch size supported by the available hardware. > […] > As long as all hyperparameters are well-tuned (especially the learning rate and regularization hyperparameters) and the number of training steps is sufficient, the same final performance should be attainable using any batch size (see Shallue et al. 2018). https://github.com/google-research/tuning_playbook#choosing-the-batch-size https://github.com/google-research/tuning_playbook#choosing-... The ideal case is full-batch with tuneable regularisation, just the hardware gets expensive.
- modeless 4y agoThe source you cite is talking about "large" batch sizes which are still a miniscule fraction of the training set. Their (good) advice has no relevance at all in a discussion of full-batch training. Full-batch training is not the ideal case. It's a recipe for overfitting. We don't optimize models for their training set performance. We need them to generalize to the true data, of which our training set may not even be a representative sample. SGD works exceptionally well for that.
- DougBTX 4y agoTake a look at the linked paper, they train using batches with about 5% of ImageNet in them (batch size = 2^16), which is pretty large from my perspective. Overall they show diminishing returns for larger batch sizes after a certain point. That fits the intuition that SGD is an approximation to GD, which gets better as batch size increases until the approximation error is negligible.