4 ms·

It improves Adam in the full-batch setting, which is not usually done or desired since minibatch should regularize and improve the loss.

• cs702 4y ago
  Ah yes, that too. I forgot to point it out. Thanks!
• DougBTX 4y ago
  Batch size can be used for regularisation, but using it for that will limit training performance. From the Google Research Tuning Playbook: > The batch size governs the training speed and shouldn't be used to directly tune the validation set performance. Often, the ideal batch size will be the largest batch size supported by the available hardware. > […] > As long as all hyperparameters are well-tuned (especially the learning rate and regularization hyperparameters) and the number of training steps is sufficient, the same final performance should be attainable using any batch size (see Shallue et al. 2018). https://github.com/google-research/tuning_playbook#choosing-the-batch-size https://github.com/google-research/tuning_playbook#choosing-... The ideal case is full-batch with tuneable regularisation, just the hardware gets expensive.
  • modeless 4y ago
    The source you cite is talking about "large" batch sizes which are still a miniscule fraction of the training set. Their (good) advice has no relevance at all in a discussion of full-batch training. Full-batch training is not the ideal case. It's a recipe for overfitting. We don't optimize models for their training set performance. We need them to generalize to the true data, of which our training set may not even be a representative sample. SGD works exceptionally well for that.
    • DougBTX 4y ago
      Take a look at the linked paper, they train using batches with about 5% of ImageNet in them (batch size = 2^16), which is pretty large from my perspective. Overall they show diminishing returns for larger batch sizes after a certain point. That fits the intuition that SGD is an approximation to GD, which gets better as batch size increases until the approximation error is negligible.