2 ms·
The source you cite is talking about "large" batch sizes which are still a miniscule fraction of the training set. Their (good) advice has no relevance at all i
by modeless 4y ago
The source you cite is talking about "large" batch sizes which are still a miniscule fraction of the training set. Their (good) advice has no relevance at all in a discussion of full-batch training.
Full-batch training is not the ideal case. It's a recipe for overfitting. We don't optimize models for their training set performance. We need them to generalize to the true data, of which our training set may not even be a representative sample. SGD works exceptionally well for that.
- DougBTX 4y agoTake a look at the linked paper, they train using batches with about 5% of ImageNet in them (batch size = 2^16), which is pretty large from my perspective. Overall they show diminishing returns for larger batch sizes after a certain point. That fits the intuition that SGD is an approximation to GD, which gets better as batch size increases until the approximation error is negligible.