Take a look at the linked paper, they train using batches with about 5% of ImageNet in them (batch size = 2^16), which is pretty large from my perspective. Over

Take a look at the linked paper, they train using batches with about 5% of ImageNet in them (batch size = 2^16), which is pretty large from my perspective. Overall they show diminishing returns for larger batch sizes after a certain point. That fits the intuition that SGD is an approximation to GD, which gets better as batch size increases until the approximation error is negligible.