6 ms·

I'd always conceptualized decreasing batch size as a performance/memory optimization to deal with the fact that datasets don't all fit into memory and to reduce

I'd always conceptualized decreasing batch size as a performance/memory optimization to deal with the fact that datasets don't all fit into memory and to reduce overall training time. You look at batch_size samples and compute the sum of the gradient of the errors to update the network weights so as to reduce the error -- shouldn't a larger batch_size inherently provide more information about the optimal direction of the update? It seems to my naive view like it should be "nice" from an accuracy perspective to look at more samples before making an adjustment to the network weights ...? In general, does changing the batch_size hyperparameter make a lot of difference on different problems ...? Does the right value for batch size tend to be problem specific and/or network architecture specific?

nullc 9y ago
[Warning: far from an expert here] No, batches also help you escape local minima.
- sxyuan 9y ago
  [Another non-expert commenting] Your comment didn't make sense to me at first, but I think I get it now. Even if you were able to fit the entire dataset into memory, batches are still a good idea, because optimizing on the entire dataset is non-convex and will likely lead you into a local minimum. However, what is a local minimum for one batch may not be a local minimum for the next batch, which helps you escape. This explains why optimization gets harder with very large batch sizes - the gradients for different batches become more similar (as they resemble the "global" gradient more closely), so you become more susceptible to local minima. I think this also explains why the learning rate scaling helps - it increases the variance across gradients, and helps you escape local minima.
  - breatheoften 9y ago
    This explanation of why larger batches might make optimization harder makes a fair amount of sense to me. This comment on Quora helps a bit as well - with a slightly different approach to characterizing the trade off of batch size/accuracy/runtime performance. https://www.quora.com/Intuitively-how-does-mini-batch-size-affect-the-performance-of-stochastic-gradient-descent https://www.quora.com/Intuitively-how-does-mini-batch-size-a... I wonder if rather than computing a single gradient for a large batch you could simultaneously compute a gradient for the batch and for several subsets of the batch -- then pick or combine the gradient subset(s) that most differ from the full batch result. Not sure if that would work out to a computational efficiency gain. Are there any optimizers that dynamically scale the batch size up/down based on an online metric?
oerpli 9y ago
One of the better articles I found about this topic when I've been learning for my exam: http://sebastianruder.com/optimizing-gradient-descent/index.html http://sebastianruder.com/optimizing-gradient-descent/index....
IanCal 9y ago
> shouldn't a larger batch_size inherently provide more information about the optimal direction of the update? Not necessarily, since a batch gradient output (as I understand it, and at least used to code it) all gets averaged together. Consider standing in a valley with two equal hills either side of you. If you were to try one direction and see that climbing that way helps, you'd take a step that way. Then the next step would keep taking you up that hill. Now, if you batched together two direction tests, what would happen? You'd average together your left and right and end up with moving nowhere. Having both at the same time doesn't give you better information about how you move if you only see the result after averaging. This interestingly maps to something we see in humans, though I'm struggling to find a decent paper on it (from the PRISM lab in Birmingham, UK if anyone else has any luck, think the person doing the research might have been called Chris). Simple adaptation tasks, in this case learning to control a joystick that has a clockwise/anticlockwise force applied to it, don't work well if you try and learn both one thing and the opposite straight away. However, sleeping in-between learning each left you able to do both well. Perhaps this was early results though. Batch tradeoffs: https://stats.stackexchange.com/questions/164876/tradeoff-batch-size-vs-number-of-iterations-to-train-a-neural-network https://stats.stackexchange.com/questions/164876/tradeoff-ba... https://arxiv.org/abs/1609.04836 https://arxiv.org/abs/1609.04836
- jononor 9y ago
  If this is the case, could one get improved learning by mixing large and small batches?

nullc 9y ago

[Warning: far from an expert here] No, batches also help you escape local minima.

sxyuan 9y ago
[Another non-expert commenting] Your comment didn't make sense to me at first, but I think I get it now. Even if you were able to fit the entire dataset into memory, batches are still a good idea, because optimizing on the entire dataset is non-convex and will likely lead you into a local minimum. However, what is a local minimum for one batch may not be a local minimum for the next batch, which helps you escape. This explains why optimization gets harder with very large batch sizes - the gradients for different batches become more similar (as they resemble the "global" gradient more closely), so you become more susceptible to local minima. I think this also explains why the learning rate scaling helps - it increases the variance across gradients, and helps you escape local minima.
- breatheoften 9y ago
  This explanation of why larger batches might make optimization harder makes a fair amount of sense to me. This comment on Quora helps a bit as well - with a slightly different approach to characterizing the trade off of batch size/accuracy/runtime performance. https://www.quora.com/Intuitively-how-does-mini-batch-size-affect-the-performance-of-stochastic-gradient-descent https://www.quora.com/Intuitively-how-does-mini-batch-size-a... I wonder if rather than computing a single gradient for a large batch you could simultaneously compute a gradient for the batch and for several subsets of the batch -- then pick or combine the gradient subset(s) that most differ from the full batch result. Not sure if that would work out to a computational efficiency gain. Are there any optimizers that dynamically scale the batch size up/down based on an online metric?

oerpli 9y ago

One of the better articles I found about this topic when I've been learning for my exam: http://sebastianruder.com/optimizing-gradient-descent/index.html http://sebastianruder.com/optimizing-gradient-descent/index....

IanCal 9y ago

> shouldn't a larger batch_size inherently provide more information about the optimal direction of the update? Not necessarily, since a batch gradient output (as I understand it, and at least used to code it) all gets averaged together. Consider standing in a valley with two equal hills either side of you. If you were to try one direction and see that climbing that way helps, you'd take a step that way. Then the next step would keep taking you up that hill. Now, if you batched together two direction tests, what would happen? You'd average together your left and right and end up with moving nowhere. Having both at the same time doesn't give you better information about how you move if you only see the result after averaging. This interestingly maps to something we see in humans, though I'm struggling to find a decent paper on it (from the PRISM lab in Birmingham, UK if anyone else has any luck, think the person doing the research might have been called Chris). Simple adaptation tasks, in this case learning to control a joystick that has a clockwise/anticlockwise force applied to it, don't work well if you try and learn both one thing and the opposite straight away. However, sleeping in-between learning each left you able to do both well. Perhaps this was early results though. Batch tradeoffs: https://stats.stackexchange.com/questions/164876/tradeoff-batch-size-vs-number-of-iterations-to-train-a-neural-network https://stats.stackexchange.com/questions/164876/tradeoff-ba... https://arxiv.org/abs/1609.04836 https://arxiv.org/abs/1609.04836

jononor 9y ago
If this is the case, could one get improved learning by mixing large and small batches?