3 ms·
This result is an amazing twofer. Not only do they give a formal probabilistic perspective on how deep nets fall into local minima, the result proposed a way to
by evrydayhustling 8y ago
This result is an amazing twofer. Not only do they give a formal probabilistic perspective on how deep nets fall into local minima, the result proposed a way to reduce two "voodoo parameters" - learning rate and batch size - to one, by showing how to pick the other in a way that is optimal for expected generalization.
- mlthoughts2018 8y agoI suspect actually there is some connection between optimal batch size and the proofs of this paper regarding how deep NN will provably converge to zero training loss with gradient descent (not stochastic gradient descent). https://arxiv.org/abs/1811.03804 https://arxiv.org/abs/1811.03804
- samgd 8y agoYou should check out our blog post related to finding the optimal learning rate, batch size, and momentum setting: https://www.myrtle.ai/2018/09/24/how-to-train-your-resnet-5/ https://www.myrtle.ai/2018/09/24/how-to-train-your-resnet-5/