7 ms·

A Bayesian Perspective on Generalization and Stochastic Gradient Descent

182 points by godelmachine 8y ago

evrydayhustling 8y ago
This result is an amazing twofer. Not only do they give a formal probabilistic perspective on how deep nets fall into local minima, the result proposed a way to reduce two "voodoo parameters" - learning rate and batch size - to one, by showing how to pick the other in a way that is optimal for expected generalization.
- mlthoughts2018 8y ago
  I suspect actually there is some connection between optimal batch size and the proofs of this paper regarding how deep NN will provably converge to zero training loss with gradient descent (not stochastic gradient descent). https://arxiv.org/abs/1811.03804 https://arxiv.org/abs/1811.03804
- samgd 8y ago
  You should check out our blog post related to finding the optimal learning rate, batch size, and momentum setting: https://www.myrtle.ai/2018/09/24/how-to-train-your-resnet-5/ https://www.myrtle.ai/2018/09/24/how-to-train-your-resnet-5/
joe_the_user 8y ago
I am increasingly amazed by the use of the term "generalization" - without modifiers or contexts - in describing what current deep learning does. Certainly, deep learning systems do generalize in certain ways when present with certain large datasets. But how can this just be "generic"? Is this just "data you find in the world" without other considerations? Shouldn't they be discussing "generalizing what qualities?" Wouldn't image-generalization be different chat-script-generalization and so-forth?
- mlthoughts2018 8y ago
  Generalization in this context is usually a very specific technical term that means after training on a training set, the fitted model continues to perform well when used on previously unseen examples (a test set) that came from the same distribution (data generating process) that Nature used to produce the training set. You can define other notions of generalization that try to be more than this, like generalizing to multiple tasks or an algorithm capable of playing any turn-based perfect information game, etc., but that is not usually what people are talking about when they talk about the formal statistics concept of generalization error.
  - deleted 8y ago
    [deleted]
  - TeMPOraL 8y ago
    I think what 'joe_the_user is saying is one should specify what exactly is the "distribution" that "Nature used to produce the training set". E.g. "images of cat faces taken indoors and cropped to square" is a pretty specific set.
    - mlthoughts2018 8y ago
      This is usually always specified to a complete degree in research papers that address this issue. I work professionally in computer vision and image processing and have never once encountered a mainstream research paper where the definition of the data generating process under study was not fully and unambiguously clear. Can you point to examples where this was problematically under-specified?
      - TeMPOraL 8y ago
        > Can you point to examples where this was problematically under-specified? Not really. My comment was about how I interpreted joe_the_user's statement; I can't think of a ML or CV research paper I've read to which this criticism would apply, but I definitely can think of such news articles and startup landing pages/marketing material.
letitgo12345 8y ago
The openreview thread -- https://openreview.net/forum?id=BJij4yg0Z https://openreview.net/forum?id=BJij4yg0Z
itissid 8y ago
This development has parallels to tuning parameters in the Montecarlo world, specifically using HCMC+NUTS one does not have to worry about finding the complete distribution and you get warned when the distribution is misbehaved[0]. HCMC+NUTS not only does better exploration of the posterior but can help you select things like step sizes, acceptance rate, simulation time(See the NUTS Sampler[1]) much more easily. [0] http://mc-stan.org/misc/warnings.html#runtime-warnings http://mc-stan.org/misc/warnings.html#runtime-warnings [1] http://mc-stan.org/workshops/ASA2016/day-1.pdf http://mc-stan.org/workshops/ASA2016/day-1.pdf

182 points by godelmachine 8y ago

This result is an amazing twofer. Not only do they give a formal probabilistic perspective on how deep nets fall into local minima, the result proposed a way to reduce two "voodoo parameters" - learning rate and batch size - to one, by showing how to pick the other in a way that is optimal for expected generalization.

mlthoughts2018 8y ago
I suspect actually there is some connection between optimal batch size and the proofs of this paper regarding how deep NN will provably converge to zero training loss with gradient descent (not stochastic gradient descent). https://arxiv.org/abs/1811.03804 https://arxiv.org/abs/1811.03804
samgd 8y ago
You should check out our blog post related to finding the optimal learning rate, batch size, and momentum setting: https://www.myrtle.ai/2018/09/24/how-to-train-your-resnet-5/ https://www.myrtle.ai/2018/09/24/how-to-train-your-resnet-5/

joe_the_user 8y ago

I am increasingly amazed by the use of the term "generalization" - without modifiers or contexts - in describing what current deep learning does. Certainly, deep learning systems do generalize in certain ways when present with certain large datasets. But how can this just be "generic"? Is this just "data you find in the world" without other considerations? Shouldn't they be discussing "generalizing what qualities?" Wouldn't image-generalization be different chat-script-generalization and so-forth?

mlthoughts2018 8y ago
Generalization in this context is usually a very specific technical term that means after training on a training set, the fitted model continues to perform well when used on previously unseen examples (a test set) that came from the same distribution (data generating process) that Nature used to produce the training set. You can define other notions of generalization that try to be more than this, like generalizing to multiple tasks or an algorithm capable of playing any turn-based perfect information game, etc., but that is not usually what people are talking about when they talk about the formal statistics concept of generalization error.
- deleted 8y ago
  [deleted]
- TeMPOraL 8y ago
  I think what 'joe_the_user is saying is one should specify what exactly is the "distribution" that "Nature used to produce the training set". E.g. "images of cat faces taken indoors and cropped to square" is a pretty specific set.
  - mlthoughts2018 8y ago
    This is usually always specified to a complete degree in research papers that address this issue. I work professionally in computer vision and image processing and have never once encountered a mainstream research paper where the definition of the data generating process under study was not fully and unambiguously clear. Can you point to examples where this was problematically under-specified?
    - TeMPOraL 8y ago
      > Can you point to examples where this was problematically under-specified? Not really. My comment was about how I interpreted joe_the_user's statement; I can't think of a ML or CV research paper I've read to which this criticism would apply, but I definitely can think of such news articles and startup landing pages/marketing material.

letitgo12345 8y ago

The openreview thread -- https://openreview.net/forum?id=BJij4yg0Z https://openreview.net/forum?id=BJij4yg0Z

itissid 8y ago

This development has parallels to tuning parameters in the Montecarlo world, specifically using HCMC+NUTS one does not have to worry about finding the complete distribution and you get warned when the distribution is misbehaved[0]. HCMC+NUTS not only does better exploration of the posterior but can help you select things like step sizes, acceptance rate, simulation time(See the NUTS Sampler[1]) much more easily. [0] http://mc-stan.org/misc/warnings.html#runtime-warnings http://mc-stan.org/misc/warnings.html#runtime-warnings [1] http://mc-stan.org/workshops/ASA2016/day-1.pdf http://mc-stan.org/workshops/ASA2016/day-1.pdf