3 ms·

Can you give a concrete example how a Bayesian concept provides practical help in ML model training or another ML task? Want to whet my appetite for your sugge

Can you give a concrete example how a Bayesian concept provides practical help in ML model training or another ML task? Want to whet my appetite for your suggestion.

bencw 6y ago
A common type of example involves relatively small or uninformative datasets. Say you flip a coin a few times and only get heads. Your maximum likelihood (frequentist) estimate is that the coin will always land heads. In a Bayesian setting, if you have a (say uniform) prior on the probability that the coin lands heads, your maximum a posteriori estimate of this probability will be non-zero, but will get continue to get smaller if you continue only seeing heads. The above example is contrived, but makes more sense in the case of language modelling. Since a bag-of-words vector, containing say counts of words seen in a document, is typically sparse (most documents only contain a limited portion of the full vocabulary), a frequentist estimate of word probability will say that certain words can never occur, just because it's never seen them. The Bayesian estimate will still assign some non-zero chance of seeing that word. Practically speaking, this leads to the idea of "smoothing" in tf-idf (text-frequency-inverse-document-frequency) vectors, by adding 1 to document frequencies. You don't need Bayesian statistics to do this, but maybe you never would have thought of it otherwise!
- astrofinch 6y ago
  >Say you flip a coin a few times and only get heads. Your maximum likelihood (frequentist) estimate is that the coin will always land heads. In a Bayesian setting, if you have a (say uniform) prior on the probability that the coin lands heads, your maximum a posteriori estimate of this probability will be non-zero, but will get continue to get smaller if you continue only seeing heads. Not quite. If you have a uniform prior, there will be no difference between MAP and MLE. >From the vantage point of Bayesian inference, MLE is a special case of maximum a posteriori estimation (MAP) that assumes a uniform prior distribution of the parameters. https://en.wikipedia.org/wiki/Maximum_likelihood_estimation https://en.wikipedia.org/wiki/Maximum_likelihood_estimation More discussion here: https://stats.stackexchange.com/questions/64259/how-does-a-uniform-prior-lead-to-the-same-estimates-from-maximum-likelihood-and https://stats.stackexchange.com/questions/64259/how-does-a-u...
- antipaul 6y ago
  Help me see how this requires a _thorough_ study of Bayesian methods. What you describe seems to be (additive/Laplace) smoothing [1], a fairly basic concept. And I don't see how this is specifically Bayesian. Then, one common critique of "Bayesian stats work well with small samples" is that with small data (weak signal), you get your prior back, so you haven't learned anything, and the only thing you're left with is your "bias", erm, prior. I am still keen to learn why one needs a thorough study of Bayesian methods to help in practical machine learning tasks (meaning, on top of a bit of statistical or data science curriculum plus on the job experience). [1] https://en.wikipedia.org/wiki/Additive_smoothing https://en.wikipedia.org/wiki/Additive_smoothing
sibmike 6y ago
The most recent example has been supply failure detection in sales timeseries data with intermittent demand. Ended up using approach described in The Longest Run of Heads by Mark F. Schilling, which is outstandingly well written stats paper and a pleasure to read.
astrophysician 6y ago
* understanding where loss functions and regularization terms come from allows you to reason about the right choice for your problem and possibly to extend/tweak them to suit your needs. Are you working directly with probabilities? Then maybe you don’t want to use L2 regularization (Gaussian prior) but a beta prior or something with the right support. Are you modeling a Poisson rate (e.g. how many people buy my product for every dollar I spend on advertising), then use a Poisson likelihood (loss function would be negative log likelihood). * do you want to have your NN model your uncertainty as well as the mean? How do you incorporate that into the loss function? Hint: loss = (yhat- y)^2/sigma_hat^2 is missing a term but you wouldn’t know that if you don’t come from Bayesian stats. * the rabbit hole goes as deep as you want. Understanding Bayesian stats removes a lot of the “ad hoc” and intuitive guesswork that goes into ML when you don’t have a solid statistical foundation for what you’re doing.

Can you give a concrete example how a Bayesian concept provides practical help in ML model training or another ML task? Want to whet my appetite for your suggestion.

bencw 6y ago

A common type of example involves relatively small or uninformative datasets. Say you flip a coin a few times and only get heads. Your maximum likelihood (frequentist) estimate is that the coin will always land heads. In a Bayesian setting, if you have a (say uniform) prior on the probability that the coin lands heads, your maximum a posteriori estimate of this probability will be non-zero, but will get continue to get smaller if you continue only seeing heads. The above example is contrived, but makes more sense in the case of language modelling. Since a bag-of-words vector, containing say counts of words seen in a document, is typically sparse (most documents only contain a limited portion of the full vocabulary), a frequentist estimate of word probability will say that certain words can never occur, just because it's never seen them. The Bayesian estimate will still assign some non-zero chance of seeing that word. Practically speaking, this leads to the idea of "smoothing" in tf-idf (text-frequency-inverse-document-frequency) vectors, by adding 1 to document frequencies. You don't need Bayesian statistics to do this, but maybe you never would have thought of it otherwise!

astrofinch 6y ago
>Say you flip a coin a few times and only get heads. Your maximum likelihood (frequentist) estimate is that the coin will always land heads. In a Bayesian setting, if you have a (say uniform) prior on the probability that the coin lands heads, your maximum a posteriori estimate of this probability will be non-zero, but will get continue to get smaller if you continue only seeing heads. Not quite. If you have a uniform prior, there will be no difference between MAP and MLE. >From the vantage point of Bayesian inference, MLE is a special case of maximum a posteriori estimation (MAP) that assumes a uniform prior distribution of the parameters. https://en.wikipedia.org/wiki/Maximum_likelihood_estimation https://en.wikipedia.org/wiki/Maximum_likelihood_estimation More discussion here: https://stats.stackexchange.com/questions/64259/how-does-a-uniform-prior-lead-to-the-same-estimates-from-maximum-likelihood-and https://stats.stackexchange.com/questions/64259/how-does-a-u...
antipaul 6y ago
Help me see how this requires a _thorough_ study of Bayesian methods. What you describe seems to be (additive/Laplace) smoothing [1], a fairly basic concept. And I don't see how this is specifically Bayesian. Then, one common critique of "Bayesian stats work well with small samples" is that with small data (weak signal), you get your prior back, so you haven't learned anything, and the only thing you're left with is your "bias", erm, prior. I am still keen to learn why one needs a thorough study of Bayesian methods to help in practical machine learning tasks (meaning, on top of a bit of statistical or data science curriculum plus on the job experience). [1] https://en.wikipedia.org/wiki/Additive_smoothing https://en.wikipedia.org/wiki/Additive_smoothing

sibmike 6y ago

The most recent example has been supply failure detection in sales timeseries data with intermittent demand. Ended up using approach described in The Longest Run of Heads by Mark F. Schilling, which is outstandingly well written stats paper and a pleasure to read.

astrophysician 6y ago

* understanding where loss functions and regularization terms come from allows you to reason about the right choice for your problem and possibly to extend/tweak them to suit your needs. Are you working directly with probabilities? Then maybe you don’t want to use L2 regularization (Gaussian prior) but a beta prior or something with the right support. Are you modeling a Poisson rate (e.g. how many people buy my product for every dollar I spend on advertising), then use a Poisson likelihood (loss function would be negative log likelihood). * do you want to have your NN model your uncertainty as well as the mean? How do you incorporate that into the loss function? Hint: loss = (yhat- y)^2/sigma_hat^2 is missing a term but you wouldn’t know that if you don’t come from Bayesian stats. * the rabbit hole goes as deep as you want. Understanding Bayesian stats removes a lot of the “ad hoc” and intuitive guesswork that goes into ML when you don’t have a solid statistical foundation for what you’re doing.