6 ms·

Bayesian statistics and machine learning: How do they differ?

105 points by magoghm 4y ago

tpoacher 4y ago
Reminds me a quote by one of my old professors: "Before we started calling it nanoparticles, we used to just call it chemistry, but then we realised we got a lot more citations that way."
317070 4y ago
This is very close to my differentiation between the two. Machine learning fundamentally cares about model performance on other data, like validation or test sets. It is looking for models that perform well, but not necessarily model the underlying process. Bayesian statistics, like most statistics, wants to accurately estimate parameters in the model. It cares most about models that are portraying the underlying data-generating process.
- lordgrenville 4y ago
  This is basically the subject of Breiman's famous paper on the "two cultures" of statistical modelling.
  - 317070 4y ago
    Great tip! Thanks for sharing that.
stfwn 4y ago
For me, the gradual build up of (1) maximum likelihood estimation to (2) maximum a posteriori estimation to (3) full posterior approximation (or posterior sampling) was helpful to understand where Bayesian methods are in machine learning. Here’s a great video series by Erik Bekkers, who is at the University of Amsterdam. It assumes solid knowledge of calculus & linear algebra and takes you through the math and intuition of all fundamental ML methods: https://youtube.com/playlist?list=PL8FnQMH2k7jzhtVYbKmvrMyXDYMmgjj_n https://youtube.com/playlist?list=PL8FnQMH2k7jzhtVYbKmvrMyXD...
isentrop1c 4y ago
Scikitlearn has a bunch of machine learning routines including knn, xdg, decision trees and so one. There is even a model zoo there. Id say bayesian mixed gaussian models to be in the same class of algorithms. Bayesian models have its own peculiarities and statistical foundation, but it isn't far fetched to imagine it inside scikitlearn and to be considered a machine learning routine.
- steppi 4y ago
  Yes, there is a BayesianGaussianMixture class in sklearn.mixture [0]. There’s also sklearn.gaussian_process [1] which offers Gaussian process classification and regression, Bayesian learning algorithms which can be thought of as analogs of the support vector machine. Rasmussen’s Gaussian Processes for Machine Learning (2006) is a great introduction despite being a little old. [2]. [0] https://scikit-learn.org/stable/modules/mixture.html https://scikit-learn.org/stable/modules/mixture.html [1] https://scikit-learn.org/stable/modules/gaussian_process.html https://scikit-learn.org/stable/modules/gaussian_process.htm... [2] http://gaussianprocess.org/gpml/chapters/ http://gaussianprocess.org/gpml/chapters/
neatze 4y ago
In my non expert opinion it is difference in what model tries to predict, at least in some contexts. For example in Reinforcement Learning P(R | A, S), eg. probability of reward given action and state, while in Bayesian network P(A | B, C), probability of even A given B, C distributions, in such networks you can do inferences. I have no idea how correct or false this explanation is.
deleted 4y ago
[deleted]
noirdujour 4y ago
This touches a little on philosophical difference, but I have found that one major difference is the quantification of uncertainty. Bayesian models, being based naturally around modeling a probability distribution, easily enables the researcher to make claims such as "I am 95% certain the mean lies between x and y, given the data." On the other hand, neural networks, decision trees, and other models more associated with ML than Bayesian statistics do not have this capability built in. (Though there are variations and techniques to build confidence intervals and such with them.)
- lalaland1125 4y ago
  This isn't true. When you fit a neural network, you are almost always fitting a probability distribution (Bernoulli for binary outcomes, normal for numeric, etc, etc) which can all provide your probability estimates. When a model for a binary outcomes returns 0.9 for a given data point, that implies a 90% probably that the value is true. Evaluating the quality of this estimates (often called measuring the calibration of the model) is even very common. (There are some exception of course. Max margin models aren't probabilistic. And sometimes people use fixed variance parameters for their normal models, etc).
  - steppi 4y ago
    This isn’t what your parent is saying. Many machine learning models are capable of producing calibrated probabilities. What Bayesian models give on top of this is that one doesn’t just predict a probability p, but a posterior distribution for p. This allows for estimating confidence bands around p that quantify ones uncertainty in the estimate. This is useful for assessing data drift and detecting anomalous examples. One can also get such uncertainty bands from non-Bayesian methods by considering ensembles of models. See the review paper by Abdar et al. [0] for more info. [0] https://www.sciencedirect.com/science/article/pii/S1566253521001081 https://www.sciencedirect.com/science/article/pii/S156625352...
    - lalaland1125 4y ago
      A "posterior probability for p" for a Bernoulli distribution is meaningless because it's equivalent to a single Bernoulli distribution.
victor106 4y ago
The Probability -Data Science-ML space is so confusing. Any resources that would help clarify?
- muraiki 4y ago
  > Books on modeling often jump right into math and methods. Drowned in detail, it can take years to appreciate the assumptions and limitations of the various modeling mindsets. > Written in a clear and concise style, Modeling Mindsets introduces approaches such as Bayesian inference, supervised learning, causal inference, and more. > After reading this book, you will have a much better understanding of the different approaches to modeling and be able to choose the right one for your problem. https://book.modeling-mindsets.com/ https://book.modeling-mindsets.com/ Edit: Ah darn, I forgot about this part: "You should feel comfortable with at least one of the mindsets in this book". So perhaps not the best start if you don't have a base in at least one method. For frequentist statistics, consider https://www.openintro.org/book/os/ https://www.openintro.org/book/os/
- sdl 4y ago
  "Probabilistic Machine Learning" by Kevin Murphy is all you need. ... only half joking ;) https://probml.github.io/pml-book/ https://probml.github.io/pml-book/
derbOac 4y ago
I've seen the term "machine learning" used without criticism to refer to such a wide range of things that it seems pretty much synonymous now with multivariate discrimination and classification. At one time I would have said "machine learning" meant "deep learning models" but then it seemed to mean "computational multivariate discrimination and classification" and now the "computational" doesn't even apply. I personally believe "machine learning" is a term that could be removed from use and nothing would suffer (and clarity might even be improved). I feel somewhat similarly about "data science" although at least that captures some intersection of database engineering and computational statistics that is useful when discussing very large data problems.
- mistrial9 4y ago
  > seen the term "machine learning" used without criticism yes > would have said "machine learning" meant "deep learning models" no not at all.. DeepLearning has gotten serious technical press since about 2015 when certain competitions yielded better-than-average-human performance. DL is also closely coupled to the business model of massive cloud providers who handle streams of digital media. Yet ML stats continue to reliably predict eighty-percent plus well, on lots of kinds of problems. > it seemed to mean "computational multivariate discrimination and classification" no, classification has always been on of two main uses for ML, the other being regression-style prediction of real values. > believe "machine learning" is a term that could be removed from use no, stats are useful and continue to be useful. It is AI that is the really wonky term, to my ear. Please consider that there is a lot of difference between academic statistical methods, and technical press product hype. You may be suggesting a change in media coverage jargon, but you know, good luck with that..
- mr_toad 4y ago
  > At one time I would have said "machine learning" meant "deep learning models" The early ML models used SVMs, Logistic Regression and Naive Bayes. Not deep at all. What made them ML models was the size of the feature sets data, and the use of automated feature selection.
brian_spiering 4y ago
I choose not to focus on the differences between Bayesian statistics and machine learning. I choose to integrate the best parts of both. If you are curious, the book "Probabilistic Machine Learning: An Introduction" by Kevin Murphy goes into greater detail.
j7ake 4y ago
Machine learning is more broad than Bayesian statistics. More Bayesian statistics methods can be also be called machine learning, but less so the other way around.
peatmoss 4y ago
For me, the difference between ML and most of what we use stats for in classical research settings (Bayesian or otherwise) has less to do with the specific methodology chosen, and more to do with the purpose of the exercise. With ML, the utility of the model is largely a function of its predictive power. You want to accomplish some task. "Make it go" With much of statistics in a research context (such as the social sciences as called out in the link), the interest is more on explanatory power of the independent variables. Most social scientists would happily trade a bit of predictive power for a more explanatory model that neatly maps back to a set of hypotheses. "Why does it go?"
- steppi 4y ago
  There’s a nice 2010 article on this distinction by Galit Shmueli [0] called To explain or to predict? that explores this distinction in depth for anyone here interested in learning more. [0] https://www.stat.berkeley.edu/~aldous/157/Papers/shmueli.pdf https://www.stat.berkeley.edu/~aldous/157/Papers/shmueli.pdf
deleted 4y ago
[deleted]
rsp1984 4y ago
The distinction that the OP is looking for is generative vs. discriminative inference. Generative inference: We know the process that generates our observed data and can model it to sufficient precision, s.t. we can establish a forward synthesis: Parameters in, synthesised data out. With that we can compare a synthesized dataset from a configuration of parameters with real observed data and, using Bayes (a.k.a. MAP estimation), infer the correct configuration of parameters that most likely generated our observed data, which is the answer we're looking for. Examples: Kalman Filters, Linear Regression. Discriminative inference: We do not know the process that generates observed data or it's too difficult to model. In that case we ask machines to take a shot at modelling it. To do that we set up parametric data transformation pipelines (e.g. Neural Nets) and feed it lots of correct input/output pairs. Over time the model will learn how to transform the input (observed data) to the output (answer we're looking for) and hopefully generalise well when we feed it new input data that it hasn't seen before. Examples: NNs for classification. Of course there are interesting mixtures and hybrids between the two and the lines are blurred in some cases but this is the general distinction.

105 points by magoghm 4y ago

Reminds me a quote by one of my old professors: "Before we started calling it nanoparticles, we used to just call it chemistry, but then we realised we got a lot more citations that way."

317070 4y ago

This is very close to my differentiation between the two. Machine learning fundamentally cares about model performance on other data, like validation or test sets. It is looking for models that perform well, but not necessarily model the underlying process. Bayesian statistics, like most statistics, wants to accurately estimate parameters in the model. It cares most about models that are portraying the underlying data-generating process.

lordgrenville 4y ago
This is basically the subject of Breiman's famous paper on the "two cultures" of statistical modelling.
- 317070 4y ago
  Great tip! Thanks for sharing that.

stfwn 4y ago

For me, the gradual build up of (1) maximum likelihood estimation to (2) maximum a posteriori estimation to (3) full posterior approximation (or posterior sampling) was helpful to understand where Bayesian methods are in machine learning. Here’s a great video series by Erik Bekkers, who is at the University of Amsterdam. It assumes solid knowledge of calculus & linear algebra and takes you through the math and intuition of all fundamental ML methods: https://youtube.com/playlist?list=PL8FnQMH2k7jzhtVYbKmvrMyXDYMmgjj_n https://youtube.com/playlist?list=PL8FnQMH2k7jzhtVYbKmvrMyXD...

isentrop1c 4y ago

Scikitlearn has a bunch of machine learning routines including knn, xdg, decision trees and so one. There is even a model zoo there. Id say bayesian mixed gaussian models to be in the same class of algorithms. Bayesian models have its own peculiarities and statistical foundation, but it isn't far fetched to imagine it inside scikitlearn and to be considered a machine learning routine.

steppi 4y ago
Yes, there is a BayesianGaussianMixture class in sklearn.mixture [0]. There’s also sklearn.gaussian_process [1] which offers Gaussian process classification and regression, Bayesian learning algorithms which can be thought of as analogs of the support vector machine. Rasmussen’s Gaussian Processes for Machine Learning (2006) is a great introduction despite being a little old. [2]. [0] https://scikit-learn.org/stable/modules/mixture.html https://scikit-learn.org/stable/modules/mixture.html [1] https://scikit-learn.org/stable/modules/gaussian_process.html https://scikit-learn.org/stable/modules/gaussian_process.htm... [2] http://gaussianprocess.org/gpml/chapters/ http://gaussianprocess.org/gpml/chapters/

neatze 4y ago

In my non expert opinion it is difference in what model tries to predict, at least in some contexts. For example in Reinforcement Learning P(R | A, S), eg. probability of reward given action and state, while in Bayesian network P(A | B, C), probability of even A given B, C distributions, in such networks you can do inferences. I have no idea how correct or false this explanation is.

deleted 4y ago

[deleted]

noirdujour 4y ago

This touches a little on philosophical difference, but I have found that one major difference is the quantification of uncertainty. Bayesian models, being based naturally around modeling a probability distribution, easily enables the researcher to make claims such as "I am 95% certain the mean lies between x and y, given the data." On the other hand, neural networks, decision trees, and other models more associated with ML than Bayesian statistics do not have this capability built in. (Though there are variations and techniques to build confidence intervals and such with them.)

lalaland1125 4y ago
This isn't true. When you fit a neural network, you are almost always fitting a probability distribution (Bernoulli for binary outcomes, normal for numeric, etc, etc) which can all provide your probability estimates. When a model for a binary outcomes returns 0.9 for a given data point, that implies a 90% probably that the value is true. Evaluating the quality of this estimates (often called measuring the calibration of the model) is even very common. (There are some exception of course. Max margin models aren't probabilistic. And sometimes people use fixed variance parameters for their normal models, etc).
- steppi 4y ago
  This isn’t what your parent is saying. Many machine learning models are capable of producing calibrated probabilities. What Bayesian models give on top of this is that one doesn’t just predict a probability p, but a posterior distribution for p. This allows for estimating confidence bands around p that quantify ones uncertainty in the estimate. This is useful for assessing data drift and detecting anomalous examples. One can also get such uncertainty bands from non-Bayesian methods by considering ensembles of models. See the review paper by Abdar et al. [0] for more info. [0] https://www.sciencedirect.com/science/article/pii/S1566253521001081 https://www.sciencedirect.com/science/article/pii/S156625352...
  - lalaland1125 4y ago
    A "posterior probability for p" for a Bernoulli distribution is meaningless because it's equivalent to a single Bernoulli distribution.

victor106 4y ago

The Probability -Data Science-ML space is so confusing. Any resources that would help clarify?

muraiki 4y ago
> Books on modeling often jump right into math and methods. Drowned in detail, it can take years to appreciate the assumptions and limitations of the various modeling mindsets. > Written in a clear and concise style, Modeling Mindsets introduces approaches such as Bayesian inference, supervised learning, causal inference, and more. > After reading this book, you will have a much better understanding of the different approaches to modeling and be able to choose the right one for your problem. https://book.modeling-mindsets.com/ https://book.modeling-mindsets.com/ Edit: Ah darn, I forgot about this part: "You should feel comfortable with at least one of the mindsets in this book". So perhaps not the best start if you don't have a base in at least one method. For frequentist statistics, consider https://www.openintro.org/book/os/ https://www.openintro.org/book/os/
sdl 4y ago
"Probabilistic Machine Learning" by Kevin Murphy is all you need. ... only half joking ;) https://probml.github.io/pml-book/ https://probml.github.io/pml-book/

derbOac 4y ago

I've seen the term "machine learning" used without criticism to refer to such a wide range of things that it seems pretty much synonymous now with multivariate discrimination and classification. At one time I would have said "machine learning" meant "deep learning models" but then it seemed to mean "computational multivariate discrimination and classification" and now the "computational" doesn't even apply. I personally believe "machine learning" is a term that could be removed from use and nothing would suffer (and clarity might even be improved). I feel somewhat similarly about "data science" although at least that captures some intersection of database engineering and computational statistics that is useful when discussing very large data problems.

mistrial9 4y ago
> seen the term "machine learning" used without criticism yes > would have said "machine learning" meant "deep learning models" no not at all.. DeepLearning has gotten serious technical press since about 2015 when certain competitions yielded better-than-average-human performance. DL is also closely coupled to the business model of massive cloud providers who handle streams of digital media. Yet ML stats continue to reliably predict eighty-percent plus well, on lots of kinds of problems. > it seemed to mean "computational multivariate discrimination and classification" no, classification has always been on of two main uses for ML, the other being regression-style prediction of real values. > believe "machine learning" is a term that could be removed from use no, stats are useful and continue to be useful. It is AI that is the really wonky term, to my ear. Please consider that there is a lot of difference between academic statistical methods, and technical press product hype. You may be suggesting a change in media coverage jargon, but you know, good luck with that..
mr_toad 4y ago
> At one time I would have said "machine learning" meant "deep learning models" The early ML models used SVMs, Logistic Regression and Naive Bayes. Not deep at all. What made them ML models was the size of the feature sets data, and the use of automated feature selection.

brian_spiering 4y ago

I choose not to focus on the differences between Bayesian statistics and machine learning. I choose to integrate the best parts of both. If you are curious, the book "Probabilistic Machine Learning: An Introduction" by Kevin Murphy goes into greater detail.

j7ake 4y ago

Machine learning is more broad than Bayesian statistics. More Bayesian statistics methods can be also be called machine learning, but less so the other way around.

peatmoss 4y ago

For me, the difference between ML and most of what we use stats for in classical research settings (Bayesian or otherwise) has less to do with the specific methodology chosen, and more to do with the purpose of the exercise. With ML, the utility of the model is largely a function of its predictive power. You want to accomplish some task. "Make it go" With much of statistics in a research context (such as the social sciences as called out in the link), the interest is more on explanatory power of the independent variables. Most social scientists would happily trade a bit of predictive power for a more explanatory model that neatly maps back to a set of hypotheses. "Why does it go?"

steppi 4y ago
There’s a nice 2010 article on this distinction by Galit Shmueli [0] called To explain or to predict? that explores this distinction in depth for anyone here interested in learning more. [0] https://www.stat.berkeley.edu/~aldous/157/Papers/shmueli.pdf https://www.stat.berkeley.edu/~aldous/157/Papers/shmueli.pdf

deleted 4y ago

[deleted]

rsp1984 4y ago

The distinction that the OP is looking for is generative vs. discriminative inference. Generative inference: We know the process that generates our observed data and can model it to sufficient precision, s.t. we can establish a forward synthesis: Parameters in, synthesised data out. With that we can compare a synthesized dataset from a configuration of parameters with real observed data and, using Bayes (a.k.a. MAP estimation), infer the correct configuration of parameters that most likely generated our observed data, which is the answer we're looking for. Examples: Kalman Filters, Linear Regression. Discriminative inference: We do not know the process that generates observed data or it's too difficult to model. In that case we ask machines to take a shot at modelling it. To do that we set up parametric data transformation pipelines (e.g. Neural Nets) and feed it lots of correct input/output pairs. Over time the model will learn how to transform the input (observed data) to the output (answer we're looking for) and hopefully generalise well when we feed it new input data that it hasn't seen before. Examples: NNs for classification. Of course there are interesting mixtures and hybrids between the two and the lines are blurred in some cases but this is the general distinction.