4 ms·

Bayesian statistics for confused data scientists

176 points by speckx 8mo ago

jhbadger 8mo ago
I think Rafael Irizarry put it best over a decade ago -- while historically there was a feud between self-declared "frequentists" and "Bayesians", people doing statistics in the modern era aren't interested in playing sides, but use a combination of techniques originating in both camps: https://simplystatistics.org/posts/2014-10-13-as-an-applied-statistician-i-find-the-frequentists-versus-bayesians-debate-completely-inconsequential/ https://simplystatistics.org/posts/2014-10-13-as-an-applied-...
- therobots927 8mo ago
  That’s Bayesian propaganda
  - jmalicki 8mo ago
    Huh? Are there really any pure frequentists post Stein's paradox? At least ones that are aware of it and maintain objections to fusing the fields?
    - kgwgk 8mo ago
      > Are there really any pure frequentists post Stein's paradox? What does that have to do with anything? If one cares about that using a shrinkage estimator is an option which maintains the frequentist purity.
      - jmalicki 8mo ago
        There is some frequentist procedure there, but it seems hard to not recognize the deep connection to Bayesian statistics and wonder if you should begin to question your baseline assumptions. Since the entire justification for using a shrinkage estimator has a whole lot more in common with the foundations of Bayesian statistics than it does with the foundations of frequentist stats. Purist frequentists using a shrinkage estimator looks a lot like heliocentric Ptolemic astronomy.
    - therobots927 8mo ago
      Downvote me all you want. Bayesianism is misapplied much more frequently than frequentism. It just makes it way too easy to fudge p values. Sorry not sorry.
      - jmalicki 8mo ago
        I do always laugh when I see a Bayesian object to p-values, then use a Bayesian procedure that is mathematically identical to treating p values as posterior probabilities. Just saying the word "Bayesian" doesn't actually make it different
        kgwgk 8mo ago
        It’s mathematically identical but conceptually different. The things that go into the calculation are different, the numbers that get out of the calculation mean different things. Laughing is healthy though.
        jmalicki 8mo ago
        You can just shortcut all of that if you're a Bayesian and just plain say "p-values are posterior probabilities under a uniform (improper) prior" and save everyone a lot of time. And if you're doing that, don't care complain that p-values can be misinterpreted, because you're basically just laundering the misinterpretation of p-values. Sure, you are mathematically pure because you made an initial assumption that it can be so, rather than being confused, but the end result is the same.
        kgwgk 8mo ago
        I have no interest in laundering the misinterpretation of p-values. I don’t know whose time is saved, frequentists don’t care about the Bayesian interpretation anyway and Bayesians don’t need to restrict themselves to a particular prior. The fact that in some cases you can choose one particular prior to get a numerical value for one probability that is equal to the numerical value for the probability of a completely different thing calculated by someone else doesn’t help anyone much. Bayesians can get the same result on their own if they want.
- jmalicki 8mo ago
  I agree... I feel like "The Elements of Statistical Learning" was possibly one of the first "postmodern" things where "well, frequentist and Bayesian are just tools in the toolbox, we now know they're not so incompatible." After Stein's paradox it became super hard to be a pure frequentist if you didn't have your head in the sand.
statskier 8mo ago
I went through grad school in a very frequentist environment. We “learned” Bayesian methods but we never used them much. In my professional life I’ve never personally worked on a problem that I felt wasn’t adequately approached with frequentist methods. I’m sure other people’s experiences are different depending on the problems you gravitate towards. In fact, I tend to get pretty frustrated with Bayesian approaches because when I do turn to them it tends to be in situations that already quite complex and large. In basically every instance of that I’ve never been able to make the Bayesian approach work. Won’t converge or the sampler says it will take days and days to run. I can almost always just resort to some resampling method that might take a few hours but it runs and gives me sensible results. I realize this is heavily biased by basically only attempting on super-complex problems, but it has sort of soured me on even trying anymore. To be clear I have no issue with Bayesian methods. Clearly they work well and many people use them with great success. But I just haven’t encountered anything in several decades of statistical work that I found really required Bayesian approaches, so I’ve really lost any motivation I had to experiment with it more.
- storus 8mo ago
  A large portion of generative AI is based on Bayesian statistics, like stable diffusion, regularization, LLM as a learned prior (though trained with frequentist MLE), variational autoencoders etc. Chain-of-thought and self-consistency can be viewed as Bayesian as well.
- nextos 8mo ago
  > I’ve never personally worked on a problem that I felt wasn’t adequately approached with frequentist methods Multilevel models are one example of problem were Bayesian methods are hard to avoid as otherwise inference is unstable, particularly when available observations are not abundant. Multilevel models should be used more often as shrinking of effect sizes is important to make robust estimates. Lots of flashy results published in Nature Medicine and similar journals turn out to be statistical noise when you look at them from a rigorous perspective with adequate shrinking. I often review for these journals, and it's a constant struggle to try to inject some rigor. From a more general perspective, many frequentist methods fall prey to Lindley's Paradox. In simple terms, their inference is poorly calibrated for large sample sizes. They often mistake a negligible deviation from the null for a "statistically significant" discovery, even when the evidence actually supports the null. This is quite typical in clinical trials. (Spiegelhalter et al, 2003) is a great read to learn more even if you are not interested in medical statistics [1]. [1] https://onlinelibrary.wiley.com/doi/book/10.1002/0470092602 https://onlinelibrary.wiley.com/doi/book/10.1002/0470092602
lottin 8mo ago
> In Bayesian statistics, on the other hand, the parameter is not a point but a distribution. To be more precise, in Bayesian statistics a parameter is random variable. But what does that mean? A parameter is a characteristic of a population (as opposed to a characteristic of a sample, which is called a statistic). A quantity, such as the average cars per household right now. That's a parameter. To think of a parameter as a random variable is like regarding reality as just one realisation of an infinite number of alternate realities that could have been. The problem is we only observe our reality. All the data samples that we can ever study come from this reality. As a result, it's impossible to infer anything about the probability distribution of the parameter. The whole Bayesian approach to statistical inference is nonsensical.
HawtAds 8mo ago
I think it would be interesting if frequentist stats can come up with more generative models. Current high level generative machine learning all rely on Bayesian modeling.
- jmalicki 8mo ago
  I'm not well versed enough, but what would a frequentist generative model even mean? The entire generative concept implicitly assumes that parameters have probability distributions themselves that naturally give rise to generative models... You could do frequentist inference on a generative model, sure, but generative modelling seems fundamentally alien to frequentist thinking?
  - HawtAds 8mo ago
    I am more familiar with Bayesian than frequentist stats, but given that they are mathematically equivalent, shouldn't frequentist stats have an answer to e.g. the loss function of a VAE? Or are generative machine learning inherently impossible to model for frequentist stats? Though if you think about it, a diffusion model is somewhat (partially) frequentist.
    - jmalicki 8mo ago
      They do! https://arxiv.org/pdf/2510.18777 https://arxiv.org/pdf/2510.18777 But that doesn't mean a frequentist views a VAE as a generative model! Putting it another way, Gaussian processes originated as a frequentist technique! But to a frequentist they are not generative.
      - HawtAds 8mo ago
        Ooh good find, thanks for the link. This will be my bedtime reading for this week :)
    - jmalicki 8mo ago
      I guess you have me thinking more... things like Parzen window estimators or other KDEs are frequentist... But while it's a probability distribution, to a frequentist they are estimating the fixed parameters of a distribution. The distribution isn't generative, it just represents uncertainty - and I think that's a bit of the deep core philosophical divide between frequentists and Bayesians - you might use all the same math, but you cannot possibly think of it as being generative.
fumeux_fume 8mo ago
As a data scientist, I find applied Bayesian methods to be incredibly straightforward for most of the common problems we see like A/B testing and online measuring of parameters. I dislike that people usually first introduce Bayesian methods theoretically, which can be a lot for beginners to wrap their head around. Why not just start from the blissful elegance of updating your parameter's prior distribution with your observed data to magically get your parameter's estimate?
- oliver236 8mo ago
  can you explain what you're saying please?
jrumbut 8mo ago
The author makes a comparison to Haskell, which I think might be a little misleading. Haskell is a little more complicated to learn but also more expressive than other programming languages, this is where the comparison works. But where it breaks down is safety. If your Haskell code runs, it's more likely to be correct because of all the type system goodness. That's the reverse of the situation with Bayesian statistics, which is more like C++. It has all kinds of cool features, but they all come with superpowered footguns. Frequentist statistics is more like Java. No one loves it but it allows you to get a lot of work done without having to track down one of the few people who really understand Haskell.
algolint 8mo ago
[flagged]
- zaik 8mo ago
  I would argue the opposite is true. It takes a long time to beat the Bayesian thinking out of students when presenting them with a confidence interval: https://link.springer.com/article/10.3758/s13423-013-0572-3 https://link.springer.com/article/10.3758/s13423-013-0572-3
  - algolint 8mo ago
    [flagged]
  - deleted 8mo ago
    [deleted]
- NeutralCrane 8mo ago
  The exact opposite is true. Virtually everyone’s intuition is aligned with the Bayesian model. That intuition has to be hammered out of people in their stats classes because for decades frequentist approaches were computationally more feasible, even if they don’t align with how most humans interpret probability.
  - algolint 8mo ago
    [flagged]
oliver236 8mo ago
Nice writeup. Something that clicked for me reading this is how much the prior/likelihood/posterior dynamic mirrors transfer learning in deep learning. The prior is basically your pre-trained weights: broad knowledge you bring to the table before seeing any task-specific data. The likelihood is your fine-tuning step. And the Bernstein-von Mises result at the end is essentially saying "with enough fine-tuning data, your pre-training washes out." Obviously the analogy isn't perfect (priors are explicit and interpretable, pre-trained weights are not), but I think it's a useful mental model for anyone coming from an ML background who finds Bayesian stats unintuitive. Regularization being secretly Bayesian was the other thing that made it click for me. If you've ever tuned a Ridge regression lambda, you were doing informal prior selection.
bvan 8mo ago
Nicely done. I have the same challenge with Bayesian stats and usually do not understand why there is such controversy. It isn’t a question of either/or, except in the minds of academics who rarely venture out into the real world, or have to balance intellectual purity with getting a job done. In the very first example, a practitioner would consciously have to decide (i.e. make the assumption) whether the number of side on the die (n) is known and deterministic. Once that decision is made, the framework with which observations are evaluated and statistical reasoning applied will forever be conditional on that assumption.. unless it is revised. Practitioners are generally OK with that, whether it leads to ‘Bayesian’ or ‘frequentist’ analysis, and move on.
smokel 8mo ago
This article made me enthusiastic to dive into Bayesian statistics (again). A quick search led me to Think Bayes [1], which also introduces the concepts using Python, and seems to have a little more depth. [1] https://allendowney.github.io/ThinkBayes2/ https://allendowney.github.io/ThinkBayes2/
7777777phil 8mo ago
Most ML practitioners use L1/L2 daily without realizing they're making Bayesian prior assumptions. Gaussian prior = Ridge, Laplace prior = Lasso. Once you see it that way, "choosing a regularization strength" is really "choosing how informative your prior is."
eimrine 8mo ago
What a stupid idea to put a pesky GIF in the middle of enough complicated article. Isn't the freaking article supposed to be readable? Play your stupid meme once than go south. It even optimized to look good in a loop-form, what a stupid mad world if some math article causes a trouble like this.
jononor 8mo ago
Highly recommend Statistical Rethinking for anyone looking for practical/applied/intuitive approach to Bayesian Statistics. For example the 2023 lecture series: https://youtu.be/FdnMWdICdRs?is=KycmwPL-cn8clOK5 https://youtu.be/FdnMWdICdRs?is=KycmwPL-cn8clOK5

176 points by speckx 8mo ago

I think Rafael Irizarry put it best over a decade ago -- while historically there was a feud between self-declared "frequentists" and "Bayesians", people doing statistics in the modern era aren't interested in playing sides, but use a combination of techniques originating in both camps: https://simplystatistics.org/posts/2014-10-13-as-an-applied-statistician-i-find-the-frequentists-versus-bayesians-debate-completely-inconsequential/ https://simplystatistics.org/posts/2014-10-13-as-an-applied-...

therobots927 8mo ago
That’s Bayesian propaganda
- jmalicki 8mo ago
  Huh? Are there really any pure frequentists post Stein's paradox? At least ones that are aware of it and maintain objections to fusing the fields?
  - kgwgk 8mo ago
    > Are there really any pure frequentists post Stein's paradox? What does that have to do with anything? If one cares about that using a shrinkage estimator is an option which maintains the frequentist purity.
    - jmalicki 8mo ago
      There is some frequentist procedure there, but it seems hard to not recognize the deep connection to Bayesian statistics and wonder if you should begin to question your baseline assumptions. Since the entire justification for using a shrinkage estimator has a whole lot more in common with the foundations of Bayesian statistics than it does with the foundations of frequentist stats. Purist frequentists using a shrinkage estimator looks a lot like heliocentric Ptolemic astronomy.
  - therobots927 8mo ago
    Downvote me all you want. Bayesianism is misapplied much more frequently than frequentism. It just makes it way too easy to fudge p values. Sorry not sorry.
    - jmalicki 8mo ago
      I do always laugh when I see a Bayesian object to p-values, then use a Bayesian procedure that is mathematically identical to treating p values as posterior probabilities. Just saying the word "Bayesian" doesn't actually make it different
      - kgwgk 8mo ago
        It’s mathematically identical but conceptually different. The things that go into the calculation are different, the numbers that get out of the calculation mean different things. Laughing is healthy though.
        jmalicki 8mo ago
        You can just shortcut all of that if you're a Bayesian and just plain say "p-values are posterior probabilities under a uniform (improper) prior" and save everyone a lot of time. And if you're doing that, don't care complain that p-values can be misinterpreted, because you're basically just laundering the misinterpretation of p-values. Sure, you are mathematically pure because you made an initial assumption that it can be so, rather than being confused, but the end result is the same.
        kgwgk 8mo ago
        I have no interest in laundering the misinterpretation of p-values. I don’t know whose time is saved, frequentists don’t care about the Bayesian interpretation anyway and Bayesians don’t need to restrict themselves to a particular prior. The fact that in some cases you can choose one particular prior to get a numerical value for one probability that is equal to the numerical value for the probability of a completely different thing calculated by someone else doesn’t help anyone much. Bayesians can get the same result on their own if they want.
jmalicki 8mo ago
I agree... I feel like "The Elements of Statistical Learning" was possibly one of the first "postmodern" things where "well, frequentist and Bayesian are just tools in the toolbox, we now know they're not so incompatible." After Stein's paradox it became super hard to be a pure frequentist if you didn't have your head in the sand.

statskier 8mo ago

I went through grad school in a very frequentist environment. We “learned” Bayesian methods but we never used them much. In my professional life I’ve never personally worked on a problem that I felt wasn’t adequately approached with frequentist methods. I’m sure other people’s experiences are different depending on the problems you gravitate towards. In fact, I tend to get pretty frustrated with Bayesian approaches because when I do turn to them it tends to be in situations that already quite complex and large. In basically every instance of that I’ve never been able to make the Bayesian approach work. Won’t converge or the sampler says it will take days and days to run. I can almost always just resort to some resampling method that might take a few hours but it runs and gives me sensible results. I realize this is heavily biased by basically only attempting on super-complex problems, but it has sort of soured me on even trying anymore. To be clear I have no issue with Bayesian methods. Clearly they work well and many people use them with great success. But I just haven’t encountered anything in several decades of statistical work that I found really required Bayesian approaches, so I’ve really lost any motivation I had to experiment with it more.

storus 8mo ago
A large portion of generative AI is based on Bayesian statistics, like stable diffusion, regularization, LLM as a learned prior (though trained with frequentist MLE), variational autoencoders etc. Chain-of-thought and self-consistency can be viewed as Bayesian as well.
nextos 8mo ago
> I’ve never personally worked on a problem that I felt wasn’t adequately approached with frequentist methods Multilevel models are one example of problem were Bayesian methods are hard to avoid as otherwise inference is unstable, particularly when available observations are not abundant. Multilevel models should be used more often as shrinking of effect sizes is important to make robust estimates. Lots of flashy results published in Nature Medicine and similar journals turn out to be statistical noise when you look at them from a rigorous perspective with adequate shrinking. I often review for these journals, and it's a constant struggle to try to inject some rigor. From a more general perspective, many frequentist methods fall prey to Lindley's Paradox. In simple terms, their inference is poorly calibrated for large sample sizes. They often mistake a negligible deviation from the null for a "statistically significant" discovery, even when the evidence actually supports the null. This is quite typical in clinical trials. (Spiegelhalter et al, 2003) is a great read to learn more even if you are not interested in medical statistics [1]. [1] https://onlinelibrary.wiley.com/doi/book/10.1002/0470092602 https://onlinelibrary.wiley.com/doi/book/10.1002/0470092602

lottin 8mo ago

> In Bayesian statistics, on the other hand, the parameter is not a point but a distribution. To be more precise, in Bayesian statistics a parameter is random variable. But what does that mean? A parameter is a characteristic of a population (as opposed to a characteristic of a sample, which is called a statistic). A quantity, such as the average cars per household right now. That's a parameter. To think of a parameter as a random variable is like regarding reality as just one realisation of an infinite number of alternate realities that could have been. The problem is we only observe our reality. All the data samples that we can ever study come from this reality. As a result, it's impossible to infer anything about the probability distribution of the parameter. The whole Bayesian approach to statistical inference is nonsensical.

HawtAds 8mo ago

I think it would be interesting if frequentist stats can come up with more generative models. Current high level generative machine learning all rely on Bayesian modeling.

jmalicki 8mo ago
I'm not well versed enough, but what would a frequentist generative model even mean? The entire generative concept implicitly assumes that parameters have probability distributions themselves that naturally give rise to generative models... You could do frequentist inference on a generative model, sure, but generative modelling seems fundamentally alien to frequentist thinking?
- HawtAds 8mo ago
  I am more familiar with Bayesian than frequentist stats, but given that they are mathematically equivalent, shouldn't frequentist stats have an answer to e.g. the loss function of a VAE? Or are generative machine learning inherently impossible to model for frequentist stats? Though if you think about it, a diffusion model is somewhat (partially) frequentist.
  - jmalicki 8mo ago
    They do! https://arxiv.org/pdf/2510.18777 https://arxiv.org/pdf/2510.18777 But that doesn't mean a frequentist views a VAE as a generative model! Putting it another way, Gaussian processes originated as a frequentist technique! But to a frequentist they are not generative.
    - HawtAds 8mo ago
      Ooh good find, thanks for the link. This will be my bedtime reading for this week :)
  - jmalicki 8mo ago
    I guess you have me thinking more... things like Parzen window estimators or other KDEs are frequentist... But while it's a probability distribution, to a frequentist they are estimating the fixed parameters of a distribution. The distribution isn't generative, it just represents uncertainty - and I think that's a bit of the deep core philosophical divide between frequentists and Bayesians - you might use all the same math, but you cannot possibly think of it as being generative.

fumeux_fume 8mo ago

As a data scientist, I find applied Bayesian methods to be incredibly straightforward for most of the common problems we see like A/B testing and online measuring of parameters. I dislike that people usually first introduce Bayesian methods theoretically, which can be a lot for beginners to wrap their head around. Why not just start from the blissful elegance of updating your parameter's prior distribution with your observed data to magically get your parameter's estimate?

oliver236 8mo ago
can you explain what you're saying please?

jrumbut 8mo ago

The author makes a comparison to Haskell, which I think might be a little misleading. Haskell is a little more complicated to learn but also more expressive than other programming languages, this is where the comparison works. But where it breaks down is safety. If your Haskell code runs, it's more likely to be correct because of all the type system goodness. That's the reverse of the situation with Bayesian statistics, which is more like C++. It has all kinds of cool features, but they all come with superpowered footguns. Frequentist statistics is more like Java. No one loves it but it allows you to get a lot of work done without having to track down one of the few people who really understand Haskell.

algolint 8mo ago

[flagged]

zaik 8mo ago
I would argue the opposite is true. It takes a long time to beat the Bayesian thinking out of students when presenting them with a confidence interval: https://link.springer.com/article/10.3758/s13423-013-0572-3 https://link.springer.com/article/10.3758/s13423-013-0572-3
- algolint 8mo ago
  [flagged]
- deleted 8mo ago
  [deleted]
NeutralCrane 8mo ago
The exact opposite is true. Virtually everyone’s intuition is aligned with the Bayesian model. That intuition has to be hammered out of people in their stats classes because for decades frequentist approaches were computationally more feasible, even if they don’t align with how most humans interpret probability.
- algolint 8mo ago
  [flagged]

oliver236 8mo ago

Nice writeup. Something that clicked for me reading this is how much the prior/likelihood/posterior dynamic mirrors transfer learning in deep learning. The prior is basically your pre-trained weights: broad knowledge you bring to the table before seeing any task-specific data. The likelihood is your fine-tuning step. And the Bernstein-von Mises result at the end is essentially saying "with enough fine-tuning data, your pre-training washes out." Obviously the analogy isn't perfect (priors are explicit and interpretable, pre-trained weights are not), but I think it's a useful mental model for anyone coming from an ML background who finds Bayesian stats unintuitive. Regularization being secretly Bayesian was the other thing that made it click for me. If you've ever tuned a Ridge regression lambda, you were doing informal prior selection.

bvan 8mo ago

Nicely done. I have the same challenge with Bayesian stats and usually do not understand why there is such controversy. It isn’t a question of either/or, except in the minds of academics who rarely venture out into the real world, or have to balance intellectual purity with getting a job done. In the very first example, a practitioner would consciously have to decide (i.e. make the assumption) whether the number of side on the die (n) is known and deterministic. Once that decision is made, the framework with which observations are evaluated and statistical reasoning applied will forever be conditional on that assumption.. unless it is revised. Practitioners are generally OK with that, whether it leads to ‘Bayesian’ or ‘frequentist’ analysis, and move on.

smokel 8mo ago

This article made me enthusiastic to dive into Bayesian statistics (again). A quick search led me to Think Bayes [1], which also introduces the concepts using Python, and seems to have a little more depth. [1] https://allendowney.github.io/ThinkBayes2/ https://allendowney.github.io/ThinkBayes2/

7777777phil 8mo ago

Most ML practitioners use L1/L2 daily without realizing they're making Bayesian prior assumptions. Gaussian prior = Ridge, Laplace prior = Lasso. Once you see it that way, "choosing a regularization strength" is really "choosing how informative your prior is."

eimrine 8mo ago

What a stupid idea to put a pesky GIF in the middle of enough complicated article. Isn't the freaking article supposed to be readable? Play your stupid meme once than go south. It even optimized to look good in a loop-form, what a stupid mad world if some math article causes a trouble like this.

jononor 8mo ago

Highly recommend Statistical Rethinking for anyone looking for practical/applied/intuitive approach to Bayesian Statistics. For example the 2023 lecture series: https://youtu.be/FdnMWdICdRs?is=KycmwPL-cn8clOK5 https://youtu.be/FdnMWdICdRs?is=KycmwPL-cn8clOK5