13 ms·

Statistical Thinking: My Journey from Frequentist to Bayesian Statistics

225 points by sebg 10y ago

gydfi 10y ago
Bayesian statistics is a fairly sensible approach to certain types of statistical problems, so I'm not sure why its proponents always seem to talk about it as if they're pushing some new religious movement with a lot of nudity, or Amway.
- Ntrails 10y ago
  In my first stats lecture at university the lecturer informed us we would not learn about Bayesian statistics and if we'd wanted to do that we should have gone to York or something.
  - gjm11 10y ago
    Why York in particular? I mean, was the subtext "York, which is our ancient enemy" or "York, which everyone knows is a grossly inferior university" or what?
    - kgwgk 10y ago
      Maybe related: https://www.york.ac.uk/maths/staff/peter-lee/ https://www.york.ac.uk/maths/staff/peter-lee/
      - keithpeter 10y ago
        http://www-users.york.ac.uk/~pml1/bayes/book.htm http://www-users.york.ac.uk/~pml1/bayes/book.htm Gives out pdfs of the exercises in his textbook on his home page. Thanks for pointer.
    - Ntrails 10y ago
      I never dared ask, actually.
- fullshark 10y ago
  How do you know a statistician is a Bayesian? - Don't worry, they'll tell you.
konschubert 10y ago
Frequentist and Bayesian statistics are different tools that answer different questions. Arguing whether one is better than the other is like arguing whether one should drink water or eat bread. I also think that a lot of the criticism against Frequentist inference comes from people who have experienced trouble applying a clean Frequentist solution in a more complex case and thus switched to Bayesian statistics as a replacement that is easier to handle.
- a_bonobo 10y ago
  To me, the popularity of frequentist approaches in the literature means that I see many misapplications and erroneous applications to get that holy p<0.05. Bayesian thinking is not that popular with these people so it's not 'tainted' yet. It's a bit like the popularity of Excel - we see many people complain about Excel's automated changing of strings to dates, for example. If we'd all switch to R to fix that problem everybody would complain about stringsAsFactors=T instead.
  - digitalzombie 10y ago
    > misapplications and erroneous applications to get that holy p<0.05. Or you can just force them to give confidence interval.
- Baeocystin 10y ago
  Seriously. It's like arguing whether the left foot or the right is more important for walking.
- stdbrouw 10y ago
  > Frequentist and Bayesian statistics are different tools that answer different questions. Arguing whether one is better than the other is like arguing whether one should drink water or eat bread. Astronomy and astrology also answer different questions, it's just that astrology doesn't answer the questions we're interested in, and it doesn't even do a great job with the questions it does purport to answer. When quantifying uncertainty, the only quantity of interest is the posterior probability, which Bayesian methods provide and frequentist methods don't. Period. The only reason we accept a likelihood instead is because it can provide a reasonable approximation and might be faster or easier to calculate, and the only reason we'd accept a p-value over the full likelihood function is when we need a quick-and-dirty metric of the importance of a variable or an intervention. These are all valid reasons, and heck, few things are more useful than a quick (frequentist) OLS regression, but to then conclude that frequentist and Bayesian methods are entirely equal and complementary is disingenuous. A good blogpost which deconstructs this line of reasoning in more detail: http://www.bayesianphilosophy.com/how-science-can-permanently-fail/ http://www.bayesianphilosophy.com/how-science-can-permanentl...
pdkl95 10y ago
"Confidence, Credibility, and why Frequentism and Science do not Mix" http://jakevdp.github.io/blog/2014/06/12/frequentism-and-bayesianism-3-confidence-credibility/ http://jakevdp.github.io/blog/2014/06/12/frequentism-and-bay...
lottin 10y ago
Wikipedia does IMO an excellent job explaining the notion of a confidence interval. On the other hand, I find the idea of being "73% certain" that something will happen much harder to understand. A percentage implies a ratio, but Bayesians never explain what the numerator and denominator are.
- paulddraper 10y ago
  They are prior observations: successes and trials.
- contravariant 10y ago
  The interpretation of the probability is the same as in frequentist statistics, except you're making statements about the model resulting from your assumptions and data, instead of some hypothetical experiment. I suppose the Bayesian approach is more about building the model whereas the frequentist approach is more about selecting the best model out of several.
  - BeetleB 10y ago
    >The interpretation of the probability is the same as in frequentist statistics Not at all. Frequentists cannot define a probability on whether it will rain in a location on a given day. They will respond that such a probability is meaningless. Bayesians can, however, give a meaning to it.
    - contravariant 10y ago
      True, but the way a Bayesianist (?) will assign meaning to it involves creating a model, based on some assumptions, which will return a probability. The Bayesian notion of probability is equivalent to the frequentist notion of probability for experiments done on that model. In that sense they are the same.
- jamii 10y ago
  One way to think of it is in terms of the expected value of betting on being correct. Even if the bet only happens once ever, I still need a way to map how confident I am to how much I think the bet is worth. If I am willing to pay up to $0.73 for a bet that pays out $1 when I'm correct then I am 73% certain.
digitalzombie 10y ago
Post talk about the con of Frequentist compare to Bayesian and end up talking about Fisher likelihood school of thought. Post argue that Likelihood is basically Bayes, it's actually Fisher school's of thought which is a half way point or a can be seen as a compromise between Frequentist and Bayesian approach. A good book that talk about this which I've been meaning to read is... In All Likelihood: Statistical Modelling and Inference Using Likelihood by Yudi Pawitan. I've read the first chapter and it's a very interesting read. I might skip bayes and go straight to likelihood coming from the school of frequency.
- kgwgk 10y ago
  You can't "skip bayes and go straight to likelihood", because as you said it's only halfway there. Fisher tried to go farther with his fiducial approach, by the way.
- stdbrouw 10y ago
  > A good book that talk about this which I've been meaning to read is... In All Likelihood: Statistical Modelling and Inference Using Likelihood by Yudi Pawitan. It's good but it's mostly a technical introduction to statistical inference (i.e. the mathematics that make statistics work), and the remarks about the differences between different schools of statistics are mostly short asides. It's not a great recommendation for non-statisticians who want more insight into the different schools of statistics. Also, both frequentist and Bayesian methods rely on likelihoods as an intermediary, so I guess you could say it's a "compromise" but only in a very uninteresting way. The likelihood L(theta|data) is simply shorthand for P(data|theta), and to get a posterior probability you then simply apply Bayes' theorem to get P(theta|data) ~ P(data|theta) * P(theta), and this last part is your prior.
nabla9 10y ago
Bayesian vs. frequentist seems to have little to do with the underlying issue. If clinical trialists use p-values wrong, how is moving to Bayesian methods going to be less misused and misunderstood? The real issue is the the established practice in the research field. It's hard to introduce new methods if peer review is not familiar with them, or if everyone in the field has problems with such basic concept as p-values. Researchers have the tendency to apply statistical tools they have learned mechanically and peer review accepts them mechanically. New tools need more thought, not less.
- stdbrouw 10y ago
  > If clinical trialists use p-values wrong, how is moving to Bayesian methods going to be less misused and misunderstood? Bayesian methods have the advantage that, while they can still be misused and misunderstood, at least when used correctly they tell us what we want to know and are easy to interpret (a posterior probability is exactly what you think it is) whereas p-values are hard to interpret correctly (a low p-value can imply a large effect size, a large sample size, or in the face of publication bias, lord knows what). That said, I do think that the advantage to Bayesian thinking in research would mostly not be about the methods but about the attitude: aside from a couple of weirdos pushing Bayes factors, Bayesian statisticians almost universally communicate results using credible intervals (HPD) instead of dichotomizing the evidence into significant or not significant. Frequentist confidence intervals will get you most of the way there and could completely replace p-values, but if you're going to advocate for better statistics and uproot established practices, might as well go all the way and encourage better methods and better ways of communicating results at the same time.
  - jamii 10y ago
    http://pcl.missouri.edu/sites/default/files/freeLunch_0.pdf http://pcl.missouri.edu/sites/default/files/freeLunch_0.pdf has a neat example of why Bayes factors might be preferred over credible intervals. > By using posterior credible intervals, we might reject the null, but by using Bayes’ rule directly we see that this rejection is made prematurely as there is no decrease in the plausibility of the zero point.
lngnmn 10y ago
What a mess. They literally produced chimeras and used them as a building blocks and premises for another ones.
- unit91 10y ago
  Stats amateur here. Care to explain?
  - lngnmn 10y ago
    Correlation, no mater how "strong" is not a causation. A causation must be proved experimentally. Statistical sampling does not constitute a valid experiment. Statistical inference is not an equivalent to a logical inference and they cannot be used interchangeably. Unproven statistical models are no better than astrological calculations or metaphysics.
  - RA_Fisher 10y ago
    Modern published "social science" as we know it is largely built on quicksand that's starting to be eroded. Get ready for a wild ride! :)
zump 10y ago
Does someone have an ELI5 for frequentist vs. bayesian?
- raverbashing 10y ago
  I'll try They both yield the same results when your knowledge of the given probabilities is exact But Frequentists will look at a "top down" view, Bayesians will look "bottom-up", more importantly, as based on Bayes's theorem, they will look at "if a then b" kind of probabilities. This sounds like a good explanation (the 1st answer) https://stats.stackexchange.com/questions/22/bayesian-and-frequentist-reasoning-in-plain-english https://stats.stackexchange.com/questions/22/bayesian-and-fr... also obligatory XKCD: https://xkcd.com/1132/ https://xkcd.com/1132/ (Though frequentists are not so naive usually)
  - Retric 10y ago
    That XKCD is a joke and not how frequentists view statistics. The confusion comes from the way you read papers. A frequentist looks at a paper as evidence but not truth. Bayesian on the other hand gets to the same place with slightly different math.
    - marcosdumay 10y ago
      The joke is about priors selection. I'd say it's spot on, since, as you say, the only reason people are not fooled like that is because they evaluate the frequentist results as evidence in an ad-hock Bayesian model. The truth is that nobody thinks exclusively on frequentist or Bayesian terms. But that's not comics-grade material, and mixing them would hide instead of surface their differences.
      - Retric 10y ago
        The problem is it's a single sample. If the output was Yes, Yes, Yes, Yes, Yes, No, Yes, then you can do frequentist statistics, but when the output is just 'Yes' then the sample size is one.
deleted 10y ago
[deleted]
LanceH 10y ago
It seems a fair percentage of stats people think frequentism is inferior.
eanzenberg 10y ago
It's always funny seeing this articles where they do 'frequentist' analysis wrong then decide Bayesian is the winner. Yes you can use prior evidence. Yes you have to be careful of multiple comparisons. Yes you have to account for reproducibility. Oh and Bayesian has p-values too.
Xcelerate 10y ago
Probability is a weird concept. It's much more philosophical than it appears at first glance, but much less philosophical than something like the interpretations of quantum mechanics. Ultimately, what is a "probability"? It's a number that can be used for making predictions about the future. Neither Bayesian methods nor frequentist methods are "ideal" in the sense of predicting (computing) the future — in fact, results from algorithmic information theory put the best bounds on what we can "hope to predict". Very loosely speaking, the best predictor is the shortest program that reproduces the data. But this is uncomputable in general, which means we use something like minimum message length (MML) or minimum description length (MDL) — which are also sometimes uncomputable, but a bit more manageable at least. In some situations, Bayesian methods can be shown to be equivalent to MDL, in the sense that sizeof(model) + sizeof(parameters) + sizeof(residual data) is a log reformulation of Bayes theorem. See this paper for more details: http://homepages.cwi.nl/~paulv/papers/mdlindbayeskolmcompl.pdf http://homepages.cwi.nl/~paulv/papers/mdlindbayeskolmcompl.p...
baybayes 10y ago
I should say that the Bayesian point of view is about thinking that all your computed probabilities are only the result of partial information, more data means changing those probabilities using the new information and Bayes' Theorem. In the Bayesian way of thinking you are always wondering if there is another source of information that can enhance your probabilities. Like in the random forest method, you try to gain new knowledge from many sources.
shurtler 10y ago
"I really like penalized maximum likelihood estimation (which is really empirical Bayes) but once we have a penalized model all of our frequentist inferential framework fails us. No one can interpret a confidence interval for a biased (shrunken; penalized) estimate." Not sure I completely understand it, but good point.

225 points by sebg 10y ago

gydfi 10y ago

Bayesian statistics is a fairly sensible approach to certain types of statistical problems, so I'm not sure why its proponents always seem to talk about it as if they're pushing some new religious movement with a lot of nudity, or Amway.

Ntrails 10y ago
In my first stats lecture at university the lecturer informed us we would not learn about Bayesian statistics and if we'd wanted to do that we should have gone to York or something.
- gjm11 10y ago
  Why York in particular? I mean, was the subtext "York, which is our ancient enemy" or "York, which everyone knows is a grossly inferior university" or what?
  - kgwgk 10y ago
    Maybe related: https://www.york.ac.uk/maths/staff/peter-lee/ https://www.york.ac.uk/maths/staff/peter-lee/
    - keithpeter 10y ago
      http://www-users.york.ac.uk/~pml1/bayes/book.htm http://www-users.york.ac.uk/~pml1/bayes/book.htm Gives out pdfs of the exercises in his textbook on his home page. Thanks for pointer.
  - Ntrails 10y ago
    I never dared ask, actually.
fullshark 10y ago
How do you know a statistician is a Bayesian? - Don't worry, they'll tell you.

konschubert 10y ago

Frequentist and Bayesian statistics are different tools that answer different questions. Arguing whether one is better than the other is like arguing whether one should drink water or eat bread. I also think that a lot of the criticism against Frequentist inference comes from people who have experienced trouble applying a clean Frequentist solution in a more complex case and thus switched to Bayesian statistics as a replacement that is easier to handle.

a_bonobo 10y ago
To me, the popularity of frequentist approaches in the literature means that I see many misapplications and erroneous applications to get that holy p<0.05. Bayesian thinking is not that popular with these people so it's not 'tainted' yet. It's a bit like the popularity of Excel - we see many people complain about Excel's automated changing of strings to dates, for example. If we'd all switch to R to fix that problem everybody would complain about stringsAsFactors=T instead.
- digitalzombie 10y ago
  > misapplications and erroneous applications to get that holy p<0.05. Or you can just force them to give confidence interval.
Baeocystin 10y ago
Seriously. It's like arguing whether the left foot or the right is more important for walking.
stdbrouw 10y ago
> Frequentist and Bayesian statistics are different tools that answer different questions. Arguing whether one is better than the other is like arguing whether one should drink water or eat bread. Astronomy and astrology also answer different questions, it's just that astrology doesn't answer the questions we're interested in, and it doesn't even do a great job with the questions it does purport to answer. When quantifying uncertainty, the only quantity of interest is the posterior probability, which Bayesian methods provide and frequentist methods don't. Period. The only reason we accept a likelihood instead is because it can provide a reasonable approximation and might be faster or easier to calculate, and the only reason we'd accept a p-value over the full likelihood function is when we need a quick-and-dirty metric of the importance of a variable or an intervention. These are all valid reasons, and heck, few things are more useful than a quick (frequentist) OLS regression, but to then conclude that frequentist and Bayesian methods are entirely equal and complementary is disingenuous. A good blogpost which deconstructs this line of reasoning in more detail: http://www.bayesianphilosophy.com/how-science-can-permanently-fail/ http://www.bayesianphilosophy.com/how-science-can-permanentl...

pdkl95 10y ago

"Confidence, Credibility, and why Frequentism and Science do not Mix" http://jakevdp.github.io/blog/2014/06/12/frequentism-and-bayesianism-3-confidence-credibility/ http://jakevdp.github.io/blog/2014/06/12/frequentism-and-bay...

lottin 10y ago

Wikipedia does IMO an excellent job explaining the notion of a confidence interval. On the other hand, I find the idea of being "73% certain" that something will happen much harder to understand. A percentage implies a ratio, but Bayesians never explain what the numerator and denominator are.

paulddraper 10y ago
They are prior observations: successes and trials.
contravariant 10y ago
The interpretation of the probability is the same as in frequentist statistics, except you're making statements about the model resulting from your assumptions and data, instead of some hypothetical experiment. I suppose the Bayesian approach is more about building the model whereas the frequentist approach is more about selecting the best model out of several.
- BeetleB 10y ago
  >The interpretation of the probability is the same as in frequentist statistics Not at all. Frequentists cannot define a probability on whether it will rain in a location on a given day. They will respond that such a probability is meaningless. Bayesians can, however, give a meaning to it.
  - contravariant 10y ago
    True, but the way a Bayesianist (?) will assign meaning to it involves creating a model, based on some assumptions, which will return a probability. The Bayesian notion of probability is equivalent to the frequentist notion of probability for experiments done on that model. In that sense they are the same.
jamii 10y ago
One way to think of it is in terms of the expected value of betting on being correct. Even if the bet only happens once ever, I still need a way to map how confident I am to how much I think the bet is worth. If I am willing to pay up to $0.73 for a bet that pays out $1 when I'm correct then I am 73% certain.

digitalzombie 10y ago

Post talk about the con of Frequentist compare to Bayesian and end up talking about Fisher likelihood school of thought. Post argue that Likelihood is basically Bayes, it's actually Fisher school's of thought which is a half way point or a can be seen as a compromise between Frequentist and Bayesian approach. A good book that talk about this which I've been meaning to read is... In All Likelihood: Statistical Modelling and Inference Using Likelihood by Yudi Pawitan. I've read the first chapter and it's a very interesting read. I might skip bayes and go straight to likelihood coming from the school of frequency.

kgwgk 10y ago
You can't "skip bayes and go straight to likelihood", because as you said it's only halfway there. Fisher tried to go farther with his fiducial approach, by the way.
stdbrouw 10y ago
> A good book that talk about this which I've been meaning to read is... In All Likelihood: Statistical Modelling and Inference Using Likelihood by Yudi Pawitan. It's good but it's mostly a technical introduction to statistical inference (i.e. the mathematics that make statistics work), and the remarks about the differences between different schools of statistics are mostly short asides. It's not a great recommendation for non-statisticians who want more insight into the different schools of statistics. Also, both frequentist and Bayesian methods rely on likelihoods as an intermediary, so I guess you could say it's a "compromise" but only in a very uninteresting way. The likelihood L(theta|data) is simply shorthand for P(data|theta), and to get a posterior probability you then simply apply Bayes' theorem to get P(theta|data) ~ P(data|theta) * P(theta), and this last part is your prior.

nabla9 10y ago

Bayesian vs. frequentist seems to have little to do with the underlying issue. If clinical trialists use p-values wrong, how is moving to Bayesian methods going to be less misused and misunderstood? The real issue is the the established practice in the research field. It's hard to introduce new methods if peer review is not familiar with them, or if everyone in the field has problems with such basic concept as p-values. Researchers have the tendency to apply statistical tools they have learned mechanically and peer review accepts them mechanically. New tools need more thought, not less.

stdbrouw 10y ago
> If clinical trialists use p-values wrong, how is moving to Bayesian methods going to be less misused and misunderstood? Bayesian methods have the advantage that, while they can still be misused and misunderstood, at least when used correctly they tell us what we want to know and are easy to interpret (a posterior probability is exactly what you think it is) whereas p-values are hard to interpret correctly (a low p-value can imply a large effect size, a large sample size, or in the face of publication bias, lord knows what). That said, I do think that the advantage to Bayesian thinking in research would mostly not be about the methods but about the attitude: aside from a couple of weirdos pushing Bayes factors, Bayesian statisticians almost universally communicate results using credible intervals (HPD) instead of dichotomizing the evidence into significant or not significant. Frequentist confidence intervals will get you most of the way there and could completely replace p-values, but if you're going to advocate for better statistics and uproot established practices, might as well go all the way and encourage better methods and better ways of communicating results at the same time.
- jamii 10y ago
  http://pcl.missouri.edu/sites/default/files/freeLunch_0.pdf http://pcl.missouri.edu/sites/default/files/freeLunch_0.pdf has a neat example of why Bayes factors might be preferred over credible intervals. > By using posterior credible intervals, we might reject the null, but by using Bayes’ rule directly we see that this rejection is made prematurely as there is no decrease in the plausibility of the zero point.

lngnmn 10y ago

What a mess. They literally produced chimeras and used them as a building blocks and premises for another ones.

unit91 10y ago
Stats amateur here. Care to explain?
- lngnmn 10y ago
  Correlation, no mater how "strong" is not a causation. A causation must be proved experimentally. Statistical sampling does not constitute a valid experiment. Statistical inference is not an equivalent to a logical inference and they cannot be used interchangeably. Unproven statistical models are no better than astrological calculations or metaphysics.
- RA_Fisher 10y ago
  Modern published "social science" as we know it is largely built on quicksand that's starting to be eroded. Get ready for a wild ride! :)

zump 10y ago

Does someone have an ELI5 for frequentist vs. bayesian?

raverbashing 10y ago
I'll try They both yield the same results when your knowledge of the given probabilities is exact But Frequentists will look at a "top down" view, Bayesians will look "bottom-up", more importantly, as based on Bayes's theorem, they will look at "if a then b" kind of probabilities. This sounds like a good explanation (the 1st answer) https://stats.stackexchange.com/questions/22/bayesian-and-frequentist-reasoning-in-plain-english https://stats.stackexchange.com/questions/22/bayesian-and-fr... also obligatory XKCD: https://xkcd.com/1132/ https://xkcd.com/1132/ (Though frequentists are not so naive usually)
- Retric 10y ago
  That XKCD is a joke and not how frequentists view statistics. The confusion comes from the way you read papers. A frequentist looks at a paper as evidence but not truth. Bayesian on the other hand gets to the same place with slightly different math.
  - marcosdumay 10y ago
    The joke is about priors selection. I'd say it's spot on, since, as you say, the only reason people are not fooled like that is because they evaluate the frequentist results as evidence in an ad-hock Bayesian model. The truth is that nobody thinks exclusively on frequentist or Bayesian terms. But that's not comics-grade material, and mixing them would hide instead of surface their differences.
    - Retric 10y ago
      The problem is it's a single sample. If the output was Yes, Yes, Yes, Yes, Yes, No, Yes, then you can do frequentist statistics, but when the output is just 'Yes' then the sample size is one.

deleted 10y ago

[deleted]

LanceH 10y ago

It seems a fair percentage of stats people think frequentism is inferior.

eanzenberg 10y ago

It's always funny seeing this articles where they do 'frequentist' analysis wrong then decide Bayesian is the winner. Yes you can use prior evidence. Yes you have to be careful of multiple comparisons. Yes you have to account for reproducibility. Oh and Bayesian has p-values too.

Xcelerate 10y ago

Probability is a weird concept. It's much more philosophical than it appears at first glance, but much less philosophical than something like the interpretations of quantum mechanics. Ultimately, what is a "probability"? It's a number that can be used for making predictions about the future. Neither Bayesian methods nor frequentist methods are "ideal" in the sense of predicting (computing) the future — in fact, results from algorithmic information theory put the best bounds on what we can "hope to predict". Very loosely speaking, the best predictor is the shortest program that reproduces the data. But this is uncomputable in general, which means we use something like minimum message length (MML) or minimum description length (MDL) — which are also sometimes uncomputable, but a bit more manageable at least. In some situations, Bayesian methods can be shown to be equivalent to MDL, in the sense that sizeof(model) + sizeof(parameters) + sizeof(residual data) is a log reformulation of Bayes theorem. See this paper for more details: http://homepages.cwi.nl/~paulv/papers/mdlindbayeskolmcompl.pdf http://homepages.cwi.nl/~paulv/papers/mdlindbayeskolmcompl.p...

baybayes 10y ago

I should say that the Bayesian point of view is about thinking that all your computed probabilities are only the result of partial information, more data means changing those probabilities using the new information and Bayes' Theorem. In the Bayesian way of thinking you are always wondering if there is another source of information that can enhance your probabilities. Like in the random forest method, you try to gain new knowledge from many sources.

shurtler 10y ago

"I really like penalized maximum likelihood estimation (which is really empirical Bayes) but once we have a penalized model all of our frequentist inferential framework fails us. No one can interpret a confidence interval for a biased (shrunken; penalized) estimate." Not sure I completely understand it, but good point.