9 ms·

Frequentism and Bayesianism: A Practical Introduction

93 points by atakan_gurkan 12y ago

lutusp 12y ago
Readers should be aware that the linked article was composed entirely in the IPython notebook environment, which means Python code blocks, Latex renderings, and graphics, can all be freely mixed in a (to me) very nice, readable article format. http://ipython.org/ http://ipython.org/
- bayesianhorse 12y ago
  And it literally can't be said enough times how awesome it is!
atakan_gurkan 12y ago
The follow-ups are also well worth reading: http://jakevdp.github.io/blog/2014/06/06/frequentism-and-bayesianism-2-when-results-differ/ http://jakevdp.github.io/blog/2014/06/06/frequentism-and-bay... http://jakevdp.github.io/blog/2014/06/12/frequentism-and-bayesianism-3-confidence-credibility/ http://jakevdp.github.io/blog/2014/06/12/frequentism-and-bay... http://jakevdp.github.io/blog/2014/06/14/frequentism-and-bayesianism-4-bayesian-in-python/ http://jakevdp.github.io/blog/2014/06/14/frequentism-and-bay...
- nkurz 12y ago
  In case it's not clear from the beginning where he really stands on the matter, in Part 3 he offers his opinion on the relative merits of both approaches: <spoiler> The moral of the story is that frequentism and Science do not mix. Let me say it directly: you should be suspicious of the use of frequentist confidence intervals and p-values in science. In a scientific setting, confidence intervals, and closely-related p-values, provide the correct answer to the wrong question. In particular, if you ever find someone stating or implying that a 95% confidence interval is 95% certain to contain a parameter of interest, do not trust their interpretation or their results. If you happen to be peer-reviewing the paper, reject it. Their data do not back-up their conclusion. </spoiler>
  - judk 12y ago
    Um, the folks at CERN don't agree with your assessment of frequentism in science. Their papers are explicitly frequentist, they celebrate results based on how many "sigmas" they get. P-value all the aym
    - nkurz 12y ago
      My interpretation? If not clear, I'm quoting the penultimate paragraph of the last post in the series, right before he thanks the reader for making it to the end: http://jakevdp.github.io/blog/2014/06/12/frequentism-and-bayesianism-3-confidence-credibility/#The-Moral-of-the-Story:-Frequentism-and-Science-Do-Not-Mix http://jakevdp.github.io/blog/2014/06/12/frequentism-and-bay...
    - Houshalter 12y ago
      Yes frequentism is common in science and the author strongly objects to it. Read the whole post: https://jakevdp.github.io/blog/2014/06/12/frequentism-and-bayesianism-3-confidence-credibility/ https://jakevdp.github.io/blog/2014/06/12/frequentism-and-ba...
    - bayesianhorse 12y ago
      I'm not a physicist, but as far as I know, CERN has access to tons of data with every intention of drowning any prior belief. In this setting, I would expect frequentist methods to shine, and bayesian method to be intractable. However I would not expect CERN papers to make the kinds of terminological / theoretical lapses about confidence the parent thread was talking about. Papers should be rejected for that kind of error, even if you are not a bayesian.
      - betatim 12y ago
        Exactly this. At CERN (and all of HEP elsewhere) the Bayes vs Freq wars were fought many years ago and are long over. The conclusion: when you have lots of data they converge (as they should!). In the case that they differ you almost always find that you have very few observations. I would argue that this 'difference' is not that exciting because it must be dominated by your assumptions, not your observations. After all once you accumulate enough observations the two methods tend to converge. Personal conclusion: if the methods disagree work on getting more data instead of fighting over which method is better.
- anon4 12y ago
  That first followup seems patently insane. "We don't know the value of p̂, so we'll assume it's 5/8". Why would you ever assume like that? The Bayesian approach appears to me exactly like the Frequentist, without the part where they assume convenient values of unknown parameters.
jmount 12y ago
Nice article. In this direction my group has been trying to help teach that you tend to need to be familiar with both frequentist and Bayesian thought (you can't always choose one or the other: http://www.win-vector.com/blog/2013/05/bayesian-and-frequentist-approaches-ask-the-right-question/ http://www.win-vector.com/blog/2013/05/bayesian-and-frequent... ) and that Bayesianism only appears to be the more complicated of the two ( http://www.win-vector.com/blog/2014/07/frequenstist-inference-only-seems-easy/ http://www.win-vector.com/blog/2014/07/frequenstist-inferenc... ).
Tarrosion 12y ago
What is the theoretical justification for taking a completely flat prior? "If we set the prior P(Ftrue)∝1 (a flat prior)," There's no probability distribution which is constant over the whole real line. Is the idea that we can pick a distribution which is constant over an arbitrarily large (but finite) interval around the observed data, and so in practice, we may get results arbitrarily close to those given?
- jmount 12y ago
  I'd say the justification is two parts. First is the Bernstein–von Mises theorem (priors don't matter once you have enough data, as long as you didn't violate Cromwell’s rule by using zeros). The second part is: improper priors are considered okay- as long as you check the posterior corresponds to a sensible distribution.
  - Houshalter 12y ago
    If I understand correctly, assuming all real numbers have equal prior probability, then no matter how much data you gather, it's still infinitely unlikely that the true value will exist within any finite range. E.g. the probability of the true value being between -100! and +100! is 0. If you draw from the distribution you will always draw "infinities".
    - jmount 12y ago
      That is why you have to check. If the P(data|param) is concentrated, then even for uniform priors on the real like you can have P(param|data) is proportional to a sensible distribution. But you have to check for specific model and data (say P(data|param) = c e^{(data-param)^2}), it isn't enough to verbally work through infinities.
- micro_cam 12y ago
  Actually kind of the opposite idea. So called "uninformative priors" can be considered a form or model regularization in that they prevent the model from fitting the data too well...they spread the posterior distribution out. Weakly informative priors like a broad normal distribution are also popular and may yield better results.
thebear 12y ago
It is perhaps worth drawing attention to this sentence in the article: "Though Bayes' theorem is where Bayesians get their name, it is not this law itself that is controversial, but the Bayesian interpretation of probability implied by the term P(F_true | D)." A widespread misunderstanding is that there is something fundamentally Bayesian about Bayes' theorem, or even that frequentists don't believe in it. It is rarely pointed out that this is not the case, and we should thank the authors for doing so.
graycat 12y ago
Here's how I relax, avoid both frequentism and bayesianism, and just love probability: I assume that there is a non-empty set, commonly called Omega, which I regard as the set of all experimental 'trials' that I might observe. But, actually, in all the history of everything in the universe, we see only one trial, only one element of this set Omega. Next, there is a non-empty collection, usually denoted by script F, of subsets of Omega. I assume that set script F is contains Omega as an element and is closed under relative complements and countable unions. By relative complements, suppose A is an element of script F. Then the relative complement of set A, maybe written A^c, is essentially set Omega - A, that is, the set of all trials in Omega and not in A. Then set script F is a sigma-algebra. Each set A in script F is an event. If our trial is in set A, then we say that event A has occurred. Next there is a function P: script F --> [0, 1]. P assigns 0 to the empty set (event) and is countably additive. Then function P is a probability measure. So for each event A in script F, P(A) is a number in [0, 1] and is the probability of event A. Now can define what is means for two events to be independent and can generalize to two sigma algebras being independent. Next, on the set R of real numbers, I consider the usual topology, that is, the collection T of open subsets of R. Then I let set B, the Borel sets, be the smallest sigma algebra such that T is a subset of B. Next I consider a function X: Omega --> R such that for each Borel set A, X^{-1}(A) is an element of script F. Then X is a random variable. Essentially anything that can have a numerical value we can regard as a random variable. Then we can state and prove the classic limit theorems -- central limit theorem, weak and strong laws of large numbers, martingale convergence theorem, law of the iterated logarithm, etc. Now we are ready to do applied probability and statistics. And we have never mentioned either frequentism or Bayesianism. For more details, in an elegant presentation, see J. Neveu, Mathematical Foundations of the Calculus of Probability.
- judk 12y ago
  OK, now try to make some claims about the real world. You will have to make frequentist confidence or Bayesian credibility claims.
  - bayesianhorse 12y ago
    Interpreting frequentist confidence in terms of the real world is actually quite difficult.
afafsd 12y ago
I don't understand why Bayesian statistics needs to be an "-ism", and still less why other statistics needs to be an "-ism" too. I don't understand why people feel the need to line up on one side or the other or get so worked up about it. Other branches of mathematics seem to avoid this kind of thing, they have no problem with the idea that there's different ways of doing the same thing. It actually discourages me from learning more about Bayesian statistics, because the whole thing sometimes comes off as a cult.
- lutusp 12y ago
  > I don't understand why Bayesian statistics needs to be an "-ism", and still less why other statistics needs to be an "-ism" too. I don't understand why people feel the need to line up on one side or the other or get so worked up about it. Don't get hung up on the terminology, instead pay attention to the ideas behind the terms. The Frequentist and Bayesian approaches are very different, and produce different outcomes, so they deserve to be understood and their differences sorted out. For example, and without providing all the technical details, using a Frequentist analysis a decision was made to recommend breast cancer screening x-rays for women over a given age. Later, after serious problems arose, a Bayesian approach showed that the false positive rate was 5 to 1 (5 false positives to 1 real cancer detection), which meant many more people were given the false news that they had cancer than those who actually did. https://www.princeton.edu/~achaney/tmve/wiki100k/docs/Bayes__theorem.html https://www.princeton.edu/~achaney/tmve/wiki100k/docs/Bayes_... This is not to exalt the Bayesian approach over the Frequentist, because both have their place, it is only to show how dramatic the difference can be.
  - pessimizer 12y ago
    It seems like you're confusing Bayes' Theorem with Bayesian statistics. Bayes probably wasn't a Bayesian, and everybody uses Bayes' theorem.
    - lutusp 12y ago
      > It seems like you're confusing Bayes' Theorem with Bayesian statistics. Yes, a tempest in a teapot. Anyone caught using Bayes' name without qualifying the use is placed in the same position as someone referring to the Victorian Era without mentioning that Victoria wasn't a Victorian. In most cases, it's not worth the digression.
eli_gottlieb 12y ago
Frequentists and Bayesians care about two different likelihood functions: * Frequentists care about p(evidence | parameters), and interpret probability as a measure over subsets of the counterfactual set of repeated trials (usually independently identically distributed) produced by their model. * Bayesians care about p(parameters | evidence), and interpret probability as "belief" or "propensity to bet". This is, of course, philosophically ridiculous, since they proceed to ground rational belief in Bayesian statistics. What they are really doing is exactly what their likelihood function says: taking a measure over subsets of the counterfactual set of possible worlds which could have produced their evidence. The frequentists have the advantage of their methods being more computationally tractable. The Bayesians have the advantages of intuitiveness and of yielding more accurate inferences from the same limited data-sets. Pick the tool you need and remember what you're taking a measure over!
droob 12y ago
"37 Ways to More Accurately Read the Bones You're Casting to Predict the Harvest"
adrianbg 12y ago
Does anyone know how to make the formulas render properly? Even using the iPython notebook viewer hasn't helped.
- walrus 12y ago
  In your JavaScript console, run: var s = document.createElement('script'); s.src = 'https://cdn.mathjax.org/mathjax/latest/MathJax.js?config=TeX-AMS_HTML'; document.body.appendChild(s); The page is pointing to the old MathJax CDN, which was decommissioned on July 31: http://www.mathjax.org/changes-to-the-mathjax-cdn/ http://www.mathjax.org/changes-to-the-mathjax-cdn/
- deleted 12y ago
  [deleted]

93 points by atakan_gurkan 12y ago

Readers should be aware that the linked article was composed entirely in the IPython notebook environment, which means Python code blocks, Latex renderings, and graphics, can all be freely mixed in a (to me) very nice, readable article format. http://ipython.org/ http://ipython.org/

bayesianhorse 12y ago
And it literally can't be said enough times how awesome it is!

atakan_gurkan 12y ago

The follow-ups are also well worth reading: http://jakevdp.github.io/blog/2014/06/06/frequentism-and-bayesianism-2-when-results-differ/ http://jakevdp.github.io/blog/2014/06/06/frequentism-and-bay... http://jakevdp.github.io/blog/2014/06/12/frequentism-and-bayesianism-3-confidence-credibility/ http://jakevdp.github.io/blog/2014/06/12/frequentism-and-bay... http://jakevdp.github.io/blog/2014/06/14/frequentism-and-bayesianism-4-bayesian-in-python/ http://jakevdp.github.io/blog/2014/06/14/frequentism-and-bay...

nkurz 12y ago
In case it's not clear from the beginning where he really stands on the matter, in Part 3 he offers his opinion on the relative merits of both approaches: <spoiler> The moral of the story is that frequentism and Science do not mix. Let me say it directly: you should be suspicious of the use of frequentist confidence intervals and p-values in science. In a scientific setting, confidence intervals, and closely-related p-values, provide the correct answer to the wrong question. In particular, if you ever find someone stating or implying that a 95% confidence interval is 95% certain to contain a parameter of interest, do not trust their interpretation or their results. If you happen to be peer-reviewing the paper, reject it. Their data do not back-up their conclusion. </spoiler>
- judk 12y ago
  Um, the folks at CERN don't agree with your assessment of frequentism in science. Their papers are explicitly frequentist, they celebrate results based on how many "sigmas" they get. P-value all the aym
  - nkurz 12y ago
    My interpretation? If not clear, I'm quoting the penultimate paragraph of the last post in the series, right before he thanks the reader for making it to the end: http://jakevdp.github.io/blog/2014/06/12/frequentism-and-bayesianism-3-confidence-credibility/#The-Moral-of-the-Story:-Frequentism-and-Science-Do-Not-Mix http://jakevdp.github.io/blog/2014/06/12/frequentism-and-bay...
  - Houshalter 12y ago
    Yes frequentism is common in science and the author strongly objects to it. Read the whole post: https://jakevdp.github.io/blog/2014/06/12/frequentism-and-bayesianism-3-confidence-credibility/ https://jakevdp.github.io/blog/2014/06/12/frequentism-and-ba...
  - bayesianhorse 12y ago
    I'm not a physicist, but as far as I know, CERN has access to tons of data with every intention of drowning any prior belief. In this setting, I would expect frequentist methods to shine, and bayesian method to be intractable. However I would not expect CERN papers to make the kinds of terminological / theoretical lapses about confidence the parent thread was talking about. Papers should be rejected for that kind of error, even if you are not a bayesian.
    - betatim 12y ago
      Exactly this. At CERN (and all of HEP elsewhere) the Bayes vs Freq wars were fought many years ago and are long over. The conclusion: when you have lots of data they converge (as they should!). In the case that they differ you almost always find that you have very few observations. I would argue that this 'difference' is not that exciting because it must be dominated by your assumptions, not your observations. After all once you accumulate enough observations the two methods tend to converge. Personal conclusion: if the methods disagree work on getting more data instead of fighting over which method is better.
anon4 12y ago
That first followup seems patently insane. "We don't know the value of p̂, so we'll assume it's 5/8". Why would you ever assume like that? The Bayesian approach appears to me exactly like the Frequentist, without the part where they assume convenient values of unknown parameters.

jmount 12y ago

Nice article. In this direction my group has been trying to help teach that you tend to need to be familiar with both frequentist and Bayesian thought (you can't always choose one or the other: http://www.win-vector.com/blog/2013/05/bayesian-and-frequentist-approaches-ask-the-right-question/ http://www.win-vector.com/blog/2013/05/bayesian-and-frequent... ) and that Bayesianism only appears to be the more complicated of the two ( http://www.win-vector.com/blog/2014/07/frequenstist-inference-only-seems-easy/ http://www.win-vector.com/blog/2014/07/frequenstist-inferenc... ).

Tarrosion 12y ago

What is the theoretical justification for taking a completely flat prior? "If we set the prior P(Ftrue)∝1 (a flat prior)," There's no probability distribution which is constant over the whole real line. Is the idea that we can pick a distribution which is constant over an arbitrarily large (but finite) interval around the observed data, and so in practice, we may get results arbitrarily close to those given?

jmount 12y ago
I'd say the justification is two parts. First is the Bernstein–von Mises theorem (priors don't matter once you have enough data, as long as you didn't violate Cromwell’s rule by using zeros). The second part is: improper priors are considered okay- as long as you check the posterior corresponds to a sensible distribution.
- Houshalter 12y ago
  If I understand correctly, assuming all real numbers have equal prior probability, then no matter how much data you gather, it's still infinitely unlikely that the true value will exist within any finite range. E.g. the probability of the true value being between -100! and +100! is 0. If you draw from the distribution you will always draw "infinities".
  - jmount 12y ago
    That is why you have to check. If the P(data|param) is concentrated, then even for uniform priors on the real like you can have P(param|data) is proportional to a sensible distribution. But you have to check for specific model and data (say P(data|param) = c e^{(data-param)^2}), it isn't enough to verbally work through infinities.
micro_cam 12y ago
Actually kind of the opposite idea. So called "uninformative priors" can be considered a form or model regularization in that they prevent the model from fitting the data too well...they spread the posterior distribution out. Weakly informative priors like a broad normal distribution are also popular and may yield better results.

thebear 12y ago

It is perhaps worth drawing attention to this sentence in the article: "Though Bayes' theorem is where Bayesians get their name, it is not this law itself that is controversial, but the Bayesian interpretation of probability implied by the term P(F_true | D)." A widespread misunderstanding is that there is something fundamentally Bayesian about Bayes' theorem, or even that frequentists don't believe in it. It is rarely pointed out that this is not the case, and we should thank the authors for doing so.

graycat 12y ago

Here's how I relax, avoid both frequentism and bayesianism, and just love probability: I assume that there is a non-empty set, commonly called Omega, which I regard as the set of all experimental 'trials' that I might observe. But, actually, in all the history of everything in the universe, we see only one trial, only one element of this set Omega. Next, there is a non-empty collection, usually denoted by script F, of subsets of Omega. I assume that set script F is contains Omega as an element and is closed under relative complements and countable unions. By relative complements, suppose A is an element of script F. Then the relative complement of set A, maybe written A^c, is essentially set Omega - A, that is, the set of all trials in Omega and not in A. Then set script F is a sigma-algebra. Each set A in script F is an event. If our trial is in set A, then we say that event A has occurred. Next there is a function P: script F --> [0, 1]. P assigns 0 to the empty set (event) and is countably additive. Then function P is a probability measure. So for each event A in script F, P(A) is a number in [0, 1] and is the probability of event A. Now can define what is means for two events to be independent and can generalize to two sigma algebras being independent. Next, on the set R of real numbers, I consider the usual topology, that is, the collection T of open subsets of R. Then I let set B, the Borel sets, be the smallest sigma algebra such that T is a subset of B. Next I consider a function X: Omega --> R such that for each Borel set A, X^{-1}(A) is an element of script F. Then X is a random variable. Essentially anything that can have a numerical value we can regard as a random variable. Then we can state and prove the classic limit theorems -- central limit theorem, weak and strong laws of large numbers, martingale convergence theorem, law of the iterated logarithm, etc. Now we are ready to do applied probability and statistics. And we have never mentioned either frequentism or Bayesianism. For more details, in an elegant presentation, see J. Neveu, Mathematical Foundations of the Calculus of Probability.

judk 12y ago
OK, now try to make some claims about the real world. You will have to make frequentist confidence or Bayesian credibility claims.
- bayesianhorse 12y ago
  Interpreting frequentist confidence in terms of the real world is actually quite difficult.

afafsd 12y ago

I don't understand why Bayesian statistics needs to be an "-ism", and still less why other statistics needs to be an "-ism" too. I don't understand why people feel the need to line up on one side or the other or get so worked up about it. Other branches of mathematics seem to avoid this kind of thing, they have no problem with the idea that there's different ways of doing the same thing. It actually discourages me from learning more about Bayesian statistics, because the whole thing sometimes comes off as a cult.

lutusp 12y ago
> I don't understand why Bayesian statistics needs to be an "-ism", and still less why other statistics needs to be an "-ism" too. I don't understand why people feel the need to line up on one side or the other or get so worked up about it. Don't get hung up on the terminology, instead pay attention to the ideas behind the terms. The Frequentist and Bayesian approaches are very different, and produce different outcomes, so they deserve to be understood and their differences sorted out. For example, and without providing all the technical details, using a Frequentist analysis a decision was made to recommend breast cancer screening x-rays for women over a given age. Later, after serious problems arose, a Bayesian approach showed that the false positive rate was 5 to 1 (5 false positives to 1 real cancer detection), which meant many more people were given the false news that they had cancer than those who actually did. https://www.princeton.edu/~achaney/tmve/wiki100k/docs/Bayes__theorem.html https://www.princeton.edu/~achaney/tmve/wiki100k/docs/Bayes_... This is not to exalt the Bayesian approach over the Frequentist, because both have their place, it is only to show how dramatic the difference can be.
- pessimizer 12y ago
  It seems like you're confusing Bayes' Theorem with Bayesian statistics. Bayes probably wasn't a Bayesian, and everybody uses Bayes' theorem.
  - lutusp 12y ago
    > It seems like you're confusing Bayes' Theorem with Bayesian statistics. Yes, a tempest in a teapot. Anyone caught using Bayes' name without qualifying the use is placed in the same position as someone referring to the Victorian Era without mentioning that Victoria wasn't a Victorian. In most cases, it's not worth the digression.

eli_gottlieb 12y ago

Frequentists and Bayesians care about two different likelihood functions: * Frequentists care about p(evidence | parameters), and interpret probability as a measure over subsets of the counterfactual set of repeated trials (usually independently identically distributed) produced by their model. * Bayesians care about p(parameters | evidence), and interpret probability as "belief" or "propensity to bet". This is, of course, philosophically ridiculous, since they proceed to ground rational belief in Bayesian statistics. What they are really doing is exactly what their likelihood function says: taking a measure over subsets of the counterfactual set of possible worlds which could have produced their evidence. The frequentists have the advantage of their methods being more computationally tractable. The Bayesians have the advantages of intuitiveness and of yielding more accurate inferences from the same limited data-sets. Pick the tool you need and remember what you're taking a measure over!

droob 12y ago

"37 Ways to More Accurately Read the Bones You're Casting to Predict the Harvest"

adrianbg 12y ago

Does anyone know how to make the formulas render properly? Even using the iPython notebook viewer hasn't helped.

walrus 12y ago
In your JavaScript console, run: var s = document.createElement('script'); s.src = 'https://cdn.mathjax.org/mathjax/latest/MathJax.js?config=TeX-AMS_HTML'; document.body.appendChild(s); The page is pointing to the old MathJax CDN, which was decommissioned on July 31: http://www.mathjax.org/changes-to-the-mathjax-cdn/ http://www.mathjax.org/changes-to-the-mathjax-cdn/
deleted 12y ago
[deleted]