7 ms·

Modern Science and the Bayesian-Frequentist Controversy (2005) [pdf]

70 points by ot 12y ago

3pt14159 12y ago
I remember how weird I felt in first year university taking frequentist based statistics. Unbeknownst to me, my statistics training in high school was completely Bayesian and when the university prof started talking about how we needed to sample a certain population set in order to come to an accurate conclusion I felt very uneasy with the whole mess. I know both systems are useful and reasonable in their own sense, and often even equivalent, but I found the Bayesian way of thinking so much more intuitive than the frequentist way.
- icelancer 12y ago
  I've taught a fair number of people statistics, and whatever they learn first is what is most intuitive - the vast majority of the time. It just so happens to be that they learn frequentist statistics (or some derivation of them) first.
- eli_gottlieb 12y ago
  I learned Bayesianism in a discrete probability and combinatorics class in undergrad, then took frequentist statistics in grad-school. The big problem is not that frequentism is somehow "just wrong", but that many people combine frequentist hypothesis testing with Popperian falsificationism (see rant above) to shoehorn frequentism into doing inference about hypotheses, which is profoundly wrong and not at all what frequentism is about, mathematically. The explanations for how frequentist inferences get used to do science feel wrong because they are: scientific induction is Bayesian. Of course, given mountains of data, either approach will work and converge to more-or-less identical results, but that's true of almost anything in statistics. Another thing is that I really wish the frequentist lectures would just up and admit that yes, likelihoods really are conditional probabilities: they're just conditional probabilities in the Bayesian mode. A likelihood (as used in frequentist statistics) is just the probability of the evidence conditioned on a certain parameter value. Even frequentists ought to admit that it talks about a counterfactual limit of the frequency ratio, but apparently calculations on counterfactuals are "too Bayesian" for them, so they pretend that "likelihoods aren't really probabilities". That confused the hell out of me in class.
tjradcliffe 12y ago
One aspect of the Bayesian/Frequentist "controversy" (which amongst working scientists isn't very controversial) is the supposedly subjective nature of probability under the Bayesian system. I think this is mostly a conceptual error on the part of Bayesians: we should be treating the "degree of belief" in the Bayesian system as precisely that, a degree of belief, and nothing more. It is not a "probability", which I'm happy to leave with a frequentist definition. That is, Bayes' theorem should read: p(h|e) = P(e|h)*p(h)/P(e) where the lower-case "p"'s indicate plausibility (degree of belief) and the upper case "P"'s indicate probability as measured by frequency of occurrence. Probabilities under this system are completely objective. Plausibilities are--naturally--subjective. As someone who advocates for an epistemology of a knowing subject (that is: there is no knowing without a subject that knows, and the central question of epistemology is how that subject knows) it is entirely natural that the degree of belief in a proposition should be subject-specific. After all, whose degree of belief is it? This is not a failure of the Bayesian approach, but a virtue: it allows us to successfully navigate the many traditional paradoxes of subject-free theories of knowledge, which put "truth" outside of the realm of the knower and pursue a false goal (certainty) much like pre-scientific alchemists pursued the false goal of the philosopher's stone.
- cscheid 12y ago
  Wait: the entire point of Bayes's idea was that under very sensible and straightforward rules, "plausibilities" behave exactly like probabilities. So why multiply concepts? What is the difference, in your proposal, between "plausibilities" and "probabilities"?
  - jerrytsai 12y ago
    You are correct: the way these mathematical creatures work, plausibilities behave like probabilties. What tjradcliife is trying to do, by differentiating between plausibilities and probabilities is to address the concern of frequentism with Bayesianism: namely, the "P(h)"-- what is usually termed as a "prior probability" in statistical jargon-- usually does not arise from objective data. Accordingly, Bayesianism can be treated with suspicion-- since the P(h) is arbitrarily specified, one could "load the dice" and obtain a P(h|e) ("posterior probability") guaranteed to suit the analyst's conclusions. By differentiating the two creatures, tjradcliffe hopes to resolve the dis-ease with calling P(h|e) a "probability". By frequentist philosophy, probabilities should be grounded in some sort of firm soil, e.g., the relative frequency of what would be observed if one could conduct an experiment ad infinitum. A Bayesian "probability" is not grounded this way, so calling it a "plausibility" instead would resolve this tension. Another way of putting this-- P(h) to a frequentist feels like a tainted probability-- really not a probability at all-- so even after updating the prior probability with data [i.e., P(e|h)/P(e)], the posterior probability P(h|e) still retains the influence of that original taint. Solution: stop calling the Bayesian probabilities "probabilities"; instead, call them "plausibilities".
tedsanders 12y ago
I don't think this paper teaches the difference between Bayesianism and Frequentism (though in fairness to the author, that may not be the goal). The paper says Bayesianism is 'aggressive' and 'subjective' while Frequentism is 'cautious' and aims for 'universally acceptable' conclusions. I wouldn't characterize them that way. Frequentism is also subjective (http://lesswrong.com/lw/1gc/frequentist_statistics_are_frequently_subjective/ http://lesswrong.com/lw/1gc/frequentist_statistics_are_frequ...) and Bayesianism can be quite cautious. To me, the real difference in Bayesian and Frequentist methods are the types of questions they aim to answer. Bayesian methods try to answer the question of "What is the probability of the model" given some observed data. Frequentist methods try to answer the question of "What is the probability of the observed data" given an assumed model. In this sense, they are somewhat complementary, which why many practicing statisticians don't actually care about the Bayesianism-Frequentism debate. I am happy that Bayesian methods are becoming more popular, though, since the questions we actually care about ("what is the probability of the model given the data?") are often best suited by Bayesian methods. Also, as a physicist I'm disappointed that the paper asserts that physicists disdained statistics in 1903. For the most part, statistical mechanics was developed in the late 1800s! (https://en.wikipedia.org/wiki/Statistical_mechanics#History https://en.wikipedia.org/wiki/Statistical_mechanics#History)
- bkcooper 12y ago
  Also, as a physicist I'm disappointed that the paper asserts that physicists disdained statistics in 1903. For the most part, statistical mechanics was developed in the late 1800s! Sure, but Boltzmann's ideas were famously controversial at the time. I don't think they represent a good measure of mainstream practice at the time.
- ced 12y ago
  It's a bit of a tangent from your main point, but: Bayesian methods try to answer the question of "What is the probability of the model" given some observed data There are people arguing that P(M|D) isn't really a meaningful quantity, because the space of models is infinite, and we can't reasonably put a prior on it. I prefer talking about P(M1|D)/P(M2|D): "The data favors model 1 three times more than it favors model 2" The philosophical problem with P(M|D) is the same as distinguishing randomness from non-randomness (cf. Popperian falsifiability): there's no test to prove that something is truly random, all we can say is "We've looked for patterns over and over again, but we haven't found one yet".
cscheid 12y ago
The paper is from 2005, so it might be worth saying that on the submission title. And it's also worth mentioning that because Gelman and Shalizi have recently written a really nice paper about the philosophy of Bayes: http://www.stat.columbia.edu/~gelman/research/published/philosophy.pdf http://www.stat.columbia.edu/~gelman/research/published/phil... (PDF) Gelman is a well-known "bayesian", and Shalizi is a well-known "not all that is interesting is bayes"-ian.
murbard2 12y ago
Frequentist statistics is so easy to use! Just put your raw data in a text file, tar it together with your research paper, and put the tarball through a hash function. If the first 7 bits are 0, then you can reject the null hypothesis with p < 0.01.
- murbard2 12y ago
  So this is obviously a joke, but I think it illustrates something important. First off, it's important to realize that the above approach is technically correct. The odds of getting 7 zeros in the hash of your research paper by chance are indeed less than 1%. However, this is obviously a useless test, its probability of rejecting the null hypothesis given that it is false is likely 1/128... We don't really expect the phenomenom that we're measuring to produce good hash preimages. In technical terms, the test has no power. However, to even talk about the "power" of a test, one needs a model of what the effect looks like. For instance, we expect the medicine will increase the rate of cures, we expect education will increase job success, etc. And yet, the entire freaking point of p-value testing is to maintain a pretense that Popperianism makes sense. Validating a hypothesis? While my simple minded fellow, that would be anathema! Don't you know that Popper explained that you can only refute hypotheses! So we'll refute the null hypothesis instead. Fine, but if you do that, the hash-function method works. The reason it sounds stupid is that Popperianism is wrong.
  - sa1 12y ago
    Can you explain the part where you are criticising Popperianism to me?(honest question) From what I gather, you seem to be criticising logical positivism more, but I only have a layman's idea of the philosophy involved.
    - murbard2 12y ago
      Popper attempts to do away with induction. He does so by insisting that the only valid form of scientific inquiry is to refute provably false hypotheses. That leaves us with the vague notion of Occam's razor to find out which hypotheses we should go by. The focus is on rejecting false hypotheses, not on proving correct ones. Translated into statistics, this gives you p-values. To pretend they're not relying on induction, people test "the null hypothesis", hopping to statistically disprove it. My point is that, in the absence of another hypothesis, there are an infinite number of ways you can disprove the null hypothesis. In real science, the actual hypothesis is always lurking in the closet, so why not actually model it and compute likelihood ratio? There is actually a much more rigorous framework for epistemology, it's algorithmic complexity. While it's not practical to use (the most common version is uncomputable), it's an ideal that helps thinking about the scientific method.
graycat 12y ago
Let's clean up the "controversy": There's some pure math due to Lebesgue, Kolmogorov, and others that defines, clearly, precisely, a 'probability space' along with trials, events, probabilities, and random variables. It's a very nice box of math with a lot of really nice theorems with rock solid proofs, e.g., the algebra of random variables, approximation of random variables by other random variables, convergence of random variables, independence, conditioning and conditional independence, the central limit theorem, the weak and strong laws of large numbers, ergodic theory, Brownian motion, martingales and the martingale convergence theorem, Markov processes, and much more. That math is clean and rock solid. Does it work in practice? Often, yes. Much of the connection with practice is easy to see. For much of the rest, keys are when we have independence or conditional independence, and there are some surprisingly good, intuitive approaches to these two. Next, it works. Now, call it done. 'Controversy' over with. For the math, see, e.g., J. Neveu, 'Mathematical Foundations of the Calculus of Probability'. 'Frequentist'? That's essentially answered by the weak/strong laws of large numbers. Bayesian? That's essentially conditional probability, with the advanced approach via the Radon-Nikodym theorem with a famous proof by von Neumann. Can also prove some theorems that say that a Bayesian start doesn't have to hurt after a while.
pcvarmint 12y ago
Lecture video (starts at 03:40): https://www.youtube.com/watch?v=DMnCFRtOITI https://www.youtube.com/watch?v=DMnCFRtOITI Interview mentioned at introduction: https://projecteuclid.org/download/pdf_1/euclid.ss/1063994981 https://projecteuclid.org/download/pdf_1/euclid.ss/106399498...
jmount 12y ago
A fun example I worked out of a situation that is a bit of a problem for frequentist inference (i.e. designed to be compatible with Bayesian inference): http://www.win-vector.com/blog/2014/07/frequenstist-inference-only-seems-easy/ http://www.win-vector.com/blog/2014/07/frequenstist-inferenc...

70 points by ot 12y ago

I remember how weird I felt in first year university taking frequentist based statistics. Unbeknownst to me, my statistics training in high school was completely Bayesian and when the university prof started talking about how we needed to sample a certain population set in order to come to an accurate conclusion I felt very uneasy with the whole mess. I know both systems are useful and reasonable in their own sense, and often even equivalent, but I found the Bayesian way of thinking so much more intuitive than the frequentist way.

icelancer 12y ago
I've taught a fair number of people statistics, and whatever they learn first is what is most intuitive - the vast majority of the time. It just so happens to be that they learn frequentist statistics (or some derivation of them) first.
eli_gottlieb 12y ago
I learned Bayesianism in a discrete probability and combinatorics class in undergrad, then took frequentist statistics in grad-school. The big problem is not that frequentism is somehow "just wrong", but that many people combine frequentist hypothesis testing with Popperian falsificationism (see rant above) to shoehorn frequentism into doing inference about hypotheses, which is profoundly wrong and not at all what frequentism is about, mathematically. The explanations for how frequentist inferences get used to do science feel wrong because they are: scientific induction is Bayesian. Of course, given mountains of data, either approach will work and converge to more-or-less identical results, but that's true of almost anything in statistics. Another thing is that I really wish the frequentist lectures would just up and admit that yes, likelihoods really are conditional probabilities: they're just conditional probabilities in the Bayesian mode. A likelihood (as used in frequentist statistics) is just the probability of the evidence conditioned on a certain parameter value. Even frequentists ought to admit that it talks about a counterfactual limit of the frequency ratio, but apparently calculations on counterfactuals are "too Bayesian" for them, so they pretend that "likelihoods aren't really probabilities". That confused the hell out of me in class.

tjradcliffe 12y ago

One aspect of the Bayesian/Frequentist "controversy" (which amongst working scientists isn't very controversial) is the supposedly subjective nature of probability under the Bayesian system. I think this is mostly a conceptual error on the part of Bayesians: we should be treating the "degree of belief" in the Bayesian system as precisely that, a degree of belief, and nothing more. It is not a "probability", which I'm happy to leave with a frequentist definition. That is, Bayes' theorem should read: p(h|e) = P(e|h)*p(h)/P(e) where the lower-case "p"'s indicate plausibility (degree of belief) and the upper case "P"'s indicate probability as measured by frequency of occurrence. Probabilities under this system are completely objective. Plausibilities are--naturally--subjective. As someone who advocates for an epistemology of a knowing subject (that is: there is no knowing without a subject that knows, and the central question of epistemology is how that subject knows) it is entirely natural that the degree of belief in a proposition should be subject-specific. After all, whose degree of belief is it? This is not a failure of the Bayesian approach, but a virtue: it allows us to successfully navigate the many traditional paradoxes of subject-free theories of knowledge, which put "truth" outside of the realm of the knower and pursue a false goal (certainty) much like pre-scientific alchemists pursued the false goal of the philosopher's stone.

cscheid 12y ago
Wait: the entire point of Bayes's idea was that under very sensible and straightforward rules, "plausibilities" behave exactly like probabilities. So why multiply concepts? What is the difference, in your proposal, between "plausibilities" and "probabilities"?
- jerrytsai 12y ago
  You are correct: the way these mathematical creatures work, plausibilities behave like probabilties. What tjradcliife is trying to do, by differentiating between plausibilities and probabilities is to address the concern of frequentism with Bayesianism: namely, the "P(h)"-- what is usually termed as a "prior probability" in statistical jargon-- usually does not arise from objective data. Accordingly, Bayesianism can be treated with suspicion-- since the P(h) is arbitrarily specified, one could "load the dice" and obtain a P(h|e) ("posterior probability") guaranteed to suit the analyst's conclusions. By differentiating the two creatures, tjradcliffe hopes to resolve the dis-ease with calling P(h|e) a "probability". By frequentist philosophy, probabilities should be grounded in some sort of firm soil, e.g., the relative frequency of what would be observed if one could conduct an experiment ad infinitum. A Bayesian "probability" is not grounded this way, so calling it a "plausibility" instead would resolve this tension. Another way of putting this-- P(h) to a frequentist feels like a tainted probability-- really not a probability at all-- so even after updating the prior probability with data [i.e., P(e|h)/P(e)], the posterior probability P(h|e) still retains the influence of that original taint. Solution: stop calling the Bayesian probabilities "probabilities"; instead, call them "plausibilities".

tedsanders 12y ago

I don't think this paper teaches the difference between Bayesianism and Frequentism (though in fairness to the author, that may not be the goal). The paper says Bayesianism is 'aggressive' and 'subjective' while Frequentism is 'cautious' and aims for 'universally acceptable' conclusions. I wouldn't characterize them that way. Frequentism is also subjective (http://lesswrong.com/lw/1gc/frequentist_statistics_are_frequently_subjective/ http://lesswrong.com/lw/1gc/frequentist_statistics_are_frequ...) and Bayesianism can be quite cautious. To me, the real difference in Bayesian and Frequentist methods are the types of questions they aim to answer. Bayesian methods try to answer the question of "What is the probability of the model" given some observed data. Frequentist methods try to answer the question of "What is the probability of the observed data" given an assumed model. In this sense, they are somewhat complementary, which why many practicing statisticians don't actually care about the Bayesianism-Frequentism debate. I am happy that Bayesian methods are becoming more popular, though, since the questions we actually care about ("what is the probability of the model given the data?") are often best suited by Bayesian methods. Also, as a physicist I'm disappointed that the paper asserts that physicists disdained statistics in 1903. For the most part, statistical mechanics was developed in the late 1800s! (https://en.wikipedia.org/wiki/Statistical_mechanics#History https://en.wikipedia.org/wiki/Statistical_mechanics#History)

bkcooper 12y ago
Also, as a physicist I'm disappointed that the paper asserts that physicists disdained statistics in 1903. For the most part, statistical mechanics was developed in the late 1800s! Sure, but Boltzmann's ideas were famously controversial at the time. I don't think they represent a good measure of mainstream practice at the time.
ced 12y ago
It's a bit of a tangent from your main point, but: Bayesian methods try to answer the question of "What is the probability of the model" given some observed data There are people arguing that P(M|D) isn't really a meaningful quantity, because the space of models is infinite, and we can't reasonably put a prior on it. I prefer talking about P(M1|D)/P(M2|D): "The data favors model 1 three times more than it favors model 2" The philosophical problem with P(M|D) is the same as distinguishing randomness from non-randomness (cf. Popperian falsifiability): there's no test to prove that something is truly random, all we can say is "We've looked for patterns over and over again, but we haven't found one yet".

cscheid 12y ago

The paper is from 2005, so it might be worth saying that on the submission title. And it's also worth mentioning that because Gelman and Shalizi have recently written a really nice paper about the philosophy of Bayes: http://www.stat.columbia.edu/~gelman/research/published/philosophy.pdf http://www.stat.columbia.edu/~gelman/research/published/phil... (PDF) Gelman is a well-known "bayesian", and Shalizi is a well-known "not all that is interesting is bayes"-ian.

murbard2 12y ago

Frequentist statistics is so easy to use! Just put your raw data in a text file, tar it together with your research paper, and put the tarball through a hash function. If the first 7 bits are 0, then you can reject the null hypothesis with p < 0.01.

murbard2 12y ago
So this is obviously a joke, but I think it illustrates something important. First off, it's important to realize that the above approach is technically correct. The odds of getting 7 zeros in the hash of your research paper by chance are indeed less than 1%. However, this is obviously a useless test, its probability of rejecting the null hypothesis given that it is false is likely 1/128... We don't really expect the phenomenom that we're measuring to produce good hash preimages. In technical terms, the test has no power. However, to even talk about the "power" of a test, one needs a model of what the effect looks like. For instance, we expect the medicine will increase the rate of cures, we expect education will increase job success, etc. And yet, the entire freaking point of p-value testing is to maintain a pretense that Popperianism makes sense. Validating a hypothesis? While my simple minded fellow, that would be anathema! Don't you know that Popper explained that you can only refute hypotheses! So we'll refute the null hypothesis instead. Fine, but if you do that, the hash-function method works. The reason it sounds stupid is that Popperianism is wrong.
- sa1 12y ago
  Can you explain the part where you are criticising Popperianism to me?(honest question) From what I gather, you seem to be criticising logical positivism more, but I only have a layman's idea of the philosophy involved.
  - murbard2 12y ago
    Popper attempts to do away with induction. He does so by insisting that the only valid form of scientific inquiry is to refute provably false hypotheses. That leaves us with the vague notion of Occam's razor to find out which hypotheses we should go by. The focus is on rejecting false hypotheses, not on proving correct ones. Translated into statistics, this gives you p-values. To pretend they're not relying on induction, people test "the null hypothesis", hopping to statistically disprove it. My point is that, in the absence of another hypothesis, there are an infinite number of ways you can disprove the null hypothesis. In real science, the actual hypothesis is always lurking in the closet, so why not actually model it and compute likelihood ratio? There is actually a much more rigorous framework for epistemology, it's algorithmic complexity. While it's not practical to use (the most common version is uncomputable), it's an ideal that helps thinking about the scientific method.

graycat 12y ago

Let's clean up the "controversy": There's some pure math due to Lebesgue, Kolmogorov, and others that defines, clearly, precisely, a 'probability space' along with trials, events, probabilities, and random variables. It's a very nice box of math with a lot of really nice theorems with rock solid proofs, e.g., the algebra of random variables, approximation of random variables by other random variables, convergence of random variables, independence, conditioning and conditional independence, the central limit theorem, the weak and strong laws of large numbers, ergodic theory, Brownian motion, martingales and the martingale convergence theorem, Markov processes, and much more. That math is clean and rock solid. Does it work in practice? Often, yes. Much of the connection with practice is easy to see. For much of the rest, keys are when we have independence or conditional independence, and there are some surprisingly good, intuitive approaches to these two. Next, it works. Now, call it done. 'Controversy' over with. For the math, see, e.g., J. Neveu, 'Mathematical Foundations of the Calculus of Probability'. 'Frequentist'? That's essentially answered by the weak/strong laws of large numbers. Bayesian? That's essentially conditional probability, with the advanced approach via the Radon-Nikodym theorem with a famous proof by von Neumann. Can also prove some theorems that say that a Bayesian start doesn't have to hurt after a while.

pcvarmint 12y ago

Lecture video (starts at 03:40): https://www.youtube.com/watch?v=DMnCFRtOITI https://www.youtube.com/watch?v=DMnCFRtOITI Interview mentioned at introduction: https://projecteuclid.org/download/pdf_1/euclid.ss/1063994981 https://projecteuclid.org/download/pdf_1/euclid.ss/106399498...

jmount 12y ago

A fun example I worked out of a situation that is a bit of a problem for frequentist inference (i.e. designed to be compatible with Bayesian inference): http://www.win-vector.com/blog/2014/07/frequenstist-inference-only-seems-easy/ http://www.win-vector.com/blog/2014/07/frequenstist-inferenc...