3 ms·

My impression was that frequentists and bayesians do actually disagree in some cases. The example I remember is, suppose you have a coin with a uniformly rando

My impression was that frequentists and bayesians do actually disagree in some cases. The example I remember is, suppose you have a coin with a uniformly random bias, except that the bias will not be zero. What is the probability that you will see heads if you flip it once? - Bayesian: 50% - Frequentist: Anything except 50% Although you'd probably find they agreed on all the "what do I expect to see" stuff, so I guess that kind of is like interpretations of QM.

pseut 13y ago
That's almost a straw-man argument. When you're making statements about pure probabilities, everyone is a bayesian. Most toy examples make "frequentists" look like morons, because essentially all of the judgement and discretion is removed from the problem, so you'd have to be an idiot not to apply Bayes's rule. The difference shows up when you actually have an interesting data set to analyze. Bayesian statistics can disagree with frequentist stats in small samples because they're (often) using different normalization strategies; and they can disagree in large samples where the CLT fails. There may be other settings where they diverge too that I'm not aware of. But neither of those scenarios is one where insisting "I'm a Bayesian, so the answer is blah" or "I'm a frequentist, so... blah blah" is likely to be a good strategy. Those are the settings where it's hard.
- delluminatus 13y ago
  If an example can illustrate a difference between two formal methodologies, the example should be as simple as possible, right? It doesn't paint one as superior to the other, but simply highlights the different methods by which they assign probabilities to events. If you make the example more complicated, the difference becomes less clear.
  - pseut 13y ago
    Right, but my point is that "as simple as possible" in this case is still "very complicated." It's sort of like trying to use "Hello world" to explain the difference between static and dynamic typing.
_dps 13y ago
A practical way to summarize the difference is that Frequentists refuse to assign the word "probability" to anything other than the output of a formal generative model, whereas Bayesians use the word more broadly and are willing to ask "What is the probability of this hypothesis being true?". The latter is often called "subjective probability" in texts that attempt to discuss the division. In response to the Bayesians' question, Frequentists say "You've specified no sampling process for hypotheses so I can't answer your question. If you want to represent your subjective belief system over hypotheses as a probability distribution you are welcome to do so by Cox's theorem. But the resulting number should not be understood as probability in the sense of fractional sampling, and we'd really be happier if you'd use a different word for it." Frequentists and Bayesians agree precisely when the Bayesian's prior is the output distribution of a generative process for the inputs to the problem under study (in this case, biases for coins). If the Bayesian's prior is "just chosen based on judgement" then the Frequentist theory says "That's no longer probability, but it may still be a useful framework for quantifying subjective beliefs and the support for such beliefs from observational data." So, in your coin example: Bayesian: anything you want, based on your prior for coin biases. If your prior is symmetric, then 50% as you say. Frequentist: the question is meaningless without a formally specified generating process for biased coins, but if you tell me the generating process then I can give you a ranking (likelihood) function over all possible biases.
- pdonis 13y ago
  if you tell me the generating process then I can give you a ranking (likelihood) function over all possible biases. But if we already know the generating process, all the hard work is already done. The case of real interest is where we don't know the generating process, but we have a bunch of data, and we're trying to figure out what the generating process is from the data.
  - _dps 13y ago
    > But if we already know the generating process, all the hard work is already done. Yes, I agree completely, and I believe that so would most frequentists! The point, even among the most strident critics, is rarely one of whether Bayesian approaches are useful; it is that Frequentists regard the proper domain of what they call "probability" to be exclusively related to generative processes and relative sampling ratios. From a purely technical point of view I see nothing wrong with that, even if it is a somewhat strict position (having trained primarily in mathematics I'm comfortable allowing people their strict definitions as long as they are recognized as such). As a natural consequence of this divergence, Bayesian modeling allows you to explicitly punt the problem of generative modeling for the tricky bits, and it puts a nice big warning label on it saying "If you botch your prior, you're going to have a bad time." The biased coin is a perfect example. Knowing the bias, the generative process for a sequence of flips is trivial. But how are you going to create a generative process that seriously engages how and where the biases emerge? It is far easier to just say "empirically, biases seem to be distributed like so." At this point the Frequentist says "Well, that assumption is not derived from a formal model and is only loosely falsifiable, but if I accept it as a substitute for a generative model then you and I will reach the same conclusions about posterior probabilities." I know of no frequentist who would disagree that getting good generative models for complex phenomena is often extraordinarily difficult. Where they disagree with Bayesians is whether doing something other than that should, strictly, be called "probability" or, whether it is more appropriate to call it something like "semi-empirical subjective-belief modeling".

pseut 13y ago

That's almost a straw-man argument. When you're making statements about pure probabilities, everyone is a bayesian. Most toy examples make "frequentists" look like morons, because essentially all of the judgement and discretion is removed from the problem, so you'd have to be an idiot not to apply Bayes's rule. The difference shows up when you actually have an interesting data set to analyze. Bayesian statistics can disagree with frequentist stats in small samples because they're (often) using different normalization strategies; and they can disagree in large samples where the CLT fails. There may be other settings where they diverge too that I'm not aware of. But neither of those scenarios is one where insisting "I'm a Bayesian, so the answer is blah" or "I'm a frequentist, so... blah blah" is likely to be a good strategy. Those are the settings where it's hard.

delluminatus 13y ago
If an example can illustrate a difference between two formal methodologies, the example should be as simple as possible, right? It doesn't paint one as superior to the other, but simply highlights the different methods by which they assign probabilities to events. If you make the example more complicated, the difference becomes less clear.
- pseut 13y ago
  Right, but my point is that "as simple as possible" in this case is still "very complicated." It's sort of like trying to use "Hello world" to explain the difference between static and dynamic typing.

_dps 13y ago

A practical way to summarize the difference is that Frequentists refuse to assign the word "probability" to anything other than the output of a formal generative model, whereas Bayesians use the word more broadly and are willing to ask "What is the probability of this hypothesis being true?". The latter is often called "subjective probability" in texts that attempt to discuss the division. In response to the Bayesians' question, Frequentists say "You've specified no sampling process for hypotheses so I can't answer your question. If you want to represent your subjective belief system over hypotheses as a probability distribution you are welcome to do so by Cox's theorem. But the resulting number should not be understood as probability in the sense of fractional sampling, and we'd really be happier if you'd use a different word for it." Frequentists and Bayesians agree precisely when the Bayesian's prior is the output distribution of a generative process for the inputs to the problem under study (in this case, biases for coins). If the Bayesian's prior is "just chosen based on judgement" then the Frequentist theory says "That's no longer probability, but it may still be a useful framework for quantifying subjective beliefs and the support for such beliefs from observational data." So, in your coin example: Bayesian: anything you want, based on your prior for coin biases. If your prior is symmetric, then 50% as you say. Frequentist: the question is meaningless without a formally specified generating process for biased coins, but if you tell me the generating process then I can give you a ranking (likelihood) function over all possible biases.

pdonis 13y ago
if you tell me the generating process then I can give you a ranking (likelihood) function over all possible biases. But if we already know the generating process, all the hard work is already done. The case of real interest is where we don't know the generating process, but we have a bunch of data, and we're trying to figure out what the generating process is from the data.
- _dps 13y ago
  > But if we already know the generating process, all the hard work is already done. Yes, I agree completely, and I believe that so would most frequentists! The point, even among the most strident critics, is rarely one of whether Bayesian approaches are useful; it is that Frequentists regard the proper domain of what they call "probability" to be exclusively related to generative processes and relative sampling ratios. From a purely technical point of view I see nothing wrong with that, even if it is a somewhat strict position (having trained primarily in mathematics I'm comfortable allowing people their strict definitions as long as they are recognized as such). As a natural consequence of this divergence, Bayesian modeling allows you to explicitly punt the problem of generative modeling for the tricky bits, and it puts a nice big warning label on it saying "If you botch your prior, you're going to have a bad time." The biased coin is a perfect example. Knowing the bias, the generative process for a sequence of flips is trivial. But how are you going to create a generative process that seriously engages how and where the biases emerge? It is far easier to just say "empirically, biases seem to be distributed like so." At this point the Frequentist says "Well, that assumption is not derived from a formal model and is only loosely falsifiable, but if I accept it as a substitute for a generative model then you and I will reach the same conclusions about posterior probabilities." I know of no frequentist who would disagree that getting good generative models for complex phenomena is often extraordinarily difficult. Where they disagree with Bayesians is whether doing something other than that should, strictly, be called "probability" or, whether it is more appropriate to call it something like "semi-empirical subjective-belief modeling".