4 ms·

I could never quite understand the divide between Bayesian statistics and frequentist statistics. Both seem to be ultimately about counting the frequency by whi

I could never quite understand the divide between Bayesian statistics and frequentist statistics. Both seem to be ultimately about counting the frequency by which something occurs and normalizing this frequency with respect to the number of all possible outcomes. Bayesian statistics essentially is concerned with the application of the Bayesian updating technique by which one can iteratively improve a distribution over the values a parameter toward the true distribution using Bayes' rule. One can prove that given sufficiently many updates, the initial distribution does not matter as long it is non-zero for all possible values. What I am not quite seeing is how this has philosophical implications which divide the field into Frequentists and Bayesians. It rather seems that Bayesian statistics is just a frequentist method that is helpful for dealing with noisy measurements as one can always recover from a bad distributions using more updates.

twanvl 10y ago
Bayesian and frequentist approaches ultimately have a different notion of probability. In the frequentist approach, a probability of 10% means that if you repeat an experiment many times, roughly 1 out of 10 times you will observe an event. In Baysian statistics, a probability of 10% means that you are that certain about the event happening. So you would be willing to bet at 10 to 1 odds on the event happening. There doesn't have to be any repetition of the experiment for the probability to make sense. And as you can hopefully see, there is always a prior, that is, your belief about the event before doing any experiments.
- hacker42 10y ago
  But it seems to me one can still define this in terms of frequencies using a more general definition of what is meant by repeating an experiment. For example, when you use a probability P as a degree of belief or certainty about whether a patient X with symptoms W has a particular disease Y, one would define the universe as containing all possible realities in which X has the same symptoms W but with different underlying causal factors that lead to the same symptoms. An observation is a uniform sample from this universe and the belief is that a fraction of P of these realities has the cause Y.
- eximius 10y ago
  Another way to look at it is this way: P(H|D) = P(D|H) P(H) / P(D) Bayesians are interested in the probablity of various hypotheses h in H given some data D. Frequentists calculate the probability of some data given a hypothesis (p-value is not strictly a probability but it can be one - it is ALWAYS a measure of extremity of data coming from the assumed hypothesis, which can be considered a relative probability). Most interesting to me is that the Bayesian formula includes P(D|H) which is basically what the frequentists are calculating. In this sense, the question Bayesians answer is far closer to what we want to ask and far more powerful. In practice, the frequentist approach is often more than enough, though. The tradeoff is computability and simplicity.
  - hacker42 10y ago
    Interesting, I've never thought about the likelihood as a confidence, but it makes sense. But sometimes the confidence also seems to reflect the opposite of extremity (e.g. for the null hypothesis).
kgwgk 10y ago
Bayesian probability cannot always be interpreted as a frequency. For example, one could assign a bayesian probability to the extra-terrestrial origin of life. It wouldn't make much sense to think of it as a frequentist probability: one can easily imagine playing the future several times, but it's not so easy when dealing with the past. And statistics is not just probability. Frequentist inference is based on procedures that "behave well" in the long term, but may or may not make sense for the particular outcome at hand. For example, a 95% confidence interval calculated using a procedure that guarantees that the interval contains the true value 95% of the time may yield an interval that cannot contain the true value (for example the interval covers only negative values and the true value is known to be positive). See http://learnbayes.org/papers/confidenceIntervalsFallacy/ http://learnbayes.org/papers/confidenceIntervalsFallacy/ for a discussion of confidence intervals. Another issue is related to how the "possible outcomes" are defined. For example, a frequentist analysis of the fairness of a coin after getting four heads and then a tail will be different depending on whether we decided to throw the coin until getting a tail or we had fixed beforehand the number of trials. Look for "stopping rules" or "optional stopping."
- hacker42 10y ago
  What would you respond to my other comment downthread? https://news.ycombinator.com/item?id=11985863 https://news.ycombinator.com/item?id=11985863
  - kgwgk 10y ago
    If you are able to think of the different realities that would lead to us having this discussion today, some of them with life being originated on planet Earth and some of them with life coming from elsewhere, and you're able to reason about the relative frequency of these two kinds of realities, you definitely have more imagination than me. And I don't know what do you gain with that. Does your frequentist interpretation have a physical meaning? I could assign a Bayesian probability to panspermia, and you could assign a different probability. Is there, according to your equivalent definition in terms of frequencies, a "correct" probability?
GregBuchholz 10y ago
Have you ever taken a look at: "Probability Theory: The Logic of Science" http://www.med.mcgill.ca/epidemiology/hanley/bios601/GaussianModel/JaynesProbabilityTheory.pdf http://www.med.mcgill.ca/epidemiology/hanley/bios601/Gaussia...
Eliezer 10y ago
Sorry, it took a while, but I wrote two Arbital nodes for you: - https://arbital.com/p/subjective_probability/ https://arbital.com/p/subjective_probability/ - https://arbital.com/p/likelihood_vs_pvalue/ https://arbital.com/p/likelihood_vs_pvalue/
- hacker42 10y ago
  This is great! Thanks.

twanvl 10y ago

Bayesian and frequentist approaches ultimately have a different notion of probability. In the frequentist approach, a probability of 10% means that if you repeat an experiment many times, roughly 1 out of 10 times you will observe an event. In Baysian statistics, a probability of 10% means that you are that certain about the event happening. So you would be willing to bet at 10 to 1 odds on the event happening. There doesn't have to be any repetition of the experiment for the probability to make sense. And as you can hopefully see, there is always a prior, that is, your belief about the event before doing any experiments.

hacker42 10y ago
But it seems to me one can still define this in terms of frequencies using a more general definition of what is meant by repeating an experiment. For example, when you use a probability P as a degree of belief or certainty about whether a patient X with symptoms W has a particular disease Y, one would define the universe as containing all possible realities in which X has the same symptoms W but with different underlying causal factors that lead to the same symptoms. An observation is a uniform sample from this universe and the belief is that a fraction of P of these realities has the cause Y.
eximius 10y ago
Another way to look at it is this way: P(H|D) = P(D|H) P(H) / P(D) Bayesians are interested in the probablity of various hypotheses h in H given some data D. Frequentists calculate the probability of some data given a hypothesis (p-value is not strictly a probability but it can be one - it is ALWAYS a measure of extremity of data coming from the assumed hypothesis, which can be considered a relative probability). Most interesting to me is that the Bayesian formula includes P(D|H) which is basically what the frequentists are calculating. In this sense, the question Bayesians answer is far closer to what we want to ask and far more powerful. In practice, the frequentist approach is often more than enough, though. The tradeoff is computability and simplicity.
- hacker42 10y ago
  Interesting, I've never thought about the likelihood as a confidence, but it makes sense. But sometimes the confidence also seems to reflect the opposite of extremity (e.g. for the null hypothesis).

kgwgk 10y ago

Bayesian probability cannot always be interpreted as a frequency. For example, one could assign a bayesian probability to the extra-terrestrial origin of life. It wouldn't make much sense to think of it as a frequentist probability: one can easily imagine playing the future several times, but it's not so easy when dealing with the past. And statistics is not just probability. Frequentist inference is based on procedures that "behave well" in the long term, but may or may not make sense for the particular outcome at hand. For example, a 95% confidence interval calculated using a procedure that guarantees that the interval contains the true value 95% of the time may yield an interval that cannot contain the true value (for example the interval covers only negative values and the true value is known to be positive). See http://learnbayes.org/papers/confidenceIntervalsFallacy/ http://learnbayes.org/papers/confidenceIntervalsFallacy/ for a discussion of confidence intervals. Another issue is related to how the "possible outcomes" are defined. For example, a frequentist analysis of the fairness of a coin after getting four heads and then a tail will be different depending on whether we decided to throw the coin until getting a tail or we had fixed beforehand the number of trials. Look for "stopping rules" or "optional stopping."

hacker42 10y ago
What would you respond to my other comment downthread? https://news.ycombinator.com/item?id=11985863 https://news.ycombinator.com/item?id=11985863
- kgwgk 10y ago
  If you are able to think of the different realities that would lead to us having this discussion today, some of them with life being originated on planet Earth and some of them with life coming from elsewhere, and you're able to reason about the relative frequency of these two kinds of realities, you definitely have more imagination than me. And I don't know what do you gain with that. Does your frequentist interpretation have a physical meaning? I could assign a Bayesian probability to panspermia, and you could assign a different probability. Is there, according to your equivalent definition in terms of frequencies, a "correct" probability?

GregBuchholz 10y ago

Have you ever taken a look at: "Probability Theory: The Logic of Science" http://www.med.mcgill.ca/epidemiology/hanley/bios601/GaussianModel/JaynesProbabilityTheory.pdf http://www.med.mcgill.ca/epidemiology/hanley/bios601/Gaussia...

Eliezer 10y ago

Sorry, it took a while, but I wrote two Arbital nodes for you: - https://arbital.com/p/subjective_probability/ https://arbital.com/p/subjective_probability/ - https://arbital.com/p/likelihood_vs_pvalue/ https://arbital.com/p/likelihood_vs_pvalue/

hacker42 10y ago
This is great! Thanks.