4 ms·

A famous Bayesian arguing for frequentist statistics? Gelman tries to steal the concept "Frequentism" from simple minded frequentist statisticians. His argume

A famous Bayesian arguing for frequentist statistics? Gelman tries to steal the concept "Frequentism" from simple minded frequentist statisticians. His argument seems to be: Simple minded frequentists statisticians perform a statistical procedure once - they do not think about performing the procedure many times. They fall into this trap (from Gelman's paper): "3. Researcher degrees of freedom without fishing: computing a single test based on the data, but in an environment where a different test would have been performed given different data"

rwilson4 4y ago
Gelman is one of the few self-proclaimed Bayesians who doesn't seem to outright hate frequentist approaches. They're complementary approaches. Bayesian methods are great for combining different sources of information. Frequentist methods are great for validating that a method is working well. (For example, Gelman often recommends running simulations to see if models give sensible predictions, but that is itself a pretty frequentist thing to do.) Frequentism is mostly about how to evaluate a methodology. It's pretty agnostic about what that methodology is. Bayesian methods are about combining different sources of information. In a situation where you only have one source of information, Bayesian and Frequentist methods usually give the same answer. People say you might as well always use Bayesian methods then. But no matter what, you should always try to validate or poke holes in your model, and Frequentist techniques are great for that. So it's best to be familiar with both!
- hackandthink 4y ago
  yes https://stats.stackexchange.com/questions/115157/what-are-posterior-predictive-checks-and-what-makes-them-useful https://stats.stackexchange.com/questions/115157/what-are-po...
- kgwgk 4y ago
  > running simulations to see if models give sensible predictions, but that is itself a pretty frequentist thing to do Is looking at probability distributions “a pretty frequentist thing to do”? Even when those models and simulations include _prior_ probability distributions? Sure, one can (re)define frequentist to include Bayesian models - as Gelman seems to want to do in that post. I just don’t see how this helps to clarify anything.
- whatshisface 4y ago
  >In a situation where you only have one source of information, Bayesian and Frequentist methods usually give the same answer. Bayesian and frequentist methods always give the same answer because they represent two different ways of translating the same mathematical ideas into English.
  - kgwgk 4y ago
    What do you call "frequentist methods"? Imagine the following question: "what's the male/female ratio in gorillas?" A frequentist method may provide the answer "[1.1 1.3] is a 95% confidence interval" based on taking a sample of zoos and asking them about the sex of the last gorilla born there. A Bayesian method will provide a different answer - maybe one difficult to reconcile. Because it's not "translating the same mathematical ideas into English". Not only the translation is different - the "mathematical ideas" considered are different as well.
    - whatshisface 4y ago
      I don't think that's a complete case, you can't just say that the bayesian answer would be difficult to reconcile without offering it.
      - kgwgk 4y ago
        A Bayesian may put a strong prior on around the 1:1 sex ratio at birth - because in addition to that data regarding a sample of births they incorporate in the calculation knowledge about the plausible ratio coming from previous observations or biological facts about giraffes or related animals - and get a 95% credible interval (which is conceptually completely different from a 95% confidence interval) like [0.99 1.01] or whatever. You can't just say that Bayesian and frequentist methods _always_ give the same answer without offering even a _single_ example. What is commonly understood as 'Bayesian methods' will give answers in the form of a probability distribution. What is commonly understood as 'frequentist methods' will never do that. How can they always give the same answer then?
AstralStorm 4y ago
The frequentist test for this attempts to see what would happen with a variety of test designs using likelihood ratio and similar statistical tests. Relaxing it you end up with Generalized Method of Moments family. A Bayesian would attempt to compute the Bayes factor using approximate Bayesian computation resulting in more or less the same thing. You end up with various information criteria. Both approaches then converge in using Monte Carlo techniques to evaluate the features of the whole experimental setup using simulated data. All of the above approaches replace the problem of choosing the test/design based on data by the researcher with one by a data driven algorithm with known properties.

rwilson4 4y ago

Gelman is one of the few self-proclaimed Bayesians who doesn't seem to outright hate frequentist approaches. They're complementary approaches. Bayesian methods are great for combining different sources of information. Frequentist methods are great for validating that a method is working well. (For example, Gelman often recommends running simulations to see if models give sensible predictions, but that is itself a pretty frequentist thing to do.) Frequentism is mostly about how to evaluate a methodology. It's pretty agnostic about what that methodology is. Bayesian methods are about combining different sources of information. In a situation where you only have one source of information, Bayesian and Frequentist methods usually give the same answer. People say you might as well always use Bayesian methods then. But no matter what, you should always try to validate or poke holes in your model, and Frequentist techniques are great for that. So it's best to be familiar with both!

hackandthink 4y ago
yes https://stats.stackexchange.com/questions/115157/what-are-posterior-predictive-checks-and-what-makes-them-useful https://stats.stackexchange.com/questions/115157/what-are-po...
kgwgk 4y ago
> running simulations to see if models give sensible predictions, but that is itself a pretty frequentist thing to do Is looking at probability distributions “a pretty frequentist thing to do”? Even when those models and simulations include _prior_ probability distributions? Sure, one can (re)define frequentist to include Bayesian models - as Gelman seems to want to do in that post. I just don’t see how this helps to clarify anything.
whatshisface 4y ago
>In a situation where you only have one source of information, Bayesian and Frequentist methods usually give the same answer. Bayesian and frequentist methods always give the same answer because they represent two different ways of translating the same mathematical ideas into English.
- kgwgk 4y ago
  What do you call "frequentist methods"? Imagine the following question: "what's the male/female ratio in gorillas?" A frequentist method may provide the answer "[1.1 1.3] is a 95% confidence interval" based on taking a sample of zoos and asking them about the sex of the last gorilla born there. A Bayesian method will provide a different answer - maybe one difficult to reconcile. Because it's not "translating the same mathematical ideas into English". Not only the translation is different - the "mathematical ideas" considered are different as well.
  - whatshisface 4y ago
    I don't think that's a complete case, you can't just say that the bayesian answer would be difficult to reconcile without offering it.
    - kgwgk 4y ago
      A Bayesian may put a strong prior on around the 1:1 sex ratio at birth - because in addition to that data regarding a sample of births they incorporate in the calculation knowledge about the plausible ratio coming from previous observations or biological facts about giraffes or related animals - and get a 95% credible interval (which is conceptually completely different from a 95% confidence interval) like [0.99 1.01] or whatever. You can't just say that Bayesian and frequentist methods _always_ give the same answer without offering even a _single_ example. What is commonly understood as 'Bayesian methods' will give answers in the form of a probability distribution. What is commonly understood as 'frequentist methods' will never do that. How can they always give the same answer then?

AstralStorm 4y ago

The frequentist test for this attempts to see what would happen with a variety of test designs using likelihood ratio and similar statistical tests. Relaxing it you end up with Generalized Method of Moments family. A Bayesian would attempt to compute the Bayes factor using approximate Bayesian computation resulting in more or less the same thing. You end up with various information criteria. Both approaches then converge in using Monte Carlo techniques to evaluate the features of the whole experimental setup using simulated data. All of the above approaches replace the problem of choosing the test/design based on data by the researcher with one by a data driven algorithm with known properties.