5 ms·

Let me introduce you to Gerd Gigerenzer's paper "The Null Ritual", in which he brilliantly shows that almost nobody understands the rituals of frequentist stati

Let me introduce you to Gerd Gigerenzer's paper "The Null Ritual", in which he brilliantly shows that almost nobody understands the rituals of frequentist statistics [1]. "How many students and teachers noticed that all of the statements were wrong? As Figure 1 shows, none of the students did. Every student endorsed one or more of the illusions about the meaning of a p-value. One might think that these students lack the right genes for statistical thinking and are stubbornly resistant to education. A glance at the performance of their teachers, however, indicates that wishful thinking might not be entirely their fault. Ninety percent of the professors and lecturers also had illusions, a proportion almost as high as among their students. Most surprisingly, 80% of the statistics teachers shared illusions with their students." [1] http://library.mpib-berlin.mpg.de/ft/gg/GG_Null_2004.pdf http://library.mpib-berlin.mpg.de/ft/gg/GG_Null_2004.pdf

civilized 4y ago
I don't disagree that many people struggle to properly interpret frequentist statistics. But adopting a Bayesian perspective isn't a magic talisman. The number of people who are competent Bayesians but don't understand frequentist statistics is zero, and no pedagogical changes will improve that.
- uniqueuid 4y ago
  I don't agree on this one. The problem with frequentism and the null ritual is that it makes statistics easier (just do this one test, read this one number) and renders some kinds of mistakes somewhat harder (publishing a single false positive). At the same time, it makes some bad mistakes much easier, most notably ignoring power, false positives, the garden of forking paths, type S errors and publication bias. The inherent problem is that the null model is not what people assume it is, and the method (or at least the established canon of approaches) don't make you think about it. If you use bayesian methods, you're pretty much forced to spend more time considering the effect size and credibility of your results, and you're basically required to report them. This means that even non-competent bayesians probably have a better contribution to cumulative science.
  - civilized 4y ago
    It sounds like you read Andrew Gelman's blog. You might be interested to review this post: https://statmodeling.stat.columbia.edu/2012/11/10/16808/ https://statmodeling.stat.columbia.edu/2012/11/10/16808/ Frequentist statistics does not force you to accept any hypothesis test with a result p < 0.05 as definitive proof of something. It does not forbid considering prior probability of a result. It just doesn't formalize the consideration of prior probabilities because it is hard to distill this consideration into a formal recipe. Everyone agrees that Bayes' Rule is valid and important, the question is when and how best to use it. > If you use bayesian methods, you're pretty much forced to spend more time considering the effect size and credibility of your results, and you're basically required to report them. You're not forced to do those things well. Any scientific method can be cargo culted.
    - uniqueuid 4y ago
      Yes I think Gelman is doing great work, as is e.g. McElreath [1]. The problem that I care about is not whether frequentist statistics can be taught and used well. They can, and I try in my teaching to do so. The problem is that empirically, frequentist statistics is a fig leaf for a ton of extremely problematic work. And pushing bayesian thinking is currently our best chance to fix this, because it's easier to do a shift in the mental framework than to fix the perception of an existing framework. [1] https://xcelab.net/rm/statistical-rethinking/ https://xcelab.net/rm/statistical-rethinking/
      - civilized 4y ago
        I agree that more Bayesian thinking is needed. But I suspect that pushing the technicalities of Bayesian analysis (MCMC etc) would perversely lead to even more cargo-culting, as people would struggle with the technicalities and look desperately for quick fixes. Bayesian methodology benefits from having relatively much more statistically sophisticated practitioners, which leads to an optimism bias when we imagine how it would scale up.
        uniqueuid 4y ago
- bart_spoon 4y ago
  No one said it’s a magic talisman. But Bayesian probability is far more in line with most peoples intuitive understanding about probability and statistics than frequentist probability. There will always be no instances where individuals err in the way they apply statistical methods. But that doesn’t mean that there is no value in moving the “default” in statistics to a more intuitive methodology from one that is so obtuse that it’s common for relatively advanced practitioners to stumble over it
- AlanYx 4y ago
  The problem is deeper than people just struggling with frequentist statistics. It's difficult IMHO to actually reason properly about frequentist results without also having a grounding in Bayesian statistics. For example, in the Gigerenzer et al. paper cited above, to properly understand why statement #6 is false, a person has to understand that they've confused p(D|H0) with 1 – p(D). That's why it's not surprising IMHO that cookbook approaches to frequentist statistics without a nontrivial Bayesian component, as are often found in social science-focused stats courses, inevitably end up with limited or skewed comprehension.
  - civilized 4y ago
    I vehemently agree with this. It's the OP's "stop learning frequentist statistics" that I vehemently disagree with.
- planede 4y ago
  The difference between Bayesian and frequentist statistics is that they answer different questions. People are often more interested in the questions that Bayesian statistics answer, and so they often misinterpret frequentist results to also answer those questions. It's possible that it can happen the other way around as well, but my impression is that it happens less often.

civilized 4y ago

I don't disagree that many people struggle to properly interpret frequentist statistics. But adopting a Bayesian perspective isn't a magic talisman. The number of people who are competent Bayesians but don't understand frequentist statistics is zero, and no pedagogical changes will improve that.

uniqueuid 4y ago
I don't agree on this one. The problem with frequentism and the null ritual is that it makes statistics easier (just do this one test, read this one number) and renders some kinds of mistakes somewhat harder (publishing a single false positive). At the same time, it makes some bad mistakes much easier, most notably ignoring power, false positives, the garden of forking paths, type S errors and publication bias. The inherent problem is that the null model is not what people assume it is, and the method (or at least the established canon of approaches) don't make you think about it. If you use bayesian methods, you're pretty much forced to spend more time considering the effect size and credibility of your results, and you're basically required to report them. This means that even non-competent bayesians probably have a better contribution to cumulative science.
- civilized 4y ago
  It sounds like you read Andrew Gelman's blog. You might be interested to review this post: https://statmodeling.stat.columbia.edu/2012/11/10/16808/ https://statmodeling.stat.columbia.edu/2012/11/10/16808/ Frequentist statistics does not force you to accept any hypothesis test with a result p < 0.05 as definitive proof of something. It does not forbid considering prior probability of a result. It just doesn't formalize the consideration of prior probabilities because it is hard to distill this consideration into a formal recipe. Everyone agrees that Bayes' Rule is valid and important, the question is when and how best to use it. > If you use bayesian methods, you're pretty much forced to spend more time considering the effect size and credibility of your results, and you're basically required to report them. You're not forced to do those things well. Any scientific method can be cargo culted.
  - uniqueuid 4y ago
    Yes I think Gelman is doing great work, as is e.g. McElreath [1]. The problem that I care about is not whether frequentist statistics can be taught and used well. They can, and I try in my teaching to do so. The problem is that empirically, frequentist statistics is a fig leaf for a ton of extremely problematic work. And pushing bayesian thinking is currently our best chance to fix this, because it's easier to do a shift in the mental framework than to fix the perception of an existing framework. [1] https://xcelab.net/rm/statistical-rethinking/ https://xcelab.net/rm/statistical-rethinking/
    - civilized 4y ago
      I agree that more Bayesian thinking is needed. But I suspect that pushing the technicalities of Bayesian analysis (MCMC etc) would perversely lead to even more cargo-culting, as people would struggle with the technicalities and look desperately for quick fixes. Bayesian methodology benefits from having relatively much more statistically sophisticated practitioners, which leads to an optimism bias when we imagine how it would scale up.
      - uniqueuid 4y ago
bart_spoon 4y ago
No one said it’s a magic talisman. But Bayesian probability is far more in line with most peoples intuitive understanding about probability and statistics than frequentist probability. There will always be no instances where individuals err in the way they apply statistical methods. But that doesn’t mean that there is no value in moving the “default” in statistics to a more intuitive methodology from one that is so obtuse that it’s common for relatively advanced practitioners to stumble over it
AlanYx 4y ago
The problem is deeper than people just struggling with frequentist statistics. It's difficult IMHO to actually reason properly about frequentist results without also having a grounding in Bayesian statistics. For example, in the Gigerenzer et al. paper cited above, to properly understand why statement #6 is false, a person has to understand that they've confused p(D|H0) with 1 – p(D). That's why it's not surprising IMHO that cookbook approaches to frequentist statistics without a nontrivial Bayesian component, as are often found in social science-focused stats courses, inevitably end up with limited or skewed comprehension.
- civilized 4y ago
  I vehemently agree with this. It's the OP's "stop learning frequentist statistics" that I vehemently disagree with.
planede 4y ago
The difference between Bayesian and frequentist statistics is that they answer different questions. People are often more interested in the questions that Bayesian statistics answer, and so they often misinterpret frequentist results to also answer those questions. It's possible that it can happen the other way around as well, but my impression is that it happens less often.