3 ms·

I don't agree on this one. The problem with frequentism and the null ritual is that it makes statistics easier (just do this one test, read this one number) an

I don't agree on this one. The problem with frequentism and the null ritual is that it makes statistics easier (just do this one test, read this one number) and renders some kinds of mistakes somewhat harder (publishing a single false positive). At the same time, it makes some bad mistakes much easier, most notably ignoring power, false positives, the garden of forking paths, type S errors and publication bias. The inherent problem is that the null model is not what people assume it is, and the method (or at least the established canon of approaches) don't make you think about it. If you use bayesian methods, you're pretty much forced to spend more time considering the effect size and credibility of your results, and you're basically required to report them. This means that even non-competent bayesians probably have a better contribution to cumulative science.

civilized 4y ago
It sounds like you read Andrew Gelman's blog. You might be interested to review this post: https://statmodeling.stat.columbia.edu/2012/11/10/16808/ https://statmodeling.stat.columbia.edu/2012/11/10/16808/ Frequentist statistics does not force you to accept any hypothesis test with a result p < 0.05 as definitive proof of something. It does not forbid considering prior probability of a result. It just doesn't formalize the consideration of prior probabilities because it is hard to distill this consideration into a formal recipe. Everyone agrees that Bayes' Rule is valid and important, the question is when and how best to use it. > If you use bayesian methods, you're pretty much forced to spend more time considering the effect size and credibility of your results, and you're basically required to report them. You're not forced to do those things well. Any scientific method can be cargo culted.
- uniqueuid 4y ago
  Yes I think Gelman is doing great work, as is e.g. McElreath [1]. The problem that I care about is not whether frequentist statistics can be taught and used well. They can, and I try in my teaching to do so. The problem is that empirically, frequentist statistics is a fig leaf for a ton of extremely problematic work. And pushing bayesian thinking is currently our best chance to fix this, because it's easier to do a shift in the mental framework than to fix the perception of an existing framework. [1] https://xcelab.net/rm/statistical-rethinking/ https://xcelab.net/rm/statistical-rethinking/
  - civilized 4y ago
    I agree that more Bayesian thinking is needed. But I suspect that pushing the technicalities of Bayesian analysis (MCMC etc) would perversely lead to even more cargo-culting, as people would struggle with the technicalities and look desperately for quick fixes. Bayesian methodology benefits from having relatively much more statistically sophisticated practitioners, which leads to an optimism bias when we imagine how it would scale up.
    - uniqueuid 4y ago
      Perhaps you don't see it this way, but I take our eventual convergence as a sign that we don't actually disagree. It's rather that the problem we're starting from is different. Thanks for taking your time for this debate!
      - civilized 4y ago
        Likewise! This is a tricky problem and requires a lot of patient thought.
cdavid 4y ago
I guess it depends on the context in which it is applied. It is anecdotal, but in the industries where I worked, people trying to introduce Bayesian statistics did not have a higher chance of their results being interpreted properly. If you do A/B testing and don't do pre-registration or power analysis, what are the chances that you can/will be able to explain the nuances of probabilistic reasoning ? I agree w/ the parent poster than the fundamental issue is probability: if you are talking w/ people w/o background in stats, you will have a really hard time to go beyond a true/false statement. Moreover, one of the most effective (in $ terms) application of statistics in recent times is A/B testing. While you can do "Bayesian A/B testing", the basic methodology is fundamentally frequentist. Mistakes there can be hedged through better tooling / UX (to avoid peeking, etc.), as effectively as using Bayesian statistics.
marcosdumay 4y ago
Hum... You are basically equating frequentism with the null ritual. The article itself is basically equating frequentism with the null ritual. Ok, that explains why people bother spend so much energy badmouthing frequentism, but it's a very bad framing anyway, bothering a lie. Frequentism is not the null ritual. In fact, it's almost completely compatible with bayesianism, the one large difference being the freedom to set priors before doing your analysis. If the article was titled "It's time to stop teaching the null ritual to scientists", nobody would even disagree.
- zozbot234 4y ago
  > Frequentism is not the null ritual. In fact, it's almost completely compatible with bayesianism, the one large difference being the freedom to set priors before doing your analysis. This may be true of "frequentism" in a strict sense, but many statistical methods in common use that are often described as such (including, arguably, NHST) are not consistent with reasonable versions of the likelihood principle https://en.wikipedia.org/wiki/Likelihood_principle https://en.wikipedia.org/wiki/Likelihood_principle . In a sense, one might feasibly argue that these methods are not even properly frequentist.

civilized 4y ago

It sounds like you read Andrew Gelman's blog. You might be interested to review this post: https://statmodeling.stat.columbia.edu/2012/11/10/16808/ https://statmodeling.stat.columbia.edu/2012/11/10/16808/ Frequentist statistics does not force you to accept any hypothesis test with a result p < 0.05 as definitive proof of something. It does not forbid considering prior probability of a result. It just doesn't formalize the consideration of prior probabilities because it is hard to distill this consideration into a formal recipe. Everyone agrees that Bayes' Rule is valid and important, the question is when and how best to use it. > If you use bayesian methods, you're pretty much forced to spend more time considering the effect size and credibility of your results, and you're basically required to report them. You're not forced to do those things well. Any scientific method can be cargo culted.

uniqueuid 4y ago
Yes I think Gelman is doing great work, as is e.g. McElreath [1]. The problem that I care about is not whether frequentist statistics can be taught and used well. They can, and I try in my teaching to do so. The problem is that empirically, frequentist statistics is a fig leaf for a ton of extremely problematic work. And pushing bayesian thinking is currently our best chance to fix this, because it's easier to do a shift in the mental framework than to fix the perception of an existing framework. [1] https://xcelab.net/rm/statistical-rethinking/ https://xcelab.net/rm/statistical-rethinking/
- civilized 4y ago
  I agree that more Bayesian thinking is needed. But I suspect that pushing the technicalities of Bayesian analysis (MCMC etc) would perversely lead to even more cargo-culting, as people would struggle with the technicalities and look desperately for quick fixes. Bayesian methodology benefits from having relatively much more statistically sophisticated practitioners, which leads to an optimism bias when we imagine how it would scale up.
  - uniqueuid 4y ago
    Perhaps you don't see it this way, but I take our eventual convergence as a sign that we don't actually disagree. It's rather that the problem we're starting from is different. Thanks for taking your time for this debate!
    - civilized 4y ago
      Likewise! This is a tricky problem and requires a lot of patient thought.

cdavid 4y ago

I guess it depends on the context in which it is applied. It is anecdotal, but in the industries where I worked, people trying to introduce Bayesian statistics did not have a higher chance of their results being interpreted properly. If you do A/B testing and don't do pre-registration or power analysis, what are the chances that you can/will be able to explain the nuances of probabilistic reasoning ? I agree w/ the parent poster than the fundamental issue is probability: if you are talking w/ people w/o background in stats, you will have a really hard time to go beyond a true/false statement. Moreover, one of the most effective (in $ terms) application of statistics in recent times is A/B testing. While you can do "Bayesian A/B testing", the basic methodology is fundamentally frequentist. Mistakes there can be hedged through better tooling / UX (to avoid peeking, etc.), as effectively as using Bayesian statistics.

marcosdumay 4y ago

Hum... You are basically equating frequentism with the null ritual. The article itself is basically equating frequentism with the null ritual. Ok, that explains why people bother spend so much energy badmouthing frequentism, but it's a very bad framing anyway, bothering a lie. Frequentism is not the null ritual. In fact, it's almost completely compatible with bayesianism, the one large difference being the freedom to set priors before doing your analysis. If the article was titled "It's time to stop teaching the null ritual to scientists", nobody would even disagree.

zozbot234 4y ago
> Frequentism is not the null ritual. In fact, it's almost completely compatible with bayesianism, the one large difference being the freedom to set priors before doing your analysis. This may be true of "frequentism" in a strict sense, but many statistical methods in common use that are often described as such (including, arguably, NHST) are not consistent with reasonable versions of the likelihood principle https://en.wikipedia.org/wiki/Likelihood_principle https://en.wikipedia.org/wiki/Likelihood_principle . In a sense, one might feasibly argue that these methods are not even properly frequentist.