4 ms·

It's also the tool. There are whole books dedicated to it (I can recommend Bernouilli's Fallacy by Clayton), but the gist is: NHST answers the wrong question (

by tgv 3y ago

It's also the tool. There are whole books dedicated to it (I can recommend Bernouilli's Fallacy by Clayton), but the gist is: NHST answers the wrong question (it should answer: what is the likelihood of the hypothesis given the data, but answers: what is the likelihood of the data under the hypothesis), and is (thus) very sensitive to the probability of the prior. To quote the old example: when you wake up with a headache, it's not usual to assume you've got a brain tumor.

GTP 3y ago
Interesting, what would be the correct tool then? Do we have a better one?
- tgv 3y ago
  Bayesian analysis overcomes this particular problem, but of course doesn't have such an easy and commonly accepted "THIS IS SIGNIFICANT AND BY GOLLY I AM RIGHT!" signal. It's conceptually harder than NHST, doesn't lead to easy conclusions (unless your effect is really strong), and many researchers have never even heard about it. And they are the reviewers. I got into a bit of a kerfuffle about a statistical test once. A critical review insisted I do an ANOVA, even though it was inadequate, and I had used a weaker test. Such is the status quo. Work has been done to make Bayesian analysis as straight-forward as NHST, but if it gets accepted with a threshold (comparable to the current p<0.05), it won't help. Some articles now publish both analyses, which is nice. And there are tools to help you, e.g. https://jasp-stats.org/ https://jasp-stats.org/
  - happy3483t583 3y ago
    > doesn't have such an easy and commonly accepted "THIS IS SIGNIFICANT AND BY GOLLY I AM RIGHT!" signal. Part of the ongoing stats revolution is abandoning the binary significance test, because it's often misaligned with the fundamental business question at hand. If, instead, you have a probability distribution over hypotheses, you can make optimal business decisions based on standard utility theory, like this https://philosophy.hku.hk/think/stat/naode1.php https://philosophy.hku.hk/think/stat/naode1.php. > It's conceptually harder than NHST From a different angle: Experimenters fundamentally want to know p(hypothesis | data): Does adding 20g sugar to 1kg cement prevent 98% of the units from crushing under 1000kg load? Below are two answers. Which would people rather tell their boss? 1) Likely yes, given our experimental data. The totality of evidence shows that there's a 70% probability that failure rates are below the limit, but we're not yet ready to commit changes to our manufacturing. We can run more experiments to nail this down and make a concrete decision (heh). 2) Sorry boss, I have no clue. The p-value was 0.15, so we can't say anything. Unfortunately old-school stats can't easily add new data to an existing experiment, so we have to run new experiments, and then combine everything with a fancy new meta-analysis. Or we could call the first experiment a sneak peek but then we have to do multiplicity corrections. But even after we do all that, I can't really tell you the probability that sugar strengthens cement. NHST doesn't work that way. From the stakeholder's perspective, #1 is conceptually lots easier.
- happy3483t583 3y ago
  Gateway drug: Everyone's favorite old-school t-test, brought into the modern world: https://www.sumsar.net/best_online/ https://www.sumsar.net/best_online/ Here's the story of the tool: https://jkkweb.sitehost.iu.edu/articles/Kruschke2013JEPG.pdf https://jkkweb.sitehost.iu.edu/articles/Kruschke2013JEPG.pdf (Kruschke's books and papers are a terrific resource for learning real-world applied stats)
happy3483t583 3y ago
> is (thus) very sensitive to the probability of the prior Standard practice is to test sensitivity of the posterior against various priors. They're often non-informative, e.g., `effectSize ~ N(0, wide)`, and the results are quite robust. Here's a typical playbook: http://www.stat.columbia.edu/~gelman/research/published/43586_2020_1_Author.pdf http://www.stat.columbia.edu/~gelman/research/published/4358...

by tgv 3y ago

GTP 3y ago

Interesting, what would be the correct tool then? Do we have a better one?

tgv 3y ago
Bayesian analysis overcomes this particular problem, but of course doesn't have such an easy and commonly accepted "THIS IS SIGNIFICANT AND BY GOLLY I AM RIGHT!" signal. It's conceptually harder than NHST, doesn't lead to easy conclusions (unless your effect is really strong), and many researchers have never even heard about it. And they are the reviewers. I got into a bit of a kerfuffle about a statistical test once. A critical review insisted I do an ANOVA, even though it was inadequate, and I had used a weaker test. Such is the status quo. Work has been done to make Bayesian analysis as straight-forward as NHST, but if it gets accepted with a threshold (comparable to the current p<0.05), it won't help. Some articles now publish both analyses, which is nice. And there are tools to help you, e.g. https://jasp-stats.org/ https://jasp-stats.org/
- happy3483t583 3y ago
  > doesn't have such an easy and commonly accepted "THIS IS SIGNIFICANT AND BY GOLLY I AM RIGHT!" signal. Part of the ongoing stats revolution is abandoning the binary significance test, because it's often misaligned with the fundamental business question at hand. If, instead, you have a probability distribution over hypotheses, you can make optimal business decisions based on standard utility theory, like this https://philosophy.hku.hk/think/stat/naode1.php https://philosophy.hku.hk/think/stat/naode1.php. > It's conceptually harder than NHST From a different angle: Experimenters fundamentally want to know p(hypothesis | data): Does adding 20g sugar to 1kg cement prevent 98% of the units from crushing under 1000kg load? Below are two answers. Which would people rather tell their boss? 1) Likely yes, given our experimental data. The totality of evidence shows that there's a 70% probability that failure rates are below the limit, but we're not yet ready to commit changes to our manufacturing. We can run more experiments to nail this down and make a concrete decision (heh). 2) Sorry boss, I have no clue. The p-value was 0.15, so we can't say anything. Unfortunately old-school stats can't easily add new data to an existing experiment, so we have to run new experiments, and then combine everything with a fancy new meta-analysis. Or we could call the first experiment a sneak peek but then we have to do multiplicity corrections. But even after we do all that, I can't really tell you the probability that sugar strengthens cement. NHST doesn't work that way. From the stakeholder's perspective, #1 is conceptually lots easier.
happy3483t583 3y ago
Gateway drug: Everyone's favorite old-school t-test, brought into the modern world: https://www.sumsar.net/best_online/ https://www.sumsar.net/best_online/ Here's the story of the tool: https://jkkweb.sitehost.iu.edu/articles/Kruschke2013JEPG.pdf https://jkkweb.sitehost.iu.edu/articles/Kruschke2013JEPG.pdf (Kruschke's books and papers are a terrific resource for learning real-world applied stats)

happy3483t583 3y ago

> is (thus) very sensitive to the probability of the prior Standard practice is to test sensitivity of the posterior against various priors. They're often non-informative, e.g., `effectSize ~ N(0, wide)`, and the results are quite robust. Here's a typical playbook: http://www.stat.columbia.edu/~gelman/research/published/43586_2020_1_Author.pdf http://www.stat.columbia.edu/~gelman/research/published/4358...