4 ms·
I'd add bad math. Fisher significance testing is on shaky grounds[0] both theoretically (significance testing violates the likelihood principle) and practically
by Jach 13y ago
I'd add bad math. Fisher significance testing is on shaky grounds[0] both theoretically (significance testing violates the likelihood principle) and practically (just by nature of having a small sample size, a lot of null hypotheses can be rejected, while with a larger sample size hardly none are rejected). Hypothesis testing is little better. The "Bayesian revolution" has barely begun and has yet to influence a majority of researchers.
[0] http://uncertainty.stat.cmu.edu/ http://uncertainty.stat.cmu.edu/ Chapter 12.
- simonster 13y agoI don't think this is quite right. If the null hypothesis is true and you conduct the same study an infinite number of times, you will reject the null hypothesis 5% of the time at alpha = 0.05 regardless of sample size, unless there is something wrong with your sampling procedure or hypothesis test. This is the point of null hypothesis significance testing. In practice, if you fix p=0.05 and increase your n, the probability that you will find a statistically significant result often increases because your power increases, and in many situations, the probability that the null hypothesis is true is close to zero. (Andrew Gelman uses the example of asking whether there are significant differences between voting patterns of men and women.) On the other hand, effect size estimates become more accurate as the sample size increases. This mitigates the above issue, provided you actually report your effect size. It also means that small sample studies that report statistically significant results are more likely to overestimate their effect size, which is especially problematic if you are applying null hypothesis significance testing when you know the null hypothesis is false.