3 ms·
Great article. I'm not sure that replication itself will solve the problem since Type 1 error rate requires asymptotics. We'd have to run many replications and
by RA_Fisher 11y ago
Great article. I'm not sure that replication itself will solve the problem since Type 1 error rate requires asymptotics. We'd have to run many replications and then show convergence. That'll be broadly cost-prohibitive for all but the most important conclusions. Lower thresholds probably won't do it either. Right now, the only solutions I see are:
a) Baysian methods
b) Fisher's single H hypothesis method
c) Tukey's Exploratory Data Analysis method.
d) All of the above.
- Fomite 11y agoRelying on effect measures and their confidence intervals, rather than relying on p-values as a "Yes/No" threshold should likely be on your list, especially as an interim step that should be easy for those who still want p-values to swallow.
- sgerrish 11y agoI don't see why (e) teaching scientists to be statistically literate so they don't abuse or misunderstand these tests, and/or (f) focusing on reproducible results and shaming researchers with sloppy methodology, wouldn't work. The hypothesis test has known limitations, but it's not clear that we should blame null hypothesis tests for people mis-using them, when researchers untrained in stats are just as likely to mis-use any method you give them.