4 ms·

Most of the objections here and in the article are not inherent problems with frequentist p-values. First, the reported p-value might be wrong. E.g. basing it

Most of the objections here and in the article are not inherent problems with frequentist p-values. First, the reported p-value might be wrong. E.g. basing it on assumptions of normality when the data is non-normal. However modern non-parametric approaches like the bootstrap can avoid this issue. Second, testing multiple hypotheses. If you test 10 hypotheses then you cannot reject the null (that all 10 null hypotheses hold) simply because one single hypothesis is rejected in isolation. But this is well known, and failing to account for it is an issue with the researcher, not with frequentist statistics. I actually think that the main practical difference between Bayesian and Frequentist statistics is whether accounting for the issue of multiple hypotheses is done formally or informally.

hootener 13y ago
The article doesn't bash the p-value as a statistical test specifically, more its use and interpretation by scientists over the years. You're absolutely correct about using non-parametric tests, and more scientists should be using them. The normality assumption is flat out laughable when using real-world data most of the time. You're also correct about multiple hypothesis testing. Accounting for familywise error (e.g., Holms adjustments) can help to keep your p-value reporting honest. That doesn't negate the underlying problem, though. A p-value is simply an indication, nothing more. The p-value never promised to be more than that. The issue isn't in the p-value's construction, the issue lies in its misuse and how easily it can be abused in statistical reporting (see: p-hacking). The p-value as a test statistic is perfectly honest in my opinion. But like many other statistical methods, it comes with its own set of baggage that I feel gets conveniently glossed over more often than it should.

hootener 13y ago

The article doesn't bash the p-value as a statistical test specifically, more its use and interpretation by scientists over the years. You're absolutely correct about using non-parametric tests, and more scientists should be using them. The normality assumption is flat out laughable when using real-world data most of the time. You're also correct about multiple hypothesis testing. Accounting for familywise error (e.g., Holms adjustments) can help to keep your p-value reporting honest. That doesn't negate the underlying problem, though. A p-value is simply an indication, nothing more. The p-value never promised to be more than that. The issue isn't in the p-value's construction, the issue lies in its misuse and how easily it can be abused in statistical reporting (see: p-hacking). The p-value as a test statistic is perfectly honest in my opinion. But like many other statistical methods, it comes with its own set of baggage that I feel gets conveniently glossed over more often than it should.