4 ms·
This is wrong in multiple ways. The p-value is not an error rate even if the null model is correct, "alpha" (the cutoff, usually 0.05) is the error rate. The p
by nonbel 9y ago
This is wrong in multiple ways. The p-value is not an error rate even if the null model is correct, "alpha" (the cutoff, usually 0.05) is the error rate.
The p-value also has nothing to do with "how likely it is we are seeing the result of a chance." As you note regarding the t-distribution, the p-value calculation assumes the null model is true (which not necessarily, but usually amounts to "chance did it").
How can a procedure both at once assume something is true and give you the probability it is true? You are also using the term "likelihood" in a strange way (it has a specific meaning when it comes to statistics). Anyway, your comments seem to indicate you hold two standard misinterpretations of p-values:
1 If P = .05, the null hypothesis has only a 5% chance of being true.
2 A nonsignificant difference (eg, P >= .05) means there is no difference between groups.
3 A statistically significant finding is clinically important.
4 Studies with P values on opposite sides of .05 are conflicting.
5 Studies with the same P value provide the same evidence against the null hypothesis.
6 P = .05 means that we have observed data that would occur only 5% of the time under the null hypothesis.
7 P = .05 and P <=.05 mean the same thing.
8 P values are properly written as inequalities (eg, “P <=.02” when P =.015)
9 P = .05 means that if you reject the null hypothesis, the probability of a type I error is only 5%.
10 With a P = .05 threshold for significance, the chance of a type I error will be 5%.
11 You should use a one-sided P value when you don’t care about a result in one direction, or a difference in
that direction is impossible.
12 A scientific conclusion or treatment policy should be based on whether or not the P value is significant.
https://www.ncbi.nlm.nih.gov/pubmed/18582619 https://www.ncbi.nlm.nih.gov/pubmed/18582619
- abecedarius 9y agoWhat's wrong about #6? Is it just blurring the difference between "less than .05" and "only .05"? #7 isn't clear to me either, though I'm guessing it's a pithy way of saying that two results that are both "significant" can easily have different evidential value.
- nonbel 9y agoThere was a problem with pasting the symbols (= vs < vs <=, etc), I think I have it fixed now. Please check again.
- nonbel 9y agoHere are the reasons given by Goodman (2008): >"Misconception #6: P = .05 means that we have observed data that would occur only 5% of the time under the null hypothesis. That this is not the case is seen immediately from the P value’s definition, the probability of the observed data, plus more extreme data, under the null hypothesis. The result with the P value of exactly .05 (or any other value) is the most probable of all the other possible results included in the “tail area” that defines the P value. The probability of any individual result is actually quite small, and Fisher said he threw in the rest of the tail area “as an approximation.” As we will see later in this chapter, the inclusion of these rarer outcomes poses serious logical and quantitative problems for the P value, and using comparative rather than single probabilities to measure evidence eliminates the need to include outcomes other than what was observed. [...] Misconception #7: P = .05 and P <= .05 mean the same thing. This misconception shows how diabolically difficult it is to either explain or understand P values. There is a big difference between these results in terms of weight of evidence, but because the same number (5%) is associated with each, that difference is literally impossible to communicate. It can be calculated and seen clearly only using a Bayesian evidence metric.16"
- abecedarius 9y agoThanks for liberating those quotes from the shackles of Elsevier (and fixing the parent). About the letter 'p': I'd ban the word 'significant' instead if we're going to go there. You could say something like 'null-improbable' if you insisted on using NHST -- such a term would make it harder to forget that it's a fiddly technical concept.
- AstralStorm 9y agoAlpha is the error rate of the binary significance test if at all. Not related to any of the observations. However, since p is a loglikelihood ratio, it is related to the observation and null themselves. To get actual likelihood you need to exponentiate, know the degrees of freedom and know one of the actual likelihoods. (Typically null is easier to come by as the other side is assumed to be null+treatment.) This exact likelihood is of course tiny. Which is why inequality is supposed to be used. The process is not valid if any of the test's assumptions is violated in a big way. Other than this, all the points are true.
- kgwgk 9y agoHow is the p-value a log-likelihood ratio?
- AstralStorm 9y agoA mathematical sleigh of hand depending on big N, chi-squared statistics and Wilks' theorem. This is obviously inaccurate for small N. Otherwise p depends on the exact test employed. (Specifically the distributions employed. For Fischer's exact, hypergeometric, notoriously hard to calculate, but if you can, you can recover likelihoods too.)
- kgwgk 9y agoA p-value can be calculated for any statistic. The p-value is often calculated using the sample average. I wouldn't say that it is an average, though.
- nonbel 9y agoThe standard meaning of likelihood is P(data|Hypothesis)[1] where, for the purposes here, hypothesis will refer to "chance/accident generated the data". Do you agree with this? (I understand you say "kind of likelihood" because we are dealing with the inequality) If so, can you clarify what you mean by "P is ... Likelihood of type 1 error (accidentally getting a result)..." ? [1] https://en.wikipedia.org/wiki/Likelihood_function https://en.wikipedia.org/wiki/Likelihood_function