4 ms·
P is a kind of likelihood with very strong assumption. Likelihood of type 1 error (accidentally getting a result) if error distribution matches student t square
by AstralStorm 9y ago
P is a kind of likelihood with very strong assumption. Likelihood of type 1 error (accidentally getting a result) if error distribution matches student t squared - if you are using the Fischer test to compute it. (ANOVA that is.) And if the hypotheses are independent which is supposed to be the case for null.
Specifically a logarithm of a likelihood ratio of two hypotheses given the assumption.
Note specific three ifs. Then read what the difference between likelihood and probability is before correcting people. Or what a difference is between likelihood and likelihood ratio is.
(Not much in case of null hypothesis testing, unless you didn't check the null to be true in untreated population. Or how true it is, as in didn't get likelihood for placebo. You get the actual likelihood approximation from Wilks' theorem after relating F distribution to chi squared.)
Now this fails if:
The actual distribution of the observations is not close to T or normal. (There are some more technical reasons, this is a shorthand.) For example multimodal or highly skewed. This is typically glossed over. (Will usually cause a false positive.) Has to be checked for null too.
The hypotheses are related. (Say multiple observations over different dosages.) There are different tests that should be used instead.
A few more technical reasons.
- nonbel 9y agoThis is wrong in multiple ways. The p-value is not an error rate even if the null model is correct, "alpha" (the cutoff, usually 0.05) is the error rate. The p-value also has nothing to do with "how likely it is we are seeing the result of a chance." As you note regarding the t-distribution, the p-value calculation assumes the null model is true (which not necessarily, but usually amounts to "chance did it"). How can a procedure both at once assume something is true and give you the probability it is true? You are also using the term "likelihood" in a strange way (it has a specific meaning when it comes to statistics). Anyway, your comments seem to indicate you hold two standard misinterpretations of p-values: 1 If P = .05, the null hypothesis has only a 5% chance of being true. 2 A nonsignificant difference (eg, P >= .05) means there is no difference between groups. 3 A statistically significant finding is clinically important. 4 Studies with P values on opposite sides of .05 are conflicting. 5 Studies with the same P value provide the same evidence against the null hypothesis. 6 P = .05 means that we have observed data that would occur only 5% of the time under the null hypothesis. 7 P = .05 and P <=.05 mean the same thing. 8 P values are properly written as inequalities (eg, “P <=.02” when P =.015) 9 P = .05 means that if you reject the null hypothesis, the probability of a type I error is only 5%. 10 With a P = .05 threshold for significance, the chance of a type I error will be 5%. 11 You should use a one-sided P value when you don’t care about a result in one direction, or a difference in that direction is impossible. 12 A scientific conclusion or treatment policy should be based on whether or not the P value is significant. https://www.ncbi.nlm.nih.gov/pubmed/18582619 https://www.ncbi.nlm.nih.gov/pubmed/18582619
- abecedarius 9y agoWhat's wrong about #6? Is it just blurring the difference between "less than .05" and "only .05"? #7 isn't clear to me either, though I'm guessing it's a pithy way of saying that two results that are both "significant" can easily have different evidential value.
- nonbel 9y agoThere was a problem with pasting the symbols (= vs < vs <=, etc), I think I have it fixed now. Please check again.
- nonbel 9y agoHere are the reasons given by Goodman (2008): >"Misconception #6: P = .05 means that we have observed data that would occur only 5% of the time under the null hypothesis. That this is not the case is seen immediately from the P value’s definition, the probability of the observed data, plus more extreme data, under the null hypothesis. The result with the P value of exactly .05 (or any other value) is the most probable of all the other possible results included in the “tail area” that defines the P value. The probability of any individual result is actually quite small, and Fisher said he threw in the rest of the tail area “as an approximation.” As we will see later in this chapter, the inclusion of these rarer outcomes poses serious logical and quantitative problems for the P value, and using comparative rather than single probabilities to measure evidence eliminates the need to include outcomes other than what was observed. [...] Misconception #7: P = .05 and P <= .05 mean the same thing. This misconception shows how diabolically difficult it is to either explain or understand P values. There is a big difference between these results in terms of weight of evidence, but because the same number (5%) is associated with each, that difference is literally impossible to communicate. It can be calculated and seen clearly only using a Bayesian evidence metric.16"
- abecedarius 9y agoThanks for liberating those quotes from the shackles of Elsevier (and fixing the parent). About the letter 'p': I'd ban the word 'significant' instead if we're going to go there. You could say something like 'null-improbable' if you insisted on using NHST -- such a term would make it harder to forget that it's a fiddly technical concept.