3 ms·

>"P value is a measure of error (And not a probability but maybe one kind of likelihood), not evidence, how likely it is we are seeing the result of a chance"

• AstralStorm 9y ago
  P is a kind of likelihood with very strong assumption. Likelihood of type 1 error (accidentally getting a result) if error distribution matches student t squared - if you are using the Fischer test to compute it. (ANOVA that is.) And if the hypotheses are independent which is supposed to be the case for null. Specifically a logarithm of a likelihood ratio of two hypotheses given the assumption. Note specific three ifs. Then read what the difference between likelihood and probability is before correcting people. Or what a difference is between likelihood and likelihood ratio is. (Not much in case of null hypothesis testing, unless you didn't check the null to be true in untreated population. Or how true it is, as in didn't get likelihood for placebo. You get the actual likelihood approximation from Wilks' theorem after relating F distribution to chi squared.) Now this fails if: The actual distribution of the observations is not close to T or normal. (There are some more technical reasons, this is a shorthand.) For example multimodal or highly skewed. This is typically glossed over. (Will usually cause a false positive.) Has to be checked for null too. The hypotheses are related. (Say multiple observations over different dosages.) There are different tests that should be used instead. A few more technical reasons.
  • nonbel 9y ago
    This is wrong in multiple ways. The p-value is not an error rate even if the null model is correct, "alpha" (the cutoff, usually 0.05) is the error rate. The p-value also has nothing to do with "how likely it is we are seeing the result of a chance." As you note regarding the t-distribution, the p-value calculation assumes the null model is true (which not necessarily, but usually amounts to "chance did it"). How can a procedure both at once assume something is true and give you the probability it is true? You are also using the term "likelihood" in a strange way (it has a specific meaning when it comes to statistics). Anyway, your comments seem to indicate you hold two standard misinterpretations of p-values: 1 If P = .05, the null hypothesis has only a 5% chance of being true. 2 A nonsignificant difference (eg, P >= .05) means there is no difference between groups. 3 A statistically significant finding is clinically important. 4 Studies with P values on opposite sides of .05 are conflicting. 5 Studies with the same P value provide the same evidence against the null hypothesis. 6 P = .05 means that we have observed data that would occur only 5% of the time under the null hypothesis. 7 P = .05 and P <=.05 mean the same thing. 8 P values are properly written as inequalities (eg, “P <=.02” when P
    =.015) 9 P = .05 means that if you reject the null hypothesis, the probability of a type I error is only 5%. 10 With a P = .05 threshold for significance, the chance of a type I error will be 5%. 11 You should use a one-sided P value when you don’t care about a result in one direction, or a difference in that direction is impossible. 12 A scientific conclusion or treatment policy should be based on whether or not the P value is significant. https://www.ncbi.nlm.nih.gov/pubmed/18582619 https://www.ncbi.nlm.nih.gov/pubmed/18582619
    • abecedarius 9y ago
      What's wrong about #6? Is it just blurring the difference between "less than .05" and "only .05"? #7 isn't clear to me either, though I'm guessing it's a pithy way of saying that two results that are both "significant" can easily have different evidential value.
      • nonbel 9y ago
        There was a problem with pasting the symbols (= vs < vs <=, etc), I think I have it fixed now. Please check again.