4 ms·
I read this article when it originally came out, and I think it's mostly a historical discussion. But, I can summarize the "leap of logic" that significance tes
by JHonaker 4y ago
I read this article when it originally came out, and I think it's mostly a historical discussion. But, I can summarize the "leap of logic" that significance testing makes for you.
If T(X) is a statistic designed to measure a quantity A, then significance testing are statements that say
P(T(X) is close to A | A) = P
In other words, the statistic T is close to the true value of A (which is fixed) with probability P. So far, I think this is pretty straightforward, but the tricky part comes when you apply the idea that "closeness" is symmetric. In the physical world, if A is close to B then B is close to A, but that isn't necessarily a valid thing to infer based on a significance test.
The thing ostensibly implied by a hypothesis test is that the these two are reasonable to exchange:
P(T(X) is close to A | A) = P (A is close to T(X) | T(X))
The interpretation of the right side is what you really want. The probability that the thing I want to know about is close to my statistic given my statistic. Unfortunately, this is not true in general. It's this misunderstanding that's at the heart of "logical fallacy" of frequentist hypothesis testing. No matter how careful we statisticians are about p-values, empirical evidence has shown that this leap is what people make time and time again (including us).
This is to say nothing of the absurdity of interpreting probability as a long run frequency for events that happen once or relying on results that formulate things that way when you're doing iterative analysis and development (as all actual inferential processes are).
- wrp 4y agoI think we agree on NHST. I expend a lot of effort to remind students (and colleagues) that the popular interpretation of significance tests gets the direction of implication backwards.
- JHonaker 4y agoYea, it's difficult, especially because the right way to interpret them is both confusing and unhelpful. Why do I care about the properties of my test if it doesn't say anything about the what I should know about the thing I'm measuring in this specific case. Covering the true parameter 95% of the time is useless if I can't make that leap to saying this one contains it 95% of the time. It's so confusing, and I say this as someone who has devoted all of their adult life to statistics.