4 ms·

It seems that his real target is significance testing and he is trying to cast it in a bad light by associating it with eugenics. I started skimming after about

by wrp 4y ago

It seems that his real target is significance testing and he is trying to cast it in a bad light by associating it with eugenics. I started skimming after about a third, so I may have missed a critical point, but I don't see a valid argument there. I'm not a fan of eugenics or significant testing, but they are orthogonal issues.

JHonaker 4y ago
I read this article when it originally came out, and I think it's mostly a historical discussion. But, I can summarize the "leap of logic" that significance testing makes for you. If T(X) is a statistic designed to measure a quantity A, then significance testing are statements that say P(T(X) is close to A | A) = P In other words, the statistic T is close to the true value of A (which is fixed) with probability P. So far, I think this is pretty straightforward, but the tricky part comes when you apply the idea that "closeness" is symmetric. In the physical world, if A is close to B then B is close to A, but that isn't necessarily a valid thing to infer based on a significance test. The thing ostensibly implied by a hypothesis test is that the these two are reasonable to exchange: P(T(X) is close to A | A) = P (A is close to T(X) | T(X)) The interpretation of the right side is what you really want. The probability that the thing I want to know about is close to my statistic given my statistic. Unfortunately, this is not true in general. It's this misunderstanding that's at the heart of "logical fallacy" of frequentist hypothesis testing. No matter how careful we statisticians are about p-values, empirical evidence has shown that this leap is what people make time and time again (including us). This is to say nothing of the absurdity of interpreting probability as a long run frequency for events that happen once or relying on results that formulate things that way when you're doing iterative analysis and development (as all actual inferential processes are).
- wrp 4y ago
  I think we agree on NHST. I expend a lot of effort to remind students (and colleagues) that the popular interpretation of significance tests gets the direction of implication backwards.
  - JHonaker 4y ago
    Yea, it's difficult, especially because the right way to interpret them is both confusing and unhelpful. Why do I care about the properties of my test if it doesn't say anything about the what I should know about the thing I'm measuring in this specific case. Covering the true parameter 95% of the time is useless if I can't make that leap to saying this one contains it 95% of the time. It's so confusing, and I say this as someone who has devoted all of their adult life to statistics.
soledades 4y ago
Not really sure how you would get the sense that his “real target” is significance testing. Seemed much more just an interesting history of how authority, power, and bad science can intertwine, to disastrous effect.
- boxed 4y ago
  > Statistical thinking and eugenicist thinking are, in fact, deeply intertwined, and many of the theoretical problems with methods like significance testing—first developed to identify racial differences—are remnants of their original purpose, to support eugenics. This is in the fourth paragraph. This is clearly a hit piece against not just science but the very idea that truth can exist.
  - giraffe_lady 4y ago
    Is the assertion not true? It does seem to be a truth that the origin of statistics was closely tied to eugenicist goals. "Science" isn't a helpless creature that can be hit or needs to be defended. It's a collection of knowledge and techniques. No technique is invalidated solely by knowledge of its origins, but an awareness of the goals and constraints under which a technique is developed can give us insight into its uses and limitations now. The idea that we shouldn't pursue those paths of knowledge, out of a responsibility to preserve the myth of science's purity from political considerations, is a radical ideology. Attempting to hide these ties, or refusing to address them, does not serve the ideals of truth that you're implying compel you.
    - dekhn 4y ago
      it is truth that the origin of statistics was closely tied to eugenicist goals, but it can be completely decoupled (IE, in an alternate universe, statistics was invented and refined entirely on crop yield predictions). The general mainstream belief is that there is nothing truly technically "wrong" with the approach that was developed then, nor does the mainstream believe that fisher and galton were the last word in statistics.
    - boxed 4y ago
jmopp 4y ago
In my opinion they're both symptoms of the same thing: a tendency to categorise in very black-and-white terms a world which is far more complex.
- atoav 4y ago
  Statistics can be great at distracting from the truth, especially for those with an agenda or an stiff image of the world as they see it. You figure out that all people of $ethnicity perform worse in $test? Very likely you just measured their socioeconomic background instead of the correlation of their ethnicity to that test. There is soo many ways in which such a statistical analysis can be methodically wrong, it is not even funny anymore. To find the truth here you need to be someone who cares about the truth more than a out finding proof for what they already believe, and those people are sadly quite rare.

by wrp 4y ago

JHonaker 4y ago

I read this article when it originally came out, and I think it's mostly a historical discussion. But, I can summarize the "leap of logic" that significance testing makes for you. If T(X) is a statistic designed to measure a quantity A, then significance testing are statements that say P(T(X) is close to A | A) = P In other words, the statistic T is close to the true value of A (which is fixed) with probability P. So far, I think this is pretty straightforward, but the tricky part comes when you apply the idea that "closeness" is symmetric. In the physical world, if A is close to B then B is close to A, but that isn't necessarily a valid thing to infer based on a significance test. The thing ostensibly implied by a hypothesis test is that the these two are reasonable to exchange: P(T(X) is close to A | A) = P (A is close to T(X) | T(X)) The interpretation of the right side is what you really want. The probability that the thing I want to know about is close to my statistic given my statistic. Unfortunately, this is not true in general. It's this misunderstanding that's at the heart of "logical fallacy" of frequentist hypothesis testing. No matter how careful we statisticians are about p-values, empirical evidence has shown that this leap is what people make time and time again (including us). This is to say nothing of the absurdity of interpreting probability as a long run frequency for events that happen once or relying on results that formulate things that way when you're doing iterative analysis and development (as all actual inferential processes are).

wrp 4y ago
I think we agree on NHST. I expend a lot of effort to remind students (and colleagues) that the popular interpretation of significance tests gets the direction of implication backwards.
- JHonaker 4y ago
  Yea, it's difficult, especially because the right way to interpret them is both confusing and unhelpful. Why do I care about the properties of my test if it doesn't say anything about the what I should know about the thing I'm measuring in this specific case. Covering the true parameter 95% of the time is useless if I can't make that leap to saying this one contains it 95% of the time. It's so confusing, and I say this as someone who has devoted all of their adult life to statistics.

soledades 4y ago

Not really sure how you would get the sense that his “real target” is significance testing. Seemed much more just an interesting history of how authority, power, and bad science can intertwine, to disastrous effect.

boxed 4y ago
> Statistical thinking and eugenicist thinking are, in fact, deeply intertwined, and many of the theoretical problems with methods like significance testing—first developed to identify racial differences—are remnants of their original purpose, to support eugenics. This is in the fourth paragraph. This is clearly a hit piece against not just science but the very idea that truth can exist.
- giraffe_lady 4y ago
  Is the assertion not true? It does seem to be a truth that the origin of statistics was closely tied to eugenicist goals. "Science" isn't a helpless creature that can be hit or needs to be defended. It's a collection of knowledge and techniques. No technique is invalidated solely by knowledge of its origins, but an awareness of the goals and constraints under which a technique is developed can give us insight into its uses and limitations now. The idea that we shouldn't pursue those paths of knowledge, out of a responsibility to preserve the myth of science's purity from political considerations, is a radical ideology. Attempting to hide these ties, or refusing to address them, does not serve the ideals of truth that you're implying compel you.
  - dekhn 4y ago
    it is truth that the origin of statistics was closely tied to eugenicist goals, but it can be completely decoupled (IE, in an alternate universe, statistics was invented and refined entirely on crop yield predictions). The general mainstream belief is that there is nothing truly technically "wrong" with the approach that was developed then, nor does the mainstream believe that fisher and galton were the last word in statistics.
  - boxed 4y ago

jmopp 4y ago

In my opinion they're both symptoms of the same thing: a tendency to categorise in very black-and-white terms a world which is far more complex.

atoav 4y ago
Statistics can be great at distracting from the truth, especially for those with an agenda or an stiff image of the world as they see it. You figure out that all people of $ethnicity perform worse in $test? Very likely you just measured their socioeconomic background instead of the correlation of their ethnicity to that test. There is soo many ways in which such a statistical analysis can be methodically wrong, it is not even funny anymore. To find the truth here you need to be someone who cares about the truth more than a out finding proof for what they already believe, and those people are sadly quite rare.