3 ms·

But if you didn't do that, you would be assuming any crazy theory you think of is true, and require overwhelming evidence to refute it. > I'd suggest starting

by beojan 8y ago

But if you didn't do that, you would be assuming any crazy theory you think of is true, and require overwhelming evidence to refute it. > I'd suggest starting with Meehl 1967 Your very quote states that this whole idea works for physics, and is describing that it doesn't work in psychology and social sciences. I'm confused by this: To take the IQ example, if you take the set of all people in a country, for instance, and randomly select two sets of 5 people, you will find that their average IQ differs by some amount. If you instead select two sets of 50000 people, the average IQs should be a lot closer (to each other, and to the true average). This is by definition true, since IQ is by definition normally distributed. If that's not what you see, you have a methodological problem.

nonbel 8y ago
By "physics" he clearly is referring to testing the quantitative predictions of models. The first sentence of the abstract: >"Because physical theories typically predict numerical values, an improvement in ex-perimental precision reduces the tolerance range and hence increases corroborability." I don't know the historical details but it would appear that since 1967 the NHST virus has proceeded to begin its infestation of physics. Now physicists are acting more and more like psychologists (focusing on testing a null model rather than the predictions of their models; which is the first phase of the illness) and will reap the same consequences of stagnation, etc. Lowering the threshold to 3e-7 instead of 5e-2 just makes this slower and more expensive. >"To take the IQ example, if you take the set of all people in a country, for instance, and randomly select two sets of 5 people, you will find that their average IQ differs by some amount. If you instead select two sets of 50000 people, the average IQs should be a lot closer (to each other, and to the true average). This is by definition true, since IQ is by definition normally distributed. If that's not what you see, you have a methodological problem." I'm not sure why you are talking about standard errors of random samples from a population... He was just giving stats 101 background for philosophers so he could get to this point: "While no competent psychologist is unaware of this obvious distinction between a substantive psychological theory T and a statistical hypothesis H implied by it, in practice there is a tendency to conflate the substan- tive theory with the statistical hypothesis, thereby illicitly conferring upon T some-what the same degree of support given H by a successful refutation of the null hypothesis. Hence the investigator, upon finding an observed difference which has an extremely small probability of occurring on the null hypothesis, gleefully records the tiny probability number “p < .001,” and there is a tendency to feel that the extreme smallness of this probability of a Type I error is somehow transferable to a small probability of “making a theoretical mistake.”
- beojan 8y ago
  I mentioned the IQ stuff because I found it to be a strange example. By definition of IQ, it shouldn't exhibit any issues so long as the random sampling is done correctly. In addition, there's a difference between saying "p is low so it is unlikely that the null is true" which is absolutely correct, and "1 - p is the probability the alternative hypothesis is correct". Certainly in physics people don't believe the second, I don't know about social sciences.

by beojan 8y ago

nonbel 8y ago

By "physics" he clearly is referring to testing the quantitative predictions of models. The first sentence of the abstract: >"Because physical theories typically predict numerical values, an improvement in ex-perimental precision reduces the tolerance range and hence increases corroborability." I don't know the historical details but it would appear that since 1967 the NHST virus has proceeded to begin its infestation of physics. Now physicists are acting more and more like psychologists (focusing on testing a null model rather than the predictions of their models; which is the first phase of the illness) and will reap the same consequences of stagnation, etc. Lowering the threshold to 3e-7 instead of 5e-2 just makes this slower and more expensive. >"To take the IQ example, if you take the set of all people in a country, for instance, and randomly select two sets of 5 people, you will find that their average IQ differs by some amount. If you instead select two sets of 50000 people, the average IQs should be a lot closer (to each other, and to the true average). This is by definition true, since IQ is by definition normally distributed. If that's not what you see, you have a methodological problem." I'm not sure why you are talking about standard errors of random samples from a population... He was just giving stats 101 background for philosophers so he could get to this point: "While no competent psychologist is unaware of this obvious distinction between a substantive psychological theory T and a statistical hypothesis H implied by it, in practice there is a tendency to conflate the substan- tive theory with the statistical hypothesis, thereby illicitly conferring upon T some-what the same degree of support given H by a successful refutation of the null hypothesis. Hence the investigator, upon finding an observed difference which has an extremely small probability of occurring on the null hypothesis, gleefully records the tiny probability number “p < .001,” and there is a tendency to feel that the extreme smallness of this probability of a Type I error is somehow transferable to a small probability of “making a theoretical mistake.”

beojan 8y ago
I mentioned the IQ stuff because I found it to be a strange example. By definition of IQ, it shouldn't exhibit any issues so long as the random sampling is done correctly. In addition, there's a difference between saying "p is low so it is unlikely that the null is true" which is absolutely correct, and "1 - p is the probability the alternative hypothesis is correct". Certainly in physics people don't believe the second, I don't know about social sciences.