3 ms·
> Were I to run 100 independently designed experiments that all tested real effects, my choice of p-value does not determine the number that erroneously find no
by lisper 9y ago
> Were I to run 100 independently designed experiments that all tested real effects, my choice of p-value does not determine the number that erroneously find no result.
That's actually not true. The problem is that you cannot define what is a "real effect" without begging the question. Let me illustrate with an example: Suppose I do what appears to be a legitimate experiment to test a well-accepted law of nature. Unbeknownst to me, my instrument is faulty (or has been tampered with) and is in fact returning essentially random results. If I choose p<=0.05 then I will get a positive result 1 time in 20, i.e. I will "erroneously find no result" 19 times out of 20. Except that I'm not really erroneously finding no result. I am correctly finding the results of a different experiment than the one I think I'm running.
And, of course, an instrument that returns random numbers is an extreme example. See:
https://arstechnica.com/science/2018/04/new-measurements-set-two-areas-of-physics-against-each-other/ https://arstechnica.com/science/2018/04/new-measurements-set...
for a current real-world example.
Now, you do have a valid point in that I made a tacit assumption when I claimed that statistics put a lower bound on the error rate. That assumption is that most experiments that are done test hypotheses that ultimately turn out to be false. I believe this assumption is actually true. This is one of the reasons making scientific discoveries is hard. But I didn't actually state this assumption, so I'll cop to that.
> At this point you're arguing against the sources I cited.
That's not true either. The first source says that the error rate is much higher than the p-value, which agrees with me. (I'm actually a little skeptical that the error rate is as high as your source says -- "At least 23% (and typically close to 50%" for p=0.05, but that's certainly not refuting my position!) I don't have time to go through your second citation right now.
- 3JPLW 9y ago> The problem is that you cannot define what is a "real effect" without begging the question That's precisely my point. We don't know the sizes of these populations, so you really can't say anything quantitative about how the p-value affects the proportion of bad results in any given journal. My example was intentionally contrived. You just have to be really careful when you talk about p-values because _so many people_ have this misunderstanding and it's actively harmful to getting the correct interpretation. That's why we keep going back and forth here — I'm not disagreeing that the situation is bad, I just want folks to recognize what the stats say and what they don't.
- lisper 9y ago> We don't know the sizes of these populations I think we can make some pretty reasonable guesses. In any case, I certainly agree with you about the big picture: it's confusing, it's important to get it right, and a lot of people don't understand it. I may even be one of those people. But at the moment I believe that our disagreement is really over something else, namely, whether it's reasonable to have a prior on the null hypothesis.