4 ms·

Were I to run 100 independently designed experiments that all tested real effects, my choice of p-value does not determine the number that erroneously find no r

by 3JPLW 9y ago

Were I to run 100 independently designed experiments that all tested real effects, my choice of p-value does not determine the number that erroneously find no result. If the effect is small and I didn't gather enough data, a p-value of 0.05 could result in only a handful of experiments accurately reflecting reality. Let's say 30 make the cut. Were I to run another 100 independently designed experiments that all tested big real effects, then a p-value of 0.05 could result in all 100 accurately reflecting reality. Were I to run 100 independently designed experiments that all tested false effects, then at a p-value level of 0.05, I should see about 5 erroneously significant results. Now I submit all these significant results to a journal and they all get published. Of the 135 results, only 5 are coming to the wrong conclusion. Look at that, an error rate lower than your lower bound. We simply don't know the sizes of these populations. It could be better or worse than my example, and it could be better or worse than 5%. There simply is no lower bound. Of course you're right that there is a problem here but there isn't a lower bound guaranteed by the stats tests. Quite a bit of the problem stems from misinterpretations of p-values. For someone who likes to condescendingly explain concepts, you might want to make sure you have the harder concept down before assuming others don't understand "ratios" and "lower bounds." At this point you're arguing against the sources I cited. You should check them out. :)

lisper 9y ago
> Were I to run 100 independently designed experiments that all tested real effects, my choice of p-value does not determine the number that erroneously find no result. That's actually not true. The problem is that you cannot define what is a "real effect" without begging the question. Let me illustrate with an example: Suppose I do what appears to be a legitimate experiment to test a well-accepted law of nature. Unbeknownst to me, my instrument is faulty (or has been tampered with) and is in fact returning essentially random results. If I choose p<=0.05 then I will get a positive result 1 time in 20, i.e. I will "erroneously find no result" 19 times out of 20. Except that I'm not really erroneously finding no result. I am correctly finding the results of a different experiment than the one I think I'm running. And, of course, an instrument that returns random numbers is an extreme example. See: https://arstechnica.com/science/2018/04/new-measurements-set-two-areas-of-physics-against-each-other/ https://arstechnica.com/science/2018/04/new-measurements-set... for a current real-world example. Now, you do have a valid point in that I made a tacit assumption when I claimed that statistics put a lower bound on the error rate. That assumption is that most experiments that are done test hypotheses that ultimately turn out to be false. I believe this assumption is actually true. This is one of the reasons making scientific discoveries is hard. But I didn't actually state this assumption, so I'll cop to that. > At this point you're arguing against the sources I cited. That's not true either. The first source says that the error rate is much higher than the p-value, which agrees with me. (I'm actually a little skeptical that the error rate is as high as your source says -- "At least 23% (and typically close to 50%" for p=0.05, but that's certainly not refuting my position!) I don't have time to go through your second citation right now.
- 3JPLW 9y ago
  > The problem is that you cannot define what is a "real effect" without begging the question That's precisely my point. We don't know the sizes of these populations, so you really can't say anything quantitative about how the p-value affects the proportion of bad results in any given journal. My example was intentionally contrived. You just have to be really careful when you talk about p-values because _so many people_ have this misunderstanding and it's actively harmful to getting the correct interpretation. That's why we keep going back and forth here — I'm not disagreeing that the situation is bad, I just want folks to recognize what the stats say and what they don't.
  - lisper 9y ago
    > We don't know the sizes of these populations I think we can make some pretty reasonable guesses. In any case, I certainly agree with you about the big picture: it's confusing, it's important to get it right, and a lot of people don't understand it. I may even be one of those people. But at the moment I believe that our disagreement is really over something else, namely, whether it's reasonable to have a prior on the null hypothesis.

by 3JPLW 9y ago

lisper 9y ago

> Were I to run 100 independently designed experiments that all tested real effects, my choice of p-value does not determine the number that erroneously find no result. That's actually not true. The problem is that you cannot define what is a "real effect" without begging the question. Let me illustrate with an example: Suppose I do what appears to be a legitimate experiment to test a well-accepted law of nature. Unbeknownst to me, my instrument is faulty (or has been tampered with) and is in fact returning essentially random results. If I choose p<=0.05 then I will get a positive result 1 time in 20, i.e. I will "erroneously find no result" 19 times out of 20. Except that I'm not really erroneously finding no result. I am correctly finding the results of a different experiment than the one I think I'm running. And, of course, an instrument that returns random numbers is an extreme example. See: https://arstechnica.com/science/2018/04/new-measurements-set-two-areas-of-physics-against-each-other/ https://arstechnica.com/science/2018/04/new-measurements-set... for a current real-world example. Now, you do have a valid point in that I made a tacit assumption when I claimed that statistics put a lower bound on the error rate. That assumption is that most experiments that are done test hypotheses that ultimately turn out to be false. I believe this assumption is actually true. This is one of the reasons making scientific discoveries is hard. But I didn't actually state this assumption, so I'll cop to that. > At this point you're arguing against the sources I cited. That's not true either. The first source says that the error rate is much higher than the p-value, which agrees with me. (I'm actually a little skeptical that the error rate is as high as your source says -- "At least 23% (and typically close to 50%" for p=0.05, but that's certainly not refuting my position!) I don't have time to go through your second citation right now.

3JPLW 9y ago
> The problem is that you cannot define what is a "real effect" without begging the question That's precisely my point. We don't know the sizes of these populations, so you really can't say anything quantitative about how the p-value affects the proportion of bad results in any given journal. My example was intentionally contrived. You just have to be really careful when you talk about p-values because _so many people_ have this misunderstanding and it's actively harmful to getting the correct interpretation. That's why we keep going back and forth here — I'm not disagreeing that the situation is bad, I just want folks to recognize what the stats say and what they don't.
- lisper 9y ago
  > We don't know the sizes of these populations I think we can make some pretty reasonable guesses. In any case, I certainly agree with you about the big picture: it's confusing, it's important to get it right, and a lot of people don't understand it. I may even be one of those people. But at the moment I believe that our disagreement is really over something else, namely, whether it's reasonable to have a prior on the null hypothesis.