4 ms·

Ah, yes, I read you as saying that "1 out of every 20 statistical tests in every journal is guaranteed to be wrong. It's just stats, people." If what you mean

• lisper 9y ago
  I meant to say what I said. Getting a significant p-value when in fact there is no effect is a wrong result, hence, at a minimum, 1 in 20 results (on average) in a journal that uses a p-value threshold of 0.05 will be wrong.
  • 3JPLW 9y ago
    Then that is incorrect. The situation is simultaneously better and worse than your interpretation: > while a low P value indicates that your data are unlikely > assuming a true null, it can’t evaluate which of two > competing cases is more likely: > * The null is true but your sample was unusual. > * The null is false. You're conflating the two populations, and indeed we don't know which is which. But the P-value is only talking about the likelihood of observing your data given a true null. It only has any effect on that first population. Without knowing the size of the second population, you cannot say anything about the fraction of erroneous results in a journal based upon the statistical test alone. http://blog.minitab.com/blog/adventures-in-statistics-2/how-to-correctly-interpret-p-values http://blog.minitab.com/blog/adventures-in-statistics-2/how-... At the same time, though, converting p-values to a bayesian posterior or conditional probability that says something about the likelihood of the _hypothesis_ (like you're trying to do) is hard and generally way more pessimistic than 1 in 20 — it's more like 1 in 3. http://www.dcscience.net/Sellke-Bayarri-Berger-calibration-of-P-2001.pdf http://www.dcscience.net/Sellke-Bayarri-Berger-calibration-o...
    • lisper 9y ago
      > it can’t evaluate which of two competing cases is more likely Of course it can. In fact, it can tell you exactly how much more likely one case is versus the other. P=0.05 means that there is a 5% chance that the null is true and hence a 95% chance the the null is false. > says something about the likelihood of the _hypothesis_ (like you're trying to do) No. I am simply saying that there is a lower bound on the rate of false positives caused by the choice of the threshold p value. If you chose P<=0.05 then you will have an error rate of at least 5%. The actual error rate is almost certainly higher (that's what "lower bound" means) for a host of other reasons.
      • 3JPLW 9y ago
        Were I to run 100 independently designed experiments that all tested real effects, my choice of p-value does not determine the number that erroneously find no result. If the effect is small and I didn't gather enough data, a p-value of 0.05 could result in only a handful of experiments accurately reflecting reality. Let's say 30 make the cut. Were I to run another 100 independently designed experiments that all tested big real effects, then a p-value of 0.05 could result in all 100 accurately reflecting reality. Were I to run 100 independently designed experiments that all tested false effects, then at a p-value level of 0.05, I should see about 5 erroneously significant results. Now I submit all these significant results to a journal and they all get published. Of the 135 results, only 5 are coming to the wrong conclusion. Look at that, an error rate lower than your lower bound. We simply don't know the sizes of these populations. It could be better or worse than my example, and it could be better or worse than 5%. There simply is no lower bound. Of course you're right that there is a problem here but there isn't a lower bound guaranteed by the stats tests. Quite a bit of the problem stems from misinterpretations of p-values. For someone who likes to condescendingly explain concepts, you might want to make sure you have the harder concept down before assuming others don't understand "ratios" and "lower bounds." At this point you're arguing against the sources I cited. You should check them out. :)
        • lisper 9y ago
          > Were I to run 100 independently designed experiments that all tested real effects, my choice of p-value does not determine the number that erroneously find no result. That's actually not true. The problem is that you cannot define what is a "real effect" without begging the question. Let me illustrate with an example: Suppose I do what appears to be a legitimate experiment to test a well-accepted law of nature. Unbeknownst to me, my instrument is faulty (or has been tampered with) and is in fact returning essentially random results. If I choose p<=0.05 then I will get a positive result 1 time in 20, i.e. I will "erroneously find no result" 19 times out of 20. Except that I'm not really erroneously finding no result. I am correctly finding the results of a different experiment than the one I think I'm running. And, of course, an instrument that returns random numbers is an extreme example. See: https://arstechnica.com/science/2018/04/new-measurements-set-two-areas-of-physics-against-each-other/ https://arstechnica.com/science/2018/04/new-measurements-set... for a current real-world example. Now, you do have a valid point in that I made a tacit assumption when I claimed that statistics put a lower bound on the error rate. That assumption is that most experiments that are done test hypotheses that ultimately turn out to be false. I believe this assumption is actually true. This is one of the reasons making scientific discoveries is hard. But I didn't actually state this assumption, so I'll cop to that. > At this point you're arguing against the sources I cited. That's not true either. The first source says that the error rate is much higher than the p-value, which agrees with me. (I'm actually a little skeptical that the error rate is as high as your source says -- "At least 23% (and typically close to 50%" for p=0.05, but that's certainly not refuting my position!) I don't have time to go through your second citation right now.