5 ms·

Then that is incorrect. The situation is simultaneously better and worse than your interpretation: > while a low P value indicates that your data are unli

by 3JPLW 9y ago

Then that is incorrect. The situation is simultaneously better and worse than your interpretation: > while a low P value indicates that your data are unlikely > assuming a true null, it can’t evaluate which of two > competing cases is more likely: > * The null is true but your sample was unusual. > * The null is false. You're conflating the two populations, and indeed we don't know which is which. But the P-value is only talking about the likelihood of observing your data given a true null. It only has any effect on that first population. Without knowing the size of the second population, you cannot say anything about the fraction of erroneous results in a journal based upon the statistical test alone. http://blog.minitab.com/blog/adventures-in-statistics-2/how-to-correctly-interpret-p-values http://blog.minitab.com/blog/adventures-in-statistics-2/how-... At the same time, though, converting p-values to a bayesian posterior or conditional probability that says something about the likelihood of the _hypothesis_ (like you're trying to do) is hard and generally way more pessimistic than 1 in 20 — it's more like 1 in 3. http://www.dcscience.net/Sellke-Bayarri-Berger-calibration-of-P-2001.pdf http://www.dcscience.net/Sellke-Bayarri-Berger-calibration-o...

lisper 9y ago
> it can’t evaluate which of two competing cases is more likely Of course it can. In fact, it can tell you exactly how much more likely one case is versus the other. P=0.05 means that there is a 5% chance that the null is true and hence a 95% chance the the null is false. > says something about the likelihood of the _hypothesis_ (like you're trying to do) No. I am simply saying that there is a lower bound on the rate of false positives caused by the choice of the threshold p value. If you chose P<=0.05 then you will have an error rate of at least 5%. The actual error rate is almost certainly higher (that's what "lower bound" means) for a host of other reasons.
- 3JPLW 9y ago
  Were I to run 100 independently designed experiments that all tested real effects, my choice of p-value does not determine the number that erroneously find no result. If the effect is small and I didn't gather enough data, a p-value of 0.05 could result in only a handful of experiments accurately reflecting reality. Let's say 30 make the cut. Were I to run another 100 independently designed experiments that all tested big real effects, then a p-value of 0.05 could result in all 100 accurately reflecting reality. Were I to run 100 independently designed experiments that all tested false effects, then at a p-value level of 0.05, I should see about 5 erroneously significant results. Now I submit all these significant results to a journal and they all get published. Of the 135 results, only 5 are coming to the wrong conclusion. Look at that, an error rate lower than your lower bound. We simply don't know the sizes of these populations. It could be better or worse than my example, and it could be better or worse than 5%. There simply is no lower bound. Of course you're right that there is a problem here but there isn't a lower bound guaranteed by the stats tests. Quite a bit of the problem stems from misinterpretations of p-values. For someone who likes to condescendingly explain concepts, you might want to make sure you have the harder concept down before assuming others don't understand "ratios" and "lower bounds." At this point you're arguing against the sources I cited. You should check them out. :)
  - lisper 9y ago
    > Were I to run 100 independently designed experiments that all tested real effects, my choice of p-value does not determine the number that erroneously find no result. That's actually not true. The problem is that you cannot define what is a "real effect" without begging the question. Let me illustrate with an example: Suppose I do what appears to be a legitimate experiment to test a well-accepted law of nature. Unbeknownst to me, my instrument is faulty (or has been tampered with) and is in fact returning essentially random results. If I choose p<=0.05 then I will get a positive result 1 time in 20, i.e. I will "erroneously find no result" 19 times out of 20. Except that I'm not really erroneously finding no result. I am correctly finding the results of a different experiment than the one I think I'm running. And, of course, an instrument that returns random numbers is an extreme example. See: https://arstechnica.com/science/2018/04/new-measurements-set-two-areas-of-physics-against-each-other/ https://arstechnica.com/science/2018/04/new-measurements-set... for a current real-world example. Now, you do have a valid point in that I made a tacit assumption when I claimed that statistics put a lower bound on the error rate. That assumption is that most experiments that are done test hypotheses that ultimately turn out to be false. I believe this assumption is actually true. This is one of the reasons making scientific discoveries is hard. But I didn't actually state this assumption, so I'll cop to that. > At this point you're arguing against the sources I cited. That's not true either. The first source says that the error rate is much higher than the p-value, which agrees with me. (I'm actually a little skeptical that the error rate is as high as your source says -- "At least 23% (and typically close to 50%" for p=0.05, but that's certainly not refuting my position!) I don't have time to go through your second citation right now.
    - 3JPLW 9y ago
      > The problem is that you cannot define what is a "real effect" without begging the question That's precisely my point. We don't know the sizes of these populations, so you really can't say anything quantitative about how the p-value affects the proportion of bad results in any given journal. My example was intentionally contrived. You just have to be really careful when you talk about p-values because _so many people_ have this misunderstanding and it's actively harmful to getting the correct interpretation. That's why we keep going back and forth here — I'm not disagreeing that the situation is bad, I just want folks to recognize what the stats say and what they don't.
- sfink 9y ago
  > P=0.05 means that there is a 5% chance that the null is true No. It simply does not mean that. It would be very convenient if it did, but it does not and can not. p=0.05 means that if the null hypothesis were true and you ran your experiment, then you would only have a 5% chance of getting the results that you did. The distinction is not at all obvious, but you'll need to understand it before you can make sense of statistics. The p-value is not the number you want; what you want is the probability that the null is false, but you can't have that unless you know the actual probability distribution. Which generally isn't something you could possibly know or figure out. (You can make educated guesses about that distribution, by making a guess and then updating it based on observed results. That's Bayesian statistics, which is great because it does give you a full probability distribution and all you have to feed into it is... uh... a probability distribution.)
  - deleted 9y ago
    [deleted]
  - lisper 9y ago
    Yes, you're right. I was making the tacit assumption that the null is true most of the time. On this assumption, the two statements are more or less equivalent. I believe this assumption is correct. If it weren't, making real scientific discoveries would be a lot easier. But it is an assumption.
    - 3JPLW 9y ago
      You're falling prey to the very common "p-value fallacy." P-values talk about the likelihood of the data, not the likelihood of the hypothesis. The two are unfortunately not linked.
      - lisper 9y ago
        Yes, I am aware of this. P-values tell you the probability that what looks like a positive result is not in fact a positive result but merely a fluke, i.e. the p-value is (a lower bound on) the probability of a false positive. So?

by 3JPLW 9y ago

lisper 9y ago

> it can’t evaluate which of two competing cases is more likely Of course it can. In fact, it can tell you exactly how much more likely one case is versus the other. P=0.05 means that there is a 5% chance that the null is true and hence a 95% chance the the null is false. > says something about the likelihood of the _hypothesis_ (like you're trying to do) No. I am simply saying that there is a lower bound on the rate of false positives caused by the choice of the threshold p value. If you chose P<=0.05 then you will have an error rate of at least 5%. The actual error rate is almost certainly higher (that's what "lower bound" means) for a host of other reasons.

3JPLW 9y ago
Were I to run 100 independently designed experiments that all tested real effects, my choice of p-value does not determine the number that erroneously find no result. If the effect is small and I didn't gather enough data, a p-value of 0.05 could result in only a handful of experiments accurately reflecting reality. Let's say 30 make the cut. Were I to run another 100 independently designed experiments that all tested big real effects, then a p-value of 0.05 could result in all 100 accurately reflecting reality. Were I to run 100 independently designed experiments that all tested false effects, then at a p-value level of 0.05, I should see about 5 erroneously significant results. Now I submit all these significant results to a journal and they all get published. Of the 135 results, only 5 are coming to the wrong conclusion. Look at that, an error rate lower than your lower bound. We simply don't know the sizes of these populations. It could be better or worse than my example, and it could be better or worse than 5%. There simply is no lower bound. Of course you're right that there is a problem here but there isn't a lower bound guaranteed by the stats tests. Quite a bit of the problem stems from misinterpretations of p-values. For someone who likes to condescendingly explain concepts, you might want to make sure you have the harder concept down before assuming others don't understand "ratios" and "lower bounds." At this point you're arguing against the sources I cited. You should check them out. :)
- lisper 9y ago
  > Were I to run 100 independently designed experiments that all tested real effects, my choice of p-value does not determine the number that erroneously find no result. That's actually not true. The problem is that you cannot define what is a "real effect" without begging the question. Let me illustrate with an example: Suppose I do what appears to be a legitimate experiment to test a well-accepted law of nature. Unbeknownst to me, my instrument is faulty (or has been tampered with) and is in fact returning essentially random results. If I choose p<=0.05 then I will get a positive result 1 time in 20, i.e. I will "erroneously find no result" 19 times out of 20. Except that I'm not really erroneously finding no result. I am correctly finding the results of a different experiment than the one I think I'm running. And, of course, an instrument that returns random numbers is an extreme example. See: https://arstechnica.com/science/2018/04/new-measurements-set-two-areas-of-physics-against-each-other/ https://arstechnica.com/science/2018/04/new-measurements-set... for a current real-world example. Now, you do have a valid point in that I made a tacit assumption when I claimed that statistics put a lower bound on the error rate. That assumption is that most experiments that are done test hypotheses that ultimately turn out to be false. I believe this assumption is actually true. This is one of the reasons making scientific discoveries is hard. But I didn't actually state this assumption, so I'll cop to that. > At this point you're arguing against the sources I cited. That's not true either. The first source says that the error rate is much higher than the p-value, which agrees with me. (I'm actually a little skeptical that the error rate is as high as your source says -- "At least 23% (and typically close to 50%" for p=0.05, but that's certainly not refuting my position!) I don't have time to go through your second citation right now.
  - 3JPLW 9y ago
    > The problem is that you cannot define what is a "real effect" without begging the question That's precisely my point. We don't know the sizes of these populations, so you really can't say anything quantitative about how the p-value affects the proportion of bad results in any given journal. My example was intentionally contrived. You just have to be really careful when you talk about p-values because _so many people_ have this misunderstanding and it's actively harmful to getting the correct interpretation. That's why we keep going back and forth here — I'm not disagreeing that the situation is bad, I just want folks to recognize what the stats say and what they don't.
sfink 9y ago
> P=0.05 means that there is a 5% chance that the null is true No. It simply does not mean that. It would be very convenient if it did, but it does not and can not. p=0.05 means that if the null hypothesis were true and you ran your experiment, then you would only have a 5% chance of getting the results that you did. The distinction is not at all obvious, but you'll need to understand it before you can make sense of statistics. The p-value is not the number you want; what you want is the probability that the null is false, but you can't have that unless you know the actual probability distribution. Which generally isn't something you could possibly know or figure out. (You can make educated guesses about that distribution, by making a guess and then updating it based on observed results. That's Bayesian statistics, which is great because it does give you a full probability distribution and all you have to feed into it is... uh... a probability distribution.)
- deleted 9y ago
  [deleted]
- lisper 9y ago
  Yes, you're right. I was making the tacit assumption that the null is true most of the time. On this assumption, the two statements are more or less equivalent. I believe this assumption is correct. If it weren't, making real scientific discoveries would be a lot easier. But it is an assumption.
  - 3JPLW 9y ago
    You're falling prey to the very common "p-value fallacy." P-values talk about the likelihood of the data, not the likelihood of the hypothesis. The two are unfortunately not linked.
    - lisper 9y ago
      Yes, I am aware of this. P-values tell you the probability that what looks like a positive result is not in fact a positive result but merely a fluke, i.e. the p-value is (a lower bound on) the probability of a false positive. So?