3 ms·

It's different because you're ignoring my second bullet point. There's a very common misconception that p-value is the percent chance the data confirm that the

by 3JPLW 9y ago

It's different because you're ignoring my second bullet point. There's a very common misconception that p-value is the percent chance the data confirm that the effect is real. That's not at all how p-values work. They are showing the _likelihood_ of generating the data were there no "effect." You make a "null hypothesis" — that is, you assume there is no effect — and you construct a distribution of the results you might see in such a case. If your results are relatively unlikely, then you say the null hypothesis is rejected at a p-value level, leading support to the fact that there is such an effect. But there's another bullet in my comment. You could _also_ see a low p-value if the null hypothesis is wrong! That would _also_ lead to a low p-value, but for a completely different reasons — reasons that are typically elucidated in the entire rest of the paper. Thus, you should be most suspicious of papers that do not have a reasonable cause of action, as that should increase your prior that the null hypothesis is actually correct. If I were to edit that post, I'd add a third bullet — that they cheated or did bad science. My objection is to you stating that the statistical tests alone guarantee at least a 1 in 20 failure rate with a p-value of .05. There are indeed many other reasons to be suspect, but the statistical tests don't guarantee this.

lisper 9y ago
I think you're just misinterpreting what I mean by "failure" or "wrong result". I mean a positive result that is due to chance rather than to the null hypothesis actually being false. On that view, statistics alone guarantee a "failure" rate of at least 1 in 20. That's what choosing a threshold of 0.05 means. The on top of that baseline rate of false results you also have self-selection and occasional outright cheating that drives the failure rate higher than 1 in 20. 1 in 20 is what you get when you use a 0.05 threshold and everything else in the system works flawlessly.
- 3JPLW 9y ago
  Ah, yes, I read you as saying that "1 out of every 20 statistical tests in every journal is guaranteed to be wrong. It's just stats, people." If what you mean to say is that "1 out of every 20 statistical tests of a false finding will demonstrate a 'significant' p-value and may subsequently get published," then we're in agreement.
  - lisper 9y ago
    I meant to say what I said. Getting a significant p-value when in fact there is no effect is a wrong result, hence, at a minimum, 1 in 20 results (on average) in a journal that uses a p-value threshold of 0.05 will be wrong.
    - 3JPLW 9y ago
      Then that is incorrect. The situation is simultaneously better and worse than your interpretation: > while a low P value indicates that your data are unlikely > assuming a true null, it can’t evaluate which of two > competing cases is more likely: > * The null is true but your sample was unusual. > * The null is false. You're conflating the two populations, and indeed we don't know which is which. But the P-value is only talking about the likelihood of observing your data given a true null. It only has any effect on that first population. Without knowing the size of the second population, you cannot say anything about the fraction of erroneous results in a journal based upon the statistical test alone. http://blog.minitab.com/blog/adventures-in-statistics-2/how-to-correctly-interpret-p-values http://blog.minitab.com/blog/adventures-in-statistics-2/how-... At the same time, though, converting p-values to a bayesian posterior or conditional probability that says something about the likelihood of the _hypothesis_ (like you're trying to do) is hard and generally way more pessimistic than 1 in 20 — it's more like 1 in 3. http://www.dcscience.net/Sellke-Bayarri-Berger-calibration-of-P-2001.pdf http://www.dcscience.net/Sellke-Bayarri-Berger-calibration-o...
      - lisper 9y ago
        > it can’t evaluate which of two competing cases is more likely Of course it can. In fact, it can tell you exactly how much more likely one case is versus the other. P=0.05 means that there is a 5% chance that the null is true and hence a 95% chance the the null is false. > says something about the likelihood of the _hypothesis_ (like you're trying to do) No. I am simply saying that there is a lower bound on the rate of false positives caused by the choice of the threshold p value. If you chose P<=0.05 then you will have an error rate of at least 5%. The actual error rate is almost certainly higher (that's what "lower bound" means) for a host of other reasons.

by 3JPLW 9y ago

lisper 9y ago

I think you're just misinterpreting what I mean by "failure" or "wrong result". I mean a positive result that is due to chance rather than to the null hypothesis actually being false. On that view, statistics alone guarantee a "failure" rate of at least 1 in 20. That's what choosing a threshold of 0.05 means. The on top of that baseline rate of false results you also have self-selection and occasional outright cheating that drives the failure rate higher than 1 in 20. 1 in 20 is what you get when you use a 0.05 threshold and everything else in the system works flawlessly.

3JPLW 9y ago
Ah, yes, I read you as saying that "1 out of every 20 statistical tests in every journal is guaranteed to be wrong. It's just stats, people." If what you mean to say is that "1 out of every 20 statistical tests of a false finding will demonstrate a 'significant' p-value and may subsequently get published," then we're in agreement.
- lisper 9y ago
  I meant to say what I said. Getting a significant p-value when in fact there is no effect is a wrong result, hence, at a minimum, 1 in 20 results (on average) in a journal that uses a p-value threshold of 0.05 will be wrong.
  - 3JPLW 9y ago
    Then that is incorrect. The situation is simultaneously better and worse than your interpretation: > while a low P value indicates that your data are unlikely > assuming a true null, it can’t evaluate which of two > competing cases is more likely: > * The null is true but your sample was unusual. > * The null is false. You're conflating the two populations, and indeed we don't know which is which. But the P-value is only talking about the likelihood of observing your data given a true null. It only has any effect on that first population. Without knowing the size of the second population, you cannot say anything about the fraction of erroneous results in a journal based upon the statistical test alone. http://blog.minitab.com/blog/adventures-in-statistics-2/how-to-correctly-interpret-p-values http://blog.minitab.com/blog/adventures-in-statistics-2/how-... At the same time, though, converting p-values to a bayesian posterior or conditional probability that says something about the likelihood of the _hypothesis_ (like you're trying to do) is hard and generally way more pessimistic than 1 in 20 — it's more like 1 in 3. http://www.dcscience.net/Sellke-Bayarri-Berger-calibration-of-P-2001.pdf http://www.dcscience.net/Sellke-Bayarri-Berger-calibration-o...
    - lisper 9y ago
      > it can’t evaluate which of two competing cases is more likely Of course it can. In fact, it can tell you exactly how much more likely one case is versus the other. P=0.05 means that there is a 5% chance that the null is true and hence a 95% chance the the null is false. > says something about the likelihood of the _hypothesis_ (like you're trying to do) No. I am simply saying that there is a lower bound on the rate of false positives caused by the choice of the threshold p value. If you chose P<=0.05 then you will have an error rate of at least 5%. The actual error rate is almost certainly higher (that's what "lower bound" means) for a host of other reasons.