3 ms·

The point of p-values is to limit false positive rates. If you only accept results with p-value <= 0.05, your false positive rate is at most 5%. Of course, if

The point of p-values is to limit false positive rates. If you only accept results with p-value <= 0.05, your false positive rate is at most 5%. Of course, if you had apriori knowledge that everything was a "null result" , then 100% of the results you accept would be false positives. By Bayes' theorem, P(H|D) = P(D|H) * P(H) / P(D) . Therefore, given a fixed "prior probability" of a hypothesis P(H), a lower p-value (P(D|H)) implies a lower probability of the null being true. I think this relationship leads to a lot of confusion when thinking of p-values.

jcranmer 11y ago
You are incorrect. False positive rates in practice appear to be about 20-50%, even with a p-value of 0.05.
- blahblah3 11y ago
  I was describing the theory behind the p-value. In practice, people slice and dice the dataset in multiple ways and try several specifications which increases the false positive rate (see the multiple comparisons problem)
- nonbel 11y ago
  The best estimate right now is 20-50% replication rate for fields that rely on p less than 0.05 to judge results. So I think you meant 50-80% false positive rate, although these are not equivalent (ie is the null hypothesis of no difference ever really exactly true...): >"Ninety-seven percent of original studies had statistically significant results. Thirty-six percent of replications had statistically significant results; 47% of original effect sizes were in the 95% confidence interval of the replication effect size; 39% of effects were subjectively rated to have replicated the original result; [...] In cell biology, two industrial laboratories reported success replicating the original results of landmark findings in only 11 and 25% of the attempted cases" http://www.ncbi.nlm.nih.gov/pubmed/26315443 http://www.ncbi.nlm.nih.gov/pubmed/26315443 Edit: For comparison, Dr. Oz was accused of fraud and called in front of congress for only being wrong half the time: https://www.washingtonpost.com/news/morning-mix/wp/2014/12/19/half-of-dr-ozs-medical-advice-is-baseless-or-wrong-study-says/ https://www.washingtonpost.com/news/morning-mix/wp/2014/12/1...
TeMPOraL 11y ago
Wait, isn't P(H) probability of the tested hypothesis being true, and therefore P(D|H) probability of seeing the data given tested hypothesis is true? The p-value would be P(D|¬H), right?
- blahblah3 11y ago
  Yeah I may have worded it confusingly, you can think of H as the null hypothesis in the equation (or replace it all with not H if you want)
  - TeMPOraL 11y ago
    Ok, thanks for the clarification!
tedsanders 11y ago
Just to clarify this correct comment for others reading it: When blahblah says the false positive rate will be at most 5%, she or he means that at most, 5% of experiments will result in a false positive, not that 5% of results will be false positive results. This tripped me up when I read it the first time.

jcranmer 11y ago

You are incorrect. False positive rates in practice appear to be about 20-50%, even with a p-value of 0.05.

blahblah3 11y ago
I was describing the theory behind the p-value. In practice, people slice and dice the dataset in multiple ways and try several specifications which increases the false positive rate (see the multiple comparisons problem)
nonbel 11y ago
The best estimate right now is 20-50% replication rate for fields that rely on p less than 0.05 to judge results. So I think you meant 50-80% false positive rate, although these are not equivalent (ie is the null hypothesis of no difference ever really exactly true...): >"Ninety-seven percent of original studies had statistically significant results. Thirty-six percent of replications had statistically significant results; 47% of original effect sizes were in the 95% confidence interval of the replication effect size; 39% of effects were subjectively rated to have replicated the original result; [...] In cell biology, two industrial laboratories reported success replicating the original results of landmark findings in only 11 and 25% of the attempted cases" http://www.ncbi.nlm.nih.gov/pubmed/26315443 http://www.ncbi.nlm.nih.gov/pubmed/26315443 Edit: For comparison, Dr. Oz was accused of fraud and called in front of congress for only being wrong half the time: https://www.washingtonpost.com/news/morning-mix/wp/2014/12/19/half-of-dr-ozs-medical-advice-is-baseless-or-wrong-study-says/ https://www.washingtonpost.com/news/morning-mix/wp/2014/12/1...

TeMPOraL 11y ago

Wait, isn't P(H) probability of the tested hypothesis being true, and therefore P(D|H) probability of seeing the data given tested hypothesis is true? The p-value would be P(D|¬H), right?

blahblah3 11y ago
Yeah I may have worded it confusingly, you can think of H as the null hypothesis in the equation (or replace it all with not H if you want)
- TeMPOraL 11y ago
  Ok, thanks for the clarification!

tedsanders 11y ago

Just to clarify this correct comment for others reading it: When blahblah says the false positive rate will be at most 5%, she or he means that at most, 5% of experiments will result in a false positive, not that 5% of results will be false positive results. This tripped me up when I read it the first time.