4 ms·
You are incorrect. False positive rates in practice appear to be about 20-50%, even with a p-value of 0.05.
by jcranmer 11y ago
You are incorrect. False positive rates in practice appear to be about 20-50%, even with a p-value of 0.05.
- blahblah3 11y agoI was describing the theory behind the p-value. In practice, people slice and dice the dataset in multiple ways and try several specifications which increases the false positive rate (see the multiple comparisons problem)
- nonbel 11y agoThe best estimate right now is 20-50% replication rate for fields that rely on p less than 0.05 to judge results. So I think you meant 50-80% false positive rate, although these are not equivalent (ie is the null hypothesis of no difference ever really exactly true...): >"Ninety-seven percent of original studies had statistically significant results. Thirty-six percent of replications had statistically significant results; 47% of original effect sizes were in the 95% confidence interval of the replication effect size; 39% of effects were subjectively rated to have replicated the original result; [...] In cell biology, two industrial laboratories reported success replicating the original results of landmark findings in only 11 and 25% of the attempted cases" http://www.ncbi.nlm.nih.gov/pubmed/26315443 http://www.ncbi.nlm.nih.gov/pubmed/26315443 Edit: For comparison, Dr. Oz was accused of fraud and called in front of congress for only being wrong half the time: https://www.washingtonpost.com/news/morning-mix/wp/2014/12/19/half-of-dr-ozs-medical-advice-is-baseless-or-wrong-study-says/ https://www.washingtonpost.com/news/morning-mix/wp/2014/12/1...