3 ms·

This paper, almost 20 years old, has plenty of follow-up work showing the claims in this original paper aren’t true. One simple angle is Ioannidis simply makes

This paper, almost 20 years old, has plenty of follow-up work showing the claims in this original paper aren’t true. One simple angle is Ioannidis simply makes up some parameters to show things could be bad. Later empirical work measuring those parameters found Ioannidis off by orders of magnitude. One example https://arxiv.org/abs/1301.3718 https://arxiv.org/abs/1301.3718 There’s ample other published papers showing other holes in the claims. https://scholar.google.com/scholar?cites=15681017780418799273&as_sdt=800005&sciodt=0,15&hl=en https://scholar.google.com/scholar?cites=1568101778041879927... Google scholar papers citing this

asgraham 2y ago
re: the arxiv link Why is it that microarray true positive p-values follow a beta distribution? Following the citations led to a lot of empirical confirmation but I couldn't find any discussion of why. More to the point of this rebuttal, though: why would we expect the amalgamation of 70k micro-array experiments' abstract-reported p-values to follow a single beta distribution? And what about modeling the bias-induced bump of barely-significant results? If there's some theoretical reason why the meta-study can use the beta-uniform model, then I could see this being only a mild underestimation of the proportion of false positives (14%), but otherwise I'm confused how we can interpret this.
mike_hearn 2y ago
That arXiv paper seems to imply that a false positive rate of 14% is "a reliable record of scientific progress". Even if we take their claim at face value, 14% of medical papers making false claims seems way too high. I'm personally very skeptical it's that low. What I found during COVID is that entire literatures exist in this sort of weird space where you can't even say if they're true or false because the basic methodologies of the field don't even get you that far. Computational epidemiology never seemed to test its predictions against reality to begin with, so the whole concept of doing P-value analysis on such papers would be meaningless. Based on personal experience I'd feel like Ioannidis is directionally correct.

asgraham 2y ago

re: the arxiv link Why is it that microarray true positive p-values follow a beta distribution? Following the citations led to a lot of empirical confirmation but I couldn't find any discussion of why. More to the point of this rebuttal, though: why would we expect the amalgamation of 70k micro-array experiments' abstract-reported p-values to follow a single beta distribution? And what about modeling the bias-induced bump of barely-significant results? If there's some theoretical reason why the meta-study can use the beta-uniform model, then I could see this being only a mild underestimation of the proportion of false positives (14%), but otherwise I'm confused how we can interpret this.

mike_hearn 2y ago

That arXiv paper seems to imply that a false positive rate of 14% is "a reliable record of scientific progress". Even if we take their claim at face value, 14% of medical papers making false claims seems way too high. I'm personally very skeptical it's that low. What I found during COVID is that entire literatures exist in this sort of weird space where you can't even say if they're true or false because the basic methodologies of the field don't even get you that far. Computational epidemiology never seemed to test its predictions against reality to begin with, so the whole concept of doing P-value analysis on such papers would be meaningless. Based on personal experience I'd feel like Ioannidis is directionally correct.