4 ms·
It is solid probabilistic reasoning if you accept: - A publication bias term u that is pulled out of thin air for different scenarios. - A parameter R that re
by cgiles 9y ago
It is solid probabilistic reasoning if you accept:
- A publication bias term u that is pulled out of thin air for different scenarios.
- A parameter R that reflects the unknowable proportion of true relationships/hypotheses in a field compared to the universe of possible hypotheses. The pre-study odds he calculates are determined entirely by R.
Actually this paper provides a good framework for determining what factors will affect the PPV of a field. In that regard it was a good contribution. But the enormous leap to the title -- which can only be achieved by massive assumptions about R and u -- was baseless fearmongering and demagoguery.
If one wants to address these issues at the 50,000 feet level, it would be far better to look at something concrete, such as reproducibility rates, as indeed many have done, rather than models based on pre-study odds. If we actually knew the pre-study odds, or IOW the proportion of hypotheses in hypothesis space that are true, then we could just correct our p-values for that and be done with it. Although I suppose that many Bayesians would not see any problem with abstracting away everything we don't know into priors R and u, making wild guesses about them, and drawing conclusions.
We do have a replication crisis on our hands. But the pre-study odds, whatever they are, are unchangeable. If this paper had been framed in terms of "the lower the pre-study odds, the higher the power will need to be to compensate to get an acceptable level of reproducibility", I would accept it wholeheartedly, although then it would have been a simple and obvious statement rather than a citation-grabber.
- AlexCoventry 9y agoOf course pre-study odds are low in fields querying complex systems on the basis of extremely weak theoretical frameworks.
- cgiles 9y agoI agree. However, nothing much better is available. I loved reading work from the Santa Fe Institute and Stuart Kauffman and so on, but the reality is that despite their best efforts, they provided nothing better. General statements about the system-as-a-whole, but no specifics about where to look for the etiology of a specific disease or process. Nobels await those who could do more. Until such time as someone does bring theory to biology, for example, wet-labbers (not me) struggle on. I am in a little doubt as to whether general theories about complex systems could provide useful predictive frameworks, but if so, great. But until then, pre-study odds are zero if you don't do the study.
- AlexCoventry 9y agoI wasn't appealing to complexity theory as the solution. I was arguing that Ioannidis's assumptions about R are reasonable.