3 ms·
Doesn't wash. If replication attempts are subject to publication bias, we are still better off because the subsequent meta-analyses overcome sampling error bett
by aab0 10y ago
Doesn't wash. If replication attempts are subject to publication bias, we are still better off because the subsequent meta-analyses overcome sampling error better and are able to show publication bias using funnel plots or other techniques like p-uniform, and sometimes can even correct the bias to give you a less biased and more accurate estimate. In contrast, you cannot show publication bias for a particular novel result nor can you easily correct for it. So given the choice between 5 studies on a single hypothesis (all afflicted by publication bias) and 5 studies on 5 hypotheses (with publication bias), you are better off in the former scenario. (In the latter scenario, you could try to use informative priors estimated from field-wide demonstrations of publication bias, but at least so far, this is an extremely unpopular approach.)
- jessriedel 10y agoYes, a much better headline would be "The value of study replications is positive but suppressed without some combination of pre-registration and result-blind refereeing".
- frozenport 10y agoPre-registration doesn't help because it simply gives the author's a narrative to fake. As pointed out in the article, result-blind refereeing is officially claimed by most journals, although from the rejection I have received this does not appear to be the case in practice.
- timroy 10y agoCan you explain what you mean by "it simply gives the author's a narrative to fake"? At the very least, pre-registration helps guard against the "file-drawer" effect, where a negative result simply goes in the trash, and no one ever knows it occurred. If you mean that pre-registration will not prevent jiggling the results until the researcher gets a significant p-value, then I agree that many forms of pre-registration which lack registration of the analysis and so on will allow the authors to fake the narrative.
- jessriedel 10y ago> Pre-registration doesn't help because it simply gives the author's a narrative to fake. Like timroy, I don't know what your objection to pre-registration is. I'm definitely talking about pre-registering the analysis, not just the data collection. > As pointed out in the article, result-blind refereeing is officially claimed by most journals, Did I miss something? That doesn't seem to be what the article is saying: > One proposed remedy is to modify the peer review process so that reviewers grade manuscripts on the quality of their introductions and methods sections rather than the novelty of the findings. Similarly, journals could assess submissions based solely on the rigor of their methods—as does Plos One. PLoS One is an anomaly. This is not normal.