4 ms·
I think fixing the peer review process may be easier than the replication crisis. All the incentives seem to align against performing work to replicated previou
by bumby 7y ago
I think fixing the peer review process may be easier than the replication crisis. All the incentives seem to align against performing work to replicated previously accepted science.
- inetknght 7y ago> I think fixing the peer review process may be easier than the replication crisis. I think fixing the two are almost one and the same. Peer review without replication is just a place for ivory tower intellectuals to review from an armchair. Replication helps to demonstrate that other people have not only "reviewed" the work but have gone through the trouble of setting up the problem and re-identifying the (potentially undocumented) pain points.
- bumby 7y agoI wish the downvotes would be accompanied with comments but I don't think your stance is completely unreasonable. I do think, however, it may not be tenable in the current structure. I think there is too much cost (both monetarily and opportunity) to expect replication to be part of the peer review process. But somehow, we still need to incentive replication.
- PeterisP 7y agoReplication often (depends on the kind of science you're doing) takes a comparable amount of work to making the paper in the first place. If I'm a reviewer, then that means that I'm asked to volunteer a bit of my spare time to facilitate the evaluation of my peers by helping the editor (who's expected to be competent but not knowledgeable in every niche) to read the paper, and answer the following questions: (a) is this topic relevant and novel in my sub-niche; (b) is the content adequately described in a way that the target audience (i.e. people in my sub-niche like me) can understand clearly; and (c) does it have any obvious flaws or ignorance of things and previous research that are (or should be) well-known in my sub-niche. So I provide my niche-expert opinion about these questions to help the editor make their decision on which papers to include and what changes to mandate. That is peer review - in some sense, establishing whether the author is "behaving as a peer". Replication is something that comes after the study is published. If the system would require me to replicate your experiment for a review (assuming that I'm interested enough in the particular topic - I review papers that are tangentially interesting, but not the direction that I'd want to do myself), then that would be unreasonable to have it done as a volunteer service, it would require months of full time work to make a review, and who would pay for that work? If the authors would have to fund a replication before publishing, that places an enormous barrier on publishing and, frankly, I would want to read results that have not passed that barrier because I can't assume that everything that's useful for my research will be able to pass it. As the reader in the target audience for scientific publishing (i.e. other researchers actively working in that field - scientific journals and conferences are made by a niche of scientists for that niche of scientists, it's how we communicate between ourselves, disseminating knowledge to outsiders is not the primary goal for the vast majority of academic publication venues, that's what textbooks and monographies are for) I want authors to publish their results without a prolonged vetting, because I'd prefer to read them sooner rather than later.
- barry-cotter 7y agoMove to post publication peer review as was the norm until after WWII. Problem solved.
- bumby 7y agoCan you elaborate on how that will fix the problem? It’s not immediately apparent to me
- barry-cotter 7y agoIf even journalists and the general public know that all getting published means is that the author(s) convinced an editor that an article probably wasn't total bullshit people would trust them less. Trust is the problem. If we can get people to trust literatures rather than articles we have progress.
- bumby 7y agoI'm not trying to be argumentative, just trying to understand. I'm not really familiar with the pre-WWII method of publication. How would moving peer-review to after publication enhance trust? It seems to me one of the problems currently is the vast amount of publication. Much of it seems derivative or of little value other than resume padding. While I agree the current peer review system isn't ideal, it does seem to at least provide some throttling of the volume of publications.