3 ms·

> I think fixing the peer review process may be easier than the replication crisis. I think fixing the two are almost one and the same. Peer review without rep

> I think fixing the peer review process may be easier than the replication crisis. I think fixing the two are almost one and the same. Peer review without replication is just a place for ivory tower intellectuals to review from an armchair. Replication helps to demonstrate that other people have not only "reviewed" the work but have gone through the trouble of setting up the problem and re-identifying the (potentially undocumented) pain points.

bumby 7y ago
I wish the downvotes would be accompanied with comments but I don't think your stance is completely unreasonable. I do think, however, it may not be tenable in the current structure. I think there is too much cost (both monetarily and opportunity) to expect replication to be part of the peer review process. But somehow, we still need to incentive replication.
PeterisP 7y ago
Replication often (depends on the kind of science you're doing) takes a comparable amount of work to making the paper in the first place. If I'm a reviewer, then that means that I'm asked to volunteer a bit of my spare time to facilitate the evaluation of my peers by helping the editor (who's expected to be competent but not knowledgeable in every niche) to read the paper, and answer the following questions: (a) is this topic relevant and novel in my sub-niche; (b) is the content adequately described in a way that the target audience (i.e. people in my sub-niche like me) can understand clearly; and (c) does it have any obvious flaws or ignorance of things and previous research that are (or should be) well-known in my sub-niche. So I provide my niche-expert opinion about these questions to help the editor make their decision on which papers to include and what changes to mandate. That is peer review - in some sense, establishing whether the author is "behaving as a peer". Replication is something that comes after the study is published. If the system would require me to replicate your experiment for a review (assuming that I'm interested enough in the particular topic - I review papers that are tangentially interesting, but not the direction that I'd want to do myself), then that would be unreasonable to have it done as a volunteer service, it would require months of full time work to make a review, and who would pay for that work? If the authors would have to fund a replication before publishing, that places an enormous barrier on publishing and, frankly, I would want to read results that have not passed that barrier because I can't assume that everything that's useful for my research will be able to pass it. As the reader in the target audience for scientific publishing (i.e. other researchers actively working in that field - scientific journals and conferences are made by a niche of scientists for that niche of scientists, it's how we communicate between ourselves, disseminating knowledge to outsiders is not the primary goal for the vast majority of academic publication venues, that's what textbooks and monographies are for) I want authors to publish their results without a prolonged vetting, because I'd prefer to read them sooner rather than later.

bumby 7y ago

I wish the downvotes would be accompanied with comments but I don't think your stance is completely unreasonable. I do think, however, it may not be tenable in the current structure. I think there is too much cost (both monetarily and opportunity) to expect replication to be part of the peer review process. But somehow, we still need to incentive replication.

PeterisP 7y ago

Replication often (depends on the kind of science you're doing) takes a comparable amount of work to making the paper in the first place. If I'm a reviewer, then that means that I'm asked to volunteer a bit of my spare time to facilitate the evaluation of my peers by helping the editor (who's expected to be competent but not knowledgeable in every niche) to read the paper, and answer the following questions: (a) is this topic relevant and novel in my sub-niche; (b) is the content adequately described in a way that the target audience (i.e. people in my sub-niche like me) can understand clearly; and (c) does it have any obvious flaws or ignorance of things and previous research that are (or should be) well-known in my sub-niche. So I provide my niche-expert opinion about these questions to help the editor make their decision on which papers to include and what changes to mandate. That is peer review - in some sense, establishing whether the author is "behaving as a peer". Replication is something that comes after the study is published. If the system would require me to replicate your experiment for a review (assuming that I'm interested enough in the particular topic - I review papers that are tangentially interesting, but not the direction that I'd want to do myself), then that would be unreasonable to have it done as a volunteer service, it would require months of full time work to make a review, and who would pay for that work? If the authors would have to fund a replication before publishing, that places an enormous barrier on publishing and, frankly, I would want to read results that have not passed that barrier because I can't assume that everything that's useful for my research will be able to pass it. As the reader in the target audience for scientific publishing (i.e. other researchers actively working in that field - scientific journals and conferences are made by a niche of scientists for that niche of scientists, it's how we communicate between ourselves, disseminating knowledge to outsiders is not the primary goal for the vast majority of academic publication venues, that's what textbooks and monographies are for) I want authors to publish their results without a prolonged vetting, because I'd prefer to read them sooner rather than later.