4 ms·

Peer review is a base level quality check, not a correctness check in most disciplines. It provides value, indeed most of my papers have been improved by going

Peer review is a base level quality check, not a correctness check in most disciplines. It provides value, indeed most of my papers have been improved by going through peer review. But it's good to be conscious of what it can and can not do. One thing that it does do is signal intent. It signals that you are willing to engage with the expert community, and specifically their feedback and criticism. Finally, there is of course a fallacy in saying "not trustworthy either". If 99% of random blog posts are wrong, but 60% of published papers are wrong, then a random paper is 40 times more likely to be correct than a random blog post and it's perfectly sensible to be hesitant about citing the latter. Finally, scientists have always cited outside of traditional sources and continue to do so [1]. I see no evidence for your assertion that "appeal to authority" has become more prevalent or important than it used to be. With all the ongoing open science initiatives it's rather the opposite. [1] Personal communication with various scientists, 2018

neuromantik8086 8y ago
> Peer review is a base level quality check Ideally this is true. At the worst though, you'll end up with a peer reviewer that gets pedantic or one who's a member of a citation cartel [0] that demands that you cite his/her friends papers unnecessarily as a quid-pro-quo leading to publication. It's difficult to say how common this is in practice since academia is generally terrible about self-reflection, but I observed concerning evidence of this when I was still in science. In some fields at least, there's ample evidence [1] that peer review provides inadequate QC controls for methods sections, making it literally impossible to replicate results due to entire processing steps being omitted. This is why post-publication services such as PubPeer have become more and more relevant, since it has become obvious as time has elapsed how woefully inadequate the initial screening process is. [0] https://academia.stackexchange.com/questions/81261/reviewer-recommending-citation-of-certain-papers-of-specific-group-of-authors-ma https://academia.stackexchange.com/questions/81261/reviewer-... [1] https://www.ncbi.nlm.nih.gov/pubmed/22796459 https://www.ncbi.nlm.nih.gov/pubmed/22796459
- Certhas 8y ago
  It's good to keep in mind that reviewers do not make the decision. They recommend the decision. The Editor, who knows the identity of the reviewer, can see if a reviewer is asking for unreasonable citations. So while this definitely does happen (and to different degrees in different fields) there are some checks and balances, too.
  - neuromantik8086 8y ago
    True, and your overall point (that it is better than the alternative; i.e. viXra) stands. It's really one of those things where ymmv depending on the field- I just happened to notice a somewhat disheartening tendency for those sorts of horse trading practices to creep in, which is ultimately one of the major reasons why I left science. I could be naive, but my impression was that there are almost too many degrees of freedom in the peer review process that could allow for this kind of chicanery, and that it would almost be preferable to have criteria for acceptance laid out in advance (i.e., exactly what steps, metrics, and citable articles were necessary) and use those the way an actuary would. In some ways I almost think that practices like preregistration serve as a better first-pass indicator of whether or not research is publishable.
    - Certhas 8y ago
      Completely agreed. I think the system is ripe for an overhaul, and has many problematic aspects. We'll see how it goes, after all it doesn't look too bad with the open access revolution. I personally would like more use of pre-registration (there is something about that on the front-page right now. But also I think deanonymzing peer review and publishing the reviews along with the papers could be very interesting. Obvious drawbacks as well of course but I think it has potentially massive upsides.

neuromantik8086 8y ago

> Peer review is a base level quality check Ideally this is true. At the worst though, you'll end up with a peer reviewer that gets pedantic or one who's a member of a citation cartel [0] that demands that you cite his/her friends papers unnecessarily as a quid-pro-quo leading to publication. It's difficult to say how common this is in practice since academia is generally terrible about self-reflection, but I observed concerning evidence of this when I was still in science. In some fields at least, there's ample evidence [1] that peer review provides inadequate QC controls for methods sections, making it literally impossible to replicate results due to entire processing steps being omitted. This is why post-publication services such as PubPeer have become more and more relevant, since it has become obvious as time has elapsed how woefully inadequate the initial screening process is. [0] https://academia.stackexchange.com/questions/81261/reviewer-recommending-citation-of-certain-papers-of-specific-group-of-authors-ma https://academia.stackexchange.com/questions/81261/reviewer-... [1] https://www.ncbi.nlm.nih.gov/pubmed/22796459 https://www.ncbi.nlm.nih.gov/pubmed/22796459

Certhas 8y ago
It's good to keep in mind that reviewers do not make the decision. They recommend the decision. The Editor, who knows the identity of the reviewer, can see if a reviewer is asking for unreasonable citations. So while this definitely does happen (and to different degrees in different fields) there are some checks and balances, too.
- neuromantik8086 8y ago
  True, and your overall point (that it is better than the alternative; i.e. viXra) stands. It's really one of those things where ymmv depending on the field- I just happened to notice a somewhat disheartening tendency for those sorts of horse trading practices to creep in, which is ultimately one of the major reasons why I left science. I could be naive, but my impression was that there are almost too many degrees of freedom in the peer review process that could allow for this kind of chicanery, and that it would almost be preferable to have criteria for acceptance laid out in advance (i.e., exactly what steps, metrics, and citable articles were necessary) and use those the way an actuary would. In some ways I almost think that practices like preregistration serve as a better first-pass indicator of whether or not research is publishable.
  - Certhas 8y ago
    Completely agreed. I think the system is ripe for an overhaul, and has many problematic aspects. We'll see how it goes, after all it doesn't look too bad with the open access revolution. I personally would like more use of pre-registration (there is something about that on the front-page right now. But also I think deanonymzing peer review and publishing the reviews along with the papers could be very interesting. Obvious drawbacks as well of course but I think it has potentially massive upsides.