5 ms·

> Scientists often abhor citing from blogs, forums, and Wikipedia, as those publications "are not peer-reviewed by trustworthy professionals". Which is sad bec

by qubax 8y ago

> Scientists often abhor citing from blogs, forums, and Wikipedia, as those publications "are not peer-reviewed by trustworthy professionals". Which is sad because most peer-reviewed material are not trustworthy either. https://bigthink.com/neurobonkers/believe-it-or-not-most-published-research-findings-are-probably-false https://bigthink.com/neurobonkers/believe-it-or-not-most-pub... https://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.0020124 https://journals.plos.org/plosmedicine/article?id=10.1371/jo... It's worrying that academia is slowly turning into a rigid "appeal to authority" church rather than a space for truth and inquiry.

Certhas 8y ago
Peer review is a base level quality check, not a correctness check in most disciplines. It provides value, indeed most of my papers have been improved by going through peer review. But it's good to be conscious of what it can and can not do. One thing that it does do is signal intent. It signals that you are willing to engage with the expert community, and specifically their feedback and criticism. Finally, there is of course a fallacy in saying "not trustworthy either". If 99% of random blog posts are wrong, but 60% of published papers are wrong, then a random paper is 40 times more likely to be correct than a random blog post and it's perfectly sensible to be hesitant about citing the latter. Finally, scientists have always cited outside of traditional sources and continue to do so [1]. I see no evidence for your assertion that "appeal to authority" has become more prevalent or important than it used to be. With all the ongoing open science initiatives it's rather the opposite. [1] Personal communication with various scientists, 2018
- neuromantik8086 8y ago
  > Peer review is a base level quality check Ideally this is true. At the worst though, you'll end up with a peer reviewer that gets pedantic or one who's a member of a citation cartel [0] that demands that you cite his/her friends papers unnecessarily as a quid-pro-quo leading to publication. It's difficult to say how common this is in practice since academia is generally terrible about self-reflection, but I observed concerning evidence of this when I was still in science. In some fields at least, there's ample evidence [1] that peer review provides inadequate QC controls for methods sections, making it literally impossible to replicate results due to entire processing steps being omitted. This is why post-publication services such as PubPeer have become more and more relevant, since it has become obvious as time has elapsed how woefully inadequate the initial screening process is. [0] https://academia.stackexchange.com/questions/81261/reviewer-recommending-citation-of-certain-papers-of-specific-group-of-authors-ma https://academia.stackexchange.com/questions/81261/reviewer-... [1] https://www.ncbi.nlm.nih.gov/pubmed/22796459 https://www.ncbi.nlm.nih.gov/pubmed/22796459
  - Certhas 8y ago
    It's good to keep in mind that reviewers do not make the decision. They recommend the decision. The Editor, who knows the identity of the reviewer, can see if a reviewer is asking for unreasonable citations. So while this definitely does happen (and to different degrees in different fields) there are some checks and balances, too.
    - neuromantik8086 8y ago
      True, and your overall point (that it is better than the alternative; i.e. viXra) stands. It's really one of those things where ymmv depending on the field- I just happened to notice a somewhat disheartening tendency for those sorts of horse trading practices to creep in, which is ultimately one of the major reasons why I left science. I could be naive, but my impression was that there are almost too many degrees of freedom in the peer review process that could allow for this kind of chicanery, and that it would almost be preferable to have criteria for acceptance laid out in advance (i.e., exactly what steps, metrics, and citable articles were necessary) and use those the way an actuary would. In some ways I almost think that practices like preregistration serve as a better first-pass indicator of whether or not research is publishable.
      - Certhas 8y ago
        Completely agreed. I think the system is ripe for an overhaul, and has many problematic aspects. We'll see how it goes, after all it doesn't look too bad with the open access revolution. I personally would like more use of pre-registration (there is something about that on the front-page right now. But also I think deanonymzing peer review and publishing the reviews along with the papers could be very interesting. Obvious drawbacks as well of course but I think it has potentially massive upsides.
pjc50 8y ago
Academia has always been built around the ""authority"" of its references because you can't re-prove everything from the ground up in one paper.
gmueckl 8y ago
I am about to get really mad about statements like this one. It contains an egregious generalization within itself, rendering it essentially untrue. The papers check results in fields like medicine and psychology which are plagued by the fact that they have to rely on various kind of effects from random sampling of study participants and statistics. I am not surprised that a study with 10 participants leads to wrong results with a significant probability. But this does not generalize to science in general. Physics and chemistry do not suffer fron this because their experiments have fundamentally different setups. So extrapolating from medicine to "most science is wrong" is just as wrong. You need to approach these fields differently when you want to know how many wrong results are published.
neuromantik8086 8y ago
> It's worrying that academia is slowly turning into a rigid "appeal to authority" church Seeing as how academia literally started out as an accessory to the various Christian denominations for educating priests, I don't see how you really could characterize it as "a space for truth and inquiry". It's only been relatively recently that "appeal to authority" has given way to more secular/scientific ways. Even within the last couple of years, I would wager that academia is no more authoritarian than it has ever been- tenured professors have almost always had a disproportionate amount of influence over their trainees' futures, etc. The head of my father's department when he was getting his PhD in the 70s actually attempted to pull my father's stipend because of an avant garde project that would today be accepted as normal. I've seen a trend towards the opposite, actually, with graduate students becoming more skeptical of the institution and more proactively addressing its shortcomings through grad student unions and advocacy of open science policies (which neither university administrations nor longstanding tenured professors are fans of in general).
- tokai 8y ago
  My advisor used to say that old professors contribute most to science by dying.

by qubax 8y ago

Certhas 8y ago

Peer review is a base level quality check, not a correctness check in most disciplines. It provides value, indeed most of my papers have been improved by going through peer review. But it's good to be conscious of what it can and can not do. One thing that it does do is signal intent. It signals that you are willing to engage with the expert community, and specifically their feedback and criticism. Finally, there is of course a fallacy in saying "not trustworthy either". If 99% of random blog posts are wrong, but 60% of published papers are wrong, then a random paper is 40 times more likely to be correct than a random blog post and it's perfectly sensible to be hesitant about citing the latter. Finally, scientists have always cited outside of traditional sources and continue to do so [1]. I see no evidence for your assertion that "appeal to authority" has become more prevalent or important than it used to be. With all the ongoing open science initiatives it's rather the opposite. [1] Personal communication with various scientists, 2018

neuromantik8086 8y ago
> Peer review is a base level quality check Ideally this is true. At the worst though, you'll end up with a peer reviewer that gets pedantic or one who's a member of a citation cartel [0] that demands that you cite his/her friends papers unnecessarily as a quid-pro-quo leading to publication. It's difficult to say how common this is in practice since academia is generally terrible about self-reflection, but I observed concerning evidence of this when I was still in science. In some fields at least, there's ample evidence [1] that peer review provides inadequate QC controls for methods sections, making it literally impossible to replicate results due to entire processing steps being omitted. This is why post-publication services such as PubPeer have become more and more relevant, since it has become obvious as time has elapsed how woefully inadequate the initial screening process is. [0] https://academia.stackexchange.com/questions/81261/reviewer-recommending-citation-of-certain-papers-of-specific-group-of-authors-ma https://academia.stackexchange.com/questions/81261/reviewer-... [1] https://www.ncbi.nlm.nih.gov/pubmed/22796459 https://www.ncbi.nlm.nih.gov/pubmed/22796459
- Certhas 8y ago
  It's good to keep in mind that reviewers do not make the decision. They recommend the decision. The Editor, who knows the identity of the reviewer, can see if a reviewer is asking for unreasonable citations. So while this definitely does happen (and to different degrees in different fields) there are some checks and balances, too.
  - neuromantik8086 8y ago
    True, and your overall point (that it is better than the alternative; i.e. viXra) stands. It's really one of those things where ymmv depending on the field- I just happened to notice a somewhat disheartening tendency for those sorts of horse trading practices to creep in, which is ultimately one of the major reasons why I left science. I could be naive, but my impression was that there are almost too many degrees of freedom in the peer review process that could allow for this kind of chicanery, and that it would almost be preferable to have criteria for acceptance laid out in advance (i.e., exactly what steps, metrics, and citable articles were necessary) and use those the way an actuary would. In some ways I almost think that practices like preregistration serve as a better first-pass indicator of whether or not research is publishable.
    - Certhas 8y ago
      Completely agreed. I think the system is ripe for an overhaul, and has many problematic aspects. We'll see how it goes, after all it doesn't look too bad with the open access revolution. I personally would like more use of pre-registration (there is something about that on the front-page right now. But also I think deanonymzing peer review and publishing the reviews along with the papers could be very interesting. Obvious drawbacks as well of course but I think it has potentially massive upsides.

pjc50 8y ago

Academia has always been built around the ""authority"" of its references because you can't re-prove everything from the ground up in one paper.

gmueckl 8y ago

I am about to get really mad about statements like this one. It contains an egregious generalization within itself, rendering it essentially untrue. The papers check results in fields like medicine and psychology which are plagued by the fact that they have to rely on various kind of effects from random sampling of study participants and statistics. I am not surprised that a study with 10 participants leads to wrong results with a significant probability. But this does not generalize to science in general. Physics and chemistry do not suffer fron this because their experiments have fundamentally different setups. So extrapolating from medicine to "most science is wrong" is just as wrong. You need to approach these fields differently when you want to know how many wrong results are published.

neuromantik8086 8y ago

> It's worrying that academia is slowly turning into a rigid "appeal to authority" church Seeing as how academia literally started out as an accessory to the various Christian denominations for educating priests, I don't see how you really could characterize it as "a space for truth and inquiry". It's only been relatively recently that "appeal to authority" has given way to more secular/scientific ways. Even within the last couple of years, I would wager that academia is no more authoritarian than it has ever been- tenured professors have almost always had a disproportionate amount of influence over their trainees' futures, etc. The head of my father's department when he was getting his PhD in the 70s actually attempted to pull my father's stipend because of an avant garde project that would today be accepted as normal. I've seen a trend towards the opposite, actually, with graduate students becoming more skeptical of the institution and more proactively addressing its shortcomings through grad student unions and advocacy of open science policies (which neither university administrations nor longstanding tenured professors are fans of in general).

tokai 8y ago
My advisor used to say that old professors contribute most to science by dying.