4 ms·

Devil's advocate: sometimes, knowing the author of a work really can allow a reviewer/reader to make a more accurate assessment of its value. As an extreme exa

Devil's advocate: sometimes, knowing the author of a work really can allow a reviewer/reader to make a more accurate assessment of its value. As an extreme example, if Stephen Hawking submits a paper that calls into question the speed of light as a universal constant, I'm much more likely to take it seriously (and in the peer-review case, would be much more likely to recommend it for publication) than the same paper from some unknown "up and coming" physicist. This is because Hawking is an expert in the field, and I trust that if there were any obvious flaws in the analysis he would be more likely to have discovered them than a randomly-selected unestablished researcher. No reviewer has time to independently investigate and verify every single claim a paper makes, and knowing that the paper comes from an elite institution might very reasonably be used as a signal that it should be given the benefit of the doubt.

greggarious 9y ago
>Devil's advocate: sometimes, knowing the author of a work really can allow a reviewer/reader to make a more accurate assessment of its value. >As an extreme example, if Stephen Hawking submits a paper that calls into question the speed of light as a universal constant, I'm much more likely to take it seriously I'm sure Mr. Hawking, being so skilled in his field, can appropriately cite related work and lay out what he sees as it's flaws without resorting to an appeal to authority. If you need such crutches to know when someone's paper is high quality, perhaps you are not fully invested in the reviewing process and should allow someone willing to take the time to fully evaluate all submitted works.
eesmith 9y ago
How are you being a devil's advocate? As far as I can tell, your viewpoint is completely described in the conclusion, quoted here: "In conclusion, the heart of our findings is that single-blind reviewers make use of information about authors and institutions. Specifically, single-blind reviewers are more likely to bid on papers from top institutions and more likely to recommend for acceptance papers from famous authors or top institutions, compared with their double-blind counterparts." "The primary ethical question is whether this behavior is acceptable. In one interpretation, single-blind reviewers make use of prior information that may allow them to make better overall judgments. As a consequence, however, it may be that other work is disadvantaged, in the sense that two contributions of roughly equal merit might be scored differently by single-blind reviewers, in favor of the one from a top school, while double-blind reviewers may not show this bias as strongly." Or are you arguing that there is no ethical concern at all? The introduction seems to cover the main pro and con arguments, as part of the justification for the need for experimental evidence.
- humanrebar 9y ago
  > How are you being a devil's advocate? Popular opinion among scientists, engineers, etc. is that science stands (and should stand) on its own without needing support from appeals to authority, bandwagon effects, etc.
  - eesmith 9y ago
    Okay, but how was kcorbitt being a devil's advocate with respect to this paper?
    - humanrebar 9y ago
      He was being devil's advocate with respect to popular opinion, not this particular paper.
      - eesmith 9y ago
        The paper was pretty clear that the popular opinion (among those with knowledge of the topic) is mixed, with advocates for both single blind and double blind, and that that vast majority of evaluations is done single blind - that is to say, aligned with the kcorbitt's position. What do you mean by "devil's advocate"? Because I can't make sense of how to apply either my understanding of the term, nor the description at Wikipedia, to kcorbitt's comment.
0xBA5ED 9y ago
A scientific review should focus on the findings alone. Using the author's name as a shortcut around (or an excuse to avoid) thoroughly reviewing the content should not be encouraged.

greggarious 9y ago

>Devil's advocate: sometimes, knowing the author of a work really can allow a reviewer/reader to make a more accurate assessment of its value. >As an extreme example, if Stephen Hawking submits a paper that calls into question the speed of light as a universal constant, I'm much more likely to take it seriously I'm sure Mr. Hawking, being so skilled in his field, can appropriately cite related work and lay out what he sees as it's flaws without resorting to an appeal to authority. If you need such crutches to know when someone's paper is high quality, perhaps you are not fully invested in the reviewing process and should allow someone willing to take the time to fully evaluate all submitted works.

eesmith 9y ago

How are you being a devil's advocate? As far as I can tell, your viewpoint is completely described in the conclusion, quoted here: "In conclusion, the heart of our findings is that single-blind reviewers make use of information about authors and institutions. Specifically, single-blind reviewers are more likely to bid on papers from top institutions and more likely to recommend for acceptance papers from famous authors or top institutions, compared with their double-blind counterparts." "The primary ethical question is whether this behavior is acceptable. In one interpretation, single-blind reviewers make use of prior information that may allow them to make better overall judgments. As a consequence, however, it may be that other work is disadvantaged, in the sense that two contributions of roughly equal merit might be scored differently by single-blind reviewers, in favor of the one from a top school, while double-blind reviewers may not show this bias as strongly." Or are you arguing that there is no ethical concern at all? The introduction seems to cover the main pro and con arguments, as part of the justification for the need for experimental evidence.

humanrebar 9y ago
> How are you being a devil's advocate? Popular opinion among scientists, engineers, etc. is that science stands (and should stand) on its own without needing support from appeals to authority, bandwagon effects, etc.
- eesmith 9y ago
  Okay, but how was kcorbitt being a devil's advocate with respect to this paper?
  - humanrebar 9y ago
    He was being devil's advocate with respect to popular opinion, not this particular paper.
    - eesmith 9y ago
      The paper was pretty clear that the popular opinion (among those with knowledge of the topic) is mixed, with advocates for both single blind and double blind, and that that vast majority of evaluations is done single blind - that is to say, aligned with the kcorbitt's position. What do you mean by "devil's advocate"? Because I can't make sense of how to apply either my understanding of the term, nor the description at Wikipedia, to kcorbitt's comment.

0xBA5ED 9y ago

A scientific review should focus on the findings alone. Using the author's name as a shortcut around (or an excuse to avoid) thoroughly reviewing the content should not be encouraged.