4 ms·

While we're at it, why not make public whoever audited or referred the paper? Why does that need to be secret?

by icebergwarrior 6y ago

apsec112 6y ago
People might retaliate against a colleague who gave a bad review.
- zapzupnz 6y ago
  Indeed. I've seen some mightily acerbic rebuttals to other researchers' articles published as articles; I dread to think what a rebuttal to a review might look like. I'm sure someone will say "well, maybe the acerbicness is the problem"; perhaps so, but I welcome the rigorous honesty with which some academics willingly write.
- bluenose69 6y ago
  True, and just imagine if the reviewer were a PhD candidate who would be seeking a job in a few years.
sinkasapa 6y ago
It might be that the person whoes paper you're reviewing will some day be able to influence whether you'll be hired.
smabie 6y ago
We should have less transparency, not more. Right now the peer review process is single-blind, but it really should be double-blind. Even if someone doesn't have a PhD or work at a research institution, they should be able to publish good science. Right now, that just isn't possible. And the opposite problem is also true: if you're a big shot in your field, you'll be able to find at least one journal that will publish whatever crap you submitted, regardless of the quality.
- derefr 6y ago
  Getting that benefit only requires double-blinding during the review process. There’s no reason that both sides of the blinding cannot be removed (and also revealed to third parties) after the review process is complete.
  - adminprof 6y ago
    You can't think of any reason? What about retaliation for being the less enthusiastic reviewer? What about the opposite, that you become known for being an easy reviewer who missed some obvious flaws?
    - derefr 6y ago
      You're still assuming that this happens right away. How about if the blinding is removed after 50 years? Then people studying the history of science would have the data, but it would have no impact on the careers of the people involved.
  - smabie 6y ago
    I can't think of any real upside of that, but can think of a lot of downside. Humans, and especially academics (I've noticed a trend in my work environments: the less money everyone makes, the more power is sublimated into inane dick measuring contests) are petty, and less personal it is, the better.
    - derefr 6y ago
      How about if the anonymity is "de-blinded" only partially, replacing it with pseudonymity? Imagine a system like this: 1. The reviewers on your paper not only have to collaborate on a decision to accept/reject, but also write an opinion (like a https://en.wikipedia.org/wiki/Legal_opinion https://en.wikipedia.org/wiki/Legal_opinion) about your paper, individual to themselves, after the consensus to accept/reject is reached (so some of them will likely be https://en.wikipedia.org/wiki/Dissenting_opinion https://en.wikipedia.org/wiki/Dissenting_opinion s.) 2. The reviewers are each assigned a global permanent pseudonymous identifier—a UUID, basically—known only to them and some "Society for the Advancement of the Scientific Process in Academia" organization. 3. Every vote a peer-reviewer makes, and also every opinion they write about a paper, must be registered with the same academic-process org, whose job is then to collate and publish them to the Internet under the reviewer's pseudonymous identifier. You'd be able to use such a website to both 1. audit the peer-review process for a given paper; and 2. cross-reference a given peer-reviewer's votes/opinions. Additionally, the standards body itself could use the cross-referencing ability to normalize peer-reviewer votes, ala how the Netflix Prize recommendation systems normalized votes by a person's interpretation of the star ratings. (They'd have to ask peer-reviewers to vote with something more fine-grained than a binary pass/fail, but that'd be an easy change.) The only thing I would worry about in such a system, is that academics might not want the negative opinions of the peer-reviewers on their paper to pop up when random other people plug the paper's DOI into Google Scholar, because a dissent on an accepted paper might unduly impact the paper's impact-factor.

While we're at it, why not make public whoever audited or referred the paper? Why does that need to be secret?

apsec112 6y ago

People might retaliate against a colleague who gave a bad review.

zapzupnz 6y ago
Indeed. I've seen some mightily acerbic rebuttals to other researchers' articles published as articles; I dread to think what a rebuttal to a review might look like. I'm sure someone will say "well, maybe the acerbicness is the problem"; perhaps so, but I welcome the rigorous honesty with which some academics willingly write.
bluenose69 6y ago
True, and just imagine if the reviewer were a PhD candidate who would be seeking a job in a few years.

sinkasapa 6y ago

It might be that the person whoes paper you're reviewing will some day be able to influence whether you'll be hired.

smabie 6y ago

We should have less transparency, not more. Right now the peer review process is single-blind, but it really should be double-blind. Even if someone doesn't have a PhD or work at a research institution, they should be able to publish good science. Right now, that just isn't possible. And the opposite problem is also true: if you're a big shot in your field, you'll be able to find at least one journal that will publish whatever crap you submitted, regardless of the quality.

derefr 6y ago
Getting that benefit only requires double-blinding during the review process. There’s no reason that both sides of the blinding cannot be removed (and also revealed to third parties) after the review process is complete.
- adminprof 6y ago
  You can't think of any reason? What about retaliation for being the less enthusiastic reviewer? What about the opposite, that you become known for being an easy reviewer who missed some obvious flaws?
  - derefr 6y ago
    You're still assuming that this happens right away. How about if the blinding is removed after 50 years? Then people studying the history of science would have the data, but it would have no impact on the careers of the people involved.
- smabie 6y ago
  I can't think of any real upside of that, but can think of a lot of downside. Humans, and especially academics (I've noticed a trend in my work environments: the less money everyone makes, the more power is sublimated into inane dick measuring contests) are petty, and less personal it is, the better.
  - derefr 6y ago
    How about if the anonymity is "de-blinded" only partially, replacing it with pseudonymity? Imagine a system like this: 1. The reviewers on your paper not only have to collaborate on a decision to accept/reject, but also write an opinion (like a https://en.wikipedia.org/wiki/Legal_opinion https://en.wikipedia.org/wiki/Legal_opinion) about your paper, individual to themselves, after the consensus to accept/reject is reached (so some of them will likely be https://en.wikipedia.org/wiki/Dissenting_opinion https://en.wikipedia.org/wiki/Dissenting_opinion s.) 2. The reviewers are each assigned a global permanent pseudonymous identifier—a UUID, basically—known only to them and some "Society for the Advancement of the Scientific Process in Academia" organization. 3. Every vote a peer-reviewer makes, and also every opinion they write about a paper, must be registered with the same academic-process org, whose job is then to collate and publish them to the Internet under the reviewer's pseudonymous identifier. You'd be able to use such a website to both 1. audit the peer-review process for a given paper; and 2. cross-reference a given peer-reviewer's votes/opinions. Additionally, the standards body itself could use the cross-referencing ability to normalize peer-reviewer votes, ala how the Netflix Prize recommendation systems normalized votes by a person's interpretation of the star ratings. (They'd have to ask peer-reviewers to vote with something more fine-grained than a binary pass/fail, but that'd be an easy change.) The only thing I would worry about in such a system, is that academics might not want the negative opinions of the peer-reviewers on their paper to pop up when random other people plug the paper's DOI into Google Scholar, because a dissent on an accepted paper might unduly impact the paper's impact-factor.