5 ms·

I'm going to respond to this in the context of academic computer science (which works off conferences), in particular, the systems subfield. YMMV if you extrapo

I'm going to respond to this in the context of academic computer science (which works off conferences), in particular, the systems subfield. YMMV if you extrapolate this to non-CS science and journals, so take this with a grain of salt. The bottom line is I, as a powerless grad student, am _very_ comfortable with the level of reviewing competence I have witnessed (and been a part of). The proposal to deanonymize reviews has a huge flaw: Namely, there's no way I (as a grad student with no political power) will write a critical review of somebody famous if I know I won't remain anonymous. The risk of blowback is too damn high. (And given the small pool, writing style would probably be sufficient to deanonymize, if my entire trail of reviews were public). There's no way I have the balls to write a critical review of a paper by, say, Robert Morris or Larry Peterson, for some prestigious venue such as SOSP, when I'm asked to write a review. Unless I know for sure that there's absolutely no risk whatsoever of being identified. Look. Reviewing is thankless and hard. Anonymity exists so that we can be critical and not risk career suicide. Last year at OSDI I received 9 reviews, and they were all exemplarily thoughtful, even the obligatory "strong reject" one of them. The problem of "petty reviews" _has_ a solution: The program committee is self policing, and the reviews are not anonymous within the PC. (For example, at SOSP this year, there are 28 PC members.) You know that most of your reviews come from this gang of 28. You might have one or two externally solicited experts in addition, but that too is not anonymous _to the PC members_. If there's a culture of meanness, the PC members and PC chairs can fix this by internal policing (and they do).

jofer 13y ago
Very good points! I completely agree that there's a lot value in anonymity when reviewing, and I don't think it should go away. Blowback over disagreeing is very, very real, and I've experienced it myself (though it was from presenting something at a conference that disagreed with part of a previous publication, rather than as a result of a review). However, I do think public and optionally non-anonymous reviews could be helpful. (Incidentally, you had _9_ reviewers for a single publication?? Holy cow... In the geosciences, I've never heard of more than 3, and it's usually only 2. That's one thing that would really help in my field. More reviewers results in a broader review, rather than a particular person's pet peeve.)
- arjunnarayan 13y ago
  To be fair, it's not common to have 9 reviews. OSDI 2012 was particularly thorough (my reviews totaled about ~4000 words) I'm more used to 4-6 reviews, around 3000 words or so. SOSP/OSDI is probably the most thorough and competently run conference that I know of in all of academia. This makes sense: they only take about 30 papers (it's restricted not just by quality, but the quality bar goes up as more submissions are made, just to keep it exclusive), it's once a year, and systems researchers sync their lives to the OSDI/SOSP submission cycle. It's the "flagship" systems conference, and a lot of advances are first described as OSDI/SOSP papers. For some perspective, the original MapReduce paper was by Jeff Dean and Sanjay Ghemawat, published in OSDI 2004. Last year's best paper was the description of the Spanner Distributed Database by Google. Incidentally, I have deanonymized several reviews already. When you go to conferences, and some researcher asks the exact same question and has the exact same opinion as reviewer C, chances are... Sometimes the recognition is mutual as they realize what they just revealed (thankfully that has only resulted in humorous chuckles, at least in my life. But I can see how it can go wrong so easily!)
  - seanmcdirmid 13y ago
    You might only get 9 reviews though if you pass into the second or third rounds. Controversial papers in the PL conferences I haunt will get more reviews out of necessity; better to add more reviewers to help break ties.
    - arjunnarayan 13y ago
      Yes. According to the OSDI 2012 proceedings introduction, all papers in the third round had 9 reviews. No paper was accepted or rejected with less than 51% of the PC member's assent.
      - seanmcdirmid 13y ago
        That is good and bad, it definitely skews toward safe non controversial papers.
scott_s 13y ago
Interesting. I don't think I do, or would have, feared blowback from giving a negative review - but I have no way of knowing for sure. I do know that I'm generally reviewing papers because a colleague asked me to, or asked me to be on the PC. So, in that instance, I'm more concerned with doing a good job in the eyes of my colleague. I do know that I have received many cursory reviews where the reviewer did not fully read the paper. I have often wondered if single-blind the other way would be better: reviewer signs their name, but paper authors are anonymous. While this may fix the crappy review problem, it is susceptible to what you fear.
- arjunnarayan 13y ago
  Some reviewers do (voluntarily) do that[1]. I would seriously consider doing so once I'm tenured (i.e. when I feel I am immune to blowback). I would hope that those cursory reviewers resulted in feedback from the other PC members that fix the issue (either through improved future reviews, or not being invited back to be on the PC.) [1]: http://se.ethz.ch/~meyer/publications/online/whysign/ http://se.ethz.ch/~meyer/publications/online/whysign/
  - scott_s 13y ago
    I've heard of others who do that, too. The difficulty is that those who self-select to sign their own name are probably not part of the problem.
    - seanmcdirmid 13y ago
      I sign my reviews unless I'm involved in some sort of post paper evaluation. It seems only fair since the reviews aren't double blind (given double blind, I would not sign). I also publish reviews on my own papers as a matter of interesting record.
jkimmel 13y ago
Your point regarding anonymity is highly valid, and I appreciate that you bring it up. There is actually a concept I'm rather fond of that seeks to address this, usually referred to as peer-reviewed peer review [1]. Essentially, anonymous experts review publications in much the manner they do now, with no risk of their identities being revealed. A separate set of review-reviewers then rates the publication-reviewers themselves, scoring them on a standardized quantitative scale (1-10) based on the validity of their claims and the depth of their assessment. In this manner, experts collect an "aggregate reviewer rating" of sorts analogous to an h-index [2], call it an R-index. This R-index can then serve to select accurate reviewers for future publications. A high R-index could also be included as a positive bullet point on a CV, providing some reward for reviewers to put a high degree of effort into their otherwise unrewarded reviewing work. A few potential flaws are evident: what is one to do if early reviews are scored poorly, and you are no longer able to provide new reviews to increase your R-index? A grace period before scoring could potentially remedy this, ie) an R-index is not assigned until a reviewer has written >10 reviews. [1] - http://www.frontiersin.org/Computational_Neuroscience/10.3389/fncom.2012.00020/full http://www.frontiersin.org/Computational_Neuroscience/10.338... [2] - http://en.wikipedia.org/wiki/H-index‎ http://en.wikipedia.org/wiki/H-index‎
- arjunnarayan 13y ago
  I like the R-index idea. It's basically Slashdot's meta-moderation system? I always thought that had some promise. However, it does add even more _work_ to a system that is extremely stressed (writing reviews is so much work! Imagine having to read papers, read reviews and then review the review on top of that!) But basically existing program committees (in computer science) do serve that meta-moderation purpose (without the empirical R-index metric). The entire PC does get to see the deanonymized reviews of the other program committee members. If someone's just being an unhelpful jerk, they're probably not going to get invited back in the future.
drjesusphd 13y ago
I agree. But why not make it double-blind? Make the author anonymous to the reviewer too? Wouldn't that make the most sense?

jofer 13y ago

Very good points! I completely agree that there's a lot value in anonymity when reviewing, and I don't think it should go away. Blowback over disagreeing is very, very real, and I've experienced it myself (though it was from presenting something at a conference that disagreed with part of a previous publication, rather than as a result of a review). However, I do think public and optionally non-anonymous reviews could be helpful. (Incidentally, you had _9_ reviewers for a single publication?? Holy cow... In the geosciences, I've never heard of more than 3, and it's usually only 2. That's one thing that would really help in my field. More reviewers results in a broader review, rather than a particular person's pet peeve.)

arjunnarayan 13y ago
To be fair, it's not common to have 9 reviews. OSDI 2012 was particularly thorough (my reviews totaled about ~4000 words) I'm more used to 4-6 reviews, around 3000 words or so. SOSP/OSDI is probably the most thorough and competently run conference that I know of in all of academia. This makes sense: they only take about 30 papers (it's restricted not just by quality, but the quality bar goes up as more submissions are made, just to keep it exclusive), it's once a year, and systems researchers sync their lives to the OSDI/SOSP submission cycle. It's the "flagship" systems conference, and a lot of advances are first described as OSDI/SOSP papers. For some perspective, the original MapReduce paper was by Jeff Dean and Sanjay Ghemawat, published in OSDI 2004. Last year's best paper was the description of the Spanner Distributed Database by Google. Incidentally, I have deanonymized several reviews already. When you go to conferences, and some researcher asks the exact same question and has the exact same opinion as reviewer C, chances are... Sometimes the recognition is mutual as they realize what they just revealed (thankfully that has only resulted in humorous chuckles, at least in my life. But I can see how it can go wrong so easily!)
- seanmcdirmid 13y ago
  You might only get 9 reviews though if you pass into the second or third rounds. Controversial papers in the PL conferences I haunt will get more reviews out of necessity; better to add more reviewers to help break ties.
  - arjunnarayan 13y ago
    Yes. According to the OSDI 2012 proceedings introduction, all papers in the third round had 9 reviews. No paper was accepted or rejected with less than 51% of the PC member's assent.
    - seanmcdirmid 13y ago
      That is good and bad, it definitely skews toward safe non controversial papers.

scott_s 13y ago

Interesting. I don't think I do, or would have, feared blowback from giving a negative review - but I have no way of knowing for sure. I do know that I'm generally reviewing papers because a colleague asked me to, or asked me to be on the PC. So, in that instance, I'm more concerned with doing a good job in the eyes of my colleague. I do know that I have received many cursory reviews where the reviewer did not fully read the paper. I have often wondered if single-blind the other way would be better: reviewer signs their name, but paper authors are anonymous. While this may fix the crappy review problem, it is susceptible to what you fear.

arjunnarayan 13y ago
Some reviewers do (voluntarily) do that[1]. I would seriously consider doing so once I'm tenured (i.e. when I feel I am immune to blowback). I would hope that those cursory reviewers resulted in feedback from the other PC members that fix the issue (either through improved future reviews, or not being invited back to be on the PC.) [1]: http://se.ethz.ch/~meyer/publications/online/whysign/ http://se.ethz.ch/~meyer/publications/online/whysign/
- scott_s 13y ago
  I've heard of others who do that, too. The difficulty is that those who self-select to sign their own name are probably not part of the problem.
  - seanmcdirmid 13y ago
    I sign my reviews unless I'm involved in some sort of post paper evaluation. It seems only fair since the reviews aren't double blind (given double blind, I would not sign). I also publish reviews on my own papers as a matter of interesting record.

jkimmel 13y ago

Your point regarding anonymity is highly valid, and I appreciate that you bring it up. There is actually a concept I'm rather fond of that seeks to address this, usually referred to as peer-reviewed peer review [1]. Essentially, anonymous experts review publications in much the manner they do now, with no risk of their identities being revealed. A separate set of review-reviewers then rates the publication-reviewers themselves, scoring them on a standardized quantitative scale (1-10) based on the validity of their claims and the depth of their assessment. In this manner, experts collect an "aggregate reviewer rating" of sorts analogous to an h-index [2], call it an R-index. This R-index can then serve to select accurate reviewers for future publications. A high R-index could also be included as a positive bullet point on a CV, providing some reward for reviewers to put a high degree of effort into their otherwise unrewarded reviewing work. A few potential flaws are evident: what is one to do if early reviews are scored poorly, and you are no longer able to provide new reviews to increase your R-index? A grace period before scoring could potentially remedy this, ie) an R-index is not assigned until a reviewer has written >10 reviews. [1] - http://www.frontiersin.org/Computational_Neuroscience/10.3389/fncom.2012.00020/full http://www.frontiersin.org/Computational_Neuroscience/10.338... [2] - http://en.wikipedia.org/wiki/H-index‎ http://en.wikipedia.org/wiki/H-index‎

arjunnarayan 13y ago
I like the R-index idea. It's basically Slashdot's meta-moderation system? I always thought that had some promise. However, it does add even more _work_ to a system that is extremely stressed (writing reviews is so much work! Imagine having to read papers, read reviews and then review the review on top of that!) But basically existing program committees (in computer science) do serve that meta-moderation purpose (without the empirical R-index metric). The entire PC does get to see the deanonymized reviews of the other program committee members. If someone's just being an unhelpful jerk, they're probably not going to get invited back in the future.

drjesusphd 13y ago

I agree. But why not make it double-blind? Make the author anonymous to the reviewer too? Wouldn't that make the most sense?