4 ms·

Very good points! I completely agree that there's a lot value in anonymity when reviewing, and I don't think it should go away. Blowback over disagreeing is v

by jofer 13y ago

Very good points! I completely agree that there's a lot value in anonymity when reviewing, and I don't think it should go away. Blowback over disagreeing is very, very real, and I've experienced it myself (though it was from presenting something at a conference that disagreed with part of a previous publication, rather than as a result of a review). However, I do think public and optionally non-anonymous reviews could be helpful. (Incidentally, you had _9_ reviewers for a single publication?? Holy cow... In the geosciences, I've never heard of more than 3, and it's usually only 2. That's one thing that would really help in my field. More reviewers results in a broader review, rather than a particular person's pet peeve.)

arjunnarayan 13y ago
To be fair, it's not common to have 9 reviews. OSDI 2012 was particularly thorough (my reviews totaled about ~4000 words) I'm more used to 4-6 reviews, around 3000 words or so. SOSP/OSDI is probably the most thorough and competently run conference that I know of in all of academia. This makes sense: they only take about 30 papers (it's restricted not just by quality, but the quality bar goes up as more submissions are made, just to keep it exclusive), it's once a year, and systems researchers sync their lives to the OSDI/SOSP submission cycle. It's the "flagship" systems conference, and a lot of advances are first described as OSDI/SOSP papers. For some perspective, the original MapReduce paper was by Jeff Dean and Sanjay Ghemawat, published in OSDI 2004. Last year's best paper was the description of the Spanner Distributed Database by Google. Incidentally, I have deanonymized several reviews already. When you go to conferences, and some researcher asks the exact same question and has the exact same opinion as reviewer C, chances are... Sometimes the recognition is mutual as they realize what they just revealed (thankfully that has only resulted in humorous chuckles, at least in my life. But I can see how it can go wrong so easily!)
- seanmcdirmid 13y ago
  You might only get 9 reviews though if you pass into the second or third rounds. Controversial papers in the PL conferences I haunt will get more reviews out of necessity; better to add more reviewers to help break ties.
  - arjunnarayan 13y ago
    Yes. According to the OSDI 2012 proceedings introduction, all papers in the third round had 9 reviews. No paper was accepted or rejected with less than 51% of the PC member's assent.
    - seanmcdirmid 13y ago
      That is good and bad, it definitely skews toward safe non controversial papers.

by jofer 13y ago

arjunnarayan 13y ago

To be fair, it's not common to have 9 reviews. OSDI 2012 was particularly thorough (my reviews totaled about ~4000 words) I'm more used to 4-6 reviews, around 3000 words or so. SOSP/OSDI is probably the most thorough and competently run conference that I know of in all of academia. This makes sense: they only take about 30 papers (it's restricted not just by quality, but the quality bar goes up as more submissions are made, just to keep it exclusive), it's once a year, and systems researchers sync their lives to the OSDI/SOSP submission cycle. It's the "flagship" systems conference, and a lot of advances are first described as OSDI/SOSP papers. For some perspective, the original MapReduce paper was by Jeff Dean and Sanjay Ghemawat, published in OSDI 2004. Last year's best paper was the description of the Spanner Distributed Database by Google. Incidentally, I have deanonymized several reviews already. When you go to conferences, and some researcher asks the exact same question and has the exact same opinion as reviewer C, chances are... Sometimes the recognition is mutual as they realize what they just revealed (thankfully that has only resulted in humorous chuckles, at least in my life. But I can see how it can go wrong so easily!)

seanmcdirmid 13y ago
You might only get 9 reviews though if you pass into the second or third rounds. Controversial papers in the PL conferences I haunt will get more reviews out of necessity; better to add more reviewers to help break ties.
- arjunnarayan 13y ago
  Yes. According to the OSDI 2012 proceedings introduction, all papers in the third round had 9 reviews. No paper was accepted or rejected with less than 51% of the PC member's assent.
  - seanmcdirmid 13y ago
    That is good and bad, it definitely skews toward safe non controversial papers.