4 ms·

Very true, but he does go on to state: > If we had, instead, a system where the review process was transparent and persisted for the useful life of a work Tha

by jofer 13y ago

Very true, but he does go on to state: > If we had, instead, a system where the review process was transparent and persisted for the useful life of a work That is a very interesting idea. (Reviews are currently confidential and usually anonymous -- only the author and editor sees the reviews, and only the editor knows who wrote them.) Not only would this reduce blatantly petty reviews, it would also give some recognition to the work that goes into reviewing a paper. It's currently an important but very thankless job. It's considered polite to acknowledge reviewers at the end of the paper (even if they remain anonymous), but it's often not done.

pizza_boy 13y ago
> it would also give some recognition to the work that goes into reviewing a paper I think you're spot on with this comment. That's exactly what we're building at publons.com. Check it out.
tigroferoce 13y ago
Completely agree. I've been into computer science research for few years and not knowing who accepted the papers and which were the reviews always bugged me. I've always advocated that the reviews behind an accepted paper SHOULD all be public and publicly distributed along with the paper.
- thaumasiotes 13y ago
  This is definitely a benefit to the readers, and potentially (though not definitely) to the community as a whole. However, it's much worse for the reviewers. I hear so much about how researchers provide papers to journals for free, and review papers for journals for free, and then the journals (are presumably evil to) charge high fees to anyone who wants to read the papers. How much can we pile on the reviewers before they stop doing it?
  - tigroferoce 13y ago
    Good point. But it is a key point that making reviews is part of your job as researched and you MUST take it very seriously. I received so many reviews poorly written, just few lines saying "OK for me". I really think that this way of reviewing is wrong. Reviewers should take care in their job and they should be ready to face a public discussion on why they decided to accept a paper. Note that IMHO is OK to be anonymous while refusing a paper because otherwise a reviewer may be put under lot of pressure.
zokier 13y ago
Maybe the general idea should shift towards papers being more of a opening for discussion, where as reviewers would be privileged to get "first posts".
- arjunnarayan 13y ago
  As any scientist will tell you, this is in fact how we (as academics) do view publication. They are the first step, and definitely not the end goal, of academia.
  - jeltz 13y ago
    There are huge differences between different fields of science. In some publication has become the end goal.
arjunnarayan 13y ago
I'm going to respond to this in the context of academic computer science (which works off conferences), in particular, the systems subfield. YMMV if you extrapolate this to non-CS science and journals, so take this with a grain of salt. The bottom line is I, as a powerless grad student, am _very_ comfortable with the level of reviewing competence I have witnessed (and been a part of). The proposal to deanonymize reviews has a huge flaw: Namely, there's no way I (as a grad student with no political power) will write a critical review of somebody famous if I know I won't remain anonymous. The risk of blowback is too damn high. (And given the small pool, writing style would probably be sufficient to deanonymize, if my entire trail of reviews were public). There's no way I have the balls to write a critical review of a paper by, say, Robert Morris or Larry Peterson, for some prestigious venue such as SOSP, when I'm asked to write a review. Unless I know for sure that there's absolutely no risk whatsoever of being identified. Look. Reviewing is thankless and hard. Anonymity exists so that we can be critical and not risk career suicide. Last year at OSDI I received 9 reviews, and they were all exemplarily thoughtful, even the obligatory "strong reject" one of them. The problem of "petty reviews" _has_ a solution: The program committee is self policing, and the reviews are not anonymous within the PC. (For example, at SOSP this year, there are 28 PC members.) You know that most of your reviews come from this gang of 28. You might have one or two externally solicited experts in addition, but that too is not anonymous _to the PC members_. If there's a culture of meanness, the PC members and PC chairs can fix this by internal policing (and they do).
- jofer 13y ago
  Very good points! I completely agree that there's a lot value in anonymity when reviewing, and I don't think it should go away. Blowback over disagreeing is very, very real, and I've experienced it myself (though it was from presenting something at a conference that disagreed with part of a previous publication, rather than as a result of a review). However, I do think public and optionally non-anonymous reviews could be helpful. (Incidentally, you had _9_ reviewers for a single publication?? Holy cow... In the geosciences, I've never heard of more than 3, and it's usually only 2. That's one thing that would really help in my field. More reviewers results in a broader review, rather than a particular person's pet peeve.)
gizmo686 13y ago
>only the editor knows who wrote [the reviews]. This is not entirely true. I've talked with several people involved in research (math and CS). The ideal is that only the editor knows who wrote the reviews. However many fields do not have that many people who are qualified to review cutting edge results. This means that it is possible to get an idea of who your reviewer is based on how he wrote. Of course, although the people I talked to said they often can figure out who reviewed them, they cann't actually confirm that they are right (and I definitely cannot).
snowwrestler 13y ago
There is a transparent review process for the useful life of a published work. Other scientists can read it, test its conclusions themselves, and publish their own papers supporting or critiquing the original work. If a paper claims a significant result, it will eventually get tested. If it claims an insignificant result, it might never get tested. That's ok, little harm done because it is insignificant.

by jofer 13y ago

pizza_boy 13y ago

> it would also give some recognition to the work that goes into reviewing a paper I think you're spot on with this comment. That's exactly what we're building at publons.com. Check it out.

tigroferoce 13y ago

Completely agree. I've been into computer science research for few years and not knowing who accepted the papers and which were the reviews always bugged me. I've always advocated that the reviews behind an accepted paper SHOULD all be public and publicly distributed along with the paper.

thaumasiotes 13y ago
This is definitely a benefit to the readers, and potentially (though not definitely) to the community as a whole. However, it's much worse for the reviewers. I hear so much about how researchers provide papers to journals for free, and review papers for journals for free, and then the journals (are presumably evil to) charge high fees to anyone who wants to read the papers. How much can we pile on the reviewers before they stop doing it?
- tigroferoce 13y ago
  Good point. But it is a key point that making reviews is part of your job as researched and you MUST take it very seriously. I received so many reviews poorly written, just few lines saying "OK for me". I really think that this way of reviewing is wrong. Reviewers should take care in their job and they should be ready to face a public discussion on why they decided to accept a paper. Note that IMHO is OK to be anonymous while refusing a paper because otherwise a reviewer may be put under lot of pressure.

zokier 13y ago

Maybe the general idea should shift towards papers being more of a opening for discussion, where as reviewers would be privileged to get "first posts".

arjunnarayan 13y ago
As any scientist will tell you, this is in fact how we (as academics) do view publication. They are the first step, and definitely not the end goal, of academia.
- jeltz 13y ago
  There are huge differences between different fields of science. In some publication has become the end goal.

arjunnarayan 13y ago

I'm going to respond to this in the context of academic computer science (which works off conferences), in particular, the systems subfield. YMMV if you extrapolate this to non-CS science and journals, so take this with a grain of salt. The bottom line is I, as a powerless grad student, am _very_ comfortable with the level of reviewing competence I have witnessed (and been a part of). The proposal to deanonymize reviews has a huge flaw: Namely, there's no way I (as a grad student with no political power) will write a critical review of somebody famous if I know I won't remain anonymous. The risk of blowback is too damn high. (And given the small pool, writing style would probably be sufficient to deanonymize, if my entire trail of reviews were public). There's no way I have the balls to write a critical review of a paper by, say, Robert Morris or Larry Peterson, for some prestigious venue such as SOSP, when I'm asked to write a review. Unless I know for sure that there's absolutely no risk whatsoever of being identified. Look. Reviewing is thankless and hard. Anonymity exists so that we can be critical and not risk career suicide. Last year at OSDI I received 9 reviews, and they were all exemplarily thoughtful, even the obligatory "strong reject" one of them. The problem of "petty reviews" _has_ a solution: The program committee is self policing, and the reviews are not anonymous within the PC. (For example, at SOSP this year, there are 28 PC members.) You know that most of your reviews come from this gang of 28. You might have one or two externally solicited experts in addition, but that too is not anonymous _to the PC members_. If there's a culture of meanness, the PC members and PC chairs can fix this by internal policing (and they do).

jofer 13y ago
Very good points! I completely agree that there's a lot value in anonymity when reviewing, and I don't think it should go away. Blowback over disagreeing is very, very real, and I've experienced it myself (though it was from presenting something at a conference that disagreed with part of a previous publication, rather than as a result of a review). However, I do think public and optionally non-anonymous reviews could be helpful. (Incidentally, you had _9_ reviewers for a single publication?? Holy cow... In the geosciences, I've never heard of more than 3, and it's usually only 2. That's one thing that would really help in my field. More reviewers results in a broader review, rather than a particular person's pet peeve.)

gizmo686 13y ago

>only the editor knows who wrote [the reviews]. This is not entirely true. I've talked with several people involved in research (math and CS). The ideal is that only the editor knows who wrote the reviews. However many fields do not have that many people who are qualified to review cutting edge results. This means that it is possible to get an idea of who your reviewer is based on how he wrote. Of course, although the people I talked to said they often can figure out who reviewed them, they cann't actually confirm that they are right (and I definitely cannot).

snowwrestler 13y ago

There is a transparent review process for the useful life of a published work. Other scientists can read it, test its conclusions themselves, and publish their own papers supporting or critiquing the original work. If a paper claims a significant result, it will eventually get tested. If it claims an insignificant result, it might never get tested. That's ok, little harm done because it is insignificant.