4 ms·

Could we make peer review even more valuable if at least some parts of the review process were eventually made available to the public? I think of this especial

Could we make peer review even more valuable if at least some parts of the review process were eventually made available to the public? I think of this especially when I hear the occasional story of fake articles making it through the process. I can imagine the list of "reviewers & comments" some day becoming as important as the list of citations. Would reviewers be better motivated if they knew that their feedback and requested changes were going to be part of the public record? Or would the whole process degenerate into writing meaningless sound bites for the the scholarly equivalent of dust jacket blurbs?

gjm11 14y ago
So now you're a young researcher and you receive a manuscript for review. It's no good and you pan it mercilessly in review. Then your review gets published, and it just so happens that the author of the paper is a senior person at the university where you apply for your next job. Oops. So more-junior academics might start being reluctant to recommend rejection of any paper. Or, if they are told who the author is (which they generally are), reluctant to recommend rejection of any paper whose author might ever be important to their career. These seem like really bad incentives.
- krutulis 14y ago
  The idea that appraisal of academic work must be made in secret does not ring true. Perhaps universities have degenerated into political posturing over pursuit of knowledge, but even this seems like an argument for opening the process in order to reform it. The young researcher should not have to be the one to tell the senior researcher the work is crap. This is why the process is called "peer" review. Of course, the senior researcher should not be submitting crap, and should be told bluntly if she or he does. But perhaps I'm missing the point and "peer review" is now doublespeak for the process of editing a paper. In this case, the academic publishing racket appears even more absurd.
  - jacquesm 14y ago
    Academia is rife with politics. I've seen some of this up close through an ex girlfriend and I was frankly shocked at the pettiness and the vindictiveness of it all. Furthering science didn't seem to be at the forefront of the minds of those participating, it seemed to be mostly about ego. This was in the field of biology, possibly other fields are better / worse.
    - arethuza 14y ago
      Hence the old saying that: "The politics of the university are so intense because the stakes are so low" Certainly in my six years working in academia I was surprised at the long term vendettas and bitterness that seemed to be common in what was, in reality, really a rather pleasant working environment.
    - impendia 14y ago
      I am an academic mathematician. Happily, I have observed little of what you described. Ego, sure, but pettiness and vindictiveness, no.
      - jacquesm 14y ago
        Math may very well be a huge exception to this because all mathematicians can test their proofs in an objective way rather than as partial opinions or bits of understanding of a greater whole. It's a discovery process with 'true' or 'false' embedded right into the core of the science.
    - javajosh 14y ago
      I'd like to get a better sense of the kinds of conflicts you observed. Did people get upset because someone panned their article? Or would they be upset if a colleague panned a friend's article? What triggered the pettiness and how was it expressed?
  - deleted 14y ago
    [deleted]
- scott_s 14y ago
  This is a valid point, but in my experience, technical people can't help from criticizing things. The typical computer science conference review process is single-blind: authors do not know the reviewers, but reviewers know who the authors are. Some CS conferences now are doing double blind: reviewers also do not know who the authors are. Personally, I would prefer reverse single blind: authors know who the reviewers are, but reviewers do not know who the authors are. Why? I think it's good to try prevent reviewers from being biased (either way) by names or institutions. But I think that reviewer anonymity has lead to some really crummy reviews; people don't read papers carefully (if they read the whole thing at all) and throw out one paragraph or two. I think that if reviewers had to sign their name of the bottom of their review, they would do a better job. I try to adhere to that when I review a paper: would I feel embarrassed if my name was attached to this?
  - stared 14y ago
    It's not that easy. From papers cited it may be easy to deduce who is the author. Typically, there are projects resulting in many papers, not single ones - so it is natural, and desired, to refer to previous works. Additionally, it may be easier to find if an author is not committing self-plagiarism (sending the almost the same thing to a few different places). More on that topic: http://academia.stackexchange.com/questions/625/open-versus-blind-reviewing-process http://academia.stackexchange.com/questions/625/open-versus-...
    - scott_s 14y ago
      That is also a problem with double-blind reviews, and I'm quite familiar with the problems (and annoyances) of anonymizing a paper. However, my main point was about making the reviewers sign their name. I am unaware of a CS community that does so.
  - cwzwarich 14y ago
    Double blind isn't really anonymous. It's pretty easy to tell who the author is from subject, citations, and writing style.
mhluongo 14y ago
There's been a recent push for optionally open reviews- I know some of the PLoS properties allow it.

gjm11 14y ago

So now you're a young researcher and you receive a manuscript for review. It's no good and you pan it mercilessly in review. Then your review gets published, and it just so happens that the author of the paper is a senior person at the university where you apply for your next job. Oops. So more-junior academics might start being reluctant to recommend rejection of any paper. Or, if they are told who the author is (which they generally are), reluctant to recommend rejection of any paper whose author might ever be important to their career. These seem like really bad incentives.

krutulis 14y ago
The idea that appraisal of academic work must be made in secret does not ring true. Perhaps universities have degenerated into political posturing over pursuit of knowledge, but even this seems like an argument for opening the process in order to reform it. The young researcher should not have to be the one to tell the senior researcher the work is crap. This is why the process is called "peer" review. Of course, the senior researcher should not be submitting crap, and should be told bluntly if she or he does. But perhaps I'm missing the point and "peer review" is now doublespeak for the process of editing a paper. In this case, the academic publishing racket appears even more absurd.
- jacquesm 14y ago
  Academia is rife with politics. I've seen some of this up close through an ex girlfriend and I was frankly shocked at the pettiness and the vindictiveness of it all. Furthering science didn't seem to be at the forefront of the minds of those participating, it seemed to be mostly about ego. This was in the field of biology, possibly other fields are better / worse.
  - arethuza 14y ago
    Hence the old saying that: "The politics of the university are so intense because the stakes are so low" Certainly in my six years working in academia I was surprised at the long term vendettas and bitterness that seemed to be common in what was, in reality, really a rather pleasant working environment.
  - impendia 14y ago
    I am an academic mathematician. Happily, I have observed little of what you described. Ego, sure, but pettiness and vindictiveness, no.
    - jacquesm 14y ago
      Math may very well be a huge exception to this because all mathematicians can test their proofs in an objective way rather than as partial opinions or bits of understanding of a greater whole. It's a discovery process with 'true' or 'false' embedded right into the core of the science.
  - javajosh 14y ago
    I'd like to get a better sense of the kinds of conflicts you observed. Did people get upset because someone panned their article? Or would they be upset if a colleague panned a friend's article? What triggered the pettiness and how was it expressed?
- deleted 14y ago
  [deleted]
scott_s 14y ago
This is a valid point, but in my experience, technical people can't help from criticizing things. The typical computer science conference review process is single-blind: authors do not know the reviewers, but reviewers know who the authors are. Some CS conferences now are doing double blind: reviewers also do not know who the authors are. Personally, I would prefer reverse single blind: authors know who the reviewers are, but reviewers do not know who the authors are. Why? I think it's good to try prevent reviewers from being biased (either way) by names or institutions. But I think that reviewer anonymity has lead to some really crummy reviews; people don't read papers carefully (if they read the whole thing at all) and throw out one paragraph or two. I think that if reviewers had to sign their name of the bottom of their review, they would do a better job. I try to adhere to that when I review a paper: would I feel embarrassed if my name was attached to this?
- stared 14y ago
  It's not that easy. From papers cited it may be easy to deduce who is the author. Typically, there are projects resulting in many papers, not single ones - so it is natural, and desired, to refer to previous works. Additionally, it may be easier to find if an author is not committing self-plagiarism (sending the almost the same thing to a few different places). More on that topic: http://academia.stackexchange.com/questions/625/open-versus-blind-reviewing-process http://academia.stackexchange.com/questions/625/open-versus-...
  - scott_s 14y ago
    That is also a problem with double-blind reviews, and I'm quite familiar with the problems (and annoyances) of anonymizing a paper. However, my main point was about making the reviewers sign their name. I am unaware of a CS community that does so.
- cwzwarich 14y ago
  Double blind isn't really anonymous. It's pretty easy to tell who the author is from subject, citations, and writing style.

mhluongo 14y ago

There's been a recent push for optionally open reviews- I know some of the PLoS properties allow it.