4 ms·

I disagree (respectfully). I actually think traditional peer review (2-4 anonymous editors, generally old-boys in their field) is really, really broken. Arxiv

I disagree (respectfully). I actually think traditional peer review (2-4 anonymous editors, generally old-boys in their field) is really, really broken. Arxiv and the like functions just fine without it because you get peer reviewed in the open by everyone; if you matter, people subscribe to your papers and try to poke holes in them or start citing them. Also, if you have a new theory that challenges the old guard it won't make it into the journals; with open discussion, if you can back up your crackpot theories with data and theorems they get accepted. (Remember that the "ether" was an obvious truth in the late 1800's in physics; this is a worse problem in the non-hard "sciences" like sociology.) I think the "peer review" offered by traditional academic publishing process sucks, and the sooner we get away from it the better. The old style peer review USED to work in that it kept complete garbage out of the printed journals that got sent around.

mhluongo 16y ago
I heard a story the other day about a paper trying to make it into CHI, a big conference in human-computer interaction. In CS and its subfields, conferences like CHI tend to be more important than journals (or so I've heard). Anyway, the paper was rejected by reviewers, and the rebuttal wasn't enough to make it into the conference. The paper was stellar, and the author knew it, so he persisted, reasoning that the paper was just too far-reaching for his reviewers. He applied to a portion of the conference called alt.chi, where highly externally reviewed papers can be accepted into the conference, despite being turned down by the review proper. He was able to garner 38 positive named reviews - a shoo-in - and was entered into the conference. On one hand, this is a great story about how the academic community can build processes that correct for conservative or short-sighted peer review. OTO, though, it's interesting to note that if he hadn't had 38 positive named reviews (or some n less than 38, but considered very strong), he wouldn't have been accepted. This is because, historically, named review is messed up- no one would negatively review his paper and attach his or her name. I can't talk too much about the details of arXiv, but I do know that in a named review system, it's easy to both a) discourage negative commentary, and b) discourage younger or less experienced reviewers, who are especially afraid of hurting their reputation against more experienced colleagues. Don't get me wrong- I do think that more open systems can replace the current peer review- but I also believe that double-blind reviews and sometimes the intervention of an editor are important for a coercion-free process. They also make it easier to cut the information clutter- instead of following individual researchers, I can follow a few peer-reviewed outlets I trust. EDIT: I guess a big part of my point is that while it doesn't need to be traditional peer review, some of the aspects of traditional peer review process need to be maintained in the community.
- aufreak3 16y ago
  You make an important point, but public named negative reviews are common in the field of writing fiction. Then apparently, the social setup that writers live in is different from the academic setup where public negative reviews may have "political" consequences for your career. Any comments on that? Is there something that the academic world can borrow from the writers' world to remove the discouraging influences you mention? I haven't thought through this since I don't know much about a writer's world.

mhluongo 16y ago

I heard a story the other day about a paper trying to make it into CHI, a big conference in human-computer interaction. In CS and its subfields, conferences like CHI tend to be more important than journals (or so I've heard). Anyway, the paper was rejected by reviewers, and the rebuttal wasn't enough to make it into the conference. The paper was stellar, and the author knew it, so he persisted, reasoning that the paper was just too far-reaching for his reviewers. He applied to a portion of the conference called alt.chi, where highly externally reviewed papers can be accepted into the conference, despite being turned down by the review proper. He was able to garner 38 positive named reviews - a shoo-in - and was entered into the conference. On one hand, this is a great story about how the academic community can build processes that correct for conservative or short-sighted peer review. OTO, though, it's interesting to note that if he hadn't had 38 positive named reviews (or some n less than 38, but considered very strong), he wouldn't have been accepted. This is because, historically, named review is messed up- no one would negatively review his paper and attach his or her name. I can't talk too much about the details of arXiv, but I do know that in a named review system, it's easy to both a) discourage negative commentary, and b) discourage younger or less experienced reviewers, who are especially afraid of hurting their reputation against more experienced colleagues. Don't get me wrong- I do think that more open systems can replace the current peer review- but I also believe that double-blind reviews and sometimes the intervention of an editor are important for a coercion-free process. They also make it easier to cut the information clutter- instead of following individual researchers, I can follow a few peer-reviewed outlets I trust. EDIT: I guess a big part of my point is that while it doesn't need to be traditional peer review, some of the aspects of traditional peer review process need to be maintained in the community.

aufreak3 16y ago
You make an important point, but public named negative reviews are common in the field of writing fiction. Then apparently, the social setup that writers live in is different from the academic setup where public negative reviews may have "political" consequences for your career. Any comments on that? Is there something that the academic world can borrow from the writers' world to remove the discouraging influences you mention? I haven't thought through this since I don't know much about a writer's world.