3 ms·

How do you get around bias in peer review? Journals use blind reviews so that reviewers feel free to express their real opinions without the risk of damaging th

by tensor 16y ago

How do you get around bias in peer review? Journals use blind reviews so that reviewers feel free to express their real opinions without the risk of damaging their professional relationships. Increasingly, double-blind review is being used as well. This additionally removes the famous author bias. I understand that when double blind reviewing was added to the area of AI research, the distribution of who got things published changed dramatically. I don't see how a circle of trust will help with that. The other part of the problem is getting people who are willing to do thorough reviews. Time is money, and with the massive pressures to publish, it has become somewhat common in CS for professors to hand off reviews to students who may not be qualified. Or alternately, reviews can be too short or not thorough enough. This is really illustrated well in conferences, which ironically are often valued more than journals. The extreme number of submissions coupled with a very limited assessment time and strict page limits leads to some horrible reviews that help no one. Even bad reviews should be useful in the sense that they should point out what is wrong or how to improve the work.

jerf 16y ago
"How do you get around bias in peer review?" It isn't possible. Much better to have multiple intersecting circles of trust than to pretend that there can be one unbiased judgment of a work. And when I say "isn't possible", I mean, "isn't possible". At all. You can't do it. I could all but prove this mathematically with enough time, using terminology and concepts from the machine learning bunch. (Their definition of "bias" is one of the most profound things I learned in my entire 7-year college education.) Double-blind reviews can still be done in the circle-of-trust regimes. Obviously there's a point where the work is unblinded and you have to deal with that, but it hardly strikes me as impossible.

by tensor 16y ago

jerf 16y ago

"How do you get around bias in peer review?" It isn't possible. Much better to have multiple intersecting circles of trust than to pretend that there can be one unbiased judgment of a work. And when I say "isn't possible", I mean, "isn't possible". At all. You can't do it. I could all but prove this mathematically with enough time, using terminology and concepts from the machine learning bunch. (Their definition of "bias" is one of the most profound things I learned in my entire 7-year college education.) Double-blind reviews can still be done in the circle-of-trust regimes. Obviously there's a point where the work is unblinded and you have to deal with that, but it hardly strikes me as impossible.