4 ms·
> What's the solution? ...admit up front that judgements of eminence are often subjective. ...read people's work and evaluate each study or proposal on its meri
by dahart 9y ago
> What's the solution? ...admit up front that judgements of eminence are often subjective. ...read people's work and evaluate each study or proposal on its merits.
> One trick I use to avoid status bias is to keep myself blind to the authors' identities as long as I can — a strategy that many journals in social and personality psychology have also adopted. Once I tried this, I realized just how much I had been using authors' identities as a short cut. Assessing research without this information — knowing that I might be harshly criticizing a famous person's work — is nerve-wracking. But I'm convinced it's the best way to evaluate science.
I'm not a full time academic, but having written and reviewed papers for SIGGRAPH and other journals for many years, I've been thinking about this problem. A lot of people I know are jaded about the paper review process and feel like it's infested with politics. SIGGRAPH has a double-blind review process, but it still has this problem, not knowing the identities of authors or reviewers doesn't solve it entirely. I don't know how much it helps to be double-blind and I don't know how to quantify how much it helps, but it's widely believed to improve things.
One of the problems, even in a double-blind system, is that it's still often fairly easy to deduce the organization that the author came from, especially for groups that have attained some prestige. The research specialty and the language and figures and examples in the paper sometimes make it blindingly obvious who's Ph.D. student wrote it, and almost everyone on the papers committee is more familiar with the various groups and their work and history than I am. As a paper author, I've been able to correctly guess and verify several of my reviewers over the years. Sometimes, reviewers are made aware of the origins of a paper behind closed doors. It's discouraged and ugly, but it still happens.
The "solution" rattling around in my head that I would like to try out is simple:
- Require reviewers to rank a set of pairs of papers against each other. For each pair, pick the paper you'd rather see accepted.
- Send papers to many more reviewers, and randomly. (SIGGRAPH is 5 reviewers assigned via social network, for example. How about 30 random reviewers instead?).
- Ask for less feedback from reviewers, instead relying on greater numbers of ranked vote pairs. This will both reduce the workload on reviewers, and allow more data, to reduce noise and make the result more stable.
Once the reviews are in, a ranking algorithm would relax all the ranked choices and produce a list of papers in order of votes, and the top N would be accepted without further debate.
Having done lots of reviews and participated in discussions over reviews, one of the problems I see is that papers are given absolute scores - they're not compared against each other for score. Some amazing papers end up with a low average score, and some mediocre papers end up with a high average score. Each paper's score ends up in it's own unit system, despite efforts to normalize the range.
Not sure if it'd work, but I've been wanting to try it for a while.