4 ms·

Double-anonymous/blind reviewing is completely standard in most areas of computer science, e.g. networking (SIGCOMM, NSDI, IMC, HotNets, MobiCom, MobiSys), syst

Double-anonymous/blind reviewing is completely standard in most areas of computer science, e.g. networking (SIGCOMM, NSDI, IMC, HotNets, MobiCom, MobiSys), systems (OSDI, SOSP, USENIX ATC, HotOS), security (Usenix Security, S&P), machine learning (NeurIPS and ICML), graphics and HCI (SIGGRAPH and CHI), and at least some top-tier theory conferences (like FOCS). So I think most computer scientists would agree with the article's conclusion that double-anonymous reviewing, while flawed, is better than the alternatives. I don't think I've done a non-anonymous submission in 10 years, and as a reviewer, usually I don't have a strong guess about who wrote the paper (and when I think I know, often I turn out to be wrong). It's a little annoying that this news article in Nature Magazine ignores the longstanding widespread prevalence of this practice in a conference-driven (but... we like to think important) academic discipline. :-) But: (1) I don't think the big benefit of double-anonymous reviewing is that a lousy paper from a Nobel laureate (or, from CMU/Berkeley/MIT/Stanford) isn't let in unfairly. To me the big benefit seems to be that reviewers have to review every paper as if it might be from their friends or a famous person, and consequently have to review each paper with due care and try sincerely to understand its contribution, which maybe they wouldn't do if they knew it's from some random place/author they've never heard of or don't think highly of. I think the way it affects judgment may be more in equalizing time/effort spent to read and understand a paper (and the generosity you give a paper because "maybe" it was written by your friend or somebody you respect), rather than a straight-up bias towards liking whatever the faculty at a famous university are writing about this year. (2) While I do think it's true, and a good thing, that double-anonymous reviewing helps "marginalized groups of authors who often struggle to have their work see the world" as the lead researcher says, we should probably acknowledge that authors are not the only beneficiaries of a scientific publication. The interests of the reader matter too -- the journal or conference has some duty to serve them. On the margin, maybe some readers would be more interested to learn what Albert Einstein is thinking about these days, or would like to see a well-balanced conference program that includes a good talk by a known-provocative speaker, instead of one more random (but adequate!) paper from a nobody. I'm not saying we should give a huge weight to this -- it's fine to make people eat their vegetables, but, I don't think we should act like scientific publication is only to give authors a line on their CV and the readers' preference is 100% irrelevant. A scientific journal shouldn't exclusively serve the authors. (Other kinds of media care way too much about what the reader wants, e.g. Facebook giving you whatever it thinks will keep you clicking things on Facebook, but there is probably a happy medium somewhere.) (3) The challenging frontier may be in grant submission and reviewing, where proposers are typically not anonymous to the reviewers, which surely leads to some biases. I have heard about government programs where they did use double-anonymous reviewing and it seemed weird to me. (Probably this is a situation where track record should matter, yet trying to summarize your own track record while remaining effectively anonymous seems really hard...)

Ma8ee 4y ago
I’m confused about your second point: isn’t the reader the one who gains the most from reading the best papers, independently of who’s the author?
- 9wzYQbTYsAIc 4y ago
  I think the parent commenter is saying in (2) that sometimes readers want to follow a particular author, like some cartoonists prefer to read XKCD rather than Dilbert, even though they generally study all cartoons of their genre. I’ve often seen celebrity authors write a “Letter to the Journal”, for those kind of readers, and I think that that might be the safer way for big names to be recognized. It may give the author space to collaborate with interested parties without necessarily overwhelming the peer-reviewed content. At the same time, some authors are truly just prolific.
- keithwinstein 4y ago
  For the most part, sure, but at the margin I'm not sure the "best paper" can be completely well-defined within its four corners, independent of the author. E.g. the reader might rationally care (to some degree) about what a paper says about the future of the field. If an underfunded non-famous researcher describes marginal preliminary results by pursing some approach, that's one thing, but if my very successful colleagues at Berkeley/Washington/Tsinghua or Google/MSR/Alibaba publish a paper that's like, "We have gotten pretty excited about this technique for a bunch of reasons, some quantified but some more conjectural, or qualitative, or just more difficult to measure so not done yet, but here are some interesting preliminary measurements that should whet your appetite, and because of who we are, our track record of success, and our many grad students/researchers digging for gold, we'll probably find more to report in due time" well, am I wrong to be a little more interested in the second paper on the grounds that I'd like to keep up with the field? (You might also reasonably believe that the job of an editor or program committee is to assemble the most edifying program or issue when considered as an ensemble, rather than each paper individually.) I also don't think it's completely crazy to consider to a tiny degree what the reader "wants," separate from their true interest in an omniscient sense. When you click around on the Internet, do you only read the "best papers"? Probably not. This is similar to why the New York Times (and Hacker News) don't use 90% of their word count to remind readers to eat healthy, stop smoking, maintain friendships and regular activity, sleep regular hours, and give away most of their money to buy malaria nets.

Ma8ee 4y ago

I’m confused about your second point: isn’t the reader the one who gains the most from reading the best papers, independently of who’s the author?

9wzYQbTYsAIc 4y ago
I think the parent commenter is saying in (2) that sometimes readers want to follow a particular author, like some cartoonists prefer to read XKCD rather than Dilbert, even though they generally study all cartoons of their genre. I’ve often seen celebrity authors write a “Letter to the Journal”, for those kind of readers, and I think that that might be the safer way for big names to be recognized. It may give the author space to collaborate with interested parties without necessarily overwhelming the peer-reviewed content. At the same time, some authors are truly just prolific.
keithwinstein 4y ago
For the most part, sure, but at the margin I'm not sure the "best paper" can be completely well-defined within its four corners, independent of the author. E.g. the reader might rationally care (to some degree) about what a paper says about the future of the field. If an underfunded non-famous researcher describes marginal preliminary results by pursing some approach, that's one thing, but if my very successful colleagues at Berkeley/Washington/Tsinghua or Google/MSR/Alibaba publish a paper that's like, "We have gotten pretty excited about this technique for a bunch of reasons, some quantified but some more conjectural, or qualitative, or just more difficult to measure so not done yet, but here are some interesting preliminary measurements that should whet your appetite, and because of who we are, our track record of success, and our many grad students/researchers digging for gold, we'll probably find more to report in due time" well, am I wrong to be a little more interested in the second paper on the grounds that I'd like to keep up with the field? (You might also reasonably believe that the job of an editor or program committee is to assemble the most edifying program or issue when considered as an ensemble, rather than each paper individually.) I also don't think it's completely crazy to consider to a tiny degree what the reader "wants," separate from their true interest in an omniscient sense. When you click around on the Internet, do you only read the "best papers"? Probably not. This is similar to why the New York Times (and Hacker News) don't use 90% of their word count to remind readers to eat healthy, stop smoking, maintain friendships and regular activity, sleep regular hours, and give away most of their money to buy malaria nets.