4 ms·
Anonymity can't fix scientific peer review, it can only replace type I errors with type II errors. Instead of suppressing criticism that ought not to be suppre
by lisper 12y ago
Anonymity can't fix scientific peer review, it can only replace type I errors with type II errors. Instead of suppressing criticism that ought not to be suppressed, anonymity can (and often does) fail to suppress criticism that really ought to be suppressed because it is in fact false and defamatory. And indeed, what this article is really about is a lawsuit that alleges that this kind of error has in fact taken place.
Ironically, the very title of this article is a model of non-scientific thinking that ought to be subject to criticism, but attempts to inoculate itself against criticism by asserting that the subjects of the piece are protagonists "fighting to fix" a broken system. They're not. They're fighting to replace one broken system with a different broken system.
There is no question that scientific peer review is broken and needs to be fixed. But anonymity is not the answer. And holding anonymity up as something that should itself be beyond criticism is certainly not the answer.
- seccess 12y agoI agree completely. In my experience as a PhD student (comp sci), all conferences make reviewers anonymous. As a result, there is very little accountability regarding the reviews you receive for your work. More than once, I have had papers rejected simply because a single reviewer barely read the paper and dismissed it. These kinds of reviews are very frustrating to receive, not only because they failed to understand the basic premises of your paper, but because these reviews contain no useful information on how to make your paper better for future submissions. Of course, the opposite can happen where mediocre works slips through, and the reviewers that allowed that should be held accountable too. Its painful to me that so much of the acceptance process for research papers (in my field at least) is based on luck. Moral of the story: reviewer anonymity is good but it comes at the expensive of accountability.
- javert 12y agoI'm also a computer science PhD student. I don't disagree with anything you said, but I am even more cynical. > Of course, the opposite can happen where mediocre works slips through Slips through? Mediocre work is the norm, or at least close to being the norm. (And, as you say, good work is often rejected.) > Its painful to me that so much of the acceptance process for research papers (in my field at least) is based on luck. I don't think it's based on luck, it's based on politics. Academic science is a racket. There is some luck too, though.
- seccess 12y ago> Slips through? Mediocre work is the norm, or at least close to being the norm. In computer science, this is likely because of the ridiculous number of conferences there are and how eager each one is to fill out their roster. I want to hear of the conference that published no papers this year because none were good enough for publication. > I don't think it's based on luck, it's based on politics. Academic science is a racket. I've heard (more than once) of people who submit a paper, have it rejected, and then get scooped by a professor who was on the PC of the conference that rejected them. There is a very real possibility of work being accepted or rejected purely because of what the reviewer thinks will be most beneficial to their career. So, I think your cynicism is well-placed.
- javert 12y ago> In computer science, this is likely because of the ridiculous number of conferences there are and how eager each one is to fill out their roster. I want to hear of the conference that published no papers this year because none were good enough for publication. Yep. But there is a reason it is this way, which is that to be successul on the academic feeding trough/career path, you have to serve on PC committees and you have to publish a lot of papers. > There is a very real possibility of work being accepted or rejected purely because of what the reviewer thinks will be most beneficial to their career. I think this is right, but it's less often so work can be scooped, and more often about building up your "camp." As a research professor, you need your area of expertise to be popular and you need to get a lot of citations for your work. You want to be a leader in an area that actually matters, where your definition of "actually matters" is actually purely herd mentality: what do other people consider to be important? So you want to reject work that bolsters your camp, cites you, etc., and reject work that poses a threat to the ascendancy of your area of expertise.
- blueMist 12y agoAgree with a lot of what is being said here. Have seen vicious anonymous reviews in CS - as we all have - both due to someone misunderstanding the work but also due to people from "a different camp" simply disliking a given approach. This must be common in other fields as well. Someone said on this thread that anonymity is not a guarantee of quality - completely agree. The move towards "open reviewing" for conferences in CS is very encouraging, however we will see how far it gets.
- tehwalrus 12y agoWe ended up with 5 peer reviewers and an Editor override for one of our papers, instead of the usual 2 (I was the PhD student who actually did the algebra, so I saw all the comments.) In fact, even after so much work to get it published, we now know of two errors in it - neither were pointed out by the reviewers of course. I can't help thinking that that one experience sums up all of your problem cases above :)
- ehurrell 12y agoI've had similar experience, and it can be tough, especially when the lack of interest from a reviewer is obvious. I still joke about the review I got that included the line "English need improving".
- deleted 12y ago[deleted]
- dluan 12y agoDebatable on whether or not you can assume litigation is grounds for defamation. Even if it is true, it's not necessary that anonymity required the scientist to sue. I think the appeal for the case to drop is not to discourage discussion of anonymity in review, but that it isn't discussion. A jury panel that decides to outcome of a paper could make science worse off than through a panel of peers.
- lisper 12y agoThe suit it a red herring. The point is that anonymity makes it more likely that bogus criticism will be heard, just as non-anonymity makes it more likely that valid criticism will not be heard. Anonymity is not a quality filter.
- deleted 12y ago[deleted]
- jessriedel 12y agoAll systems are imperfect, but one imperfect system can be vastly better than another.
- Florin_Andrei 12y agoI fear anonymity will just raise the level of irrelevant noise a whole lot.