4 ms·

Unfortunate as it is, peer review is quite broken and it is not quite clear how to fix it. Even if the intended goal is otherwise, in practice, peer review in C

by srean 12y ago

Unfortunate as it is, peer review is quite broken and it is not quite clear how to fix it. Even if the intended goal is otherwise, in practice, peer review in CS turns into an adversarial exercise, with some differences in levels. Rather than inspecting a submission and thinking if there is anything in the paper that might be of interest to the community, could the paper spur real progress, the review process usually boils down to: can I quickly skim to find something / anything with which I can shoot down this paper and get away with it and get done with this review. This of course is a generalization, not implying that this is what happened here. Unfortunately, the prototypical reviewer has become more like a prosecutor. There are various reasons why this has become the norm, people are aware of it, some are even trying hard to figure out ways to make it better, whereas some believe this is exactly how it should be. If accuracy of prediction was a concern, Copernican model would never have been accepted. It took years of polish to make the then new model attain the accuracy of the then prevalent model. Some would argue that such models (the Copernican for example) should remain unpublished and under covers till they beat state of the art. I personally do not agree with this view. I would rather have a reasonably well baked but not yet perfect model out in the ether soon so that the model can enlist / recruit other people to work on it. It really boils down to where in the spectrum should the community operate: (i) spammed by advances of questionable merit, with the community spending effort to winnow the good from the bad or (ii) making sure that only the rock solid ones get through, at the risk of delaying or extinguishing useful advance and waiting for a rediscovery.

seanmcdirmid 12y ago
I'm guessing the paper was submitted to PLDI or POPL (wordpress is blocked here, so I can't read the context), the tier one PL conferences. The communities are not adversarial so much as they are enamored with science. I think from that perspective, rejection is understandable: they want more numbers and comparisons, and might accept the paper even if these numbers are not very enlightening (call it going through the motions). You also don't get many points just for wild experimentation. For the same reason, many of the papers accepted seem very incremental and disappointing from a "new idea" perspective; and once you get the paper in, you have to go to the conference (I'm not a PLDI/POPL person)! One can always publish in a "new idea" PL conference (like my personal choice, Onward!, and there is even a new one called SNAPL from the people that brought us PLDI and POPL), but the points you get for publishing at those venues aren't as much as the true technical conferences (I don't care, since I have a job, but for a grad student, its a big deal). My advice would be to suck it up and write a good PLDI paper. Be very thorough and honest with numbers and comparisons, and try to do it "right" (no pointless numbers that don't contribute to the story even if they might "work"). Then with a little bit of luck in PC reviewer selection (most PLDI reviewers are reasonable, some are not), the paper would probably get in. And if they do it right, it could be a really good paper that are lacking these days. Actually, that is another point: sometimes rejection is good, because the paper that gets in later is much much better. Patience.
- scott_s 12y ago
  This, I think, is a major weakness of the typical conference publication process, which is one-pass reject or accept. Journals have a much more humane process. Take the potential sentence from a reviewer: "This work has significant novelty, but is lacking a comparison against technique XYZ." For a Tier 1 conference submission, that's a rejection. For a journal, that's a revision request. Of course, in computer science, conferences are where the action is. But, there are attempts to split the difference. VLDB has moved to a more journal acceptance process (http://www.vldb.org/2015/submission-guidelines.html; http://www.vldb.org/2015/submission-guidelines.html; rolling deadline at the start of each month through the year, reviewers can request revisions), which myself and some co-authors are currently going through. It is much more reasonable.
  - seanmcdirmid 12y ago
    SIGPLAN is a conference-oriented community where the journals are not very interesting :). They also seem to be playing with more shepherding, but in my experience, once a paper has been tentatively accepted, it is pretty much going to get in unless the authors royally screw it up. There are also papers that are never going to make it in no matter what, and rolling deadlines can provide some false sense of hope there.

by srean 12y ago

seanmcdirmid 12y ago

I'm guessing the paper was submitted to PLDI or POPL (wordpress is blocked here, so I can't read the context), the tier one PL conferences. The communities are not adversarial so much as they are enamored with science. I think from that perspective, rejection is understandable: they want more numbers and comparisons, and might accept the paper even if these numbers are not very enlightening (call it going through the motions). You also don't get many points just for wild experimentation. For the same reason, many of the papers accepted seem very incremental and disappointing from a "new idea" perspective; and once you get the paper in, you have to go to the conference (I'm not a PLDI/POPL person)! One can always publish in a "new idea" PL conference (like my personal choice, Onward!, and there is even a new one called SNAPL from the people that brought us PLDI and POPL), but the points you get for publishing at those venues aren't as much as the true technical conferences (I don't care, since I have a job, but for a grad student, its a big deal). My advice would be to suck it up and write a good PLDI paper. Be very thorough and honest with numbers and comparisons, and try to do it "right" (no pointless numbers that don't contribute to the story even if they might "work"). Then with a little bit of luck in PC reviewer selection (most PLDI reviewers are reasonable, some are not), the paper would probably get in. And if they do it right, it could be a really good paper that are lacking these days. Actually, that is another point: sometimes rejection is good, because the paper that gets in later is much much better. Patience.

scott_s 12y ago
This, I think, is a major weakness of the typical conference publication process, which is one-pass reject or accept. Journals have a much more humane process. Take the potential sentence from a reviewer: "This work has significant novelty, but is lacking a comparison against technique XYZ." For a Tier 1 conference submission, that's a rejection. For a journal, that's a revision request. Of course, in computer science, conferences are where the action is. But, there are attempts to split the difference. VLDB has moved to a more journal acceptance process (http://www.vldb.org/2015/submission-guidelines.html; http://www.vldb.org/2015/submission-guidelines.html; rolling deadline at the start of each month through the year, reviewers can request revisions), which myself and some co-authors are currently going through. It is much more reasonable.
- seanmcdirmid 12y ago
  SIGPLAN is a conference-oriented community where the journals are not very interesting :). They also seem to be playing with more shepherding, but in my experience, once a paper has been tentatively accepted, it is pretty much going to get in unless the authors royally screw it up. There are also papers that are never going to make it in no matter what, and rolling deadlines can provide some false sense of hope there.