4 ms·

I don't consider myself an expert in computer networks research but i still have 10 accepted (and ~20 rejected) papers with ~100 citations in journals, conferen

I don't consider myself an expert in computer networks research but i still have 10 accepted (and ~20 rejected) papers with ~100 citations in journals, conferences and workshops within my 3 years working on R&D. Hence my comments below derive from my non-academical and more development-oriented background. Peer review is completely broken nowadays for the following reasons: -Reviewers might not have relevant expertise to judge whether a paper is good or not. I am not talking about top-notch conferences, mainly lower-tier ones and workshops. I have seen a lot of professors pushing paper reviews to their PhDs, or even students. I was also invited to review a paper as soon as i submitted my first one (and it was rejected btw)! In another workshop i was invited to review papers just because i was a colleague of one of the organisers. -Old-school academia. This might not apply to all fields or all academia, but i have had good papers rejected because they didn't have a lot of math or simulations! My paper was examining an actual implementation of an SDN (software defined networking) protocol or a strategy with platforms and orchestrators which require weeks to setup and implement (Opendaylight, Openstack, OpenMANO, etc) with actual experiments on real users' video activity, yet to be rejected because i didn't provide any simulation. Jeez, novelty does not come only from theory, somebody has to implement and test these things in actual machines.. -Politics. I won't say much on this aspect, other than a colleague of mine had 1000 citations within the same time just because there was a "mafia" (not my words but his) of professors, lecturers, PhDs reviewing and accepting all the group's papers because they were overloading them with each other's references and co-inviting them to each other's conferences and workshops.

JohnStrangeII 6y ago
I'm in philosophy, though personally working in formal philosophy, so this is quite different from CS. In my area practically all top journals reject more than 95% of all submissions, and if you get accepted, it takes about a year until the article is published. Many journals get so many submissions that they do not provide any reviews or feedback - not even a paragraph or line - if they reject a paper. Although I have been able to stay a fully paid postdoc for more than 10 years without problems and get decent amount of publications out, I'd agree that the peer review system is broken and seems to be strongly biased in favor of mediocrity over originality. It's particularly bad in fields like philosophy in the humanities, where you could argue that there are no clear-cut criteria for evaluating the content other than merely formal issues. (Philosophy is not a science.) There are lots of people who game the system, I'd say many of my colleagues are doing it, though not (yet) the majority. By gaming the system I mean writing articles they barely endorse themselves, tailored to get past reviews. I've literally had a very successful colleague say "Who cares? It's one more publication!" when I raised doubts about a joint paper, because it involved too much hand-waving and didn't seem to work. Long story short, I have the impression that in the humanities whether you're accepted or not mainly depends on the writing style and on whether you win the "reviewer lottery." One and the same paper can get accepted in one journal and rejected in another, for journals with similar reputation. I also believe that there are certain niche areas in the humanities that are (almost) pseudo-sciences and based on a small group of researchers colluding with each other and passing opportunities for publication to each other. I won't mention the disciplines, though. I'm a bit dismayed that you describe CS in a similar way, I've always thought its much more rigid there and easier to evaluate CS papers. Having said all that, I don't think there is any alternative to the peer review system. Any alternative I've seen so far or come up with myself seems to be even more prone to manipulation and gaming the system. I believe that it's not the peer review system itself, but over-reliance on objective indicators by university administrators that has led to most of the problems. But the humanities have certainly changed to the negative, at least in philosophy I'm certain that many seminal and famous articles from the 70s of last century and earlier would not make it past the peer review today.
- currymj 6y ago
  sadly, CS is also about the reviewer lottery. however because it is conferences rather than journals, turnaround is much faster at least. and there is a culture of preprints, so having a paper rejected once doesn’t delay getting it out into the world.
- deleted 6y ago
  [deleted]
papeda 6y ago
Your points are mostly fair, though I'll push back on one. For context, I'm a late-year PhD student in ML, but I primarily do theory. > Reviewers might not have relevant expertise to judge whether a paper is good or not... Finding enough quality reviewers is hard. There are so many papers (heck, you put out 30 papers in 3 years!) and there is not much incentive to review. I review maybe 5x more papers than I write each year. It's doubly hard to get good reviews out of senior people. For this reason, I think grad student reviews are not a bad solution. In my experience, grad students puts in more effort and write decent, careful reviews. The very worst, low-effort reviews seem to come from established people who can't be bothered. A lower reviewer bar at workshops is fairly common and reasonable. Workshops are not the main publication venue (at least in my areas), so they mostly just want to filter out papers that are bad rather than only accept papers that are good. > Politics... I can't say too much about this since I primarily work in theory, but my perception from the outside is that the more "lab"-dominated a field is (i.e., accepted papers tend to come from some group with 20 PhD students and 10 postdocs) the more likely there are to be "mafias". Maybe this is a function of the kind of people who want to head up a lab, maybe it's a function of the kind of work that gets produced in factory environments, or maybe it's baseless prejudice from me.
- higeorge13 6y ago
  You are probably right about postgrads. Although i would not consider myself the best reviewer i was really investing time and effort to produce a fair and honest review with the "limited" knowledge of the field i had, and i was probably a better reviewer than "experts" who would produce 1 line reviews or generic comments "add more references". However i would beg for highly-technical reviews, providing harsh theoretical and technical feedback, which i guess would come from more experienced or expert reviewers than postgrads or myself. And by expert i mean people who have knowledge and publications on that specific paper topic, e.g. you might not be able to provide expert feedback on P2P or distributed networks just because you are an expert in QoS (network buzzwords here, please ignore :) ). Which is probably an effect of what you mentioned; the amount of conferences/workshops/journals and papers submitted and lack of incentive from people to review. The ~30 papers were an outcome of collaborative R&D projects with universities and companies, most of those just having my name in the authors and requiring me to quickly read and comment them, instead of spending effort to produce the results presented. From a third person perspective this somehow justifies the 3rd point of my original post on how these mafias are created and maintained. The thing is that i didn't care for citations or more paper invitations because my position or wage didn't depend on them (i was just doing my research for the implementation and the trips in the conferences my papers were accepted) and eventually i am not doing R&D anymore, however i can imagine how unfair this whole setup is for people who want to follow an academic path.
jseliger 6y ago
From the original piece: I don’t have a solution, but I hope my observations here might be interesting to some. I have a partial solution: researchers "publish" papers to arXiv or similar, then "submit" them to the journal, which conducts peer review. The "journal" is a list of links to papers that it has accepted or verified. That way, the paper is available to those who find it useful. If a researcher really thinks the peer reviewers are wrong, he/she can state why, and why they're leaving it up, despite the critiques. Peer-review reports can be kept anonymous but can also be appended to the paper, so that readers can decide for themselves whether the peer reviewers' comments are useful or accurate.
- laretluval 6y ago
  If the paper is on the arXiv then it will not be anonymous to the reviewers.
- currymj 6y ago
  This is how many CS conferences already work (even with public anonymous peer review, go to openreview.net to see it in action). It does seem to be better, although the frustration level for researchers is not really less. Edit: also, in CS there is a norm that reviewers will not go out of their way to figure out if a submitted paper has been previously posted on arXiv. I don't think it's possible to have perfect anonymity without forbidding researchers from publicly talking about work in progress, so this seems like a reasonable compromise.
yqx 6y ago
While peer review has its problems, I have the feeling that the problems that we're talking about here are not primarily caused by peer review per se, but by a set of perverse incentives that surround academia these days. I'm talking about the "publish or perish" culture that is results from the (sometimes) automated metrics based on which promotions are decided, grants are handed out and academic staff is assessed. Isn't it these incentives that can create attitudes like "Who cares? It's one more publication!" (as quoted by another commenter in this thread). And isn't an overwhelming amount of mediocre journal submissions that result from this one of the reasons peer review is under pressure, qualified reviewers are rare and politics are so common?

JohnStrangeII 6y ago

I'm in philosophy, though personally working in formal philosophy, so this is quite different from CS. In my area practically all top journals reject more than 95% of all submissions, and if you get accepted, it takes about a year until the article is published. Many journals get so many submissions that they do not provide any reviews or feedback - not even a paragraph or line - if they reject a paper. Although I have been able to stay a fully paid postdoc for more than 10 years without problems and get decent amount of publications out, I'd agree that the peer review system is broken and seems to be strongly biased in favor of mediocrity over originality. It's particularly bad in fields like philosophy in the humanities, where you could argue that there are no clear-cut criteria for evaluating the content other than merely formal issues. (Philosophy is not a science.) There are lots of people who game the system, I'd say many of my colleagues are doing it, though not (yet) the majority. By gaming the system I mean writing articles they barely endorse themselves, tailored to get past reviews. I've literally had a very successful colleague say "Who cares? It's one more publication!" when I raised doubts about a joint paper, because it involved too much hand-waving and didn't seem to work. Long story short, I have the impression that in the humanities whether you're accepted or not mainly depends on the writing style and on whether you win the "reviewer lottery." One and the same paper can get accepted in one journal and rejected in another, for journals with similar reputation. I also believe that there are certain niche areas in the humanities that are (almost) pseudo-sciences and based on a small group of researchers colluding with each other and passing opportunities for publication to each other. I won't mention the disciplines, though. I'm a bit dismayed that you describe CS in a similar way, I've always thought its much more rigid there and easier to evaluate CS papers. Having said all that, I don't think there is any alternative to the peer review system. Any alternative I've seen so far or come up with myself seems to be even more prone to manipulation and gaming the system. I believe that it's not the peer review system itself, but over-reliance on objective indicators by university administrators that has led to most of the problems. But the humanities have certainly changed to the negative, at least in philosophy I'm certain that many seminal and famous articles from the 70s of last century and earlier would not make it past the peer review today.

currymj 6y ago
sadly, CS is also about the reviewer lottery. however because it is conferences rather than journals, turnaround is much faster at least. and there is a culture of preprints, so having a paper rejected once doesn’t delay getting it out into the world.
deleted 6y ago
[deleted]

papeda 6y ago

Your points are mostly fair, though I'll push back on one. For context, I'm a late-year PhD student in ML, but I primarily do theory. > Reviewers might not have relevant expertise to judge whether a paper is good or not... Finding enough quality reviewers is hard. There are so many papers (heck, you put out 30 papers in 3 years!) and there is not much incentive to review. I review maybe 5x more papers than I write each year. It's doubly hard to get good reviews out of senior people. For this reason, I think grad student reviews are not a bad solution. In my experience, grad students puts in more effort and write decent, careful reviews. The very worst, low-effort reviews seem to come from established people who can't be bothered. A lower reviewer bar at workshops is fairly common and reasonable. Workshops are not the main publication venue (at least in my areas), so they mostly just want to filter out papers that are bad rather than only accept papers that are good. > Politics... I can't say too much about this since I primarily work in theory, but my perception from the outside is that the more "lab"-dominated a field is (i.e., accepted papers tend to come from some group with 20 PhD students and 10 postdocs) the more likely there are to be "mafias". Maybe this is a function of the kind of people who want to head up a lab, maybe it's a function of the kind of work that gets produced in factory environments, or maybe it's baseless prejudice from me.

higeorge13 6y ago
You are probably right about postgrads. Although i would not consider myself the best reviewer i was really investing time and effort to produce a fair and honest review with the "limited" knowledge of the field i had, and i was probably a better reviewer than "experts" who would produce 1 line reviews or generic comments "add more references". However i would beg for highly-technical reviews, providing harsh theoretical and technical feedback, which i guess would come from more experienced or expert reviewers than postgrads or myself. And by expert i mean people who have knowledge and publications on that specific paper topic, e.g. you might not be able to provide expert feedback on P2P or distributed networks just because you are an expert in QoS (network buzzwords here, please ignore :) ). Which is probably an effect of what you mentioned; the amount of conferences/workshops/journals and papers submitted and lack of incentive from people to review. The ~30 papers were an outcome of collaborative R&D projects with universities and companies, most of those just having my name in the authors and requiring me to quickly read and comment them, instead of spending effort to produce the results presented. From a third person perspective this somehow justifies the 3rd point of my original post on how these mafias are created and maintained. The thing is that i didn't care for citations or more paper invitations because my position or wage didn't depend on them (i was just doing my research for the implementation and the trips in the conferences my papers were accepted) and eventually i am not doing R&D anymore, however i can imagine how unfair this whole setup is for people who want to follow an academic path.

jseliger 6y ago

From the original piece: I don’t have a solution, but I hope my observations here might be interesting to some. I have a partial solution: researchers "publish" papers to arXiv or similar, then "submit" them to the journal, which conducts peer review. The "journal" is a list of links to papers that it has accepted or verified. That way, the paper is available to those who find it useful. If a researcher really thinks the peer reviewers are wrong, he/she can state why, and why they're leaving it up, despite the critiques. Peer-review reports can be kept anonymous but can also be appended to the paper, so that readers can decide for themselves whether the peer reviewers' comments are useful or accurate.

laretluval 6y ago
If the paper is on the arXiv then it will not be anonymous to the reviewers.
currymj 6y ago
This is how many CS conferences already work (even with public anonymous peer review, go to openreview.net to see it in action). It does seem to be better, although the frustration level for researchers is not really less. Edit: also, in CS there is a norm that reviewers will not go out of their way to figure out if a submitted paper has been previously posted on arXiv. I don't think it's possible to have perfect anonymity without forbidding researchers from publicly talking about work in progress, so this seems like a reasonable compromise.

yqx 6y ago

While peer review has its problems, I have the feeling that the problems that we're talking about here are not primarily caused by peer review per se, but by a set of perverse incentives that surround academia these days. I'm talking about the "publish or perish" culture that is results from the (sometimes) automated metrics based on which promotions are decided, grants are handed out and academic staff is assessed. Isn't it these incentives that can create attitudes like "Who cares? It's one more publication!" (as quoted by another commenter in this thread). And isn't an overwhelming amount of mediocre journal submissions that result from this one of the reasons peer review is under pressure, qualified reviewers are rare and politics are so common?