26 ms·

Peer Review

188 points by haltingproblem 6y ago

simonkafan 6y ago
Peer review is broken. More than once I've learned that papers get easier through peer review if you cite the paper of a reviewer (of course you don't know in advance who reviews it so it's basically luck). I often got comments about missing related work "from relevant authors".
- _emacsomancer_ 6y ago
  I'm not sure that's exactly the reason it's broken. Usually an editor will give a paper to people who are specialists on your topic, and they will have thus likely written papers related to the topic as well. One major issue is that, because of this, they will sometimes have a vested interest (conscious or not) in blocking the publication of your paper if it shows flaws in their own.
  - jhrmnn 6y ago
    Many journals I'm familiar with allow you to suggest referees, as well as suggest which researchers should be avoided as referees. If I'm publishing something that is competing in some sense with the work of other people, I will usually suggest to the editor to leave them out from the peer review process.
    - grodeni 6y ago
      And now you're ability to exclude reviewers is a significant factor in the acceptation of your paper... Not sure if this is really what we should whish for.
    - _emacsomancer_ 6y ago
      This has only been occasionally available in my experience, and itself introduces perhaps worse problems. If I'm explicitly arguing against someone's theory, they are likely to be a relevant judge of the merits/demerits of the arguments. Having a good editor who can discern the difference between a reviewer's relevant responses and 'turf warfare' is probably best one can wish for.
- goostavos 6y ago
  The quickest path towards realize peer review means almost nothing is... reading peer reviewed papers. Of course, take my semi-anonymous random internet opinion with a grain of salt (I've dropped out of college not once, but _twice_, so what do I know?). However, I'm moderately good at computer science topics, and have spent the last year reading and implementing papers in detail in pursuit of a fairly niche topic, and, what I was dismayed to find, is that most papers are absolute, complete and utter garbage. Like, honesty embarrassing. And that's before you even get into the fact that _you can't reproduce their results_. Encountering this has been something of a crisis in my entire belief system, to be honest. Although, maybe I've just been exposed to what "real" science looks like: a big, gross, often wrong mess of assumptions about the world that takes a long, long time to correct.
  - sideshowb 6y ago
    I suspect if you think this of most papers you've either stumbled into a really bad cluster, or you've missed something about the purpose of the papers you've read. I would agree that some papers are terrible though, I've seen a few I would never have let through review including in the very top journals. I like the Plos mega journal approach but we can't do away with peer review altogether otherwise what separates science from the conspiracy posts that circulate on Facebook ...
- sideshowb 6y ago
  "From relevant authors" is such a cowardly review. Either give me the citation to the work you want me to cite or keep quiet. So often as a reviewer I have avoided asking for citation of my own work, even if highly relevant, as I perceive that to be an abuse of power. I've also started a comedic collection of terrible reviews I've received, though it took me 15 years to gain the confidence to know when these are an indication of the reviewer's failings, not mine.
  - Veen 6y ago
    > often as a reviewer I have avoided asking for citation of my own work, even if highly relevant, as I perceive that to be an abuse of power. That's noble, but isn't it counter-productive? If you're reviewing a paper whose readers would ultimately benefit from a citation of your work, then would the paper not be improved by citing it?
    - sideshowb 6y ago
      Sure if it's worth breaching anonymity for. Last time this happened I recommended about 5 things one of which was mine. I guess it's the more borderline cases I lose sleep over.
      - papeda 6y ago
        I'm surprised that many people automatically assume "you should cite this paper" means "you should cite my paper". I tell reviewers to cite papers all the time. They're almost never my own papers. They're just relevant papers. I think there is some aspect here of just finding reasons to dislike the reviewer (though I've certainly had reviewers where those reasons were pretty easy to find).
        sideshowb 6y ago
        Agreed, I certainly don't think any request to cite X means the reviewer wrote it. But when there's a vague request to cite more without any indication of what (and sometimes a vague request to cite more on a rather niche topic tangential to the paper being reviewed) I would suspect the reviewer of trying to get their own work mentioned without saying so directly. Because if you know the field well enough to know what literature is missing, then it doesn't take you a moment to paste a couple of citations into your review.
chrisco255 6y ago
Any ideas for how the system might work differently? Could peer review learn anything from the open source community?
- WoodenChair 6y ago
  > Could peer review learn anything from the open source community? I think that's a great question. Imagine papers with pull requests, issues, and additional experiments contributed from anyone with relevant experience. And imagine those experiments being open data and up for third-party verification before publication. That could be powerful.
  - chrisco255 6y ago
    Yes, and a reputation system built in but still democratic enough that any modern-day "Einstein" could break through.
  - Vinnl 6y ago
    An article detailing this and similar ideas can be found here, for those interested: https://doi.org/10.12688/f1000research.12037.3 https://doi.org/10.12688/f1000research.12037.3
- pdonis 6y ago
  The ostensible purpose of peer review is filtering: making sure that the signal to noise ratio in published journals remains high enough for the journals to be useful to other scientists. That logic probably made sense back when journals were the primary means of communicating scientific results. But the Internet has changed the situation drastically. Scientists don't need to wait for a journal to publish things; they just go to arxiv.org. Nor do they need to wait for a journal to filter things through peer review; they have plenty of other ways of judging papers for themselves. So at this point I don't think we need a "system" at all for what peer review used to accomplish.
  - rland 6y ago
    What other ways are you referring to?
BurningFrog 6y ago
Science did well before radical ideas could be voted down by peer review. Would Einstein and Darwin have made it through peer review? I doubt it. What current day people like that are held up by their "peers"? I fear we'll never know.
- DoreenMichele 6y ago
  Would Einstein and Darwin have made it through peer review? I doubt it. Einstein worked in a patent office for years, unable to get a job in academia like he wanted. There were two attempts to prove his Theory of Relativity. They occurred years apart because it involved photographing a total eclipse of the sun and while eclipses typically happen every six months in pairs (one solar and one lunar, though sometimes there are three in a row), most of them are partial eclipses, not total eclipses. It had to be a total eclipse to prove his theory. The first attempt to make the photograph failed. There were only two teams and it was during World War I. One was stopped or arrested or something because they were assumed to be spies. The other failed due to rain. After this failure, Einstein realized there was an error in his math and redid his formula. Had the first attempt at taking photographs succeeded, we might have never heard of the man because the world might have decided he was a quack since his math didn't hold up, so, "obviously," his theory was bunk. He probably had one and only one shot at this unusual peer review process. By the time they made the second attempt, it was something like six or eight teams of photographers trying to get a photograph of the total eclipse of the sun. Only one succeeded. I think the rest were again rained out. Nicely, it was a team that had also tried several years earlier that actually succeeded. Once there was photographic evidence supporting his theory, he became world famous overnight and then was finally able to get a job in academia, IIRC. What current day people like that are held up by their "peers"? I fear we'll never know. Me. But just saying that is likely to get me mocked. No one is trying to prove I know anything. No one wants to peer review (or otherwise check) my ideas. People mostly just want me to shut up and go away.
  - pdonis 6y ago
    > Einstein worked in a patent office for years, unable to get a job in academia like he wanted. Actually, while the patent office was his day job, he was working on his Ph.D. when he published his classic 1905 papers including the one on Special Relativity. He got the Ph. D that same year, and got his first academic appointment three years later, in 1908. He didn't start working on General Relativity in earnest until about 1911 (although he had the initial ideas as early as 1907). > There were two attempts to prove his Theory of Relativity. More precisely, two attempts to verify one particular classic prediction of General Relativity, that of bending of light by the Sun. (The second, in 1919, btw, is now generally agreed to not have actually been a valid proof: the observations weren't actually good enough to see the effect when you take into account the appropriate error bars. But they were accepted at the time, probably in large part because of the influence of Eddington, who spearheaded the process.) A second classic prediction, the extra precession of the perihelion of Mercury, was already known in the 19th century, but was a mystery to physicists until General Relativity explained it. The third classic prediction of General Relativity, gravitational redshift, had to wait until the 1950s before reasonably accurate experiments were done to confirm it. > Once there was photographic evidence supporting his theory, he became world famous overnight and then was finally able to get a job in academia, IIRC. As noted above, he actually started working in academia well before the 1919 eclipse expeditions. But the results of those expeditions were indeed what made him world famous.
deleted 6y ago
[deleted]
haltingproblem 6y ago
Rodney Brooks subsumption architecture revolutionized robotics. Back in the 90s you could replicate his work with Attila and Genghis and recreate hexapod robots for under $1000 which was an pittance for a robot. He was at MIT when he submitted those papers. The churlishness of those reviews is staggering. What if he had given up and gone on to something else?
- p1esk 6y ago
  What if he had given up and gone on to something else? Maybe he would have accomplished even more?
  - haltingproblem 6y ago
    What would have happened if he had moved on to another field? Subsumption architecture would eventually have been discovered by someone else, standing on the shoulders of giants and eventuality of all discoveries etc. Or it might have taken another 50-100 years. Fundamentally unknowable.
- buboard 6y ago
  > What if he had given up and gone on to something else? Science isn't supposed to be easy and it's the faithful ones that thrive. If anything, i wish reviewers were more critical of stuff before it's published.
  - gilbetron 6y ago
    Why shouldn't science be easier? We should work at making the doing of science as easy as possible. Getting ideas out there is a boon, suppressing them is of no value. Great ideas should percolate up because they work well, not because someone that has never tested them doesn't like the author. There is no moral superiority in making ideas harder to propagate. Science itself is tough enough as is.
    - canjobear 6y ago
INGELRII 6y ago
There are two types of peer review. 1. Pre-publication peer review Reviewers examine the work before it's presented to the rest of the people. 2. Post-publication peer review Paper on an online forum where others can read and comment and cite them. Papers can be accepted into journals and go trough peer-review later. Post-publication peer review does not mean that authors can't use pre-publication peer review. Scientists can (and should be encouraged) to ask their colleagues to review papers before posting online to spot errors. Some fields have long tradition to publishing and circulating working papers, manuscripts, drafts, posters etc. years before they submit the final paper for publication and pre-publication peer-review. That works as well.
ChrisMarshallNY 6y ago
I think peer review is quite important. Note the word “peer,” though. I would not peer review papers on robotics or AI, as those are not my forté. I could review papers on some generalizations of software engineering or the Swift programming language. Editing is a different matter. My mother was a scientific editor, and absolutely brutal. She edited a book I wrote on writing software for a certain specialty of Web site, and it was quite humbling. She knew nothing at all about the technology, yet provided an immensely valuable service. The book is long past it’s “sell-by” date, yet stands as maybe the only “perfect” artifact of my life, thanks to her. Then there’s comment threads in venues like this one, which some might equate to “peer review.” There was an old Dilbert comic on Peer review. I don’t think it can be found online, but was in “Build a Better Life by Stealing Office Supplies.” It was entitled “The Joy of Feedback,” and probably applies to comment threads.
higeorge13 6y ago
I don't consider myself an expert in computer networks research but i still have 10 accepted (and ~20 rejected) papers with ~100 citations in journals, conferences and workshops within my 3 years working on R&D. Hence my comments below derive from my non-academical and more development-oriented background. Peer review is completely broken nowadays for the following reasons: -Reviewers might not have relevant expertise to judge whether a paper is good or not. I am not talking about top-notch conferences, mainly lower-tier ones and workshops. I have seen a lot of professors pushing paper reviews to their PhDs, or even students. I was also invited to review a paper as soon as i submitted my first one (and it was rejected btw)! In another workshop i was invited to review papers just because i was a colleague of one of the organisers. -Old-school academia. This might not apply to all fields or all academia, but i have had good papers rejected because they didn't have a lot of math or simulations! My paper was examining an actual implementation of an SDN (software defined networking) protocol or a strategy with platforms and orchestrators which require weeks to setup and implement (Opendaylight, Openstack, OpenMANO, etc) with actual experiments on real users' video activity, yet to be rejected because i didn't provide any simulation. Jeez, novelty does not come only from theory, somebody has to implement and test these things in actual machines.. -Politics. I won't say much on this aspect, other than a colleague of mine had 1000 citations within the same time just because there was a "mafia" (not my words but his) of professors, lecturers, PhDs reviewing and accepting all the group's papers because they were overloading them with each other's references and co-inviting them to each other's conferences and workshops.
- JohnStrangeII 6y ago
  I'm in philosophy, though personally working in formal philosophy, so this is quite different from CS. In my area practically all top journals reject more than 95% of all submissions, and if you get accepted, it takes about a year until the article is published. Many journals get so many submissions that they do not provide any reviews or feedback - not even a paragraph or line - if they reject a paper. Although I have been able to stay a fully paid postdoc for more than 10 years without problems and get decent amount of publications out, I'd agree that the peer review system is broken and seems to be strongly biased in favor of mediocrity over originality. It's particularly bad in fields like philosophy in the humanities, where you could argue that there are no clear-cut criteria for evaluating the content other than merely formal issues. (Philosophy is not a science.) There are lots of people who game the system, I'd say many of my colleagues are doing it, though not (yet) the majority. By gaming the system I mean writing articles they barely endorse themselves, tailored to get past reviews. I've literally had a very successful colleague say "Who cares? It's one more publication!" when I raised doubts about a joint paper, because it involved too much hand-waving and didn't seem to work. Long story short, I have the impression that in the humanities whether you're accepted or not mainly depends on the writing style and on whether you win the "reviewer lottery." One and the same paper can get accepted in one journal and rejected in another, for journals with similar reputation. I also believe that there are certain niche areas in the humanities that are (almost) pseudo-sciences and based on a small group of researchers colluding with each other and passing opportunities for publication to each other. I won't mention the disciplines, though. I'm a bit dismayed that you describe CS in a similar way, I've always thought its much more rigid there and easier to evaluate CS papers. Having said all that, I don't think there is any alternative to the peer review system. Any alternative I've seen so far or come up with myself seems to be even more prone to manipulation and gaming the system. I believe that it's not the peer review system itself, but over-reliance on objective indicators by university administrators that has led to most of the problems. But the humanities have certainly changed to the negative, at least in philosophy I'm certain that many seminal and famous articles from the 70s of last century and earlier would not make it past the peer review today.
albertzeyer 6y ago
This topic comes up again and again. (Here some references from a quick search: https://news.ycombinator.com/item?id=10531374 https://news.ycombinator.com/item?id=10531374 https://news.ycombinator.com/item?id=18523847 https://news.ycombinator.com/item?id=18523847 https://news.ycombinator.com/item?id=8731271 https://news.ycombinator.com/item?id=8731271 https://news.ycombinator.com/item?id=18595074 https://news.ycombinator.com/item?id=18595074 https://news.ycombinator.com/item?id=22251079 https://news.ycombinator.com/item?id=22251079) But there are not many good solutions. Most seem to agree that we should move over to a more open system, like OpenReview, and also that all publications should be open for everyone afterwards, and not behind some paywall. But these are only two aspects of the system, and this is less about the peer reviewing itself. The community already has the trend to publish more and more directly on ArXiv. Just in addition to submitting it to a conference. If the conference peer review filters it out for some unlucky reason, it's still open for everyone to see. That might be already an improvement of what we had before. But probably it's not really the optimum yet. But what is the optimum?
- jseliger 6y ago
  There are good solutions, but existing publishers like Elsevier like the current situation because it makes them rich (https://www.universityofcalifornia.edu/press-room/uc-terminates-subscriptions-worlds-largest-scientific-publisher-push-open-access-publicly https://www.universityofcalifornia.edu/press-room/uc-termina...). Any existing individual academic is strongly incentivized to work within the rotten system, even though most know its pathologies and want to defect—but one person defecting alone will screw up their own career while likely leaving the system intact. The equilibrium is bad even though the solutions aren't complicated. https://news.ycombinator.com/item?id=23291869 https://news.ycombinator.com/item?id=23291869
buboard 6y ago
That's peer review from 3+ decades ago. Things have changed substantially. PIs dont have enough time to review all that stuff, lot of it is delegated to students, reviews can be hasty, with reviewers who may latch on to a specific detail and let the rest pass through. It's not even that they are not trying hard enough - Peer review is not doable in 2020. If you honestly want to scrutinize and improve every detail of a manuscript, you re better off posting it in reddit or twitter. > clamor for double blind anonymous review That's not even possible in most fields. you can easily tell which lab it comes from (you can even often guess who the reviewer is)
specialist 6y ago
I love this topic of peer review. Feels like arguing about methodologies, software QA & test, learning organizations back in the 90s. Peer review and the reproducibility crisis are the absolute bleeding edge of the human experience. Science, governance, transparency, accountability, progress, group decision making. All of it. I encourage all the critics, thought leaders to also consider how to apply these lessons elsewhere. Every thing said about peer review also applies to policy making, product development, news reporting, investigative journalism. Peter Drucker, way back when, stated that management is a technology. Same applies to peer review and reproducibility. For whatever reason, this post prompted me to think "sociology of scientific progress". I encourage critics to also factor in the collective human psychology. Covered by so many books, like the classics Structure of Scientific Revolutions and The Diffusion of Innovation. (Sorry, none of the more contemporary titles, a la Predictably Irrational, are jumping into my head at the moment.)
papeda 6y ago
Peer review is flawed, but I don't think it's broken. For context, I'm a late-year PhD student in machine learning theory, and I submit almost entirely to conferences, which is the norm in my field. To me, the big problem is that there is little incentive for experts to review papers. I write a few papers every year, and I review 5x that number. Slightly more senior people -- postdocs and junior faculty -- may review more like 30-40 papers. Most papers I review are not good papers. This is especially true at the big machine learning conferences like NeurIPS, ICML, and ICLR, where a disconcertingly large fraction (1/3?) of submissions are outright bad. Most of them are at best "meh". So there's not much benefit to the reviewer beyond some notion of community service. I think my ~colleagues (PhD students, postdocs, young faculty) have similar opinions. We've certainly discussed these issues in the past. One positive step NeurIPS has taken is giving "best reviewer" awards that come from with a free registration. That's a nice benefit, even if it won't on its own incentivize good reviews. Some people have suggested outright paying people for reviews at some quasi-competitive rate, maybe $100 per review (note that this is not great -- a good review takes at least several hours if the paper is not terrible). NeurIPS also added a requirement this year that all submitting authors must review, which is an interesting experiment. My impractical wish is that we also have some kind of "worst reviewer" sticker for all the reviewers who just paraphrase the abstract, add some milquetoast feelings on novelty, and weak reject. Less facetiously, some kind of StackOverflow reputation system might be useful. As is there's so little external signal about being a good or bad reviewer that the incentives are pretty wrong. Some kind of points system might help. But my overall point might be: reviewers by and large aren't malicious, jealous people who are only rejecting your work because they want to protect their own turf. They are more likely good faith actors who are trying to balance time spent on a pretty thankless but societally useful job (reviewing) with time spent on much more personally enjoyable and rewarding activities (research, family, hobbies, etc). There aren't really obvious solutions to tilt the balance toward better reviews that can still handle the volume of papers.
- JoeAltmaier 6y ago
  Dunno about paying. Remember Amazon Turk? Folks just spammed it with zero-effort submissions and 'closed' thousands of contracts with bots. Instantly gamed, so some jerk could walk off with thousands, $1 at a time. The shaming idea, that could work.
guptaneil 6y ago
Peer review is something I've been thinking about a lot lately as an outsider. I've never gone through a formal peer review process or worked in academia, but it's pretty obvious that the process has significant problems. However, the problem that peer reviews are trying to solve (conferring legitimacy on a paper by the scientific community) is so important that we can't give up on it. Sure, pre-print servers mean science can move faster, but it also means anything can be "science." If we normalize publishing papers as preprints, there's very little stopping conspiracy theorists from publishing a paper and having it look plausible. The sudden rush of media coverage for COVID research has really exasperated the problem. Journalists should responsibly cover preprint studies, but frankly most don't, and who can blame them when even the scientific community is struggling to review the papers itself. I volunteer with a group[0] that's trying to address this problem in a small way by creating visual summaries/reviews of popular preprints for mainstream readers. This is a side project, so the review process is just a shared Google doc for each paper, but what I'd like to see: As a reviewer: * View diffs between paper revisions * Comment on papers inline, like a pull request * +1/-1 a submission, or just leave questions without a decision * Gain "karma" for good reviews to build my reputation in the community (like Stack OverFlow) As a paper author: * Reply to comments to understand and explain context * Post revisions As a reader: * See visual summaries of papers, with inlined reviews (like Research Explained) * Vote on both studies and reviews to surface the good ones * Browse profiles of authors and reviewers to understand their qualifications/contributions I do think https://distill.pub https://distill.pub is on the right track. Their papers are significantly more visual/readable than most, and they require contributors to open a pull request against the distill repo. However, their process isn't approachable for non-technical fields. [0]: https://www.researchexplained.org https://www.researchexplained.org
- jkhdigital 6y ago
  I have only the slightest introduction to actual peer review (pre-qual computer science PhD student) but I have to say that my views have shifted dramatically after moving from “outsider” to “insider”. The problems you mention seem more about how the public consumes and interprets scientific research, rather than how scientific communities share knowledge and keep their fields moving forward. These are two different challenges and it seems unreasonable to expect the peer review process to solve both of them.
haltingproblem 6y ago
Brooks initial papers like "Intelligence without representation", "Elephants don't place Chess" and robots like Ghengis and Attila changed the robotics landscape. It was a leap akin to going from a Mainframes to an Apple II. Brooks was extraordinarily well placed to lead this switch in Robotics. He was at MIT, a place with very talented engineers - mechanical, electrical - etc. who could work on the robots. MIT also had limitless funding and a culture for disruptive research with long payoffs. This might not have been possible at other places like CMU where robotics has more industrial applications. I remember watching the movies of Atilla and Genghis over and over again, downloaded over an FTP connection! https://www.youtube.com/watch?v=-6piNZBFdfY https://www.youtube.com/watch?v=-6piNZBFdfY
LockAndLol 6y ago
How is it that peer review hasn't changed in 30 years? With the amount of people afflicted has nobody had the time and/or influence to improve the process? There are no doubt lots of smart people concerned by this...

Peer review is broken. More than once I've learned that papers get easier through peer review if you cite the paper of a reviewer (of course you don't know in advance who reviews it so it's basically luck). I often got comments about missing related work "from relevant authors".

_emacsomancer_ 6y ago
I'm not sure that's exactly the reason it's broken. Usually an editor will give a paper to people who are specialists on your topic, and they will have thus likely written papers related to the topic as well. One major issue is that, because of this, they will sometimes have a vested interest (conscious or not) in blocking the publication of your paper if it shows flaws in their own.
- jhrmnn 6y ago
  Many journals I'm familiar with allow you to suggest referees, as well as suggest which researchers should be avoided as referees. If I'm publishing something that is competing in some sense with the work of other people, I will usually suggest to the editor to leave them out from the peer review process.
  - grodeni 6y ago
    And now you're ability to exclude reviewers is a significant factor in the acceptation of your paper... Not sure if this is really what we should whish for.
  - _emacsomancer_ 6y ago
    This has only been occasionally available in my experience, and itself introduces perhaps worse problems. If I'm explicitly arguing against someone's theory, they are likely to be a relevant judge of the merits/demerits of the arguments. Having a good editor who can discern the difference between a reviewer's relevant responses and 'turf warfare' is probably best one can wish for.
goostavos 6y ago
The quickest path towards realize peer review means almost nothing is... reading peer reviewed papers. Of course, take my semi-anonymous random internet opinion with a grain of salt (I've dropped out of college not once, but _twice_, so what do I know?). However, I'm moderately good at computer science topics, and have spent the last year reading and implementing papers in detail in pursuit of a fairly niche topic, and, what I was dismayed to find, is that most papers are absolute, complete and utter garbage. Like, honesty embarrassing. And that's before you even get into the fact that _you can't reproduce their results_. Encountering this has been something of a crisis in my entire belief system, to be honest. Although, maybe I've just been exposed to what "real" science looks like: a big, gross, often wrong mess of assumptions about the world that takes a long, long time to correct.
- sideshowb 6y ago
  I suspect if you think this of most papers you've either stumbled into a really bad cluster, or you've missed something about the purpose of the papers you've read. I would agree that some papers are terrible though, I've seen a few I would never have let through review including in the very top journals. I like the Plos mega journal approach but we can't do away with peer review altogether otherwise what separates science from the conspiracy posts that circulate on Facebook ...
sideshowb 6y ago
"From relevant authors" is such a cowardly review. Either give me the citation to the work you want me to cite or keep quiet. So often as a reviewer I have avoided asking for citation of my own work, even if highly relevant, as I perceive that to be an abuse of power. I've also started a comedic collection of terrible reviews I've received, though it took me 15 years to gain the confidence to know when these are an indication of the reviewer's failings, not mine.
- Veen 6y ago
  > often as a reviewer I have avoided asking for citation of my own work, even if highly relevant, as I perceive that to be an abuse of power. That's noble, but isn't it counter-productive? If you're reviewing a paper whose readers would ultimately benefit from a citation of your work, then would the paper not be improved by citing it?
  - sideshowb 6y ago
    Sure if it's worth breaching anonymity for. Last time this happened I recommended about 5 things one of which was mine. I guess it's the more borderline cases I lose sleep over.
    - papeda 6y ago
      I'm surprised that many people automatically assume "you should cite this paper" means "you should cite my paper". I tell reviewers to cite papers all the time. They're almost never my own papers. They're just relevant papers. I think there is some aspect here of just finding reasons to dislike the reviewer (though I've certainly had reviewers where those reasons were pretty easy to find).
      - sideshowb 6y ago
        Agreed, I certainly don't think any request to cite X means the reviewer wrote it. But when there's a vague request to cite more without any indication of what (and sometimes a vague request to cite more on a rather niche topic tangential to the paper being reviewed) I would suspect the reviewer of trying to get their own work mentioned without saying so directly. Because if you know the field well enough to know what literature is missing, then it doesn't take you a moment to paste a couple of citations into your review.

chrisco255 6y ago

Any ideas for how the system might work differently? Could peer review learn anything from the open source community?

WoodenChair 6y ago
> Could peer review learn anything from the open source community? I think that's a great question. Imagine papers with pull requests, issues, and additional experiments contributed from anyone with relevant experience. And imagine those experiments being open data and up for third-party verification before publication. That could be powerful.
- chrisco255 6y ago
  Yes, and a reputation system built in but still democratic enough that any modern-day "Einstein" could break through.
- Vinnl 6y ago
  An article detailing this and similar ideas can be found here, for those interested: https://doi.org/10.12688/f1000research.12037.3 https://doi.org/10.12688/f1000research.12037.3
pdonis 6y ago
The ostensible purpose of peer review is filtering: making sure that the signal to noise ratio in published journals remains high enough for the journals to be useful to other scientists. That logic probably made sense back when journals were the primary means of communicating scientific results. But the Internet has changed the situation drastically. Scientists don't need to wait for a journal to publish things; they just go to arxiv.org. Nor do they need to wait for a journal to filter things through peer review; they have plenty of other ways of judging papers for themselves. So at this point I don't think we need a "system" at all for what peer review used to accomplish.
- rland 6y ago
  What other ways are you referring to?

BurningFrog 6y ago

Science did well before radical ideas could be voted down by peer review. Would Einstein and Darwin have made it through peer review? I doubt it. What current day people like that are held up by their "peers"? I fear we'll never know.

DoreenMichele 6y ago
Would Einstein and Darwin have made it through peer review? I doubt it. Einstein worked in a patent office for years, unable to get a job in academia like he wanted. There were two attempts to prove his Theory of Relativity. They occurred years apart because it involved photographing a total eclipse of the sun and while eclipses typically happen every six months in pairs (one solar and one lunar, though sometimes there are three in a row), most of them are partial eclipses, not total eclipses. It had to be a total eclipse to prove his theory. The first attempt to make the photograph failed. There were only two teams and it was during World War I. One was stopped or arrested or something because they were assumed to be spies. The other failed due to rain. After this failure, Einstein realized there was an error in his math and redid his formula. Had the first attempt at taking photographs succeeded, we might have never heard of the man because the world might have decided he was a quack since his math didn't hold up, so, "obviously," his theory was bunk. He probably had one and only one shot at this unusual peer review process. By the time they made the second attempt, it was something like six or eight teams of photographers trying to get a photograph of the total eclipse of the sun. Only one succeeded. I think the rest were again rained out. Nicely, it was a team that had also tried several years earlier that actually succeeded. Once there was photographic evidence supporting his theory, he became world famous overnight and then was finally able to get a job in academia, IIRC. What current day people like that are held up by their "peers"? I fear we'll never know. Me. But just saying that is likely to get me mocked. No one is trying to prove I know anything. No one wants to peer review (or otherwise check) my ideas. People mostly just want me to shut up and go away.
- pdonis 6y ago
  > Einstein worked in a patent office for years, unable to get a job in academia like he wanted. Actually, while the patent office was his day job, he was working on his Ph.D. when he published his classic 1905 papers including the one on Special Relativity. He got the Ph. D that same year, and got his first academic appointment three years later, in 1908. He didn't start working on General Relativity in earnest until about 1911 (although he had the initial ideas as early as 1907). > There were two attempts to prove his Theory of Relativity. More precisely, two attempts to verify one particular classic prediction of General Relativity, that of bending of light by the Sun. (The second, in 1919, btw, is now generally agreed to not have actually been a valid proof: the observations weren't actually good enough to see the effect when you take into account the appropriate error bars. But they were accepted at the time, probably in large part because of the influence of Eddington, who spearheaded the process.) A second classic prediction, the extra precession of the perihelion of Mercury, was already known in the 19th century, but was a mystery to physicists until General Relativity explained it. The third classic prediction of General Relativity, gravitational redshift, had to wait until the 1950s before reasonably accurate experiments were done to confirm it. > Once there was photographic evidence supporting his theory, he became world famous overnight and then was finally able to get a job in academia, IIRC. As noted above, he actually started working in academia well before the 1919 eclipse expeditions. But the results of those expeditions were indeed what made him world famous.

deleted 6y ago

[deleted]

haltingproblem 6y ago

Rodney Brooks subsumption architecture revolutionized robotics. Back in the 90s you could replicate his work with Attila and Genghis and recreate hexapod robots for under $1000 which was an pittance for a robot. He was at MIT when he submitted those papers. The churlishness of those reviews is staggering. What if he had given up and gone on to something else?

p1esk 6y ago
What if he had given up and gone on to something else? Maybe he would have accomplished even more?
- haltingproblem 6y ago
  What would have happened if he had moved on to another field? Subsumption architecture would eventually have been discovered by someone else, standing on the shoulders of giants and eventuality of all discoveries etc. Or it might have taken another 50-100 years. Fundamentally unknowable.
buboard 6y ago
> What if he had given up and gone on to something else? Science isn't supposed to be easy and it's the faithful ones that thrive. If anything, i wish reviewers were more critical of stuff before it's published.
- gilbetron 6y ago
  Why shouldn't science be easier? We should work at making the doing of science as easy as possible. Getting ideas out there is a boon, suppressing them is of no value. Great ideas should percolate up because they work well, not because someone that has never tested them doesn't like the author. There is no moral superiority in making ideas harder to propagate. Science itself is tough enough as is.
  - canjobear 6y ago

INGELRII 6y ago

There are two types of peer review. 1. Pre-publication peer review Reviewers examine the work before it's presented to the rest of the people. 2. Post-publication peer review Paper on an online forum where others can read and comment and cite them. Papers can be accepted into journals and go trough peer-review later. Post-publication peer review does not mean that authors can't use pre-publication peer review. Scientists can (and should be encouraged) to ask their colleagues to review papers before posting online to spot errors. Some fields have long tradition to publishing and circulating working papers, manuscripts, drafts, posters etc. years before they submit the final paper for publication and pre-publication peer-review. That works as well.

ChrisMarshallNY 6y ago

I think peer review is quite important. Note the word “peer,” though. I would not peer review papers on robotics or AI, as those are not my forté. I could review papers on some generalizations of software engineering or the Swift programming language. Editing is a different matter. My mother was a scientific editor, and absolutely brutal. She edited a book I wrote on writing software for a certain specialty of Web site, and it was quite humbling. She knew nothing at all about the technology, yet provided an immensely valuable service. The book is long past it’s “sell-by” date, yet stands as maybe the only “perfect” artifact of my life, thanks to her. Then there’s comment threads in venues like this one, which some might equate to “peer review.” There was an old Dilbert comic on Peer review. I don’t think it can be found online, but was in “Build a Better Life by Stealing Office Supplies.” It was entitled “The Joy of Feedback,” and probably applies to comment threads.

higeorge13 6y ago

I don't consider myself an expert in computer networks research but i still have 10 accepted (and ~20 rejected) papers with ~100 citations in journals, conferences and workshops within my 3 years working on R&D. Hence my comments below derive from my non-academical and more development-oriented background. Peer review is completely broken nowadays for the following reasons: -Reviewers might not have relevant expertise to judge whether a paper is good or not. I am not talking about top-notch conferences, mainly lower-tier ones and workshops. I have seen a lot of professors pushing paper reviews to their PhDs, or even students. I was also invited to review a paper as soon as i submitted my first one (and it was rejected btw)! In another workshop i was invited to review papers just because i was a colleague of one of the organisers. -Old-school academia. This might not apply to all fields or all academia, but i have had good papers rejected because they didn't have a lot of math or simulations! My paper was examining an actual implementation of an SDN (software defined networking) protocol or a strategy with platforms and orchestrators which require weeks to setup and implement (Opendaylight, Openstack, OpenMANO, etc) with actual experiments on real users' video activity, yet to be rejected because i didn't provide any simulation. Jeez, novelty does not come only from theory, somebody has to implement and test these things in actual machines.. -Politics. I won't say much on this aspect, other than a colleague of mine had 1000 citations within the same time just because there was a "mafia" (not my words but his) of professors, lecturers, PhDs reviewing and accepting all the group's papers because they were overloading them with each other's references and co-inviting them to each other's conferences and workshops.

JohnStrangeII 6y ago
I'm in philosophy, though personally working in formal philosophy, so this is quite different from CS. In my area practically all top journals reject more than 95% of all submissions, and if you get accepted, it takes about a year until the article is published. Many journals get so many submissions that they do not provide any reviews or feedback - not even a paragraph or line - if they reject a paper. Although I have been able to stay a fully paid postdoc for more than 10 years without problems and get decent amount of publications out, I'd agree that the peer review system is broken and seems to be strongly biased in favor of mediocrity over originality. It's particularly bad in fields like philosophy in the humanities, where you could argue that there are no clear-cut criteria for evaluating the content other than merely formal issues. (Philosophy is not a science.) There are lots of people who game the system, I'd say many of my colleagues are doing it, though not (yet) the majority. By gaming the system I mean writing articles they barely endorse themselves, tailored to get past reviews. I've literally had a very successful colleague say "Who cares? It's one more publication!" when I raised doubts about a joint paper, because it involved too much hand-waving and didn't seem to work. Long story short, I have the impression that in the humanities whether you're accepted or not mainly depends on the writing style and on whether you win the "reviewer lottery." One and the same paper can get accepted in one journal and rejected in another, for journals with similar reputation. I also believe that there are certain niche areas in the humanities that are (almost) pseudo-sciences and based on a small group of researchers colluding with each other and passing opportunities for publication to each other. I won't mention the disciplines, though. I'm a bit dismayed that you describe CS in a similar way, I've always thought its much more rigid there and easier to evaluate CS papers. Having said all that, I don't think there is any alternative to the peer review system. Any alternative I've seen so far or come up with myself seems to be even more prone to manipulation and gaming the system. I believe that it's not the peer review system itself, but over-reliance on objective indicators by university administrators that has led to most of the problems. But the humanities have certainly changed to the negative, at least in philosophy I'm certain that many seminal and famous articles from the 70s of last century and earlier would not make it past the peer review today.

albertzeyer 6y ago

This topic comes up again and again. (Here some references from a quick search: https://news.ycombinator.com/item?id=10531374 https://news.ycombinator.com/item?id=10531374 https://news.ycombinator.com/item?id=18523847 https://news.ycombinator.com/item?id=18523847 https://news.ycombinator.com/item?id=8731271 https://news.ycombinator.com/item?id=8731271 https://news.ycombinator.com/item?id=18595074 https://news.ycombinator.com/item?id=18595074 https://news.ycombinator.com/item?id=22251079 https://news.ycombinator.com/item?id=22251079) But there are not many good solutions. Most seem to agree that we should move over to a more open system, like OpenReview, and also that all publications should be open for everyone afterwards, and not behind some paywall. But these are only two aspects of the system, and this is less about the peer reviewing itself. The community already has the trend to publish more and more directly on ArXiv. Just in addition to submitting it to a conference. If the conference peer review filters it out for some unlucky reason, it's still open for everyone to see. That might be already an improvement of what we had before. But probably it's not really the optimum yet. But what is the optimum?

jseliger 6y ago
There are good solutions, but existing publishers like Elsevier like the current situation because it makes them rich (https://www.universityofcalifornia.edu/press-room/uc-terminates-subscriptions-worlds-largest-scientific-publisher-push-open-access-publicly https://www.universityofcalifornia.edu/press-room/uc-termina...). Any existing individual academic is strongly incentivized to work within the rotten system, even though most know its pathologies and want to defect—but one person defecting alone will screw up their own career while likely leaving the system intact. The equilibrium is bad even though the solutions aren't complicated. https://news.ycombinator.com/item?id=23291869 https://news.ycombinator.com/item?id=23291869

buboard 6y ago

That's peer review from 3+ decades ago. Things have changed substantially. PIs dont have enough time to review all that stuff, lot of it is delegated to students, reviews can be hasty, with reviewers who may latch on to a specific detail and let the rest pass through. It's not even that they are not trying hard enough - Peer review is not doable in 2020. If you honestly want to scrutinize and improve every detail of a manuscript, you re better off posting it in reddit or twitter. > clamor for double blind anonymous review That's not even possible in most fields. you can easily tell which lab it comes from (you can even often guess who the reviewer is)

specialist 6y ago

I love this topic of peer review. Feels like arguing about methodologies, software QA & test, learning organizations back in the 90s. Peer review and the reproducibility crisis are the absolute bleeding edge of the human experience. Science, governance, transparency, accountability, progress, group decision making. All of it. I encourage all the critics, thought leaders to also consider how to apply these lessons elsewhere. Every thing said about peer review also applies to policy making, product development, news reporting, investigative journalism. Peter Drucker, way back when, stated that management is a technology. Same applies to peer review and reproducibility. For whatever reason, this post prompted me to think "sociology of scientific progress". I encourage critics to also factor in the collective human psychology. Covered by so many books, like the classics Structure of Scientific Revolutions and The Diffusion of Innovation. (Sorry, none of the more contemporary titles, a la Predictably Irrational, are jumping into my head at the moment.)

papeda 6y ago

Peer review is flawed, but I don't think it's broken. For context, I'm a late-year PhD student in machine learning theory, and I submit almost entirely to conferences, which is the norm in my field. To me, the big problem is that there is little incentive for experts to review papers. I write a few papers every year, and I review 5x that number. Slightly more senior people -- postdocs and junior faculty -- may review more like 30-40 papers. Most papers I review are not good papers. This is especially true at the big machine learning conferences like NeurIPS, ICML, and ICLR, where a disconcertingly large fraction (1/3?) of submissions are outright bad. Most of them are at best "meh". So there's not much benefit to the reviewer beyond some notion of community service. I think my ~colleagues (PhD students, postdocs, young faculty) have similar opinions. We've certainly discussed these issues in the past. One positive step NeurIPS has taken is giving "best reviewer" awards that come from with a free registration. That's a nice benefit, even if it won't on its own incentivize good reviews. Some people have suggested outright paying people for reviews at some quasi-competitive rate, maybe $100 per review (note that this is not great -- a good review takes at least several hours if the paper is not terrible). NeurIPS also added a requirement this year that all submitting authors must review, which is an interesting experiment. My impractical wish is that we also have some kind of "worst reviewer" sticker for all the reviewers who just paraphrase the abstract, add some milquetoast feelings on novelty, and weak reject. Less facetiously, some kind of StackOverflow reputation system might be useful. As is there's so little external signal about being a good or bad reviewer that the incentives are pretty wrong. Some kind of points system might help. But my overall point might be: reviewers by and large aren't malicious, jealous people who are only rejecting your work because they want to protect their own turf. They are more likely good faith actors who are trying to balance time spent on a pretty thankless but societally useful job (reviewing) with time spent on much more personally enjoyable and rewarding activities (research, family, hobbies, etc). There aren't really obvious solutions to tilt the balance toward better reviews that can still handle the volume of papers.

JoeAltmaier 6y ago
Dunno about paying. Remember Amazon Turk? Folks just spammed it with zero-effort submissions and 'closed' thousands of contracts with bots. Instantly gamed, so some jerk could walk off with thousands, $1 at a time. The shaming idea, that could work.

guptaneil 6y ago

Peer review is something I've been thinking about a lot lately as an outsider. I've never gone through a formal peer review process or worked in academia, but it's pretty obvious that the process has significant problems. However, the problem that peer reviews are trying to solve (conferring legitimacy on a paper by the scientific community) is so important that we can't give up on it. Sure, pre-print servers mean science can move faster, but it also means anything can be "science." If we normalize publishing papers as preprints, there's very little stopping conspiracy theorists from publishing a paper and having it look plausible. The sudden rush of media coverage for COVID research has really exasperated the problem. Journalists should responsibly cover preprint studies, but frankly most don't, and who can blame them when even the scientific community is struggling to review the papers itself. I volunteer with a group[0] that's trying to address this problem in a small way by creating visual summaries/reviews of popular preprints for mainstream readers. This is a side project, so the review process is just a shared Google doc for each paper, but what I'd like to see: As a reviewer: * View diffs between paper revisions * Comment on papers inline, like a pull request * +1/-1 a submission, or just leave questions without a decision * Gain "karma" for good reviews to build my reputation in the community (like Stack OverFlow) As a paper author: * Reply to comments to understand and explain context * Post revisions As a reader: * See visual summaries of papers, with inlined reviews (like Research Explained) * Vote on both studies and reviews to surface the good ones * Browse profiles of authors and reviewers to understand their qualifications/contributions I do think https://distill.pub https://distill.pub is on the right track. Their papers are significantly more visual/readable than most, and they require contributors to open a pull request against the distill repo. However, their process isn't approachable for non-technical fields. [0]: https://www.researchexplained.org https://www.researchexplained.org

jkhdigital 6y ago
I have only the slightest introduction to actual peer review (pre-qual computer science PhD student) but I have to say that my views have shifted dramatically after moving from “outsider” to “insider”. The problems you mention seem more about how the public consumes and interprets scientific research, rather than how scientific communities share knowledge and keep their fields moving forward. These are two different challenges and it seems unreasonable to expect the peer review process to solve both of them.

haltingproblem 6y ago

Brooks initial papers like "Intelligence without representation", "Elephants don't place Chess" and robots like Ghengis and Attila changed the robotics landscape. It was a leap akin to going from a Mainframes to an Apple II. Brooks was extraordinarily well placed to lead this switch in Robotics. He was at MIT, a place with very talented engineers - mechanical, electrical - etc. who could work on the robots. MIT also had limitless funding and a culture for disruptive research with long payoffs. This might not have been possible at other places like CMU where robotics has more industrial applications. I remember watching the movies of Atilla and Genghis over and over again, downloaded over an FTP connection! https://www.youtube.com/watch?v=-6piNZBFdfY https://www.youtube.com/watch?v=-6piNZBFdfY

LockAndLol 6y ago

How is it that peer review hasn't changed in 30 years? With the amount of people afflicted has nobody had the time and/or influence to improve the process? There are no doubt lots of smart people concerned by this...