10 ms·
Peer review: the end of an error?
- kelnage 9y agoThe initial examples the author uses really do reflect the magnitude of the third problem. All of them were big breakthrough results in their fields - and hence people cared enough to do proper reviews - especially in the latter two cases. As an academic, I have participated in reviews of papers that I would have never otherwise read, but could still contribute to the process. Without something pushing all academics to participate in the system, then a more open solution risks moving science even closer to “only interesting problems/solutions/results get published”.
- aldoushuxley001 9y ago"Only interesting problems/solutions/results get published" I'm afraid we're already there. Journals are very much strongly incentivized to publish only interesting or novel results.
- kelnage 9y agoI agree there is definitely a bias towards such results - but I fear that completely free-for-all open access reviewing will make the effect even greater.
- improbable22 9y agoIsn't this the point of journals? They communicate new knowledge, which changes the status of what is known. There's a balance to be struck in how certain these things should be before being published, for sure. The p-hacking plague is a symptom of setting this certainty bar too low, or equivalently, the required level of novelty too high. Things which stay constant are the job of standards bodies, and textbook writers... these are important too, but different.
- squiggleblaz 9y agoI don't think so. If you work in a hard science like medicine or psychology, where problems are still so hard we can't produce clear answers as to results of a paper, we shouldn't really be trusting papers until they've been reproduced by some unrelated team. But if these results don't contradict the previous papers, they're not really communicating new knowledge. Sometimes this isn't a big deal (medicine is as far as I know regulated enough that reproduction is a part of the problem), but most science is characterised by not enough money distributed without caring about scientific outcomes. So we do have a problem at the moment. A team who does important but thankless scientific work can't get published. Too much science is just first draft status. And the only way to determine that is that people keep trying and trying to build on it and eventually decades later the tower of cards collapses when clever new PhD students start down new research avenues.
- gowld 9y agohttp://www.jnr-eeb.org/index.php/jnr http://www.jnr-eeb.org/index.php/jnr
- woliveirajr 9y ago> Remarkably, we have arrived at a system where academics feel a moral obligation to perform the thankless task of reviewing the work of other academics, anonymously and unpaid. This undoubtedly makes the literature better than it would otherwise have been, and ensures at least one reader for each paper. I'm afraid it doesn't ensure that all papers will be read. Some papers can attract more attention, some papers can even receive too much comments with relevant comments being drown in the sea of observations.
- John_KZ 9y agoExactly. It's the same fallacy with open source code. Just because it could be reviewed, doesn't mean it has been. Formal or semi-formal groups that professionally review papers are essential. How are they going to get paid? If you want research to be publicly available (as it should be) you can either put the reviewers on government dime (which isn't a great idea but it's better than nothing) or you can follow the system that was pretty much followed until now: Research institutions that produce papers to be reviewed, pay up the reviewers, and then we make the research publicly available, because everyone is paid and happy. No need to double-pay the reviewers via per-paper costs etc. The cost of reviewing a paper is fixed. Hosting costs are trivial compared to that. The uni can take them on. In a sense sci-hub fixed the peer-reviewed industry. Formalize it (make all journals open-access) and we're done.
- isolli 9y agoPull-request reviews work the same as peer review: it ensures that at least one person (the reviewer) reads the code or article.
- phalangion 9y agoI think the "one reader" refers to the reviewer, who ostensibly should read as a part of the review process.
- chrisseaton 9y agoThis article says that peer review is unpaid - it isn't. If you are an academic or work in industrial research, it's part of your paid job to do peer review as part of the community. You are paid for it - it's part of your job and it's normal to do it during work hours.
- improbable22 9y agoThe point is that it's done largely out of a feeling of obligation to the community. There's a chance that if you declined all of it you'd get a reputation for being an asshole, but there would be no other consequence for you. Whoever pays your salary would probably not notice. I have actually been paid by some journals, but only a token amount, like $30/paper.
- heckless 9y agoAlthough reviewership is a useful thing that gets put on CVs, I can't think of a position I've ever seen where "review X papers" a year is part of the job requirement. A more salient point about why it is important to note that peer review is unpaid is that academics, as authors, submit their work---for free---to a journal, where reviewers review it for free, and then the journal turns around and sells the article for $35 apiece.
- electricslpnsld 9y ago> I can't think of a position I've ever seen where "review X papers" a year is part of the job requirement. I work in an industry research lab in the bay, and while there isn't a requirement to review X papers a year, it is expected that you keep up with the literature, and actively engaging in peer review is one way to do so. A few of my colleagues are on the paper committee for our major academic conference, and they end up reviewing 40+ papers a year and using a week or so to take part in the committee meetings.
- chrisseaton 9y agoThis is such an adversarial way to look at it. I see it as a community. We write and share our papers as part of our jobs, we review each others' papers as part of our jobs, we help to run the conference as part of our jobs. Everyone contributes to run the system. Nobody buys articles for $35 a piece. All major universities and companies subscribe to the ACM or whatever your field's organisation is for you. And guess who runs the ACM and other organisations? The community does! People from the community serve as leadership and help run things. Yes the ACM gets some money to run their website, conferences and outreach programs. I don't think that's crazy.
- jseliger 9y agoOne hopes the present system will go away: http://sappingattention.blogspot.com/2017/09/peer-review-is-younger-than-you-think.html http://sappingattention.blogspot.com/2017/09/peer-review-is-... The problems are in some ways even worse in the humanities: http://www.chronicle.com/article/The-New-Intellectuals/238354?key=dCUViR9a-EOigReN8KYScI_cuA0Bu5caKtkfgTzhUtPxUY_oOHNXK3YWfjxQpQr4VlF0LURGdjBDZFZTdUQwMUR3VkI5b2lWRW55Q1hicXUyc0pMWDF5eTFZUQ http://www.chronicle.com/article/The-New-Intellectuals/23835... than the sciences. Or see https://jakeseliger.com/2013/02/12/a-lot-of-academic-research-may-barely-count-as-research-by-many-definitions https://jakeseliger.com/2013/02/12/a-lot-of-academic-researc... .
- improbable22 9y agoA point not mentioned in the article is that this formal peer review system is itself quite young. I think that sending every paper out to several reviewers only became the default once we had xerox machines -- 1960s? We did a lot of serious science before that. A bit earlier, the famous anecdote is about Einstein's offence that the Phys Rev editor would dare to send his paper for review at all -- apparently this would not have happened at a German journal of that era: http://www.preposterousuniverse.com/blog/2005/09/16/einstein-vs-physical-review/ http://www.preposterousuniverse.com/blog/2005/09/16/einstein...
- noobermin 9y agoIf anything, that works as a great defense of peer review given Einstein was wrong.
- throwawayyx96 9y agoAuthor of the article is the same guy who, some years ago, organized a boycott of Elsevier journals due to their extortionate charges. https://en.wikipedia.org/wiki/The_Cost_of_Knowledge https://en.wikipedia.org/wiki/The_Cost_of_Knowledge He has practiced what he preaches with respect to publishing in open access journals.
- jwilk 9y agoArchived copy, which can be read without JS enabled: https://archive.is/iPwXi https://archive.is/iPwXi
- jwilk 9y ago(2017)
- probably_wrong 9y agoThere are two points that I wished were addressed more often. 1. New researchers. The examples he mentions include researchers who were already important in the field, and whose prestige guarantees review. But if you're a new researcher, good luck getting noticed. Blind review doesn't have this problem. 2. The cost of journals. People say "journal" and think of Elsevier, but there are several journals and conferences that publish all papers for free. The ACL, the largest conference on computational linguistics, finances itself with the entry fee to their conference, and you can read all of their papers right now (disclaimer: I'm currently doing work for them). There is a middle ground between Elsevier and Arxiv.
- femto 9y agoOne way to look at journals are that they are "ready made" reputation for those without their own. A new author can leverage the reputation of the journal to establish their own reputation. On that basis, maybe the mark of a truly prestigious researcher is one who can post directly to arXiv? Rather than the highest accolade being for a paper in Nature, the highest accolade should be for a widely reviewed paper on arXiv?
- CharlesMerriam2 9y agoAn interesting parallel can be found in the anonymous review of grant proposals. See "Fund ideas, not pedigree, to find fresh insight", https://bit.ly/2Fy64cP https://bit.ly/2Fy64cP. It ends with "peer review may be the worst system, except for all the others". It can be found in Nature, a peer reviewed journal.
- xamuel 9y agoNot to defend peer review as it is now, but one of the things I've come to value from editorial gatekeepers is that they put the brakes on me publishing things that in later years I might regret. I wrote some pretty mediocre papers as a grad student and I'm grateful some of them were rejected, because I wouldn't want them forever part of my legacy. As a matter of fact, I wrote some "papers" when I was a teenager, which, if I had been allowed to freely publish at that time, might have become permanent albatrosses around my neck :P
- danieltillett 9y agoDidn’t your supervisor step in and say this is not up to scratch? When I was a supervisor a good chunk of my time was spent going through my student’s and postdoc’s manuscripts to ensure nothing was in them that might embarrass us both.
- xamuel 9y agoI was reckless and wrote papers unrelated to my dissertation and submitted without going through my advisor. Fortunately, the bad ones were rejected. The good ones ended up making me well-recognized in some extremely tiny niches, and ultimately did way more for my career than my PhD work did :D I'm really glad some of the bad ones didn't see the light of day :P
- danieltillett 9y agommm not wise :) I guess it depends on the field, but in mine (biological sciences) I can’t think of an example where a student would submit a paper without their supervisor knowing. Anyone thinking of doing the same as you should try and draw on their colleagues in the department to proofread the manuscript. I used to proof lots of manuscripts from my colleagues and their students.
- tom_mellior 9y ago> I was reckless and wrote papers unrelated to my dissertation and submitted without going through my advisor. This is the opposite of reckless, this is how you make a name for yourself, and how you prove that you can develop an independent research agenda. As you say, it's good for your career, and it's good for science too (if the work is good, of course). Good for you!
- kolpa 9y agoThe academic peer review debate has strong parallels to the social-media news debate -- As traditional newspapers/ TV channels are no longer gatekeepers of publication, we have given up a level of vetting in exchange for freedom-from-censorship. But that puts an onus on consumers -- it's not right for consumer to accept everything they see with the same level of default trust they had before. Consumers still must "consider the source", and society to create an effective replacement for the editorial filters we are discarding. see also: "web of trust"
- danieltillett 9y agoThe big problem with peer review is almost no reviewing takes place and it is at best is either a way for your enemies to get revenge or a cheap proofreading service. The solution is to offer a largish reward to the most constructive peer review for each manuscript chosen by the editor, or better by other reviewers. If I (or more likely one of my starving grad students) knew that I would get $1000 for providing the most constructive peer review then I would put in the required effort to make peer review a worthwhile exercise. One thing that would help the peer review process is a data dump of the raw data. Many times I have seen some figure or table in a manuscript and felt I could not provide a constructive critic without seeing the underlying data. Going back through the editor to get this data is just too much of a hassle most of the time.
- wyattpeak 9y agoBut who decides what's constructive? It's all very well if you're choosing amongst people who basically agree with you - what if they're telling you it's going to take another year of experiments to get it up to standard? Which reviewers, too, do you respond to? If it's a popular paper you probably can't respond to all criticism, but picking which criticism to interact with isn't going to lead to good science. There's a lot of value, even if it's not perfect, in writers being forced to respond to one or two reviewers' comments.
- danieltillett 9y agoI suggested in the post either the editor or the other reviewers makes the decision of which was the most constructive review. The suggestion of going back and doing more work exists under the current peer review system. With my proposal you might even get suggestions of what work needs to be done and why you should do it. In all the papers I have published I have only had a few reviewers agree on anything. The normal approach is you agree to all the reasonable demands even if you don’t personally like them and argue over the important one. Most of the time if the reviewers see you have moved some of the way towards their arbitrary demands they will let you hold onto the important ones.
- 9y ago
- arca_vorago 9y agoI've said it before and I'll say it again. The focus should be on reproducing studies, not on peer-review. Of course peer review can be of assistance in selecting studies worth putting the effort and funding into reproducing, but peer-review itself is extremely corrupted as a process, but since so many peoples careers depend on it , many of them will not talk about those issues. I used to hold the peer-review and journal process on a pedestal. Then when working at a genetics lab I ended up having to read a lot of papers to get my sysadmin work done, and started realizing just how bad so many papers are, even though published in "reputable" journals. For example, since publishing is seen as so vital to some people, I noticed many of them would "consult" on another groups paper just to get their name on the paper, even though they did almost no work, and then those same people would turn around and say stuff like "I've been published X times!" as a sort of reinforcement of authoritativeness (and as advertisement). People are gaming the system because hardly anyone is actually checking them. Beyond that, the actual peer-review process itself seems to be very lackluster in rigor on it own. I think there should be a certain amount of public research funding that should be earmarked to go directly to independent reproducability labs. On top of it all, all this research that the public is funding is often behind a fucking paywall... and that's not just bullshit, it's theft and misuse of public funds. It should be required that publicly funded research should be publicly published, and therefore available for more eyes to see and review.
- dragonwriter 9y ago> The focus should be on reproducing studies, not on peer-review The two aren't separable, since a key part of peer review is assuring that a published study is, in principle, reproducible. You can't focus on reproduction withoit focussing on peer review.
- arca_vorago 9y agoI agree, I just worded it slightly poorly. My point is that the biggest focus should be on reproducing studies, not that peer-review shouldn't be revamped also. I mean, isn't reproducing a study the ultimate peer-review?
- 9y ago
- jhbadger 9y agoI think the mathematical sciences see peer review as less of a requirement than the experimental -- in theory, anyone can verify a proof for themselves. But in experimental science you really need someone familiar with the techniques to make sure that they did all the needed controls and didn't do any shortcuts and that's what peer review is useful for.
- pmiller2 9y agoI disagree. Even with the current peer review system in mathematics, false proofs have made it into the literature. Who knows how many more would creep in if we did away with peer review?
- tzahola 9y agoOne can argue that false proofs made it into the literature because people trusted the reviewers - compare with: appeal to authority.
- noobermin 9y agoWas going to comment to say the exact same thing. OP used a paper by medical professionals to poo-poo formal peer review and then used math papers as examples of successful arxiv papers scrutinized by other scientists. After realizing that, I stopped reading the article then Googled his name, and--surprise, surprise--he's a mathematician. It will work for mathy and theoretical sciences like CS and theoretical physics, but good luck getting experimentalists to review papers for free (or at least without being asked by a journal). I'm not sure whether peer review is necessary in those cases, but the culture around more applied sciences definitely considers peer review more important, even though we routinely post to arxiv anyway.
- gowld 9y agoIt sounds like you're saying that experimentalists rely on peer review as a way to fraudulently cover up their shoddy experimental methods.
- pizlonator 9y agoI've published in CS conferences, CS journals, and on a blog. Publishing in CS conferences/journals means you get reviews from 3-6 "smart people", and then maybe 50-100 people will read your paper, and most of those people will never contact you. OTOH if you publish a blog and advertise on hackernews and twitter, you get instant feedback, including from some of the same experts that would have done reviews for conferences/journals.
- tincholio 9y agoThe difference is that more often than not, you can't go to the same level of detail and depth in a blog post as you would in a scientific article (and if you do, not that many people will read it). Reading science is hard work, and properly reviewing a "journal length" paper can take upwards of a day. I don't think you can have it both ways. If you write a good vulgarization, or technical blog post, and submit it here, you will likely get many eyeballs. If you post serious science, you'll get much less feedback, if any.
- pizlonator 9y agoMy blog posts typically have more scientific content than a CS paper.
- dredmorbius 9y agoCare to link? Now I'm curious.
- tincholio 9y agoI'd be happy to see some, please link
- wbillingsley 9y agoIn some fields, such as HCI (and research through design) I think there could be value in moving to a system where instead you version papers and also publish the criticisms that led to its refinement. If you've explored a problem, designed a piece of technology, and taken it back to that audience to gain further insights into the problem, what I can do in reviewing it isn't the same kind of analysis as a maths proof. Even if I you were to give me the tech or the code, I wouldn't have the same people to test it with, and would not expect to get exactly the same results. Instead, peer review tends to criticise the writing, missing elements in the argument, pieces of literature that also need considering, aspects of the methodology, and suggest more things you could do. All of which is very valuable, and sometimes peer review is the best (only) mentoring you'll get if you're in a small university. But it's not quite the same as "we have to review this or a wrong claim might get the imprimatur of being science". Design questions don't make the same claims of completeness or context-independence. It's more of an argument based in the kind of muddy evidence you have to put up with when tackling muddy problems. Which makes me think that instead of peer review being a hurdle to publication, perhaps we should just make the whole argument and refinement public. Give the reviewers more kudos as instead of being invisible entities whose thoughts are only received by the author and editor, their role in the development of that idea becomes a published item too.
- dunk010 9y agoI am afraid this piece is missing the most massive reason that articles are still published in journals - Impact Factor. Academics live and die not just by the number of articles that they get published, but, crucially, where they get published. The entire academic system of progression is basically controlled by the journals. The reason that the splashy websites never succeed is that they have targeted the wrong problem. If you want to fix the system then you have to come up with a new way of rating academic's publications, which would then be fed back into the career progression system in their individual institutions. Good luck with that.
- iandanforth 9y agoWhile this is true in many institutions there are now useful proxies, specifically a Google Scholar profile or an individual h-index which can be calculated without publication impact factor. You are correct that it will take a long time to update documentation and policies that reference impact scores, but informally, at least in Machine Learning, people know who the contributors are with high actual impact. In addition because a lot of impactful work is coming out of industry and there is rapid turnover between academia and industry the culture as a whole is evolving.
- dunk010 9y agoOf course there's an untruth in what I said. Ostensibly it's all about Impact Factor, but in reality it's all about how much money you are bringing in. Academia is in a sad state.
- freech 9y agoReddit but only academics can participate? Probably better then peer review, but it won't have the aura.
- lowkeyokay 9y agoI really think we should have something like Stack Overflow for academic papers. The comments system works so well. Also, the so called impact factor of journals, which in many cases are directly or indirectly linked with academics and universities financial outcome, could be replaced by the points system from SO. Of course I'm not proposing using SO directly for academic research but something inspired by it.
- hvidgaard 9y agoI think a hybrid between SO and Discourse for papers would be more fitting. There really needs to be an open but moderated discussion for the papers - that'll serve as the peer review.