4 ms·
Peer review is broken. The assumption with peer review is that the review filters out bad research in a fair way. The reality is that it is neither fair nor par
by jillesvangurp 8y ago
Peer review is broken. The assumption with peer review is that the review filters out bad research in a fair way. The reality is that it is neither fair nor particularly effective in preventing bad research slipping through.
Part of the problem is the review quality is almost as inconsistent as the article quality.
From experience I can tell you that rejecting papers is a lot more work than accepting them. It requires arguing things in such a way that it survives editor scrutiny. This in turn requires actually reading the paper, doing some background checks, etc. Of course editors getting involved also means work for them. So negative reviews tend to trigger a lot of work. So much easier to just rubber stamp the reviews that give a thumbs up.
And editors have the problem of needing to fill their journals with stuff. Nobody reads these things cover to cover. So, there's a lot of filler content that will never be cited that got rubber stamped by reviewers taking a five minute glance at the text. As long as the article is not too blatantly bad, nobody cares. A common practice is to invite accepted authors back as reviewers. So, there's a notion of bad articles leading to equally bad reviews.
It is not surprising that there are researchers that game the system by publishing in friendly publications where the reviewers/editors are known friendly. They all need to keep their numbers up.
A possible fix could be to review out in the open. Publish reviews along with the article. Allow scientists to challenge these as well. It's one thing to say "this is probably fine" anonymously than it is to say, please publish this with a message stating that I approve and endorse the content of this article.
- fjsolwmv 8y agoLike the rest of economics supply and demand, it depends on if the journal is elite and has more submissions than space (Nature) or is struggling to fill space. Also, journals catre more about important results than correct ones, until they get burned too much. But much of modern science is too hard to disprove, because the claims are too subtle and complicated.
- snarf21 8y agoI am not an insider but could the reviewer turn into more of an endorsement? Meaning the paper is published and the endorsers are listed as well, e.g. "Reviewed and endorsed by Jill Jones of Cornell University." Organizations could then have clauses that if you endorsed papers that you didn't review and test that you would lose your job. I know that seems harsh but we need to find a way to get "skin in the game". I don't think having a paid organization would work and I think having something at the government level would be worse.
- nicoburns 8y agoI like this idea, but I think it might make the whole system collapse, with few people willing to review papers at all.
- pacala 8y agoThe ‘nobody wants to do job X’ problem has the usual solution: compensation.
- bdamm 8y agoThat compensation would come at the expense of research grants. This is not the same world as commercial development.
- dash2 8y agoSome journals do offer compensation - I think the American Economic Review used to offer £100. The solution hasn't been widely adopted, though, perhaps because academics enjoy the kudos of being a reviewer (and the chance to savage other people's work anonymously :-P... ). And while you can pay for reviews, it is hard to pay for review quality, which is very subjective.
- solveit 8y agoAlso because a price that would significantly incentivise a senior academic is a price that is way too high for most journals to pay. If you have prestige, a decent if not exorbitant salary, bulletproof job security, a project you're passionate about and want to spend time on, and way too many obligations to actually spend time on your project, how much would you take in exchange for adding yet another obligation to the pile, taking time away from what you really want to spend your time doing?
- icebraining 8y agoMaybe that's OK? Going a bit slower but with more solid footing seems like a reasonable tradeoff to me, though I'm a layman.
- SubiculumCode 8y agoWhether peer review is broken or not is not the problem highlighted in this article. This professor is publishing in junk journals that are not rigorous and are pay to play. I receive offers for these SPAM journals in the tens to hundreds everyday that somehow skip my inbox. The problem highlighted in this article is about why Cornell is not doing something about this professor.
- jillesvangurp 8y agoThis is just an extreme form of what is in fact common practice in the academic world. There are a lot of second and third rate publications that aren't completely horrible that happily accommodate a lot of mediocre articles peddled by mediocre professors on behalf of their mediocre graduate students. We don't all get to publish Nobel prize winning articles in Nature. Universities are typically more concerned with their professor's abilities to raise funding than their academic integrity. If the numbers look good (money coming in, publications going out) they are not likely to complain or apply much scrutiny. Their assumption here is that peer review should be sufficient. As I was arguing it is not. So that is in fact the core problem. Universities don't care, reviewers don't care, journal editors don't care, publishers don't care. They all benefit from inflated numbers. The system does not work. Cornell is complicit here and part of the problem.
- SubiculumCode 8y agoIf this was the case then I'd see these low tier / scam journals in the cv's of my colleagues. I do not. Perhaps this is only because I am at a well respected/ranked institution. In terms of funding, NIH for example, only has you list 4 or 5 most important publications in your biosketch..so quality, not quantity would matter there in terms of securing funding. Yes, lower tier (but not scam journals) can be the homes of lower quality work, but most often they are there because 1)it is a journal for a subdiscipline (Child Development, for example), 2)useful but not groundbreaking research (e.g., establishing validity of lab's protocol), or 3)research that was conducted well, but the results were not clear or are complicated. The latter is what often irks me the most about science: The need for clear results to get in a good journal. The journal quality should be determined not on results but on the quality of methods and research questions. That is why I am a proponent of pre-registration.
- samfisher83 8y agoWhy is "X is broken." People say this about interviewing and other stuff too. Yes it is broken, but that is the game. You either play it or if you chose not to you won't get the fancy Professorships and title, or in the professional world you won't get the job you want, the promotion you want etc.
- nonbel 8y agoThe difference between that and job interviews is that we are forced to pay for this crap via taxes. It is literally a black hole for tax-dollars. I still say cut government funding for research by 90% and watch it get fixed automatically as all the bad actors stop joining (you still probably need to wait for the current ones to die/retire).
- lsc 8y agoWhen I hear "X is broken" I translate it to "We haven't really figured out X, it must be hard" And usually then things make more sense. We haven't fully figured out interviewing. Interviewing is hard. We haven't fully figured out vetting academic papers, perhaps vetting academic papers is hard