6 ms·
PREPUBLICATION REVIEW IS BAD! STOP TRYING TO REINTRODUCE IT. Sorry for the all caps. Publishing papers without “peer review” isn’t some radical new concept—it’
by adastra22 3y ago
PREPUBLICATION REVIEW IS BAD! STOP TRYING TO REINTRODUCE IT.
Sorry for the all caps. Publishing papers without “peer review” isn’t some radical new concept—it’s how all scientific fields operated prior to ca. 1970. That’s about when the pace of article writing outstripped available pages in journals and this system of pre-publication review was adopted and formalized. For the first 300 years of science you published papers by sending it off as a letter to the editor (sometimes via a sponsor if you were new to the journal), and they either accepted or rejected it as-is.
The idea of having your intellectual competitors review your work and potentially sabotage your publication prospects as a standard process is a relatively recent addition. And one that has not been shown to actually be effective.
The rise of Arxiv is a recognition by researchers that we don’t need or want that system, and we should do away with it entirely in this era of digital print where page counts don’t matter. So please stop trying to force it back on us!
- freedomben 3y ago> The idea of having your intellectual competitors review your work potentially sabotage your publication prospects as a standard process is a relatively recent addition. And one that has not been shown to actually be effective. If this is true (and I'm not doubting you, just acknowledging that I'm taking your word for it) then why abandon the entire system? Why not just roll it back to the state before we added the intellectual competitor review?
- frozenport 3y ago>> Why not just roll it back to the state before we added the intellectual competitor review? Journals don't add much value outside of their peer review. Most researcher don't care about the paper copies, or pagination, or document formatting services provided by publishers. Their old paper based distribution channels are simply not used.
- adastra22 3y agoThe prior state was a situation of no pre-publication review other than the editorial staff of the journal. We should go back to that, yes. By disbanding entirely the “peer review” system that currently exists.
- bumby 3y ago>Why not just roll it back to the state before we added the intellectual competitor review? The only thing I can currently think of is that the pace of research* has grown so much that a small group of editors may be unable to handle the amount of submissions. This could result in a) an inability of the editors to thoroughly vet the submissions, b) difficulty in "good" submissions being found (ie, separating the wheat from the chafe), or c) a further devolution into very, very niche journals just to make the scope manageable for the editors * I would concede that a very, very large proportion of current research is either heavily derivative or auto-cited, so overall growth isn't to be conflated with growth in quality research
- adastra22 3y agoYou are mistaken in thinking that editors are supposed to vet submissions. That's not their job! Their job is only to weed out crackpot pseudoscience, which is a much easier task generally and also mostly solved by reputation, or papers outside of the journal's scope. And to edit the remaining papers into a constantly formatted journal publication. And that's done easy enough these days with LaTeX and online printing. It is NOT the job of journal editors to rate or vet sincere submissions they receive that are on topic for their journal. This only started happening about half a century ago, when demand for publication started to significantly outstrip supply of journal pages, back when journals were actually printed on dead trees and had limited number of pages to keep costs down. The idea then was "we're getting 50 submissions but can only print 12, so let's rate them and pick the best ones." So they started the 'peer review' [sic] process to externalize that vetting cost. It largely didn't exist before then. Only now we can accept all 50, because why not? The marginal cost of one more PDF is practically nil.
- bumby 3y ago>Only now we can accept all 50, because why not? Because the downside to creating an ever growing haystack is that it becomes increasingly difficult to find a needle. Making it easier to create a deluge of bad research won’t help me find the worthwhile research that would actually help me in my job. If I had the choice between collecting “all the data” and just collecting “the really good and relevant data” I’m opting for the latter. You are also contradicting yourself by saying it’s not the editors job to vet submissions, yet also say they weed out “crackpot” work. All you’re saying is they lazily/loosely vet submissions. I’m saying the overall system (not just the editors) have a role in providing a reasonably sized haystack (and would also admit the current system is not great at this, but it’s better than a wild-west approach)
- salty_biscuits 3y agoThe whole process comes from a time when publishing was expensive, should be cheap as chips now. The system needs a rethink so "quality" can somehow bubble up to the surface given mass publication is so simple.
- BlueTemplar 3y agoAlso, the author IMHO failed to clearly explain how "preprint review" wasn't a contradiction in terms (though they do seem to gesture towards the main issue being commercialization of journals, in the first post). In the same vein, the positive mention of Github and "open source platform" (another contradiction in terms) were at first red flags in the third article, but at least they then mentioned the threat of corporate takeover.
- SkyPuncher 3y agoI've found the entire peer review concept in academia to be extremely odd. I'm not entirely sure what problem it solves. It seems like you have one of two types of people reading these articles: * People who are already specialists/familiar enough with concepts. They'll either call BS upfront or run experimentation themselves to validate results. * People who aren't specialists and will need to corroborate evidence against other sources. My entire life as a software engineer has been built blogs, forums, and discussion from "random" people doing exactly the above.
- cqqxo4zV46cp 3y agoMost of the world isn’t ‘about’ software engineering. Software has some attributes that not very many other fields do. And software people have a habit of thinking that their experiencing writing software grants them some sort of transferable expertise, which it certainly does not.
- kelipso 3y ago> run experimentation themselves to validate results Let's just breeze through an that like it's nothing haha. Also the people reading these peer reviewed articles range from new grad students to researchers with decades of experience. There are many ways to see whether an article is high quality of not, which includes peer review, the journal it's published in, the research lab that wrote the paper. Reading a paper itself is a multi hour ordeal and you want to have a decent idea that it's not a crap paper before diving in. Believing what the paper says is something of a gamble because you really cannot just replicate an experiment that easily. And you need to read many many many papers before you can start doing your own research. So you want lots of assurances that a paper is good quality.
- abhgh 3y agoAgree. With more compute-intensive experiments in ML, I am not sure if any reviewer re-runs experiments. The reviewing is already voluntary - who's going to pay out of their pocket for the compute? It's not feasible.
- nextos 3y agoYou are right. It is often a problem that established competitors get to review articles that contradict their work and, unsurprisingly, try to sabotage them. Incentives are not well aligned. A good mid-ground is something like the non-profit journal ELife, where articles are accepted by an editor and published before review, then reviewed publicly by selected reviewers. Very transparant, and also leaves room for article updates. See the whole process here: https://elifesciences.org/about/peer-review https://elifesciences.org/about/peer-review.
- adastra22 3y agoThat’s a better system. But why involve the journal in review at all? Journals should go back to just publishing papers and any unsolicited letter-to-the-editor reviews, reproductions, or commentary they receive in response. Why add a burden of unpaid work reviewing every single paper that comes through?
- nextos 3y agoI believe ELife may eventually get deeper paid reviews. That is a reason to involve the journal in this process. The way reviews at ELife work can be seen as solicited letter-to-the editor reviews, as these do not influence the outcome. Your article is already published.
- adastra22 3y agoMaybe I'm not explaining my point very well. I think the fact that some papers don't get reviewed, even post-publication, is a feature not a bug. Otherwise it starts to present an unnecessary and damaging hurdle to publication, even if the reviews are post-publication because it is an implicit promise of future work. A proper scientific journal is an efficient clearinghouse for information, in the same way that Hacker News is an efficient clearinghouse for tech news and commentary. Journal editors play the role of dang in this setup, applying a minimal but necessary amount of moderation and editorial decisions. But imagine if you weren't allowed to post a comment to HN unless you first lined up 2-3 high-karma individuals to provide a thoughtful reply. No doubt the result would be high quality discussion, but there'd only be a handful of comments on even front-page posts, and I suspect that the overall value of the site would be vastly less than the present HN. There should be as few as possible editorial hurdles to clear in order to publish in a journal, e.g. one of the co-authors having published in the journal before or being sponsored by a respected person in the field. And the review that occurs before publication should consist of (1) spellchecks and such, and (2) formatting. This is how science worked for hundreds of years, and there's no reason we can't continue operating this way now that journal page lengths are a non-issue. (I'm not a reactionary though. I'd like to improve upon the old format in many ways. In particular I'd want journals to provide specific support for publishing reproductions of existing work or 3rd party submitted supporting documents, and much better methods for retracting or correcting a paper.)
- coolhand2120 3y agohttps://en.wikipedia.org/wiki/Replication_crisis https://en.wikipedia.org/wiki/Replication_crisis The reputation of scientific researchers has been greatly harmed by the current system. Please, help find a way to fix it, or at the very least don't hinder people trying to fix it. Thanks to the way we do things now a coin flip is _better_ than peer review. Public trust in science is at an all time low. I really hope you don't think "this is fine".
- somethingsome 3y agoIMO 'public trust in science is at an all time low' is because of bad journalism more than bad papers. Papers are written by academy-type individuals for academy-type individuals, not for consumption by non experts. An academic is usually pretty fast to determine if a paper is to be trusted. So interpreting and extrapolating to the extreme the results of a minor paper in an obscure journal is more bad journalism than bad science. Then we wonder why people don't trust science..
- adastra22 3y ago> Thanks to the way we do things now a coin flip is _better_ than peer review. Public trust in science is at an all time low. I really hope you don't think "this is fine". I don't think you read my post? I'm advocating we get rid of the "peer review" [sic] system entirely. The sibling post is right though that this problem is with bad journalism (and bad institutions), not bad science. People think that "peer review" is actually some sort of scientific hurdle that strengthens the paper. It is not, it was never meant tho fill that role, and has been totally morphed by journalists into something it has no business being.
- dbingham 3y agoHi! Author here. Preprint review as it is being discussed here is post-publication. The preprint is shared first, and review layered on top of it later. Click through to the paper[1] I'm responding to and give it a read. But, also, prepublication review doesn't need to be "reintroduced". It's still the standard for the vast majority of scholarship. By some estimates there are around 5 million scholarly papers published per year. There are only about 10 - 20 million preprints published total over the past 30 years since Arxiv's introduction. There are a bunch of layered institutions and structures that are maintaining it as the standard. I don't have data for it to hand, but my understanding is that the vast majority of preprints go on to be published in a journal with pre-publication review. And as far as most of the institutions are concerned, papers aren't considered valid until they have published in a journal with prepublication review. There is a significant movement pushing for the adoption of preprint review as an alternative to journal publishing with the hope that it can begin a change to this situation. The idea is that preprint review offers a similar level of quality control as journal review (which, most reformers would agree is not much) and could theoretically replace it in those institutional structures. That would, at least invert the current process: with papers being shared immediately and review coming later after the results were shared openly. [1] https://journals.plos.org/plosbiology/article?id=10.1371/journal.pbio.3002502 https://journals.plos.org/plosbiology/article?id=10.1371/jou...
- adastra22 3y agoThanks for your charitable response to a reader who quite obviously misunderstood what you were writing about. You might want to update your article to make that clear, as "preprint review" isn't a term I had ever encountered before, and a simple reading of your article doesn't obviously indicate that you were talking about post-publication review. (Note that outside of academia, upload to arXiv is publication, so yes your "preprint review" would be a post-publication review.)
- dbingham 3y agoNo worries. I was really writing for an audience of academics and, in particular, people involved in the various open science and publication reform movements. Sharing here was a bit of an after thought, so it wasn't written for an audience unfamiliar with the intricacies of academic publishing. It's a complicated enough space as it is that this is already a 4-5 article series with the potential to grow, even when I'm writing with the assumption of quite of a bit of knowledge. But yeah, "preprint review" is considered post-publication review both in and outside of academia. There are nuances to what is consider "publication" in academia. A preprint is not a "Version of Record", meaning it doesn't count towards tenure and promotion. The movement pushing for preprint review is attempting to layer review on top of already public preprints in the hopes that reviewed preprints can begin to count as VORs. It's unclear whether that will work. Some models, like eLife's, seem more promising than others. But eLife got a ton of backlash when they switched to their new reviewed preprint model, so it remains to be seen whether it will work in the long run.