8 ms·
Transparent peer review to be extended to all of Nature's research papers
- cperez081890 1y ago[dead]
- jostmey 1y agoThis only addresses a small part of the problem with peer-review. The real problem is that peer reviewers can’t possibly replicate the study, and so are forced to look for inconsistencies in the papers. If the paper doesn’t fit what is expected, it will often be rejected. This can also lead to self-reinforcing views that ignore contrarian data. Also, the data can be made up, and if it makes sense to the reviewers, it is generally not questioned
- jxjnskkzxxhx 1y agoReading your comment makes me think that you believe that the point of peer-review is to ensure that a paper is correct, or at least that specific aspects of it are correct. Is that the case? What do you think the point of peer-review is?
- yupitsme123 1y agoI'm not the person you replied to, but I think that in the lay world, people do indeed think that peer review is as you've described. If it's not, then maybe it should be? Research gets cited constantly in public debates and is used for policy decisions, so the public should be able to quickly separate the good from the bad, the "maybe this is true" from the "this is empirically proven." The public has lost a lot of trust in Science because research papers have been used to push political agendas, which can then never be questioned because doing so means arguing with a supposed peer-reviewed scientific consensus.
- thomasfedb 1y agoNothing is ever “proven”. There is simply more or less support for a theory or proposition. Replication and meta-analysis are an important part of this. Most scientists are in fact very conservative with how they claim their results - less so university PR departments and “study shows” clickbaiters.
- jxjnskkzxxhx 1y agoI approve of this comment.
- blix 1y agoI wish this comment was more representative of my personal experience in science. Instead I got PIs happy to say that weak evidence "proved" their theory and to try suppress evidence that negatively impacted "fundablity". The most successful scientists I worked with were the ones who always talked like a PR puff piece.
- jxjnskkzxxhx 1y agoWhat field, may I ask?
- blix 1y agoApplied physics. I'd prefer not to get too specific. Most of my peers are working for the US DoD or DoE now.
- jxjnskkzxxhx 1y agoI wonder if what you described is due to the money incentive? I did theoretical physics (no money) and my experience totally matches what the other person described.
- JumpCrisscross 1y ago> public has lost a lot of trust in Science because research papers have been used to push political agendas, which can then never be questioned because doing so means arguing with a supposed peer-reviewed scientific consensus The public has lost trust in science because 10 to 30% of it is scientifically illiterate [1]. (Tens of millions of American adults are literally illiterate [2].) That's what lets activists and politicians cherrypick bad science that supports their position or cast a scientific consensus as unquestionable. [1] https://carsey.unh.edu/publication/conspiracy-vs-science-survey-us-public-beliefs https://carsey.unh.edu/publication/conspiracy-vs-science-sur... [2] https://nces.ed.gov/pubs2019/2019179/index.asp https://nces.ed.gov/pubs2019/2019179/index.asp
- aDyslecticCrow 1y agoBad quality work is a much bigger problem than dishonest work. Systematically well-done research with fake results or data is much rarer than just... lazy bad science. There is a massive incentive to publish. Inflate the value, inflate the results, and stretch out projects to multiple smaller papers, fake results to make it seem important. This is lazy and fast, and can be caught by a stricter review and scrutiny. Papers that are properly done all the way through, but with faked data meant to push an agenda, can be disproven by counter research.
- fc417fc802 1y ago> Systematically well-done research with fake results or data is much rarer than just... lazy bad science. Genuine mistakes, logical errors, and other oversights are even more common than that. For all the issues it has, peer review is quite good at catching the things that it's intended to catch.
- yummypaint 1y agoHave you personally reviewed a non-zero number of papers? What is this statement based on? For a thread ostensibly about science, the comments are disappointingly lacking in evidence and heavy on vibes. Maybe people could learn about what peer review is before posting their strong feelings about it? The purpose certainly isn't to replicate people's experiments, that happens after publication and not by referees. One of a reviewer's duties is to look at whether the study could be replicated given the included information. That is a very different thing. Also, just because something has made it past peer review also doesn't mean it isn't controversial in the field.
- jostmey 1y agoI’ve only peer reviewed a few papers. I’ve submitted my share of papers for review This is my academic profile https://scholar.google.com/citations?user=bxn78bkAAAAJ&hl=en https://scholar.google.com/citations?user=bxn78bkAAAAJ&hl=en
- mike_hearn 1y agoHe isn't saying the purpose of peer review is to replicate work. And yes I've personally reviewed quite a lot of papers (from outside of academia), also occasionally reviewed peer reviews. What he is saying is that peer review is treated by academia as a close to gold standard, when it's really more like a bronze standard. It's a bit like finding a software company that exclusively uses volunteer post-commit code review with no unit tests or static typing, and in which the only testing process is to push to production and see if anyone inside the company complains. It's not useless as a concept, and it's generally better to have a paper that's reviewed than one that isn't unless the field has been captured by ideologues. But there are so many common problems it can't fix, not even in principle.
- kergonath 1y ago> The real problem is that peer reviewers can’t possibly replicate the study, and so are forced to look for inconsistencies in the papers. This is a misunderstanding of the role of peer review. The point is not to prove that a paper is correct, the point is to ensure a minimum level of quality. You are entirely right that most reviewers cannot hope to reproduce the results presented, and very often for very good reasons. If I write 3 proposals over the course of a year to get some beam line on a neutron source, it is completely unrealistic to expect a referee to have the same level of commitment. I think this hints at a more profound problem, which is that a lot of studies are not replicated. This is where the robustness of a scientific result comes from: anybody can make the same observation and reach the same conclusions under the same conditions. This is the real test, not whether you convinced 3 or 5 referees. The real value of an article is not in whether it was peer reviewed (though the absence of peer review is a red flag). Instead, it is in whether different people confirmed its main results over the years that follow publication.
- mike_hearn 1y agoThere are no silver bullets. Replication also isn't worth much: https://blog.plan99.net/replication-studies-cant-fix-science-0e195234a280 https://blog.plan99.net/replication-studies-cant-fix-science... Merely replicating a study doesn't tell you anything about whether the claims are correct, although failure to replicate can be a signal that the claims probably aren't correct.
- dr_dshiv 1y agoWe need more scientific societies. Modern peer review is super modern. Go back to the origins of science and it was all about a real community—setting high standards and having just a few people decide what to publish. It wasn’t trying to be “objective” — it was just high standards, determined invisibly by the members of the society. “Should we publish this?” asked the society. Alas, scale ruins everything. Nevertheless, I strongly believe “science is friendship”
- aDyslecticCrow 1y agoThat model had some major issues. Too many opposing theories to the "established norm" were dismissed, among things we now know were wrong all along. Although it could raise the standard of the process itself (methodology, writing) to very high levels, it restricted innovative ideas or unpopular outcomes.
- deleted 1y ago[deleted]
- striking 1y agoSounds like a recipe for serious inequity to me.
- JumpCrisscross 1y ago> Sounds like a recipe for serious inequity to me If it works better, it works better. The problem with the society method is it didn't work better than a decentralised scientific system.
- aspenmayer 1y agoOne of the weaknesses to such a system is human nature. We want to believe, which leads to farces like Piltdown man, which was a farce by a man called Charles Dawson (not Darwin, not the natural selection guy, though I did a double take at first), against the Geological Society and society at large. The farce wasn’t definitively disproven until 41 years later, which is quite the downside risk of the gentlemen’s club research group gatekeeping scholarly research, but the more things change, the more they stay the same. This is a good thing which seems like it’s in response to grumblings about irreproducibility and bias, but those issues will likely always be with us and must be considered anew each time an experiment is designed, and each time a print run is cut. I’m having trouble finding a downside besides vote buying or voting rings, now that which way one has voted is now attributable. Can you think of any risks under the new system?
- eig 1y agoA good step forward. Reading the referee reports and rebuttals from papers that previously opted in to transparent reviewing was incredibly useful to my own paper publishing process. In many ways Nature is ahead of Science on this. I hope that making things transparent will help reduce the situation where big labs have an easier time getting their work into high impact journals through relationships with the editor.
- aDyslecticCrow 1y agoThis is amazing. The trust of peer review as a stamp of quality among academics is dwindling, and distrust of science among the population is growing (within the increasing politicisation of some areas of research). Transparent peer review raises the bar for both academics and enhances the potential trust in the process. This is desperately needed as AI could further deteriorate the quality of science if the publishing process is not made more strict. This represents a significant step forward in rigorous science. I hope other publications follow suit.
- thfuran 1y ago>and enhances the potential trust in the process. I highly doubt it's a meaningful factor in public trust.
- Ar-Curunir 1y agoThe general public does not distrust science because peer reviews are not public.
- kergonath 1y agoRight. The general public cannot understand the points that are being made during peer review.
- jfengel 1y agoOf the people who distrust science, how many of them have ever read a scientific paper? I suspect the number is low. If that's the case, they're unlikely to be more convinced by the presence of published peer review, either.
- oerpli 1y agoI would go further: Anyone who has published a "scientific paper" in the last decade or so either "distrusts science" or is more likely than not a mid-wit at best. Your posting doesn't give me the impression that you're very familiar with "science".
- AStonesThrow 1y ago[dead]
- probably_wrong 1y agoI want to stick my neck out to say that this has the potential of being very bad for science. Imagine saying "no" to a researcher with a big social media profile. Imagine 4chan coming at you with style-detection and deanonymization tools simply because their favorite racist or antivaxer got their nonsense rejected and sent their followers after you. And this is not just me feeling this way - quoting myself from a previous comment, and according to the ACL's 2019 survey [1], "female respondents were less likely to support public review than male respondents" and "support for public review inversely correlated with reviewing experience". A measure that women ~~and inexperienced researchers~~[2] do not support is a measure that favors only those who are already part of the club. [1] Original here (currently offline): http://acl2019pcblog.fileli.unipi.it/wp-content/uploads/2019/07/ReportACL2019ReviewingSurvey.pdf http://acl2019pcblog.fileli.unipi.it/wp-content/uploads/2019..., summary here: https://www.aclweb.org/adminwiki/images/f/f5/ACL_Reviewing_Survey_Summary.pdf https://www.aclweb.org/adminwiki/images/f/f5/ACL_Reviewing_S... [2] This part has been correctly pointed out as being wrong.
- JumpCrisscross 1y ago> Imagine saying "no" to a researcher with a big social media profile "The identity of the reviewers will remain anonymous, unless they choose otherwise — as happens now." (Also "support for public review [being] inversely correlated with reviewing experience" means inexperienced reviewers are more likely to support it. Not less.)
- deleted 1y ago[deleted]
- probably_wrong 1y agoYou are correct about the second point - I'll strike it through once I find out how. As for the anonymous part, that's why I wrote "with style-detection and deanonymization tools". If the Internet could find Shia Labeouf's flag in a day [1], could they find a reviewer based on their writing? [1] https://www.dailydot.com/unclick/4chan-shia-labeouf-secret-location/ https://www.dailydot.com/unclick/4chan-shia-labeouf-secret-l...
- lenerdenator 1y agoI only got as far as undergraduate research assistant in the academia racket, so maybe I'm not "with it" in the way serious researchers are, but is this to suggest that this wasn't being done, by default, on everything already?
- fc417fc802 1y agoAny reputable journal had a reasonably robust peer review process (exact details vary of course) but historically it was strictly confidential. Only over the past perhaps 2 decades has a process where reviews are made public begun to gain mainstream support.
- deleted 1y ago[deleted]
- trelane 1y ago> You have full access to this article via your institution. I would suggest to Nature that this phrase hints at much larger a problem than showing authors arguing with Reviewer 2.
- WhatsTheBigIdea 1y agoA positive step! Bravo Nature! To little, to late? Maybe. Surely very little, surprisingly late. There are so many great reasons to distrust "The Science"... perhaps the greatest is that without powerful and persistent skepticism, science simply isn't science. Questioning and skepticism is profoundly fundamental to the scientific process. "Trust the science" or worse yet "Believe the science" are statements that are about as destructive to the foundation of science as one can get. The whole point of science was that everyone has access to the true nature of the universe which we can discover through theorizing and experimentation. Empirical experience is the great equalizer that puts the Arch Duke of Where-and-What or the Highest of High Priests the on the same footing as, say, Joseph Preistly or Benjamin Franklyn. Questioning and skepticism must be accepted from all quarters. To say that only the most select, distinguished and credentialed can be admitted to the discussion smacks of priesthood and aristocracy. The argument that we need to re-persuade the people to trust in science is missing the point by such a wide margin as to be a symptom of the problem. It is more or less the same as fretting that people aren't going to church so much any more. The science should not be trusted. The science should be tested. Those who say "Trust the science" have completely lost the thread.
- stefantalpalaru 1y ago[dead]
- knappa 1y agoI feel like the main thing that this will accomplish is to show that the quality of peer review is quite spotty. Usually 1 of 3 truly read and understood the paper enough to critique from a place of knowledge. Entirely miscategorizing a paper is not uncommon.