12 ms·
there are many separate problems of scientific fraud. I think these issues sometimes get confused which is unhelpful. 1. apparently-legitimate papers in presti
by currymj 1y ago
there are many separate problems of scientific fraud. I think these issues sometimes get confused which is unhelpful.
1. apparently-legitimate papers in prestigious journals with fraudulent data. extremely bad.
2. legitimate papers in legitimate journals which, innocently or not, just used bad methods and have wrong conclusions. this is "the replication crisis".
3. totally fake papers in paper mills with no meaningful peer review. it's really easy to spot these, no one is individually getting taken in by the results, but...
3a. sometimes they wind up in a meta-analysis, which is really bad because people might trust the meta analysis.
Problem 1 is morally worst and much more common than one would hope. Outright fabricated data in a Nature or NEJM publication (as has happened) is a disaster.
Problem 2 is amenable to reform for the most part (fields are already doing this).
Problem 3 isn't a problem at all for scientific knowledge per se, although universities and funding bodies might not be pleased their scientists are buying fake papers. You can just ignore the paper mills.
But Problem 3a can actually alter policy, which is pretty serious.
- libraryatnight 1y agoIs 3/3a about to be more serious with LLMs in the mix?
- Fomite 1y ago"about to" it is. In a recent conversation with the editor-in-chief of a journal I am on the editorial board of, a substantial bulk of the submissions we get are LLM written papers that essentially randomly look for associations in accessible data, which are then sold to faculty (primarily in China).
- mike_hearn 1y agoArguably "randomly look for associations in accessible data" is like 90% of academic publishing even before LLMs though ;)
- IronyMan100 1y agoNo, probably Not. Nobody is reading these Journals anyway. It's only good one resumes. i think even 3a is Not a problem because fake papers will follow a specific pattern in Meta Analysis. Should be catched in one of the "filter" stages during Meta Analysis.
- currymj 1y agoprobably not. the writing quality will improve but LLM-generated papers will still be ignored. to the extent they aren't ignored, but seem so plausible that they are taken seriously, eventually people will want to talk to the researchers about their results, invite them to give talks, and so on. at which point it becomes problem 1.
- spookie 1y agoAbout 3a: Never touch MDPI, the amount of fake data I've seen is ridiculous
- zdw 1y agoWould you consider the Reinhart-Rogoff paper: https://en.wikipedia.org/wiki/Growth_in_a_Time_of_Debt#Methodological_criticism https://en.wikipedia.org/wiki/Growth_in_a_Time_of_Debt#Metho... Where a mistake in the Excel spreadsheet was used by many politicians to justify austerity measures to be a #2 or #3a problem (or both)?
- flobosg 1y ago4. Plagiarism
- asciident 1y agoOr cheating in other ways, like collusion rings [1], reviewers using LLMs to review, or authors putting text in their papers so reviewers using LLMs to review will give them a favorable review [2] [1] https://cacm.acm.org/opinion/collusion-rings-threaten-the-integrity-of-computer-science-research/ https://cacm.acm.org/opinion/collusion-rings-threaten-the-in... [2] https://www.theguardian.com/technology/2025/jul/14/scientists-reportedly-hiding-ai-text-prompts-in-academic-papers-to-receive-positive-peer-reviews https://www.theguardian.com/technology/2025/jul/14/scientist...
- aaron695 1y ago[dead]
- caseysoftware 1y agoMaliciousness, incompetence, and accidents all look EXACTLY THE SAME from a replication perspective. We can't tell the researchers' intent. Until the "industry" (defined vaguely as scientists, their institutions, universities, funding entities, etc, etc) cleans house and punishes those researchers, we're quickly approaching a time where we'll have to take EVERY study skeptically until it can be replicated. * Punishment could range from "no, we won't publish your stuff without data+methodology" to ratcheting back funding to "we publicly document your lying/incompetence" (hardest and riskiest) to a variety of other things.
- OnlineGladiator 1y ago> we're quickly approaching a time where we'll have to take EVERY study skeptically until it can be replicated I've always felt like this should be the norm. Why would you trust something before it can be replicated? Even if it's unintentional, people make mistakes.
- caseysoftware 1y agoIndividually, yes I've been there a LONG time. Unfortunately, we have a media and political structure that uses the most recent study/model/whatever to advocate for, design, and enact policy before that review.
- SJC_Hacker 1y ago> I've always felt like this should be the norm. Why would you trust something before it can be replicated? Even if it's unintentional, people make mistakes. If you are close enough with a scientist, generally they will admit they don’t trust a single study Some fields also have guardrails, such as the LIGO being two separate detectors a with two independent teams
- mensetmanusman 1y agoThere isn’t really a way to “clean house” on a large scale, everyone has to somehow be more virtuous and not lie to themselves when p hacking or publishing data they know someone wouldn’t be able to reproduce.
- nextos 1y ago> Problem 1 is morally worst and much more common than one would hope. Outright fabricated data in a Nature or NEJM publication (as has happened) is a disaster. In my experience, fabricated data is rare. But cherry-picking data and misuse of statistics to prove the desired hypotheses is extremely common in top journals from e.g. the Nature Portfolio journal family. I'd say at least 1/5 of the articles in biology and medicine have serious methodological errors. IMHO, the main reason are perverse incentives and power imbalance. If you are a student or a postdoc and you get pressure from your supervisor to obtain certain result and you refuse to engage in academic misconduct, you will get pushed out of most projects and this might be the end of your academic career. In contrast, there are few, if any, consequences for the senior scientists and principal investigators who perpetuate this culture. There's no oversight, no integrity checks, nothing. Universities tend to ignore even clear cases of data fabrication and fraud, where e.g. images derived from experiments are fake. Ultimately, this creates selective pressure for misconduct, as those willing to bend the rules are rewarded.
- nothercastle 1y agoThe advisor gets all the credit for publishing and none of the blame for retracting. That’s why some labs have been hit by multiple retractions (clearly a indicator of fraud culture) but only the low level researchers are punished
- echelon 1y agoThe problem is that academic publishing isn't tied to any real fitness function. Unlike a startup or company or some other thing where you vote with dollars on utility, many papers can't be quantitatively evaluated on how much they contribute to science. Some papers are obvious and groundbreaking, but a lot of research is at such a frontier that literally no one else can evaluate it. You just nod your head and trust it. That, coupled with the fact that negative results just aren't published, is incredibly inefficient and disheartening.
- KolibriFly 1y agoThat dynamic is deeply broken
- lmm 1y agoIt really doesn't matter what the motive is when the effect is the same. Treating it as an individual moral failing isn't going to work when the incentives to fake papers (wilfully or not) are so strong. Any fix that works for 1 is going to work for 2 as well and vice versa.
- homieg33 1y agoYou can’t legislate morality.
- lukas099 1y agoWhy not? Set up the laws to incentivize morality.
- lisper 1y agoYou don't even have to change the laws. Just changing the social norms would be enough. At the moment our social norms are that the ends justify the means. If you make money, or produce something that looks like a breakthrough, that is all that matters. How you did it, or whether the thing you did has actual value, matters not at all. As long as you've provided yourself with plausible deniability, that's good enough. I know this because I've been the beneficiary of this system on multiple occasions, and it makes me sick. This is not the kind of world I want to live in.
- shmeeed 1y agoMost if not all legislation originated in morality at some point. We've just collectively forgotten that's still an option today.
- scrubs 1y agoNo you can't legislate morality. But operationally we want accountability commensurate to control. As others have pointed out, there must be consequences to bs. With that in place, we don't have to legislate other than point out consequences. Now to be a pedantic, consequences are a kind of legislation, but at least it's about behavior and outcomes not intent. It would elides certain issues out of scope.
- ocschwar 1y agoProblem #3 is relatively easy to address: shift from journals to conferences. Organizing fake conferences is a lot harder than setting up paper mill journals.
- SJC_Hacker 1y agoConferences aren’t double blind. They aren’t even single blind. Fundamentally they can’t be Peer review should be but sometimes isn’t. It always can be though, unlike conferences
- kergonath 1y agoNot really. I see many, many fake conferences adjacent to my field each year. I don’t really see why setting up one would be harder than creating a fake journal.
- aoki 1y agoI think they mean predatory journals and conferences. Setting up a predatory journal costs nothing, whereas actually holding an in-person conference of any kind costs money. A good time to repost about the famous VIDEA conference: https://users.cg.tuwien.ac.at/wp/videa.html https://users.cg.tuwien.ac.at/wp/videa.html https://users.cg.tuwien.ac.at/wp/videa-paper.html https://users.cg.tuwien.ac.at/wp/videa-paper.html
- kergonath 1y agoThese conferences have fees, just like predatory journals. They tend to be held in dodgy places in otherwise touristy areas, and most of them are partly or fully online anyway. The cherry on the cake is that attendees can use them as holidays funded by their institution. I don’t dispute that there are probably orders of magnitude more fake papers than fake conference presentations, just that shifting from journal articles to conference talks won’t help at all. If the incentives shift, current paper mills will just shift with the market.
- lisper 1y agoI can tell you from personal experience that organizing a "fake" conference is beyond trivial. All you need is a topic that is narrow enough that all of the attendees know and like each other. I was an attendee at several such conferences back when I made my living publishing papers, and I was astonished at how easy it was to publish bullshit as long as it was the right venue, and how borderline impossible it was to publish anything that I considered to have actual value. (In my career I published dozens of conference papers, but only one journal paper and one book chapter.) In actual practice what I found was that the principle driver of publishing success has absolutely nothing to do with the quality of the work, it has to do with how much your reviewers think that you might some day be in a position to review a paper of theirs. This is the fundamental problem with peer review when: career success is measured by quantity of papers published, the resulting dynamic is governed by game theory, not scientific merit.
- throwawaymaths 1y ago> Problem 2 is amenable to reform for the most part (fields are already doing this) how are they?
- rapnie 1y agoProblem 3 has another effect. There's a whole industry of conspiracy theorists who will point to that paper to give credence to their own BS, while they create Youtube videos and wild pop-sci articles to influence the general public on whatever their agenda is. I have a friend who recently has 'taken an interest in science' and is now passing Youtube videos all of the time that are total crap, but make reference to 'scientific' papers. I 'learned' how earth is already doomed because Betelgeuse went supernova, and that there are deep shafts below the pyramid of Gizeh that only an alien race could have dug. Stuff like that. Net outcome where it comes to science may be that this helps erode people's trust in science.
- KolibriFly 1y agoIf we can't rely on journals like Nature or NEJM to catch blatant fraud, then what hope is there further down the chain?
- mike_hearn 1y agoYou can't rely on journals because they have bad incentives. Their customers are the universities they're supposed to monitor for fraud, and their prestige comes from the same place news/magazine prestige does - publishing incredible claims. Catching fraud further down the chain is easy. Other professions know how to do it. Just adopt the same framework as used in financial controls. External auditors, written standards, randomized spot checks, lots and lots of paperwork to provide audit trails back to real experiments with hard-to-forge evidence. And severe punishments for those found to have engaged in intentional fraud.
- jltsiren 1y agoThat sounds expensive and difficult. Science is not a single field with a single set of practices. Regulating it is more like trying to draft a single set of regulations for finance, agriculture, pharma, and software. You can get a glimpse of that when applying for grants or publishing in more prestigious journals. There are often endless checklists and questions about regulatory and ethical compliance, data availability, reproducibility, and whatever else. And more often than not, many of the questions are category errors or otherwise make no sense in the context of your work. Even the people who are supposed to regulate science lack the ability to do so in a way applicable to the entire breadth of science.
- mike_hearn 1y agoI don't think the differences between fields in (academic) science are as big as the differences between agriculture and software. They all take grants to publish papers, in journals, and they are measured via similar metrics like citations, papers published, impact factors and so on. Academia is highly homogenous across institutions too. Yes, for sure a lot of controls implemented via audit aren't perfect fits for what is being regulated, that's the nature of trying to fix bad incentives via regulation. It's better to just have aligned incentives to start with, but that can't be done in academia, it's regulation or nothing. Also, my experience has been that sometimes academics like to claim their field is special and shouldn't be expected to play by the same rules as other fields. But that's often not true. They should in fact be held to the same standards as everyone else is and their justifications for why they're an exception don't hold water.
- selfmodruntime 1y agoI just recently found an incredibly cool paper that solved a problem I was having. It was published in a tier A conference. They claimed they would make the data and code public at a later point. Problem is, I could find nothing of it when I googled. O tried to get ahold of the authors and I even wrote to their department. The only Mail back I received concluded that neither would be available any time soon. So there‘s that. A paper that could be completely made up, with no data to support it. It might as well be make believe. I would lie if I said this didn't make me lose faith in my own work, since it justifiably takes me ages to gather data and write code for my experiments. Great.
- hnmullany 1y agoI did a meta-analysis for my masters thesis and I was convinced that 2 of the 39 papers in my dataset had fraudulent data.
- mfld 1y agoUnfortunately the line between 2 and 3 is very blurry. Like paper mill output submitted to legimate medium-to-low impact journals. Some of it gets inevitably published, others just puts a strain on peer reviewers.
- j45 1y agoThanks for the breakdown.