5 ms·

There's an article that explores the metrics here: https://fantasticanachronism.com/2020/08/11/how-many-undetected-frauds-in-science/ https://fantasticanachron

There's an article that explores the metrics here: https://fantasticanachronism.com/2020/08/11/how-many-undetected-frauds-in-science/ https://fantasticanachronism.com/2020/08/11/how-many-undetec... > 0.04% of papers are retracted. At least 1.9% of papers have duplicate images "suggestive of deliberate manipulation". About 2.5% of scientists admit to fraud, and they estimate that 10% of other scientists have committed fraud. 27% of postdocs said they were willing to select or omit data to improve their results. More than 50% of published findings in psychology are false. The ORI, which makes about 13 misconduct findings per year, gives a conservative estimate of over 2000 misconduct incidents per year. Although publishing untrue claims isn't the same thing as fraud, editors of well known journals like The Lancet or the New England Journal of Medicine have estimated that maybe half or more of the claims they publish are wrong. Statistical consistency detectors run over psych papers find that ~50% fail such checks (e.g. that computed means are possible given the input data). The authors don't care, when asked to share their data so the causes of the check failures can be explored they just refuse or ignore the request, even if they signed a document saying they'd share. You don't have these sorts of problems in cryptography but a lot of fields are rife with it, especially if you use a definition of fraud that includes pseudoscientific practices. The article goes into some of the issues and arguments with how to define and measure it.

matthewdgreen 2y ago
0.04% is an extremely small number and (it needs to be said) also includes papers retracted due to errors and other good-faith corrections. Remember that we want people to retract flawed papers! treating it as evidence of fraud is not only a mischaracterization of the result but also a choice that is bad for a society that wants quality scientific results. The other two metrics seem pretty weak. 1.9% of papers in a vast database containing 40 journals show signs of duplication. But then dig into the details: apparently a huge fraction of those are in one journal and in two specific years. Look at Figure 1 and it just screams “something very weird is going on here, let’s look closely at this methodology before we accept the top line results.” The final result is a meta-survey based on surveys done across scientists all over the world, including surveys that are written in other languages, presumably based on scientists also publishing in smaller local journals. Presumably this covers a vast range of scientists with different reputations. As I said before, if you cast a wide net that includes everyone doing science in the entire world, I bet you’ll find tons of fraud. This study just seems to do that.
- mike_hearn 2y ago
  The point about 0.04% is not that it's low, it's that it should be much higher. Getting even obviously fraudulent papers retracted is difficult and the image duplications are being found by unpaid volunteers, not via some comprehensive process so the numbers are lower bounds, not upper. You can find academic fraud in bulk with a tool as simple as grep and yet papers found that way are typically not retracted. Example, select the tortured phrases section of this database. It's literally nothing fancier than a big regex: https://dbrech.irit.fr/pls/apex/f?p=9999:24 https://dbrech.irit.fr/pls/apex/f?p=9999:24:::::: Randomly chosen paper: https://link.springer.com/article/10.1007/s11042-025-20660-1 https://link.springer.com/article/10.1007/s11042-025-20660-1 "A novel approach on heart disease prediction using optimized hybrid deep learning approach", published in Multimedia Tools and Applications. This paper has been run through a thesaurus spinner yielding garbage text like "To advance the expectation exactness of the anticipated heart malady location show" (heart disease -> heart malady). It also has nothing to do with the journal it's published in. Now you might object that the paper in question comes from India and not an R1 American university, which is how you're defining reputable. The journal itself does, though. It's edited by an academic in the Dept. of Computer Science and Engineering, Florida Atlantic University, which is an R1. It also has many dozens of people with the title of editor at other presumably reputable western universities like Brunel in the UK, the University of Salerno, etc: https://link.springer.com/journal/11042/editorial-board https://link.springer.com/journal/11042/editorial-board Clearly, none of the so-called editors of the journal can be reading what's submitted to it. Zombie journals run by well known publishers like Spring Nature are common. They auto-publish blatant spam yet always have a gazillion editors at well known universities. This stuff is so basic both generation and detection predate LLMs entirely, but it doesn't get fixed. Then you get into all the papers that aren't trivially fake but fake in advanced undetectable ways, or which are merely using questionable research practices... the true rate of retraction if standards were at the level laymen imagine would be orders of magnitude higher.
  - fc417fc802 2y ago
    > found by unpaid volunteers, not via some comprehensive process "Unpaid volunteers" describes the majority of the academic publication process so I'm not sure what you're point is. It's also a pretty reasonable approach - readers should report issues. This is exactly how moderation works the web over. Mind that I'm not arguing in favor of the status quo. Merely pointing out that this isn't some smoking gun. > you might object that the paper in question comes from India and not an R1 American university Yes, it does rather seem that you're trying to argue one thing (ie the mainstream scientific establishment of the western world is full of fraud) while selecting evidence from a rather different bucket (non-R1 institutions, journals that aren't mainstream, papers that aren't widely cited and were probably never read by anyone). > The journal itself does, though. It's edited by an academic in ... That isn't how anyone I've ever worked with assessed journal reputability. At a glance that journal doesn't look anywhere near high end to me. Remember that, just as with books, anyone can publish any scientific writeup that they'd like. By raw numbers, most published works of fiction aren't very high quality.[0] That doesn't say anything about the skilled fiction authors or the industry as a whole though. > but it doesn't get fixed. Is there a problem to begin with? People are publishing things. Are you seriously suggesting that we attempt to regulate what people are permitted to publish or who academics are permitted to associate with on the basis of some magical objective quality metric that doesn't currently exist? If you go searching for trash you will find trash. Things like industry and walk of life have little bearing on it. Trash is universal. You are lumping together a bunch of different things that no professional would ever consider to belong to the same category. If you want to critique mainstream scientific research then you need to present an analysis of sources that are widely accepted as being mainstream. [0] https://www.goodreads.com/book/show/18628458-taken-by-the-t-rex https://www.goodreads.com/book/show/18628458-taken-by-the-t-...
robotresearcher 2y ago
> More than 50% of published findings in psychology are false Wrong is not the same as fraudulent. 100% of Physics papers before Quantum Mechanics are false[1]. But not on purpose. [1] hyperbole, but you know what I mean.
- mike_hearn 2y ago
  It's hyperbole to the level that obfuscates, unfortunately. 50% of psych findings being wrong doesn't mean "right all the time except in exotic edge cases" like pre-quantum physics, it means they have no value at all and can't be salvaged. And very often the cause turns out to be fraud, which is why there is such a high rate of refusing to share the raw data from experiments - even when they signed agreements saying they'd do so on demand.
derbOac 2y ago
Not trying to be hostile but as a source on metrics, that one is grossly misleading in several ways. There's lots of problems with scientific publication but gish gallop is not the way to have an honest conversation about them.

matthewdgreen 2y ago

0.04% is an extremely small number and (it needs to be said) also includes papers retracted due to errors and other good-faith corrections. Remember that we want people to retract flawed papers! treating it as evidence of fraud is not only a mischaracterization of the result but also a choice that is bad for a society that wants quality scientific results. The other two metrics seem pretty weak. 1.9% of papers in a vast database containing 40 journals show signs of duplication. But then dig into the details: apparently a huge fraction of those are in one journal and in two specific years. Look at Figure 1 and it just screams “something very weird is going on here, let’s look closely at this methodology before we accept the top line results.” The final result is a meta-survey based on surveys done across scientists all over the world, including surveys that are written in other languages, presumably based on scientists also publishing in smaller local journals. Presumably this covers a vast range of scientists with different reputations. As I said before, if you cast a wide net that includes everyone doing science in the entire world, I bet you’ll find tons of fraud. This study just seems to do that.

mike_hearn 2y ago
The point about 0.04% is not that it's low, it's that it should be much higher. Getting even obviously fraudulent papers retracted is difficult and the image duplications are being found by unpaid volunteers, not via some comprehensive process so the numbers are lower bounds, not upper. You can find academic fraud in bulk with a tool as simple as grep and yet papers found that way are typically not retracted. Example, select the tortured phrases section of this database. It's literally nothing fancier than a big regex: https://dbrech.irit.fr/pls/apex/f?p=9999:24 https://dbrech.irit.fr/pls/apex/f?p=9999:24:::::: Randomly chosen paper: https://link.springer.com/article/10.1007/s11042-025-20660-1 https://link.springer.com/article/10.1007/s11042-025-20660-1 "A novel approach on heart disease prediction using optimized hybrid deep learning approach", published in Multimedia Tools and Applications. This paper has been run through a thesaurus spinner yielding garbage text like "To advance the expectation exactness of the anticipated heart malady location show" (heart disease -> heart malady). It also has nothing to do with the journal it's published in. Now you might object that the paper in question comes from India and not an R1 American university, which is how you're defining reputable. The journal itself does, though. It's edited by an academic in the Dept. of Computer Science and Engineering, Florida Atlantic University, which is an R1. It also has many dozens of people with the title of editor at other presumably reputable western universities like Brunel in the UK, the University of Salerno, etc: https://link.springer.com/journal/11042/editorial-board https://link.springer.com/journal/11042/editorial-board Clearly, none of the so-called editors of the journal can be reading what's submitted to it. Zombie journals run by well known publishers like Spring Nature are common. They auto-publish blatant spam yet always have a gazillion editors at well known universities. This stuff is so basic both generation and detection predate LLMs entirely, but it doesn't get fixed. Then you get into all the papers that aren't trivially fake but fake in advanced undetectable ways, or which are merely using questionable research practices... the true rate of retraction if standards were at the level laymen imagine would be orders of magnitude higher.
- fc417fc802 2y ago
  > found by unpaid volunteers, not via some comprehensive process "Unpaid volunteers" describes the majority of the academic publication process so I'm not sure what you're point is. It's also a pretty reasonable approach - readers should report issues. This is exactly how moderation works the web over. Mind that I'm not arguing in favor of the status quo. Merely pointing out that this isn't some smoking gun. > you might object that the paper in question comes from India and not an R1 American university Yes, it does rather seem that you're trying to argue one thing (ie the mainstream scientific establishment of the western world is full of fraud) while selecting evidence from a rather different bucket (non-R1 institutions, journals that aren't mainstream, papers that aren't widely cited and were probably never read by anyone). > The journal itself does, though. It's edited by an academic in ... That isn't how anyone I've ever worked with assessed journal reputability. At a glance that journal doesn't look anywhere near high end to me. Remember that, just as with books, anyone can publish any scientific writeup that they'd like. By raw numbers, most published works of fiction aren't very high quality.[0] That doesn't say anything about the skilled fiction authors or the industry as a whole though. > but it doesn't get fixed. Is there a problem to begin with? People are publishing things. Are you seriously suggesting that we attempt to regulate what people are permitted to publish or who academics are permitted to associate with on the basis of some magical objective quality metric that doesn't currently exist? If you go searching for trash you will find trash. Things like industry and walk of life have little bearing on it. Trash is universal. You are lumping together a bunch of different things that no professional would ever consider to belong to the same category. If you want to critique mainstream scientific research then you need to present an analysis of sources that are widely accepted as being mainstream. [0] https://www.goodreads.com/book/show/18628458-taken-by-the-t-rex https://www.goodreads.com/book/show/18628458-taken-by-the-t-...

robotresearcher 2y ago

> More than 50% of published findings in psychology are false Wrong is not the same as fraudulent. 100% of Physics papers before Quantum Mechanics are false[1]. But not on purpose. [1] hyperbole, but you know what I mean.

mike_hearn 2y ago
It's hyperbole to the level that obfuscates, unfortunately. 50% of psych findings being wrong doesn't mean "right all the time except in exotic edge cases" like pre-quantum physics, it means they have no value at all and can't be salvaged. And very often the cause turns out to be fraud, which is why there is such a high rate of refusing to share the raw data from experiments - even when they signed agreements saying they'd do so on demand.

derbOac 2y ago

Not trying to be hostile but as a source on metrics, that one is grossly misleading in several ways. There's lots of problems with scientific publication but gish gallop is not the way to have an honest conversation about them.