7 ms·

I don’t actually believe that this is true if “academia” is defined as the set of reputable researchers from R1 schools and similar. If you define Academia as “

I don’t actually believe that this is true if “academia” is defined as the set of reputable researchers from R1 schools and similar. If you define Academia as “anyone anywhere in the world who submits research papers” then yes, it has vast amounts of fraud in the same way that most email is spam. Within the reputable set, as someone convinced that fraud is out of control, have you ever tried to calculate the fraud rate as a percentage with numerator and denominator (either number of papers published or number of reputable researchers. I would be very interested and stunned if it was over .1% or even .01%.

azan_ 2y ago
There is lots of evidence that p-hacking is widespread (some estimate that up to 20% are p-hacked). This problem also exists in top instutions, in fact in some fields it appears that this problem is WORSE in higher ranking unis - https://mitsloan.mit.edu/sites/default/files/inline-files/P-hacking%20in%20Experimental%20Accounting%20Studies.pdf https://mitsloan.mit.edu/sites/default/files/inline-files/P-...
- cycomanic 2y ago
  Where is that evidence? The paper you cite suggests that p hacking is done in experimental accounting studies but not archival. Generally speaking, evidence suggests that fraud rates are low ( lower than in most other human endeavours). This study cites 2% [1]. This is similar to numbers that Elizabeth Bik reports. For comparison self reported doping rates were between 6 and 9% here [2] [1] https://pmc.ncbi.nlm.nih.gov/articles/PMC5723807/ https://pmc.ncbi.nlm.nih.gov/articles/PMC5723807/ [2] https://pmc.ncbi.nlm.nih.gov/articles/PMC11102888/ https://pmc.ncbi.nlm.nih.gov/articles/PMC11102888/
  - mike_hearn 2y ago
    The 2% figure isn't a study of the fraud rate, it's just a survey asking academics if they've committed fraud themselves. Ask them to estimate how many other academics commit fraud and they say more like 10%-15%.
    - signatoremo 2y ago
      So which figure is more accurate in your opinion?
      - mike_hearn 2y ago
        See my other reply to Matthew. It's very dependent on how you define fraud, which field you look at, which country you look at, and a few other things. Depending on what you choose for those variables it can range from a few percent up to 100%.
    - cycomanic 2y ago
      Those 15% is actually if they know someone who has committed academic misconduct not fraud (although there is an overlap it's not the same), and it is across all levels (I.e. from PI to PhD student). So this will very likely overestimate fraud, as we would be double counting (I.e. Multiple reporters will know the same person). Imporantly the paper also says if people reported the misconduct it had consequences in the majority of cases. And just again for comparison >30% of elite athlete say that they know someone who doped.
mike_hearn 2y ago
There's an article that explores the metrics here: https://fantasticanachronism.com/2020/08/11/how-many-undetected-frauds-in-science/ https://fantasticanachronism.com/2020/08/11/how-many-undetec... > 0.04% of papers are retracted. At least 1.9% of papers have duplicate images "suggestive of deliberate manipulation". About 2.5% of scientists admit to fraud, and they estimate that 10% of other scientists have committed fraud. 27% of postdocs said they were willing to select or omit data to improve their results. More than 50% of published findings in psychology are false. The ORI, which makes about 13 misconduct findings per year, gives a conservative estimate of over 2000 misconduct incidents per year. Although publishing untrue claims isn't the same thing as fraud, editors of well known journals like The Lancet or the New England Journal of Medicine have estimated that maybe half or more of the claims they publish are wrong. Statistical consistency detectors run over psych papers find that ~50% fail such checks (e.g. that computed means are possible given the input data). The authors don't care, when asked to share their data so the causes of the check failures can be explored they just refuse or ignore the request, even if they signed a document saying they'd share. You don't have these sorts of problems in cryptography but a lot of fields are rife with it, especially if you use a definition of fraud that includes pseudoscientific practices. The article goes into some of the issues and arguments with how to define and measure it.
- matthewdgreen 2y ago
  0.04% is an extremely small number and (it needs to be said) also includes papers retracted due to errors and other good-faith corrections. Remember that we want people to retract flawed papers! treating it as evidence of fraud is not only a mischaracterization of the result but also a choice that is bad for a society that wants quality scientific results. The other two metrics seem pretty weak. 1.9% of papers in a vast database containing 40 journals show signs of duplication. But then dig into the details: apparently a huge fraction of those are in one journal and in two specific years. Look at Figure 1 and it just screams “something very weird is going on here, let’s look closely at this methodology before we accept the top line results.” The final result is a meta-survey based on surveys done across scientists all over the world, including surveys that are written in other languages, presumably based on scientists also publishing in smaller local journals. Presumably this covers a vast range of scientists with different reputations. As I said before, if you cast a wide net that includes everyone doing science in the entire world, I bet you’ll find tons of fraud. This study just seems to do that.
refulgentis 2y ago
I agree and am disappointed to see you in gray text. I'm old enough to have seen too many pendulum swings from new truth to thought-terminating cliche, and am increasingly frustrated by a game of telephone, over years, leading to it being common wisdom that research fraud is done all the time and its shrugged off. There's some real irony in that, as we wouldn't have gotten to this point a ton of self-policing over years where it was exposed with great consequence.

azan_ 2y ago

There is lots of evidence that p-hacking is widespread (some estimate that up to 20% are p-hacked). This problem also exists in top instutions, in fact in some fields it appears that this problem is WORSE in higher ranking unis - https://mitsloan.mit.edu/sites/default/files/inline-files/P-hacking%20in%20Experimental%20Accounting%20Studies.pdf https://mitsloan.mit.edu/sites/default/files/inline-files/P-...

cycomanic 2y ago
Where is that evidence? The paper you cite suggests that p hacking is done in experimental accounting studies but not archival. Generally speaking, evidence suggests that fraud rates are low ( lower than in most other human endeavours). This study cites 2% [1]. This is similar to numbers that Elizabeth Bik reports. For comparison self reported doping rates were between 6 and 9% here [2] [1] https://pmc.ncbi.nlm.nih.gov/articles/PMC5723807/ https://pmc.ncbi.nlm.nih.gov/articles/PMC5723807/ [2] https://pmc.ncbi.nlm.nih.gov/articles/PMC11102888/ https://pmc.ncbi.nlm.nih.gov/articles/PMC11102888/
- mike_hearn 2y ago
  The 2% figure isn't a study of the fraud rate, it's just a survey asking academics if they've committed fraud themselves. Ask them to estimate how many other academics commit fraud and they say more like 10%-15%.
  - signatoremo 2y ago
    So which figure is more accurate in your opinion?
    - mike_hearn 2y ago
      See my other reply to Matthew. It's very dependent on how you define fraud, which field you look at, which country you look at, and a few other things. Depending on what you choose for those variables it can range from a few percent up to 100%.
  - cycomanic 2y ago
    Those 15% is actually if they know someone who has committed academic misconduct not fraud (although there is an overlap it's not the same), and it is across all levels (I.e. from PI to PhD student). So this will very likely overestimate fraud, as we would be double counting (I.e. Multiple reporters will know the same person). Imporantly the paper also says if people reported the misconduct it had consequences in the majority of cases. And just again for comparison >30% of elite athlete say that they know someone who doped.

mike_hearn 2y ago

There's an article that explores the metrics here: https://fantasticanachronism.com/2020/08/11/how-many-undetected-frauds-in-science/ https://fantasticanachronism.com/2020/08/11/how-many-undetec... > 0.04% of papers are retracted. At least 1.9% of papers have duplicate images "suggestive of deliberate manipulation". About 2.5% of scientists admit to fraud, and they estimate that 10% of other scientists have committed fraud. 27% of postdocs said they were willing to select or omit data to improve their results. More than 50% of published findings in psychology are false. The ORI, which makes about 13 misconduct findings per year, gives a conservative estimate of over 2000 misconduct incidents per year. Although publishing untrue claims isn't the same thing as fraud, editors of well known journals like The Lancet or the New England Journal of Medicine have estimated that maybe half or more of the claims they publish are wrong. Statistical consistency detectors run over psych papers find that ~50% fail such checks (e.g. that computed means are possible given the input data). The authors don't care, when asked to share their data so the causes of the check failures can be explored they just refuse or ignore the request, even if they signed a document saying they'd share. You don't have these sorts of problems in cryptography but a lot of fields are rife with it, especially if you use a definition of fraud that includes pseudoscientific practices. The article goes into some of the issues and arguments with how to define and measure it.

matthewdgreen 2y ago
0.04% is an extremely small number and (it needs to be said) also includes papers retracted due to errors and other good-faith corrections. Remember that we want people to retract flawed papers! treating it as evidence of fraud is not only a mischaracterization of the result but also a choice that is bad for a society that wants quality scientific results. The other two metrics seem pretty weak. 1.9% of papers in a vast database containing 40 journals show signs of duplication. But then dig into the details: apparently a huge fraction of those are in one journal and in two specific years. Look at Figure 1 and it just screams “something very weird is going on here, let’s look closely at this methodology before we accept the top line results.” The final result is a meta-survey based on surveys done across scientists all over the world, including surveys that are written in other languages, presumably based on scientists also publishing in smaller local journals. Presumably this covers a vast range of scientists with different reputations. As I said before, if you cast a wide net that includes everyone doing science in the entire world, I bet you’ll find tons of fraud. This study just seems to do that.

refulgentis 2y ago

I agree and am disappointed to see you in gray text. I'm old enough to have seen too many pendulum swings from new truth to thought-terminating cliche, and am increasingly frustrated by a game of telephone, over years, leading to it being common wisdom that research fraud is done all the time and its shrugged off. There's some real irony in that, as we wouldn't have gotten to this point a ton of self-policing over years where it was exposed with great consequence.