4 ms·

Estimating from my own reading, 1% of research is fraud and 80% is worthless for other reasons. Numbers vary between fields. If that's the case, what's the arg

by tlb 6y ago

Estimating from my own reading, 1% of research is fraud and 80% is worthless for other reasons. Numbers vary between fields. If that's the case, what's the argument for why we should spend time doing something about the 1%? Solving 100% of the 1% wouldn't change the overall situation much. Possible arguments include: - Fixing the 80% is hard, but fixing the 1% is satisfying (to the aggressively conventional-minded, at least.) - The 1% is wrong in a more harmful way than the 80%. Perhaps falsifying data is worse than hand-waving conclusions. So if the maximum upside is 1% of wrong research removed, and the downside is quenching some fraction of the good 19%, it's probably better to leave it alone.

deleted 6y ago
[deleted]
ameliaquining 6y ago
The author is arguing for an expansive definition of fraud that includes things like p-hacking. Probably a lot more than 1% of research is p-hacked; that general category of problem might even account for a majority of your 80%.
- opportune 6y ago
  I'm guessing throwing out "bad data" happens in a lot more than 1% of cases as well.
ketzo 6y ago
I think the disproportionate harm argument is a big deal. A single high-profile case of fraud undermines trust in the entire field, and that’s critical to maintain.
- anonthrowaway3 6y ago
  Why is that critical to maintain? No one in the public should trust academic science given what a complete cesspool academia is like.
- galimaufry 6y ago
  I don't think that's true. There have been frauds in materials science, but people still trust it. The fields that are untrusted are the ones that are so hard that it's plausible that the conventional wisdom might be wrong even if everyone is honest.
sillysaurusx 6y ago
It's hard to know that a worthless research paper will always be worthless.
- deleted 6y ago
  [deleted]
- TuringNYC 6y ago
  Also, who decides what is worthless?
moralestapia 6y ago
>1% of research is fraud You are way, waaaaaay underestimating the extent of the problem.
- exmadscientist 6y ago
  I think you didn't understand the grandparent poster's point. They're arguing, basically, that only 1% of research is deliberate fraud, but 80% is simply useless garbage. The reasons for the garbage are varied, but any way you slice it, it's worthless. So why worry so much about the maliciously useless when there's so much incompetent uselessness? That matches my experience back when I was an active researcher. Intentional fraud is rare (and highly damaging). Garbage research and garbage papers are everywhere. And even most of the non-garbage research is useless! The core differentiator there is that we just don't know which things will be useful without hindsight. But we can identify plenty of things that were not helpful to anyone and we knew it... yet we still funded those programs. I think we're mostly in agreement that the size of the problem is vast, even if we might disagree on labelling.
  - moralestapia 6y ago
    Perhaps I was in an extremely corrupt environment (which eventually made me leave), but I would put the "intentional fraud" rate as high as 30% or so. I pointed out one of such instances once and that was the beginning of the end for me in that place. More: My field was Biology. No one double checks anyone or anything. A measurement comes back and you can just pretty much ignore it, in the end you just write whatever you need for your thesis to hold (I mean not me, but that's pretty much everybody's dirty secret). In the rare case that someone actually wants to check it out (1 in 1,000), you can just say that the "original data was lost". Sometimes the only thing you can trace back is some sort of written journal, in which anybody could write anything and that doesn't make it true/false. Not long ago, there was a group of scientists that went on to try to replicate some landmark results on cancer research (I'll update with a good reference if I find it) and found out that only a very low number of them matched their observations. Some of those were not even close ... like, waaaay outside a very generous experimental margin of error.

by tlb 6y ago

deleted 6y ago

[deleted]

ameliaquining 6y ago

The author is arguing for an expansive definition of fraud that includes things like p-hacking. Probably a lot more than 1% of research is p-hacked; that general category of problem might even account for a majority of your 80%.

opportune 6y ago
I'm guessing throwing out "bad data" happens in a lot more than 1% of cases as well.

ketzo 6y ago

I think the disproportionate harm argument is a big deal. A single high-profile case of fraud undermines trust in the entire field, and that’s critical to maintain.

anonthrowaway3 6y ago
Why is that critical to maintain? No one in the public should trust academic science given what a complete cesspool academia is like.
galimaufry 6y ago
I don't think that's true. There have been frauds in materials science, but people still trust it. The fields that are untrusted are the ones that are so hard that it's plausible that the conventional wisdom might be wrong even if everyone is honest.

sillysaurusx 6y ago

It's hard to know that a worthless research paper will always be worthless.

deleted 6y ago
[deleted]
TuringNYC 6y ago
Also, who decides what is worthless?

moralestapia 6y ago

>1% of research is fraud You are way, waaaaaay underestimating the extent of the problem.

exmadscientist 6y ago
I think you didn't understand the grandparent poster's point. They're arguing, basically, that only 1% of research is deliberate fraud, but 80% is simply useless garbage. The reasons for the garbage are varied, but any way you slice it, it's worthless. So why worry so much about the maliciously useless when there's so much incompetent uselessness? That matches my experience back when I was an active researcher. Intentional fraud is rare (and highly damaging). Garbage research and garbage papers are everywhere. And even most of the non-garbage research is useless! The core differentiator there is that we just don't know which things will be useful without hindsight. But we can identify plenty of things that were not helpful to anyone and we knew it... yet we still funded those programs. I think we're mostly in agreement that the size of the problem is vast, even if we might disagree on labelling.
- moralestapia 6y ago
  Perhaps I was in an extremely corrupt environment (which eventually made me leave), but I would put the "intentional fraud" rate as high as 30% or so. I pointed out one of such instances once and that was the beginning of the end for me in that place. More: My field was Biology. No one double checks anyone or anything. A measurement comes back and you can just pretty much ignore it, in the end you just write whatever you need for your thesis to hold (I mean not me, but that's pretty much everybody's dirty secret). In the rare case that someone actually wants to check it out (1 in 1,000), you can just say that the "original data was lost". Sometimes the only thing you can trace back is some sort of written journal, in which anybody could write anything and that doesn't make it true/false. Not long ago, there was a group of scientists that went on to try to replicate some landmark results on cancer research (I'll update with a good reference if I find it) and found out that only a very low number of them matched their observations. Some of those were not even close ... like, waaaay outside a very generous experimental margin of error.