5 ms·

How Much Science Is Fake?

29 points by doener 2y ago

ZYbCRq22HbJ2y7 2y ago
> In this preprint, I try to quantify how much science presently being published is fake (that is, fabricated or falsified). The working estimate is approximately 1 in 7. This should make a lot of people very angry and will be widely regarded as a bad move. Sounds incredibly legit, really motivates me to read the article https://osf.io/s4gce https://osf.io/s4gce
- thoroughburro 2y ago
  It’s a Douglas Adams quote.
- zellyn 2y ago
  As another comment pointed out, it's a Douglas Adams quote. Also, are we not hackers? If we can't sprinkle jokes into discussions of serious things without thinking that makes the whole discussion unserious, what have we become? Might as well go join HR or become a politician or something…
  - mewpmewp2 2y ago
    It's a quote that initially was in a hilarious context, but here it's seemingly out of place, and not really funny, almost feels forced in there randomly so it comes off pretentious. Maybe if it was something like "Once upon a time humankind discovered the scientific method, the most accurate tool that they had so far. This has made a lot of people very angry and has been widely regarded as a bad move. To help with that here we have spotted some inaccurate articles to appease the angry."
Isamu 2y ago
Using OSF storage as a proxy. The title is a bit clickbait, that alone gives you an idea of the incentives that cause people to publish when they don’t have something. Not to take away from the premise, but people already know there are some fields where faking is rampant. An interesting paper would take the existing LLMs and see how well they can do flagging problematic work. A solution that works well at scale is needed. Heck, an AI assistant to aid peer review would be great.
- LinuxBender 2y ago
  An interesting paper would take the existing LLMs and see how well they can do flagging problematic work. I would like to see this as well. I would be curious if LLM's start to see the same patterns I see having read half of nih.gov going down many rabbit holes of researchers scratching each others backs, pencil whipping reviews and such. Can LLM's pick up on such behavior or might that require something closer to human intelligence? I do not know enough about LLM's to even guess. Could it perhaps at least pick up on entirely fake papers from Youtubers pranking the system to see if they can get their silliest papers published? I report them but by the time they are taken down the Youtubers have had a large number of views and laughs.
mbreese 2y ago
I don’t think the issue is how much of “science” is fake. It’s how you define “science”. There have always been lower quality work and journals that are inherently given lower weight in the scientific community. Many of these lower quality papers just aren’t considered by working scientists. It’s a bit like email… once you start seeing a significant amount of spam, you just get used to ignoring it. This is a feature of the system. Scientific consensus can change over time, but it takes a lot of evidence to get there. But some papers get more weight than others. For me, there are two real issues - first, public trust in science. When the public thinks that 14% or “science” is fake, they are likely to discount all scientific findings, regardless of source. This can be dangerous, especially for a public who is already distrustful of science/academia in general. So, instead of looking at results with a critical eye, the danger is that a large fraction of the public will just ignore all results. The second issue is in polluting AI training. When you feed extra garbage into a system, you’ll get garbage out of the system. As LLMs get more specialized having high quality inputs for training is critically important. A contributing force here is that lower tier journals (those most susceptible to faked data), are also the most likely to make the full text of papers widely available. So, the easiest data to get for training is also the most likely to have faked data.
- matusp 2y ago
  It also how you define "fake", eg in machine learning it can be everything from fabricated results to data leakage, low quality data, bugs, p-hacking, weak baseline design, insufficient handling of randomness, etc etc.
- mgamache 2y ago
  14% is a lower bound. I suspect if you include studies that can't be replicated it's well over 50%. And yes with that much garbage the public should dismiss all new published scientific findings. Modern scientific publishing has been gamed it's nearly useless for learning anything directly because you can't trust it. Some fields more than others of course, but it's all bad. Yes this includes tier 1 journals.
  - QuadmasterXLII 2y ago
eig 2y ago
I would not be surprised by this number given the sheer volume of crap journals that publish anything. However I would expect that the proportion of fakery in papers with >20 citations is much lower than 1 in 7, which should be somewhat reassuring. Remember that a p-value of 0.05 means that 1 in 20 studies cannot be replicated by random chance alone without any malicious intent. Side note: the “1 in 7” claim from this paper is based on a straw-poll of N=12 existing forensic metascientific papers. I find it somewhat ironic that the author makes some strong conclusions based on this number which to me, though very plausible, is itself lacking scientific rigor.
derbOac 2y ago
Although identifying outright fraud is important, I think collectively the bigger, more systemic damage is done by other things that are more subtle: material that falls in a gray area (of questionable research practices, or even less obvious than that), studies that repeat what is obvious at some point (and I don't mean replication per se, which is important, but something more basic than that), the way credit is asserted and inferred, and so forth and so on. Identifying fraud is critical but I worry a bit that baserates of such statistics get processed as an estimate of "what proportion of science is problematic", when the problems are broader than that.
Bellf 2y ago
I’m a computer science student at a major university in my country, and I completely agree. I’ve come across many papers that seem questionable, and there are powerful individuals in the scientific community who only accept work from a select group. It feels like these people are building their reputations on what often appears to be fake or unsubstantiated research.
arminiusreturns 2y ago
Lots. I am just a graybeard sysadmin type, but I got taught how to read papers critically by a world-class geneticist because we were taking papers to implementation in prod. The biggest takeaway for me was that citation padding networks and lack of reproducibility were the biggest issues and were prevalent, and this was in the top journals, you can't just say this stuff happens in the "lesser" ones!
cynicalpeace 2y ago
40-60% of scientific studies are not replicable, therefore fake as far as I'm concerned. That's why "show me the study" is such a midwit line of reasoning. We need to go back to thinking more with first principles, received wisdom, and personal experimentation. There's a place for scientific studies, but it's gone so far that many won't take an obvious point without a study behind it. There are many such examples that are quite obvious but are hard to prove: processed foods are bad, screen time is bad for kids, starship won't fall on a shark etc. But people love to poke holes in obvious truths, especially in well educated spaces like HN.
- bloated5048 2y ago
  I agree. If we relied on anecdotal evidence more often we'd have resorts on Mars by now.
  - cynicalpeace 2y ago
    "Please respond to the strongest plausible interpretation of what someone says, not a weaker one that's easier to criticize." Assuming you're being sarcastic. I never said anecdotal evidence. And indeed all 3 (first principles, received wisdom, and personal experimentation) are helping get to space, arguably more than research papers
- davisoneee 2y ago
  I think describing it as 'fake' is rather antagonistic. You could just as easily say 'poor quality' or 'unjustified'. I would say that more accurately describes the situation. It could be... - unjustified and sloppy (not 'fake', but also not considered reliable evidence) - unjustified and malicious (this i would consider 'fake') - unjustified and gamed (again, 'fake') - ...or just unjustified and under-specified (and would result in 'true' results if the conditions for replication were better studied and defined) Lots of people like to think themselves smart for following 'first principles'...and then often end up falling in the same ditches. First principles + received wisdom is a bit of a contradiction... if it's 'wisdom' rather than evidence, you are skipping your principles to go with the received starting point...
deleted 2y ago
[deleted]
lapcat 2y ago
1) The paper defines "fake" in this first sentence: > ‘Fake’ science is either intentionally fabricated-where quantitative elements are invented-or intentionally falsified-where results are dishonestly engineered from real data. 2) The title "How much science is fake?" is clickbaity. The actual assertion of the paper: > a preliminary approximation is that 1 in 7 published papers have serious errors commensurate with being untrustworthy. 3) The paper is a meta-analysis of 12 other forensic metascience papers. The approximation of 1 in 7 (14%) is an average of estimates derived from those other papers. 4) It's important to note that publishing papers is just one small subset of the massive endeavor that we call science. Moreover, as far as I'm aware, "publish or perish" appears to be a relatively recent historical phenomena of professional academia. We could probably do without many published papers, which exist merely to advance the careers of the authors. 5) In any human endeavor that involves money and/or power—which is almost everything—there will be corruption. It's unavoidable, given human nature. Thus, there's nothing uniquely untrustworthy about science. And science was never intended to be a religion: self-criticism is supposed to be a feature, not a bug, so my personal view is that the open study of fake papers is a sign of health rather than a sign of disease. 6) Is 1 in 7 really that bad? The converse is that 86% of published papers do not have serious errors commensurate with being untrustworthy.
- mewpmewp2 2y ago
  Ironically if this paper intentionally uses an incorrect and misleading term "fake", to label and then claim that 1 in 7 are fake, would it make the paper itself "fake" according to their standards? Because they find 1 in 7 with serious errors, but their definition of "fake" requires it to be intentional. Where's the evidence it's intentional for all of these 1 in 7?
  - mike_hearn 2y ago
    It says "serious errors commensurate with being untrustworthy", which I think is meant to be an alternative way to say commensurate with being fake. Arguments about intentionality are a bit of a distraction. Academics routinely try and excuse obvious fakery by claiming to be incredibly incompetent. At some point you have to treat unlimited incompetence as harshly as deliberate deception.
johnea 2y ago
The space lasers guiding hurricanes along the chem-trails is totally real! Just ask anyone in the braintrust states...

29 points by doener 2y ago

> In this preprint, I try to quantify how much science presently being published is fake (that is, fabricated or falsified). The working estimate is approximately 1 in 7. This should make a lot of people very angry and will be widely regarded as a bad move. Sounds incredibly legit, really motivates me to read the article https://osf.io/s4gce https://osf.io/s4gce

thoroughburro 2y ago
It’s a Douglas Adams quote.
zellyn 2y ago
As another comment pointed out, it's a Douglas Adams quote. Also, are we not hackers? If we can't sprinkle jokes into discussions of serious things without thinking that makes the whole discussion unserious, what have we become? Might as well go join HR or become a politician or something…
- mewpmewp2 2y ago
  It's a quote that initially was in a hilarious context, but here it's seemingly out of place, and not really funny, almost feels forced in there randomly so it comes off pretentious. Maybe if it was something like "Once upon a time humankind discovered the scientific method, the most accurate tool that they had so far. This has made a lot of people very angry and has been widely regarded as a bad move. To help with that here we have spotted some inaccurate articles to appease the angry."

Isamu 2y ago

Using OSF storage as a proxy. The title is a bit clickbait, that alone gives you an idea of the incentives that cause people to publish when they don’t have something. Not to take away from the premise, but people already know there are some fields where faking is rampant. An interesting paper would take the existing LLMs and see how well they can do flagging problematic work. A solution that works well at scale is needed. Heck, an AI assistant to aid peer review would be great.

LinuxBender 2y ago
An interesting paper would take the existing LLMs and see how well they can do flagging problematic work. I would like to see this as well. I would be curious if LLM's start to see the same patterns I see having read half of nih.gov going down many rabbit holes of researchers scratching each others backs, pencil whipping reviews and such. Can LLM's pick up on such behavior or might that require something closer to human intelligence? I do not know enough about LLM's to even guess. Could it perhaps at least pick up on entirely fake papers from Youtubers pranking the system to see if they can get their silliest papers published? I report them but by the time they are taken down the Youtubers have had a large number of views and laughs.

mbreese 2y ago

I don’t think the issue is how much of “science” is fake. It’s how you define “science”. There have always been lower quality work and journals that are inherently given lower weight in the scientific community. Many of these lower quality papers just aren’t considered by working scientists. It’s a bit like email… once you start seeing a significant amount of spam, you just get used to ignoring it. This is a feature of the system. Scientific consensus can change over time, but it takes a lot of evidence to get there. But some papers get more weight than others. For me, there are two real issues - first, public trust in science. When the public thinks that 14% or “science” is fake, they are likely to discount all scientific findings, regardless of source. This can be dangerous, especially for a public who is already distrustful of science/academia in general. So, instead of looking at results with a critical eye, the danger is that a large fraction of the public will just ignore all results. The second issue is in polluting AI training. When you feed extra garbage into a system, you’ll get garbage out of the system. As LLMs get more specialized having high quality inputs for training is critically important. A contributing force here is that lower tier journals (those most susceptible to faked data), are also the most likely to make the full text of papers widely available. So, the easiest data to get for training is also the most likely to have faked data.

matusp 2y ago
It also how you define "fake", eg in machine learning it can be everything from fabricated results to data leakage, low quality data, bugs, p-hacking, weak baseline design, insufficient handling of randomness, etc etc.
mgamache 2y ago
14% is a lower bound. I suspect if you include studies that can't be replicated it's well over 50%. And yes with that much garbage the public should dismiss all new published scientific findings. Modern scientific publishing has been gamed it's nearly useless for learning anything directly because you can't trust it. Some fields more than others of course, but it's all bad. Yes this includes tier 1 journals.
- QuadmasterXLII 2y ago

eig 2y ago

I would not be surprised by this number given the sheer volume of crap journals that publish anything. However I would expect that the proportion of fakery in papers with >20 citations is much lower than 1 in 7, which should be somewhat reassuring. Remember that a p-value of 0.05 means that 1 in 20 studies cannot be replicated by random chance alone without any malicious intent. Side note: the “1 in 7” claim from this paper is based on a straw-poll of N=12 existing forensic metascientific papers. I find it somewhat ironic that the author makes some strong conclusions based on this number which to me, though very plausible, is itself lacking scientific rigor.

derbOac 2y ago

Although identifying outright fraud is important, I think collectively the bigger, more systemic damage is done by other things that are more subtle: material that falls in a gray area (of questionable research practices, or even less obvious than that), studies that repeat what is obvious at some point (and I don't mean replication per se, which is important, but something more basic than that), the way credit is asserted and inferred, and so forth and so on. Identifying fraud is critical but I worry a bit that baserates of such statistics get processed as an estimate of "what proportion of science is problematic", when the problems are broader than that.

Bellf 2y ago

I’m a computer science student at a major university in my country, and I completely agree. I’ve come across many papers that seem questionable, and there are powerful individuals in the scientific community who only accept work from a select group. It feels like these people are building their reputations on what often appears to be fake or unsubstantiated research.

arminiusreturns 2y ago

Lots. I am just a graybeard sysadmin type, but I got taught how to read papers critically by a world-class geneticist because we were taking papers to implementation in prod. The biggest takeaway for me was that citation padding networks and lack of reproducibility were the biggest issues and were prevalent, and this was in the top journals, you can't just say this stuff happens in the "lesser" ones!

cynicalpeace 2y ago

40-60% of scientific studies are not replicable, therefore fake as far as I'm concerned. That's why "show me the study" is such a midwit line of reasoning. We need to go back to thinking more with first principles, received wisdom, and personal experimentation. There's a place for scientific studies, but it's gone so far that many won't take an obvious point without a study behind it. There are many such examples that are quite obvious but are hard to prove: processed foods are bad, screen time is bad for kids, starship won't fall on a shark etc. But people love to poke holes in obvious truths, especially in well educated spaces like HN.

bloated5048 2y ago
I agree. If we relied on anecdotal evidence more often we'd have resorts on Mars by now.
- cynicalpeace 2y ago
  "Please respond to the strongest plausible interpretation of what someone says, not a weaker one that's easier to criticize." Assuming you're being sarcastic. I never said anecdotal evidence. And indeed all 3 (first principles, received wisdom, and personal experimentation) are helping get to space, arguably more than research papers
davisoneee 2y ago
I think describing it as 'fake' is rather antagonistic. You could just as easily say 'poor quality' or 'unjustified'. I would say that more accurately describes the situation. It could be... - unjustified and sloppy (not 'fake', but also not considered reliable evidence) - unjustified and malicious (this i would consider 'fake') - unjustified and gamed (again, 'fake') - ...or just unjustified and under-specified (and would result in 'true' results if the conditions for replication were better studied and defined) Lots of people like to think themselves smart for following 'first principles'...and then often end up falling in the same ditches. First principles + received wisdom is a bit of a contradiction... if it's 'wisdom' rather than evidence, you are skipping your principles to go with the received starting point...

deleted 2y ago

[deleted]

lapcat 2y ago

1) The paper defines "fake" in this first sentence: > ‘Fake’ science is either intentionally fabricated-where quantitative elements are invented-or intentionally falsified-where results are dishonestly engineered from real data. 2) The title "How much science is fake?" is clickbaity. The actual assertion of the paper: > a preliminary approximation is that 1 in 7 published papers have serious errors commensurate with being untrustworthy. 3) The paper is a meta-analysis of 12 other forensic metascience papers. The approximation of 1 in 7 (14%) is an average of estimates derived from those other papers. 4) It's important to note that publishing papers is just one small subset of the massive endeavor that we call science. Moreover, as far as I'm aware, "publish or perish" appears to be a relatively recent historical phenomena of professional academia. We could probably do without many published papers, which exist merely to advance the careers of the authors. 5) In any human endeavor that involves money and/or power—which is almost everything—there will be corruption. It's unavoidable, given human nature. Thus, there's nothing uniquely untrustworthy about science. And science was never intended to be a religion: self-criticism is supposed to be a feature, not a bug, so my personal view is that the open study of fake papers is a sign of health rather than a sign of disease. 6) Is 1 in 7 really that bad? The converse is that 86% of published papers do not have serious errors commensurate with being untrustworthy.

mewpmewp2 2y ago
Ironically if this paper intentionally uses an incorrect and misleading term "fake", to label and then claim that 1 in 7 are fake, would it make the paper itself "fake" according to their standards? Because they find 1 in 7 with serious errors, but their definition of "fake" requires it to be intentional. Where's the evidence it's intentional for all of these 1 in 7?
- mike_hearn 2y ago
  It says "serious errors commensurate with being untrustworthy", which I think is meant to be an alternative way to say commensurate with being fake. Arguments about intentionality are a bit of a distraction. Academics routinely try and excuse obvious fakery by claiming to be incredibly incompetent. At some point you have to treat unlimited incompetence as harshly as deliberate deception.

johnea 2y ago

The space lasers guiding hurricanes along the chem-trails is totally real! Just ask anyone in the braintrust states...