6 ms·

While we're on the subject, how many studies period are faked or flawed to the point of being useless? It seems to me that the scientific community's reaction t

While we're on the subject, how many studies period are faked or flawed to the point of being useless? It seems to me that the scientific community's reaction to the replication crisis has been to ignore it.

thereisnospork 3y ago
> It seems to me that the scientific community's reaction to the replication crisis has been to ignore it. They've always known so there hasn't actually been any new information from which to spur action. In the academic circles I've run in there has always been a strong mistrust of reported results and procedures based on past difficulties with internal efforts replicating results. Basically a right of passage for a grad student to be tasked with replicating work from an impossible paper.
- HarryHirsch 3y ago
  That's the correct answer, but what is baffling is that this is news to so many people frequenting this website. Most everyone posting here has been though a university, but hardly anyone has been involved in the pursuit of research.
onthecanposting 3y ago
PIs may not be able to raise money for replication studies. Some of this is a consequence of guidelines for federal funds to prevent waste on duplication of effort.
- some_random 3y ago
  This goes far beyond there not being enough replication studies, the problem is that when replication studies are done the results don't reproduce because the results were bogus. https://en.wikipedia.org/wiki/Replication_crisis https://en.wikipedia.org/wiki/Replication_crisis
  - whatshisface 3y ago
    If the replication studies were done at a reasonable rate there would be no incentive to produce bogus results because you'd be caught before you could go through an entire career as a "successful scientist."
    - mike_hearn 3y ago
      There is no being caught :( Who is going to do the catching, exactly? There's no science police. Universities don't seem to care about anything except plagiarism. Many papers aren't even "looks legit but doesn't replicate", they are self-evidently wrong just from reading them or the associated data. Just peer review done properly would solve that, but the problem is widespread because there's no real incentive to rock the boat. Once a bad technique or invalid approach gets through to publication a few times it becomes a new standard within the field because it lets people publish more papers.
      - whatshisface 3y ago
        I think the most natural place for reform is from the people who are paying for the research, if they realized that what they are spending money for has no value if it's being made up. Although there are a lot of scientists who by the principle of self-interest seem willing to waste their life in exchange for a small salary it makes less sense why the NSF would take that offer.
        mike_hearn 3y ago
        But that can't happen at the moment. The core problem is that civil servants and politicians are easily bullied by claims they aren't experts and thus can't judge if "expert" work is any good or not. So they just dish out money to anyone who works at a university, expecting them to self-police, but they don't. There are only two ways to fix that, as far as I can see: 1. Have highly technical, scientifically trained and skeptical politicians who police academia and ensure research money is well spent. 2. Stop governments funding research in the first place. (1) just doesn't seem feasible. There are so many problems there. (2) is feasible. So I suspect you're going to start seeing calls from the right to defund universities over the next ten years or so. The underlying motivation may be that universities are strongholds of the left, but the stated justification will be the high levels of research fraud.
- bee_rider 3y ago
  This points to the real problem and also where the responsibility and interest ought to be to fix it. There’s no replication crisis for academics because they have a meatspace social network of academics; they go to conferences together and know each other. You can just ignore a paper if you know the author is an idiot. If medical studies are faked, is it a problem? Presumable regulatory agencies are using these studies or something, right? Looks like the FDA and NSF need to fund some more replication studies.
  - jonlucc 3y ago
    At least in pre-clinical research, there’s also a mechanism by which regulatory agencies can sometimes force industry to pay for replication studies. It’s a little complicated, because sometimes there aren’t many groups who can reliably do the study, and clinical trials are so expensive and time-consuming to run, it would be difficult to do at that scale. Rodent studies are generally under $1M and monkey studies are generally under $5M (up a lot recently). Clinical trials are in the hundreds of millions of dollars and take years to run.
systemvoltage 3y ago
> has been to ignore it This is putting it very mildly.
RugnirViking 3y ago
> the scientific community's reaction to the replication crisis has been to ignore it What can they do? It's an incredibly hard problem to solve. It's like asking why the buisness community has done nothing to adress the housing crisis. Large scale culture changes, or the entire structure of the way science is conducted and funded, would be the only solutions
- colechristensen 3y ago
  If a university was actually in the business of seeking truth, there is much they could do to solve this. * working groups "red teams" whose whole job it is to find weaknesses in papers published at the university * post mortems after finding papers with serious flaws exposing the problem and coming up with constructive corrective actions * funding / forcing researchers to devote a certain amount of time to replicating significant results * working groups of experts in statistics and study design available to consult * systems to register studies, methodologies, data sets, etc. with the aim of eliminating error and preventing post-hoc fishing expeditions for results The whole-ass purpose of a university is seeking knowledge. They are fully capable of doing a better job of it but they don't because what they actually focus on are things like fundraising, professional sports teams, constructing attractive buildings, and advancing ranking. Most universities would be better off just firing and not replacing 90% of their administration.
  - jltsiren 3y ago
    Universities can't do that, because they lack the expertise. Most of the time, the only people in a university capable of judging a study are the people who did it. That's because universities are in the business of teaching. Apart from a few rare exceptions, universities don't have the money to hire redundant people. Instead of hiring many experts in the same topic, they prefer hiring a wider range of expertise, in order to provide better learning opportunities for the students.
    - mkl 3y ago
      > Universities can't do that, because they lack the expertise. Most of the time, the only people in a university capable of judging a study are the people who did it. Statisticians can find faults in many studies. Universities even employ statisticians to act as consultants for research in other departments, as that raises the quality of the institution's publications.
tppiotrowski 3y ago
> While we're on the subject, how many studies period are faked or flawed to the point of being useless? Or how many studies are useless, period? It's like publishing a memoir to Amazon. You can now say "author" on your resume, or when you're introduced or at cocktail parties but nobody finds any value in what you have to say. You can also use ChatGPT because people might not notice.
- some_random 3y ago
  There is always value in expanding the breadth and depth of human knowledge, even if it doesn't seem useful to you, right now. That of course assumes that knowledge is true, which is the crux of the problem now.
  - yellowcake0 3y ago
    Unfortunately, the preponderance of very low value research in the literature puts a significant burden on the scientists who have to sift through a lot of garbage to find what they're looking for. Even if the work is ostensibly correct (much of it is not), it really doesn't do anyone much good, except for the authors of course. But now every undergraduate and every parent's vanity project at Andover wants a first author contrib., so here we are.
    - tppiotrowski 3y ago
      I think language models will help us sift through the garbage. I have a hard time finding answers using Google (too much SEO garbage), but ChatGPT does a pretty good job of filtering the noise. It's not perfect, but it saves lots of time vs. Google.
adamsb6 3y ago
The replication crisis is even a thing because of very strong career incentives. You might think that publishing about the replication crisis itself would be great for your career, but perhaps not. Maybe the incentives to be able to bullshit your way to a professorship are so great that no one wants to rock the boat.
- rqtwteye 3y ago
  "Maybe the incentives to be able to bullshit your way to a professorship are so great that no one wants to rock the boat." Our whole economy is fueled by people bullshitting each other.
  - wizzwizz4 3y ago
    Our economy is fuelled by people doing actual work. Large-ish, visible parts are shaped by bullshitters, but that's a different thing.
- spicybright 3y ago
  Talking with a few students in science programs this seems to be the case, even when few higher ups want to admit it. Proving something is false is less valued than proving it right. It's silly because if we valued the quality of science it should be the exact opposite.
treis 3y ago
My internal filter is: Sociology/Psych/Economics are almost all junk. Their conclusions may or may not be correct. Medical studies are mostly junk. There's way too much financial incentive to show marginal improvement. Theraflu and anti-depressents come to mind. Both show a small effect in studies and launched billion dollar businesses. Hard science stuff tends to be pretty good. Mostly just outright fraud and they usually end up getting caught.
- fnikacevic 3y ago
  Chemistry still has a ton of issues with people exaggerating numbers that can't be contested like yield and purity of reactions even among the top journals. Straight up fraud is rarer yes.
- mike_hearn 3y ago
  I used to think like that. Soft science = dubious, hard science = ok. Not any more though. Just recently there was LK99 in materials science though, which was pretty much the textbook definition of an exciting but non-replicable claim. Also a lot of stuff is tough to classify. Are epidemiology or climatology "hard" sciences?
a123b456c 3y ago
The scientific community ignored the replication crisis? Dude. The scientific community created the replication crisis. We are not ignoring it, I promise you. The reaction is nuanced by no one who matters is ignoring it.
thenickdude 3y ago
Check out https://retractionwatch.com/ https://retractionwatch.com/ , a continuous stream of exposés on authors publishing flawed or fraudulent works.

thereisnospork 3y ago

> It seems to me that the scientific community's reaction to the replication crisis has been to ignore it. They've always known so there hasn't actually been any new information from which to spur action. In the academic circles I've run in there has always been a strong mistrust of reported results and procedures based on past difficulties with internal efforts replicating results. Basically a right of passage for a grad student to be tasked with replicating work from an impossible paper.

HarryHirsch 3y ago
That's the correct answer, but what is baffling is that this is news to so many people frequenting this website. Most everyone posting here has been though a university, but hardly anyone has been involved in the pursuit of research.

onthecanposting 3y ago

PIs may not be able to raise money for replication studies. Some of this is a consequence of guidelines for federal funds to prevent waste on duplication of effort.

some_random 3y ago
This goes far beyond there not being enough replication studies, the problem is that when replication studies are done the results don't reproduce because the results were bogus. https://en.wikipedia.org/wiki/Replication_crisis https://en.wikipedia.org/wiki/Replication_crisis
- whatshisface 3y ago
  If the replication studies were done at a reasonable rate there would be no incentive to produce bogus results because you'd be caught before you could go through an entire career as a "successful scientist."
  - mike_hearn 3y ago
    There is no being caught :( Who is going to do the catching, exactly? There's no science police. Universities don't seem to care about anything except plagiarism. Many papers aren't even "looks legit but doesn't replicate", they are self-evidently wrong just from reading them or the associated data. Just peer review done properly would solve that, but the problem is widespread because there's no real incentive to rock the boat. Once a bad technique or invalid approach gets through to publication a few times it becomes a new standard within the field because it lets people publish more papers.
    - whatshisface 3y ago
      I think the most natural place for reform is from the people who are paying for the research, if they realized that what they are spending money for has no value if it's being made up. Although there are a lot of scientists who by the principle of self-interest seem willing to waste their life in exchange for a small salary it makes less sense why the NSF would take that offer.
      - mike_hearn 3y ago
        But that can't happen at the moment. The core problem is that civil servants and politicians are easily bullied by claims they aren't experts and thus can't judge if "expert" work is any good or not. So they just dish out money to anyone who works at a university, expecting them to self-police, but they don't. There are only two ways to fix that, as far as I can see: 1. Have highly technical, scientifically trained and skeptical politicians who police academia and ensure research money is well spent. 2. Stop governments funding research in the first place. (1) just doesn't seem feasible. There are so many problems there. (2) is feasible. So I suspect you're going to start seeing calls from the right to defund universities over the next ten years or so. The underlying motivation may be that universities are strongholds of the left, but the stated justification will be the high levels of research fraud.
bee_rider 3y ago
This points to the real problem and also where the responsibility and interest ought to be to fix it. There’s no replication crisis for academics because they have a meatspace social network of academics; they go to conferences together and know each other. You can just ignore a paper if you know the author is an idiot. If medical studies are faked, is it a problem? Presumable regulatory agencies are using these studies or something, right? Looks like the FDA and NSF need to fund some more replication studies.
- jonlucc 3y ago
  At least in pre-clinical research, there’s also a mechanism by which regulatory agencies can sometimes force industry to pay for replication studies. It’s a little complicated, because sometimes there aren’t many groups who can reliably do the study, and clinical trials are so expensive and time-consuming to run, it would be difficult to do at that scale. Rodent studies are generally under $1M and monkey studies are generally under $5M (up a lot recently). Clinical trials are in the hundreds of millions of dollars and take years to run.

systemvoltage 3y ago

> has been to ignore it This is putting it very mildly.

RugnirViking 3y ago

> the scientific community's reaction to the replication crisis has been to ignore it What can they do? It's an incredibly hard problem to solve. It's like asking why the buisness community has done nothing to adress the housing crisis. Large scale culture changes, or the entire structure of the way science is conducted and funded, would be the only solutions

colechristensen 3y ago
If a university was actually in the business of seeking truth, there is much they could do to solve this. * working groups "red teams" whose whole job it is to find weaknesses in papers published at the university * post mortems after finding papers with serious flaws exposing the problem and coming up with constructive corrective actions * funding / forcing researchers to devote a certain amount of time to replicating significant results * working groups of experts in statistics and study design available to consult * systems to register studies, methodologies, data sets, etc. with the aim of eliminating error and preventing post-hoc fishing expeditions for results The whole-ass purpose of a university is seeking knowledge. They are fully capable of doing a better job of it but they don't because what they actually focus on are things like fundraising, professional sports teams, constructing attractive buildings, and advancing ranking. Most universities would be better off just firing and not replacing 90% of their administration.
- jltsiren 3y ago
  Universities can't do that, because they lack the expertise. Most of the time, the only people in a university capable of judging a study are the people who did it. That's because universities are in the business of teaching. Apart from a few rare exceptions, universities don't have the money to hire redundant people. Instead of hiring many experts in the same topic, they prefer hiring a wider range of expertise, in order to provide better learning opportunities for the students.
  - mkl 3y ago
    > Universities can't do that, because they lack the expertise. Most of the time, the only people in a university capable of judging a study are the people who did it. Statisticians can find faults in many studies. Universities even employ statisticians to act as consultants for research in other departments, as that raises the quality of the institution's publications.

tppiotrowski 3y ago

> While we're on the subject, how many studies period are faked or flawed to the point of being useless? Or how many studies are useless, period? It's like publishing a memoir to Amazon. You can now say "author" on your resume, or when you're introduced or at cocktail parties but nobody finds any value in what you have to say. You can also use ChatGPT because people might not notice.

some_random 3y ago
There is always value in expanding the breadth and depth of human knowledge, even if it doesn't seem useful to you, right now. That of course assumes that knowledge is true, which is the crux of the problem now.
- yellowcake0 3y ago
  Unfortunately, the preponderance of very low value research in the literature puts a significant burden on the scientists who have to sift through a lot of garbage to find what they're looking for. Even if the work is ostensibly correct (much of it is not), it really doesn't do anyone much good, except for the authors of course. But now every undergraduate and every parent's vanity project at Andover wants a first author contrib., so here we are.
  - tppiotrowski 3y ago
    I think language models will help us sift through the garbage. I have a hard time finding answers using Google (too much SEO garbage), but ChatGPT does a pretty good job of filtering the noise. It's not perfect, but it saves lots of time vs. Google.

adamsb6 3y ago

The replication crisis is even a thing because of very strong career incentives. You might think that publishing about the replication crisis itself would be great for your career, but perhaps not. Maybe the incentives to be able to bullshit your way to a professorship are so great that no one wants to rock the boat.

rqtwteye 3y ago
"Maybe the incentives to be able to bullshit your way to a professorship are so great that no one wants to rock the boat." Our whole economy is fueled by people bullshitting each other.
- wizzwizz4 3y ago
  Our economy is fuelled by people doing actual work. Large-ish, visible parts are shaped by bullshitters, but that's a different thing.
spicybright 3y ago
Talking with a few students in science programs this seems to be the case, even when few higher ups want to admit it. Proving something is false is less valued than proving it right. It's silly because if we valued the quality of science it should be the exact opposite.

treis 3y ago

My internal filter is: Sociology/Psych/Economics are almost all junk. Their conclusions may or may not be correct. Medical studies are mostly junk. There's way too much financial incentive to show marginal improvement. Theraflu and anti-depressents come to mind. Both show a small effect in studies and launched billion dollar businesses. Hard science stuff tends to be pretty good. Mostly just outright fraud and they usually end up getting caught.

fnikacevic 3y ago
Chemistry still has a ton of issues with people exaggerating numbers that can't be contested like yield and purity of reactions even among the top journals. Straight up fraud is rarer yes.
mike_hearn 3y ago
I used to think like that. Soft science = dubious, hard science = ok. Not any more though. Just recently there was LK99 in materials science though, which was pretty much the textbook definition of an exciting but non-replicable claim. Also a lot of stuff is tough to classify. Are epidemiology or climatology "hard" sciences?

a123b456c 3y ago

The scientific community ignored the replication crisis? Dude. The scientific community created the replication crisis. We are not ignoring it, I promise you. The reaction is nuanced by no one who matters is ignoring it.

thenickdude 3y ago

Check out https://retractionwatch.com/ https://retractionwatch.com/ , a continuous stream of exposés on authors publishing flawed or fraudulent works.