4 ms·

PIs may not be able to raise money for replication studies. Some of this is a consequence of guidelines for federal funds to prevent waste on duplication of eff

PIs may not be able to raise money for replication studies. Some of this is a consequence of guidelines for federal funds to prevent waste on duplication of effort.

some_random 3y ago
This goes far beyond there not being enough replication studies, the problem is that when replication studies are done the results don't reproduce because the results were bogus. https://en.wikipedia.org/wiki/Replication_crisis https://en.wikipedia.org/wiki/Replication_crisis
- whatshisface 3y ago
  If the replication studies were done at a reasonable rate there would be no incentive to produce bogus results because you'd be caught before you could go through an entire career as a "successful scientist."
  - mike_hearn 3y ago
    There is no being caught :( Who is going to do the catching, exactly? There's no science police. Universities don't seem to care about anything except plagiarism. Many papers aren't even "looks legit but doesn't replicate", they are self-evidently wrong just from reading them or the associated data. Just peer review done properly would solve that, but the problem is widespread because there's no real incentive to rock the boat. Once a bad technique or invalid approach gets through to publication a few times it becomes a new standard within the field because it lets people publish more papers.
    - whatshisface 3y ago
      I think the most natural place for reform is from the people who are paying for the research, if they realized that what they are spending money for has no value if it's being made up. Although there are a lot of scientists who by the principle of self-interest seem willing to waste their life in exchange for a small salary it makes less sense why the NSF would take that offer.
      - mike_hearn 3y ago
        But that can't happen at the moment. The core problem is that civil servants and politicians are easily bullied by claims they aren't experts and thus can't judge if "expert" work is any good or not. So they just dish out money to anyone who works at a university, expecting them to self-police, but they don't. There are only two ways to fix that, as far as I can see: 1. Have highly technical, scientifically trained and skeptical politicians who police academia and ensure research money is well spent. 2. Stop governments funding research in the first place. (1) just doesn't seem feasible. There are so many problems there. (2) is feasible. So I suspect you're going to start seeing calls from the right to defund universities over the next ten years or so. The underlying motivation may be that universities are strongholds of the left, but the stated justification will be the high levels of research fraud.
        whatshisface 3y ago
        Or working from historical analogies, 3) find scientists with a public and obvious track record of replication, and maybe even a few of the fraud hunters, and put them in charge of the NSF.
        mike_hearn 3y ago
        You could try to build a science police but it won't work. Congress tried that in the 70s with the ORI. After a few failed prosecutions they basically gave up and are now forgotten - still there and they mount a few prosecutions a year but clearly it has no effect. The sheer scale of the problem cannot be fixed by trying to find people breaking "the rules" because there are no rules to begin with. Attempting to fix this within the framework of government would be incredibly distracting for the state, as it'd lead to endless fights and debates. For example, a reasonable person appointed to run the NSF might conclude they should just defund whole subfields of the social sciences, because there's no way to make them scientific at all, but good luck lasting in the job if you do that. A big part of the problem is that scientists are in hoc to a specific political ideology, so they will have powerful friends in Washington who want to see them stick around producing useful propaganda.
bee_rider 3y ago
This points to the real problem and also where the responsibility and interest ought to be to fix it. There’s no replication crisis for academics because they have a meatspace social network of academics; they go to conferences together and know each other. You can just ignore a paper if you know the author is an idiot. If medical studies are faked, is it a problem? Presumable regulatory agencies are using these studies or something, right? Looks like the FDA and NSF need to fund some more replication studies.
- jonlucc 3y ago
  At least in pre-clinical research, there’s also a mechanism by which regulatory agencies can sometimes force industry to pay for replication studies. It’s a little complicated, because sometimes there aren’t many groups who can reliably do the study, and clinical trials are so expensive and time-consuming to run, it would be difficult to do at that scale. Rodent studies are generally under $1M and monkey studies are generally under $5M (up a lot recently). Clinical trials are in the hundreds of millions of dollars and take years to run.

PIs may not be able to raise money for replication studies. Some of this is a consequence of guidelines for federal funds to prevent waste on duplication of effort.

some_random 3y ago

This goes far beyond there not being enough replication studies, the problem is that when replication studies are done the results don't reproduce because the results were bogus. https://en.wikipedia.org/wiki/Replication_crisis https://en.wikipedia.org/wiki/Replication_crisis

whatshisface 3y ago
If the replication studies were done at a reasonable rate there would be no incentive to produce bogus results because you'd be caught before you could go through an entire career as a "successful scientist."
- mike_hearn 3y ago
  There is no being caught :( Who is going to do the catching, exactly? There's no science police. Universities don't seem to care about anything except plagiarism. Many papers aren't even "looks legit but doesn't replicate", they are self-evidently wrong just from reading them or the associated data. Just peer review done properly would solve that, but the problem is widespread because there's no real incentive to rock the boat. Once a bad technique or invalid approach gets through to publication a few times it becomes a new standard within the field because it lets people publish more papers.
  - whatshisface 3y ago
    I think the most natural place for reform is from the people who are paying for the research, if they realized that what they are spending money for has no value if it's being made up. Although there are a lot of scientists who by the principle of self-interest seem willing to waste their life in exchange for a small salary it makes less sense why the NSF would take that offer.
    - mike_hearn 3y ago
      But that can't happen at the moment. The core problem is that civil servants and politicians are easily bullied by claims they aren't experts and thus can't judge if "expert" work is any good or not. So they just dish out money to anyone who works at a university, expecting them to self-police, but they don't. There are only two ways to fix that, as far as I can see: 1. Have highly technical, scientifically trained and skeptical politicians who police academia and ensure research money is well spent. 2. Stop governments funding research in the first place. (1) just doesn't seem feasible. There are so many problems there. (2) is feasible. So I suspect you're going to start seeing calls from the right to defund universities over the next ten years or so. The underlying motivation may be that universities are strongholds of the left, but the stated justification will be the high levels of research fraud.
      - whatshisface 3y ago
        Or working from historical analogies, 3) find scientists with a public and obvious track record of replication, and maybe even a few of the fraud hunters, and put them in charge of the NSF.
        mike_hearn 3y ago
        You could try to build a science police but it won't work. Congress tried that in the 70s with the ORI. After a few failed prosecutions they basically gave up and are now forgotten - still there and they mount a few prosecutions a year but clearly it has no effect. The sheer scale of the problem cannot be fixed by trying to find people breaking "the rules" because there are no rules to begin with. Attempting to fix this within the framework of government would be incredibly distracting for the state, as it'd lead to endless fights and debates. For example, a reasonable person appointed to run the NSF might conclude they should just defund whole subfields of the social sciences, because there's no way to make them scientific at all, but good luck lasting in the job if you do that. A big part of the problem is that scientists are in hoc to a specific political ideology, so they will have powerful friends in Washington who want to see them stick around producing useful propaganda.

bee_rider 3y ago

This points to the real problem and also where the responsibility and interest ought to be to fix it. There’s no replication crisis for academics because they have a meatspace social network of academics; they go to conferences together and know each other. You can just ignore a paper if you know the author is an idiot. If medical studies are faked, is it a problem? Presumable regulatory agencies are using these studies or something, right? Looks like the FDA and NSF need to fund some more replication studies.

jonlucc 3y ago
At least in pre-clinical research, there’s also a mechanism by which regulatory agencies can sometimes force industry to pay for replication studies. It’s a little complicated, because sometimes there aren’t many groups who can reliably do the study, and clinical trials are so expensive and time-consuming to run, it would be difficult to do at that scale. Rodent studies are generally under $1M and monkey studies are generally under $5M (up a lot recently). Clinical trials are in the hundreds of millions of dollars and take years to run.