4 ms·

I understand that Melinda Gates is looking for different charities to invest in. I wonder if she'd fund researchers to attempt to replicate crucial publication

I understand that Melinda Gates is looking for different charities to invest in. I wonder if she'd fund researchers to attempt to replicate crucial publications / studies. I expect that she has enough money to put a serious dent in this problem, at least if she focused on some specific area.

HarryHirsch 5y ago
It doesn't work like that. When you do actual science at the bench you find out fairly quickly whose work is solid and whose is like weak reeds. The trouble is the misaligned incentives, everyone has good reason to keep silent for fear of attracting the attention of the funding agency. If something is afoot (the rumours are usually true), soon you are looking at 7-figure damages.
- CoastalCoder 5y ago
  I don't understand why that invalidates my idea though. In this case, Gates would be the funding agency. And its published findings would hopefully steer much academic teaching and research away from bad assumptions.
  - HarryHirsch 5y ago
    The problem isn't discovering who is a shady type, everyone knows who they are already. But they just won't get outed or dealt with. The Hellinga scandal at Duke is very typical. This is what is looks like from outside: https://www.buffalo.edu/content/dam/www/news/imported/pdf/May08/NatureChemistryPaperWithdrawn.pdf https://www.buffalo.edu/content/dam/www/news/imported/pdf/Ma... And this is an inside account: https://caffeinatedchemist.medium.com/hellinga-66449cf4699e https://caffeinatedchemist.medium.com/hellinga-66449cf4699e It's exclusively an accountability issue.
  - joe_the_user 5y ago
    Just to elaborate on the "it doesn't work that way" part. Science is hard. Replicating an experiment is costly in time and money. The papers in scientific field are organized like a tree. A good paper by a good and credible researcher will be a base node that other papers branch off of. A poor or not credible paper will be leaf that won't be cited by others. Testing all the poor papers would be tremendous waste of time. And it wouldn't catch fraudsters as such because not being replicable isn't proof of fraud.
dekhn 5y ago
https://www.cos.io/about/news/reproducibility-initiative-receives-grant-validate-50-landmark-cancer-studies https://www.cos.io/about/news/reproducibility-initiative-rec... (an example, but from another rich person). Most rich people are less interested in funding replication studies than they are discoveries, but sometimes they do invest heavily in infrastructure development like Chan-Zuckerberg.
joe_the_user 5y ago
Talking about replicating this stuff misses the point. Science can't survive if the answer to faking claims is to replicate every study. The reason is that replicating a study done in complete seriousness and honesty is hard and is not guaranteed to yield the same result. So "can't be replicated" shouldn't be a black mark - "was basically made up" should be a black mark. And also, if faking stuff becomes standard with replication the main guard, an easy way to get points is announce the fake replication of a given study.
- cipheredStones 5y ago
  > replicating a study done in complete seriousness and honesty is hard and is not guaranteed to yield the same result Yes, a study failing to replicate is not in itself evidence of fraud. Nor is it definitive evidence that the effect shown in the original doesn't exist, since a replication can be messed up too. But science needs replication because we're ultimately trying to learn about the world, and an effect we can't reliably produce is usually an effect we're not sure exists. Replication sometimes catching dishonesty is a bonus: failure to replicate calls the effect into question, which calls the method used to find that effect into question, which sometimes calls into question whether that method was actually used. (Or in the case of bad statistics, whether it was chosen to dishonestly show a certain result.) It's the appropriate propagation of doubt.
  - kergonath 5y ago
    > Yes, a study failing to replicate is not in itself evidence of fraud. Nor is it definitive evidence that the effect shown in the original doesn't exist, since a replication can be messed up too. It’s not a trial to determine the guilt of a supposed fraudster. This is what institutions should do because fraud is a breach of contract (in all sane institutions). However, the main problem is made up results in the scientific record. This is what we attempt to solve by retracting articles. And from the point of view of the scientific record, there is no meaningful difference between a made up result and something that cannot be replicated.
    - 5y ago

HarryHirsch 5y ago

It doesn't work like that. When you do actual science at the bench you find out fairly quickly whose work is solid and whose is like weak reeds. The trouble is the misaligned incentives, everyone has good reason to keep silent for fear of attracting the attention of the funding agency. If something is afoot (the rumours are usually true), soon you are looking at 7-figure damages.

CoastalCoder 5y ago
I don't understand why that invalidates my idea though. In this case, Gates would be the funding agency. And its published findings would hopefully steer much academic teaching and research away from bad assumptions.
- HarryHirsch 5y ago
  The problem isn't discovering who is a shady type, everyone knows who they are already. But they just won't get outed or dealt with. The Hellinga scandal at Duke is very typical. This is what is looks like from outside: https://www.buffalo.edu/content/dam/www/news/imported/pdf/May08/NatureChemistryPaperWithdrawn.pdf https://www.buffalo.edu/content/dam/www/news/imported/pdf/Ma... And this is an inside account: https://caffeinatedchemist.medium.com/hellinga-66449cf4699e https://caffeinatedchemist.medium.com/hellinga-66449cf4699e It's exclusively an accountability issue.
- joe_the_user 5y ago
  Just to elaborate on the "it doesn't work that way" part. Science is hard. Replicating an experiment is costly in time and money. The papers in scientific field are organized like a tree. A good paper by a good and credible researcher will be a base node that other papers branch off of. A poor or not credible paper will be leaf that won't be cited by others. Testing all the poor papers would be tremendous waste of time. And it wouldn't catch fraudsters as such because not being replicable isn't proof of fraud.

dekhn 5y ago

https://www.cos.io/about/news/reproducibility-initiative-receives-grant-validate-50-landmark-cancer-studies https://www.cos.io/about/news/reproducibility-initiative-rec... (an example, but from another rich person). Most rich people are less interested in funding replication studies than they are discoveries, but sometimes they do invest heavily in infrastructure development like Chan-Zuckerberg.

joe_the_user 5y ago

Talking about replicating this stuff misses the point. Science can't survive if the answer to faking claims is to replicate every study. The reason is that replicating a study done in complete seriousness and honesty is hard and is not guaranteed to yield the same result. So "can't be replicated" shouldn't be a black mark - "was basically made up" should be a black mark. And also, if faking stuff becomes standard with replication the main guard, an easy way to get points is announce the fake replication of a given study.

cipheredStones 5y ago
> replicating a study done in complete seriousness and honesty is hard and is not guaranteed to yield the same result Yes, a study failing to replicate is not in itself evidence of fraud. Nor is it definitive evidence that the effect shown in the original doesn't exist, since a replication can be messed up too. But science needs replication because we're ultimately trying to learn about the world, and an effect we can't reliably produce is usually an effect we're not sure exists. Replication sometimes catching dishonesty is a bonus: failure to replicate calls the effect into question, which calls the method used to find that effect into question, which sometimes calls into question whether that method was actually used. (Or in the case of bad statistics, whether it was chosen to dishonestly show a certain result.) It's the appropriate propagation of doubt.
- kergonath 5y ago
  > Yes, a study failing to replicate is not in itself evidence of fraud. Nor is it definitive evidence that the effect shown in the original doesn't exist, since a replication can be messed up too. It’s not a trial to determine the guilt of a supposed fraudster. This is what institutions should do because fraud is a breach of contract (in all sane institutions). However, the main problem is made up results in the scientific record. This is what we attempt to solve by retracting articles. And from the point of view of the scientific record, there is no meaningful difference between a made up result and something that cannot be replicated.
  - 5y ago