3 ms·

I think one of the big things that you can do is split up the data gathering and data analysis. Kind of like we don’t trust the companies to audit themselves,

I think one of the big things that you can do is split up the data gathering and data analysis. Kind of like we don’t trust the companies to audit themselves, instead we have an outside firm. In this model, a researcher would create a hypothesis and collect the data. Their team would write the background and methods sections of the paper. Then the entirety of raw data would be sent off to a third party for data analysis and they would write the results part of the paper. The original team would then write the discussion part discussing the implications of the study. All papers would be required to be made public. The idea is that there would be specialized firms that do the analysis on the raw data for everyone. These would be carefully audited and certified by the government. They have no incentive to play statistical games. If they get caught cheating, then they have to pay for all the analysis on all the papers to be done again by a competitor and if any errors are found in a paper analysis it is automatically retracted. While the this re-analysis is going on, all papers would be quarantined with a note stating the paper is having its analysis redone. This incentivizes the analysis firm to be ethical, as well as incentivizes researchers to pick ethical analysis firms. Separating data collection from data analysis would help align incentives better.

wjn0 6y ago
That's an interesting idea that could stop very specific types of fraud, certainly in the life sciences. But it's not feasible for all kinds of research, and in fact could hinder lots of research. > All papers would be required to be made public. This is more universally feasible. Publicizing the data and analysis tools (scripts, software) falls into the same category, and would go a long way to help without the need for such strong separation.
OminousWeapons 6y ago
You would just get people massaging or generating the data to fit a conclusion prior to sending it out for statistical analysis. If people want to cheat they can find a way. The only way to really do what you are getting at is to construct core labs or CROs to run all experiments on behalf of the investigators. This is not infeasible in many cases (and is already done in narrow ways) but it requires hiring staff scientists to run every experiment rather than grad students or post docs and costs / complexity will explode. The real way to defend against a lot of fraud is to force people to submit actually detailed methods sections so experiments are legitimately reproducible (they largely aren't now). This would catch a lot more fraud quickly, although even this won't fully work as some experiments are simply too costly to reproduce for validation purposes (eg animal studies).
- guerby 6y ago
  This. And also actually fund researchers who do reproducibility work. May be even fund specialized teams that do only reproducibility work.

wjn0 6y ago

That's an interesting idea that could stop very specific types of fraud, certainly in the life sciences. But it's not feasible for all kinds of research, and in fact could hinder lots of research. > All papers would be required to be made public. This is more universally feasible. Publicizing the data and analysis tools (scripts, software) falls into the same category, and would go a long way to help without the need for such strong separation.

OminousWeapons 6y ago

You would just get people massaging or generating the data to fit a conclusion prior to sending it out for statistical analysis. If people want to cheat they can find a way. The only way to really do what you are getting at is to construct core labs or CROs to run all experiments on behalf of the investigators. This is not infeasible in many cases (and is already done in narrow ways) but it requires hiring staff scientists to run every experiment rather than grad students or post docs and costs / complexity will explode. The real way to defend against a lot of fraud is to force people to submit actually detailed methods sections so experiments are legitimately reproducible (they largely aren't now). This would catch a lot more fraud quickly, although even this won't fully work as some experiments are simply too costly to reproduce for validation purposes (eg animal studies).

guerby 6y ago
This. And also actually fund researchers who do reproducibility work. May be even fund specialized teams that do only reproducibility work.