3 ms·

I've had an idea about this problem that I'd be happy to hear your thoughts about: Simply put, a promise for an independant future reproduction study should be

by air7 8y ago

I've had an idea about this problem that I'd be happy to hear your thoughts about: Simply put, a promise for an independant future reproduction study should be part of the published paper. Once a researcher achieves a publishable result, she looks for a peer-researcher that will commit to perform a pre-determined reproduction study in the near future. This promise is written in the original paper. This ensures that a negative reproduction would definitely be published. It incentivises the original researcher to not mess too much with the data post-hoc, and to be as helpful as possible to their reproducing peer. The peer gets a citing before writing the paper, and all the help they'd want to get the study done as quickly and easily as possible (q&a, analysis code etc).

Balgair 8y ago
Lets look at an example to ferret out some stuff. Assume that the experiment is looking at some disease, like Gastroesophageal reflux disease (GERD) [0]. It's not a glamorous disease, but it's worth looking into. Say you are looking at the way the nervous system interacts with the sphincter to un/control it. Further, you are looking at some specific synapse. Say, also, that you have a mouse model for this as a test bed (incredibly unlikely, btw). You induce GERD in the mice, wait for erosion of the esophagus, sac the mice, and study the synapses, get the results, negative or positive. How long do you think it'll take a grad student to do this, to get to that point? 1st year is basically wasted, research-wise, as you've got classes and exams. 2nd year is a little better, as you finally understand all the issues with GERD in mice v. humans, the vagal nerves, etc. Still you may be taking classes and you are prepping for your quals. 3rd year is finally when you can get into the nitty-gritty of the experiment. You finally learn how to sac the mice properly, dissect, etc. You fail, a LOT. Lets say, research-wise, you finally have 1 year's worth of 'real' research under your belt. 4th-7th years are ~80% research work, 20% funding search (at ~90hrs/wk of work). The mice cooperate alright, there are breeding issues, random infections in the vivarium, minimal genetic drift, etc. Finally, you can get 'clean' data that other people may actually believe. You've controlled for sex, for food intake, for stomach acidity, for the phase of the moon, for how much coffee is in the room, etc. You've gone half mad getting controls done. You've learned how to dissect the mice carefully, that you can't have coffee for 3 days beforehand to calm your hands down, how to properly positions and craft the micro-pipette tips, how to run all the custom software you've hacked together badly. You've learned a LOT about this tiny portion of a mouse and how to uncover it's secrets. A lot of what you have learned is only applicable to your hands in your lab with your mice. After, say, 5 years or 'real' research, you find that those neurons that command that particular sphincter of those mice (experimental controls to hell and back) have something to do with GERD. But only to p=0.0482. A positive result, but juuuuust so. You find something to write about and publish. 7 years of your life, and it's just there if you squint at the data a lot. You get out of grad school with a PhD and open a bakery. The research, what you found, is largely forgotten all the same. Mind you, this is in a stable lab with a grad student with a stable life. No pregnancies, no marriages, no advisor conflict, no sexual harassment, no funding issues, etc. Now, the parent comment suggests that we should have some other lab come along and replicate the experiment, do the same thing, just to be certain that the 7 years of that grad studen't life were really doing something, not a statistical fluke. How is that going to actually be accomplished? Say, yeah, you have the funding already set-up via the NIH, fine. Who is going to do that second round of experiments? What grad student is going to get trained up to do someone else's experiments again? How long will it take them to get to that same point, can they even do it the same way? What about their own original research and their career path? Will their advisors be similarly alright with this? Will their life in grad school be just as stable? Who will teach them how to position the electrodes, to dissect the mice, to deal with vivarium issues, to pass their quals? It's not that the scientists don't want to re-run experiments, it's that doing then is HARD. Technically, yeah, science is not easy. But logistically? Man, even getting one student through grad school is an accomplishment. If you want to re-run each experiment and paper, you somehow have to deal with re-running all the extra jetsam and flotsam too, and I don't think we have any idea how to do that successfully. [0]https://www.mayoclinic.org/diseases-conditions/gerd/symptoms-causes/syc-20361940 https://www.mayoclinic.org/diseases-conditions/gerd/symptoms...
GauntletWizard 8y ago
This will significantly bias the second study towards getting the same result. Providing help and assistance beyond the paper itself will create experimental controls that are undocumented - assumptions that were shared between researchers but not published in the papers themselves, because humans are bad at that. Further, giving any sort of relationship between the experimenter and the reproducer will taint them - people are friendly and want to see their friends "succeed" - they will taint their own work out of empathy.
Bartweiss 8y ago
This is a really interesting idea. Adversarial collaboration has produced some interesting results, and is probably our best bet for settling arguments on topics where results are consistently rejected over methodology. It's done good work on ESP, and shows some promise on priming if anyone will actually sign on. But that basically requires finding fields with conflicting viewpoints and well-understood methodology spats, which means established debates. This idea would get the same effects - experiment design that's accessible and verifiable - on untested topics, while simultaneously baking replication attempts into initial publication. The more I think about this, the more impressed I am. It guarantees replications, it guarantees data and methodology availability, and it makes replications a publication-worthy step by making them part of the initial 'success'. It doesn't solve the file drawer or salami slicing problems, but it does huge work to sidestep them by forcing another p<.05 which isn't subject to them. And wildest of all, it might even be acceptable to journals in a way that "publish replications and null results" isn't. Thanks for the most creative approach to the replication crisis I've heard in ages!

by air7 8y ago

Balgair 8y ago

Lets look at an example to ferret out some stuff. Assume that the experiment is looking at some disease, like Gastroesophageal reflux disease (GERD) [0]. It's not a glamorous disease, but it's worth looking into. Say you are looking at the way the nervous system interacts with the sphincter to un/control it. Further, you are looking at some specific synapse. Say, also, that you have a mouse model for this as a test bed (incredibly unlikely, btw). You induce GERD in the mice, wait for erosion of the esophagus, sac the mice, and study the synapses, get the results, negative or positive. How long do you think it'll take a grad student to do this, to get to that point? 1st year is basically wasted, research-wise, as you've got classes and exams. 2nd year is a little better, as you finally understand all the issues with GERD in mice v. humans, the vagal nerves, etc. Still you may be taking classes and you are prepping for your quals. 3rd year is finally when you can get into the nitty-gritty of the experiment. You finally learn how to sac the mice properly, dissect, etc. You fail, a LOT. Lets say, research-wise, you finally have 1 year's worth of 'real' research under your belt. 4th-7th years are ~80% research work, 20% funding search (at ~90hrs/wk of work). The mice cooperate alright, there are breeding issues, random infections in the vivarium, minimal genetic drift, etc. Finally, you can get 'clean' data that other people may actually believe. You've controlled for sex, for food intake, for stomach acidity, for the phase of the moon, for how much coffee is in the room, etc. You've gone half mad getting controls done. You've learned how to dissect the mice carefully, that you can't have coffee for 3 days beforehand to calm your hands down, how to properly positions and craft the micro-pipette tips, how to run all the custom software you've hacked together badly. You've learned a LOT about this tiny portion of a mouse and how to uncover it's secrets. A lot of what you have learned is only applicable to your hands in your lab with your mice. After, say, 5 years or 'real' research, you find that those neurons that command that particular sphincter of those mice (experimental controls to hell and back) have something to do with GERD. But only to p=0.0482. A positive result, but juuuuust so. You find something to write about and publish. 7 years of your life, and it's just there if you squint at the data a lot. You get out of grad school with a PhD and open a bakery. The research, what you found, is largely forgotten all the same. Mind you, this is in a stable lab with a grad student with a stable life. No pregnancies, no marriages, no advisor conflict, no sexual harassment, no funding issues, etc. Now, the parent comment suggests that we should have some other lab come along and replicate the experiment, do the same thing, just to be certain that the 7 years of that grad studen't life were really doing something, not a statistical fluke. How is that going to actually be accomplished? Say, yeah, you have the funding already set-up via the NIH, fine. Who is going to do that second round of experiments? What grad student is going to get trained up to do someone else's experiments again? How long will it take them to get to that same point, can they even do it the same way? What about their own original research and their career path? Will their advisors be similarly alright with this? Will their life in grad school be just as stable? Who will teach them how to position the electrodes, to dissect the mice, to deal with vivarium issues, to pass their quals? It's not that the scientists don't want to re-run experiments, it's that doing then is HARD. Technically, yeah, science is not easy. But logistically? Man, even getting one student through grad school is an accomplishment. If you want to re-run each experiment and paper, you somehow have to deal with re-running all the extra jetsam and flotsam too, and I don't think we have any idea how to do that successfully. [0]https://www.mayoclinic.org/diseases-conditions/gerd/symptoms-causes/syc-20361940 https://www.mayoclinic.org/diseases-conditions/gerd/symptoms...

GauntletWizard 8y ago

This will significantly bias the second study towards getting the same result. Providing help and assistance beyond the paper itself will create experimental controls that are undocumented - assumptions that were shared between researchers but not published in the papers themselves, because humans are bad at that. Further, giving any sort of relationship between the experimenter and the reproducer will taint them - people are friendly and want to see their friends "succeed" - they will taint their own work out of empathy.

Bartweiss 8y ago

This is a really interesting idea. Adversarial collaboration has produced some interesting results, and is probably our best bet for settling arguments on topics where results are consistently rejected over methodology. It's done good work on ESP, and shows some promise on priming if anyone will actually sign on. But that basically requires finding fields with conflicting viewpoints and well-understood methodology spats, which means established debates. This idea would get the same effects - experiment design that's accessible and verifiable - on untested topics, while simultaneously baking replication attempts into initial publication. The more I think about this, the more impressed I am. It guarantees replications, it guarantees data and methodology availability, and it makes replications a publication-worthy step by making them part of the initial 'success'. It doesn't solve the file drawer or salami slicing problems, but it does huge work to sidestep them by forcing another p<.05 which isn't subject to them. And wildest of all, it might even be acceptable to journals in a way that "publish replications and null results" isn't. Thanks for the most creative approach to the replication crisis I've heard in ages!