3 ms·

How useful is meta analysis if half or more of the experiments don't replicate? As we increasingly discover with the replication crisis.

dekhn 5y ago
Medical studies can't be replicated in experiments. They are almost always one-shot. That's not how medical studies (that we do meta analysis) work. It's too expensive and too complicated.
buescher 5y ago
Bingo. Also the problem with meta-analysis of certain kinds of studies (say, observational ones) is the file-drawer effect: people don't report negative results. So you can go from a bunch of studies that show an effect (usually small) at marginal statistical significance to a meta-analysis that shows very high statistical significance when there is really no effect there. I don't think we have to worry about the file-drawer effect much with controversial COVID treatments, though. Do you?
dekhn 5y ago
I said they had different goals, not that they don't overlap. If you're in a class teaching medical studies, and you tell people "go replicate the original experiment instead of doing meta analysis", you're doing your students a disservice. Instead say: "It's always important to be able to replicate experiments and I encourage everybody in the class who is going on to do science or medicine spend at least some time in the lab replicating a basic experiment. But medical studies are so large and complicated that we can't typically repeat them in a reproducible way. Instead, we use statistics and probability to make reasonable decisions based on the data we have. Sometimes that means removing a suspect paper from a meta-analysis because we lack confidence in its reproducibility." For some of the most sophisticated medical science experiments we have today, the only way to replicate an experiment is to be a postdoc and join the lab that does the research, master the technique there, then take your reagents and other material to your professor job, and then get your local set up to replicate what you did in the original lab. This is how it normally works, for example in labs like Mina Bissell's where few people in the work even have the skills to replicate her experiments.

dekhn 5y ago

Medical studies can't be replicated in experiments. They are almost always one-shot. That's not how medical studies (that we do meta analysis) work. It's too expensive and too complicated.

buescher 5y ago

Bingo. Also the problem with meta-analysis of certain kinds of studies (say, observational ones) is the file-drawer effect: people don't report negative results. So you can go from a bunch of studies that show an effect (usually small) at marginal statistical significance to a meta-analysis that shows very high statistical significance when there is really no effect there. I don't think we have to worry about the file-drawer effect much with controversial COVID treatments, though. Do you?

dekhn 5y ago

I said they had different goals, not that they don't overlap. If you're in a class teaching medical studies, and you tell people "go replicate the original experiment instead of doing meta analysis", you're doing your students a disservice. Instead say: "It's always important to be able to replicate experiments and I encourage everybody in the class who is going on to do science or medicine spend at least some time in the lab replicating a basic experiment. But medical studies are so large and complicated that we can't typically repeat them in a reproducible way. Instead, we use statistics and probability to make reasonable decisions based on the data we have. Sometimes that means removing a suspect paper from a meta-analysis because we lack confidence in its reproducibility." For some of the most sophisticated medical science experiments we have today, the only way to replicate an experiment is to be a postdoc and join the lab that does the research, master the technique there, then take your reagents and other material to your professor job, and then get your local set up to replicate what you did in the original lab. This is how it normally works, for example in labs like Mina Bissell's where few people in the work even have the skills to replicate her experiments.