5 ms·

I've suspected that for years. I remember my undergrad biology prof coming down on students depending on meta-analysis and insisting they were no substitute for

I've suspected that for years. I remember my undergrad biology prof coming down on students depending on meta-analysis and insisting they were no substitute for attempting to replicate the experiments themselves.

dekhn 5y ago
that doesn't make scientific sense. Replicating the original experiment and doing a meta-analysis have two completely different goals. I would have argued that back to the professor (and been right).
- gorwell 5y ago
  How useful is meta analysis if half or more of the experiments don't replicate? As we increasingly discover with the replication crisis.
  - dekhn 5y ago
    Medical studies can't be replicated in experiments. They are almost always one-shot. That's not how medical studies (that we do meta analysis) work. It's too expensive and too complicated.
  - buescher 5y ago
    Bingo. Also the problem with meta-analysis of certain kinds of studies (say, observational ones) is the file-drawer effect: people don't report negative results. So you can go from a bunch of studies that show an effect (usually small) at marginal statistical significance to a meta-analysis that shows very high statistical significance when there is really no effect there. I don't think we have to worry about the file-drawer effect much with controversial COVID treatments, though. Do you?
  - dekhn 5y ago
    I said they had different goals, not that they don't overlap. If you're in a class teaching medical studies, and you tell people "go replicate the original experiment instead of doing meta analysis", you're doing your students a disservice. Instead say: "It's always important to be able to replicate experiments and I encourage everybody in the class who is going on to do science or medicine spend at least some time in the lab replicating a basic experiment. But medical studies are so large and complicated that we can't typically repeat them in a reproducible way. Instead, we use statistics and probability to make reasonable decisions based on the data we have. Sometimes that means removing a suspect paper from a meta-analysis because we lack confidence in its reproducibility." For some of the most sophisticated medical science experiments we have today, the only way to replicate an experiment is to be a postdoc and join the lab that does the research, master the technique there, then take your reagents and other material to your professor job, and then get your local set up to replicate what you did in the original lab. This is how it normally works, for example in labs like Mina Bissell's where few people in the work even have the skills to replicate her experiments.
- marcodiego 5y ago
  Hmmm... How about meta-analysis considering only results that were successfully replicated? Why not consider an index that is higher with the number of successful replications and lower of number of replication tries?
simonh 5y ago
Ok you replicate the original study. Presumably you want to use this new result to increase your statistical power compared to the original result, combining the results of the two studies.... which... is.. er... meta analysis. The whole point of meta analysis is that you have multiple studies of the same experiment.

dekhn 5y ago

that doesn't make scientific sense. Replicating the original experiment and doing a meta-analysis have two completely different goals. I would have argued that back to the professor (and been right).

gorwell 5y ago
How useful is meta analysis if half or more of the experiments don't replicate? As we increasingly discover with the replication crisis.
- dekhn 5y ago
  Medical studies can't be replicated in experiments. They are almost always one-shot. That's not how medical studies (that we do meta analysis) work. It's too expensive and too complicated.
- buescher 5y ago
  Bingo. Also the problem with meta-analysis of certain kinds of studies (say, observational ones) is the file-drawer effect: people don't report negative results. So you can go from a bunch of studies that show an effect (usually small) at marginal statistical significance to a meta-analysis that shows very high statistical significance when there is really no effect there. I don't think we have to worry about the file-drawer effect much with controversial COVID treatments, though. Do you?
- dekhn 5y ago
  I said they had different goals, not that they don't overlap. If you're in a class teaching medical studies, and you tell people "go replicate the original experiment instead of doing meta analysis", you're doing your students a disservice. Instead say: "It's always important to be able to replicate experiments and I encourage everybody in the class who is going on to do science or medicine spend at least some time in the lab replicating a basic experiment. But medical studies are so large and complicated that we can't typically repeat them in a reproducible way. Instead, we use statistics and probability to make reasonable decisions based on the data we have. Sometimes that means removing a suspect paper from a meta-analysis because we lack confidence in its reproducibility." For some of the most sophisticated medical science experiments we have today, the only way to replicate an experiment is to be a postdoc and join the lab that does the research, master the technique there, then take your reagents and other material to your professor job, and then get your local set up to replicate what you did in the original lab. This is how it normally works, for example in labs like Mina Bissell's where few people in the work even have the skills to replicate her experiments.
marcodiego 5y ago
Hmmm... How about meta-analysis considering only results that were successfully replicated? Why not consider an index that is higher with the number of successful replications and lower of number of replication tries?

simonh 5y ago

Ok you replicate the original study. Presumably you want to use this new result to increase your statistical power compared to the original result, combining the results of the two studies.... which... is.. er... meta analysis. The whole point of meta analysis is that you have multiple studies of the same experiment.