5 ms·

Meta-analysis is not a form of rigorous independent verification in the vast majority of cases. It is usually a statistical veneer placed, lazily, on top of a

Meta-analysis is not a form of rigorous independent verification in the vast majority of cases. It is usually a statistical veneer placed, lazily, on top of a smear of vastly different experimental results, giving the appearance of rigor. Replication is a rigorous form of independent verification.

dfsegoat 10y ago
How should the average researcher go about replicating something like 3-5 clinical trials that a drug company may have run for a human drug? Should we replicate all that work? Or perhaps just obtain the raw, patient-level data from said company and analyze the data ourselves? I am just curious which you view as more efficacious?
- jackmott 10y ago
  Better to replicate it. Science ain't easy.
  - jmde 10y ago
    Then you have to summarize those replications and decide whether or not the replication studies replicated findings... which means meta-analysis.
    - ekianjo 10y ago
      No, you don't need to do meta-analysis. Without combining the results of multiple trials together, you can see if an observed effect in one trial is found in other trials through independent analysis.
      - Fomite 10y ago
        But explaining whether or not "We didn't get the same answer" is the result of variation in the effect estimate, or because of a genuine failure to replicate, is one aspect of meta-analysis. The purpose of meta-analysis goes beyond pooling. Indeed, one of the first steps in meta-analysis is "Is pooling even a good idea?"
- carsongross 10y ago
  If you want to claim to be a scientist, you replicate the work. If you want to be a statistician of increasingly obvious limited social utility, you rerun the statistics.
  - dfsegoat 10y ago
    Indeed - the Cochrane Collaboration is of quite limited utility: http://www.cochrane.org/evidence http://www.cochrane.org/evidence. They are only the gold-standard for systematic reviews of therapeutic interventions.
    - carsongross 10y ago
      https://en.wikipedia.org/wiki/Replication_crisis https://en.wikipedia.org/wiki/Replication_crisis
  - epistasis 10y ago
    Question: why should we take your declarations of who is and isn't a scientist more seriously than the scientists themselves? Lets take a typical cancer drug. It goes through safty, dosing, and efficiacy clinical trials. In lucky cases, it will show efficacy in a multi-year clinical trial across many geographic locations with hundreds of patients. If the trial is successful, the FDA will eventually approve the drug, the drug can now be prescribed. The FDA mandates that follow up study is done continuously to learn more about the drug; better indications for use, contraindications for when it won't work, etc. Where does "replication" come into any of this? Why in the world would somebody replicate a dosing study and generate the same data? That would be unethical, dangerous, and counterproductive. At best, one would throw out bad data that was improperly collected. At worst, one would just abandon the drug and move on to a different candidate drug. When it comes to human studies that come at real cost to human life, not using all the best available information is unscientific, unethical, should result in civil penalties, and should probably result in criminal penalties as well. This is the situation that the parent was asking about; by making short blanket statements about what a scientist is and is not, without considering the real issues at hand, makes it seem like you're not engaging the issue. "Scientists replicate data" is a simple thing to say if you're looking at stars or running a particle collider or working on a new synthetic compound; taking that simple minded attitude is not appropriate for much of the most expensive research out there. The question of what to replicate and when is a difficult one; it's the tradeoff between new discovery and making sure you're on the right path. If you can make a new discovery that simultaneously proves or disproves that you're on the right path, that's a smarter move, but it's not "replication."
- ekianjo 10y ago
  > Or perhaps just obtain the raw, patient-level data from said company and analyze the data ourselves? Data can be faked, manipulated, unless you can trace it back to its origin and ensure it was recorded properly. There are multiple cases (even recent ones) where investigators modified the results of their trials in order to make them look better than they were. Only replication can alleviate that concern.
  - Twisell 10y ago
    Though critical reading of raw data can reveal fake entry, witch is equally interesting. I agree that replication will always be better and should ideally be mandatory. But meta-review is still way better than nothing. And sometimes you just can't replicate for various reasons.
    - dangerlibrary 10y ago
      I don't think it's obvious that meta-review is better than nothing. It can lend credence to incorrect/fraudulent results, making actual replication appear less important.
Angostura 10y ago
It can be rigorous and useful. Let's not throw the baby out.

dfsegoat 10y ago

How should the average researcher go about replicating something like 3-5 clinical trials that a drug company may have run for a human drug? Should we replicate all that work? Or perhaps just obtain the raw, patient-level data from said company and analyze the data ourselves? I am just curious which you view as more efficacious?

jackmott 10y ago
Better to replicate it. Science ain't easy.
- jmde 10y ago
  Then you have to summarize those replications and decide whether or not the replication studies replicated findings... which means meta-analysis.
  - ekianjo 10y ago
    No, you don't need to do meta-analysis. Without combining the results of multiple trials together, you can see if an observed effect in one trial is found in other trials through independent analysis.
    - Fomite 10y ago
      But explaining whether or not "We didn't get the same answer" is the result of variation in the effect estimate, or because of a genuine failure to replicate, is one aspect of meta-analysis. The purpose of meta-analysis goes beyond pooling. Indeed, one of the first steps in meta-analysis is "Is pooling even a good idea?"
carsongross 10y ago
If you want to claim to be a scientist, you replicate the work. If you want to be a statistician of increasingly obvious limited social utility, you rerun the statistics.
- dfsegoat 10y ago
  Indeed - the Cochrane Collaboration is of quite limited utility: http://www.cochrane.org/evidence http://www.cochrane.org/evidence. They are only the gold-standard for systematic reviews of therapeutic interventions.
  - carsongross 10y ago
    https://en.wikipedia.org/wiki/Replication_crisis https://en.wikipedia.org/wiki/Replication_crisis
- epistasis 10y ago
  Question: why should we take your declarations of who is and isn't a scientist more seriously than the scientists themselves? Lets take a typical cancer drug. It goes through safty, dosing, and efficiacy clinical trials. In lucky cases, it will show efficacy in a multi-year clinical trial across many geographic locations with hundreds of patients. If the trial is successful, the FDA will eventually approve the drug, the drug can now be prescribed. The FDA mandates that follow up study is done continuously to learn more about the drug; better indications for use, contraindications for when it won't work, etc. Where does "replication" come into any of this? Why in the world would somebody replicate a dosing study and generate the same data? That would be unethical, dangerous, and counterproductive. At best, one would throw out bad data that was improperly collected. At worst, one would just abandon the drug and move on to a different candidate drug. When it comes to human studies that come at real cost to human life, not using all the best available information is unscientific, unethical, should result in civil penalties, and should probably result in criminal penalties as well. This is the situation that the parent was asking about; by making short blanket statements about what a scientist is and is not, without considering the real issues at hand, makes it seem like you're not engaging the issue. "Scientists replicate data" is a simple thing to say if you're looking at stars or running a particle collider or working on a new synthetic compound; taking that simple minded attitude is not appropriate for much of the most expensive research out there. The question of what to replicate and when is a difficult one; it's the tradeoff between new discovery and making sure you're on the right path. If you can make a new discovery that simultaneously proves or disproves that you're on the right path, that's a smarter move, but it's not "replication."
ekianjo 10y ago
> Or perhaps just obtain the raw, patient-level data from said company and analyze the data ourselves? Data can be faked, manipulated, unless you can trace it back to its origin and ensure it was recorded properly. There are multiple cases (even recent ones) where investigators modified the results of their trials in order to make them look better than they were. Only replication can alleviate that concern.
- Twisell 10y ago
  Though critical reading of raw data can reveal fake entry, witch is equally interesting. I agree that replication will always be better and should ideally be mandatory. But meta-review is still way better than nothing. And sometimes you just can't replicate for various reasons.
  - dangerlibrary 10y ago
    I don't think it's obvious that meta-review is better than nothing. It can lend credence to incorrect/fraudulent results, making actual replication appear less important.

Angostura 10y ago

It can be rigorous and useful. Let's not throw the baby out.