5 ms·

The further Science, Inc. gets from the scientific method and independent verification, the more of a joke it becomes. Vox Day (no endorsement) makes the disti

The further Science, Inc. gets from the scientific method and independent verification, the more of a joke it becomes. Vox Day (no endorsement) makes the distinction between scientistry and scientody, which I think is an increasingly useful one.

Fomite 10y ago
One should note that meta-analysis is a form of rigorous independent verification. There's also some flaws in his analysis: https://www.ncbi.nlm.nih.gov/pubmed/27620683#cm27620683_26950 https://www.ncbi.nlm.nih.gov/pubmed/27620683#cm27620683_2695...
- carsongross 10y ago
  Meta-analysis is not a form of rigorous independent verification in the vast majority of cases. It is usually a statistical veneer placed, lazily, on top of a smear of vastly different experimental results, giving the appearance of rigor. Replication is a rigorous form of independent verification.
  - dfsegoat 10y ago
    How should the average researcher go about replicating something like 3-5 clinical trials that a drug company may have run for a human drug? Should we replicate all that work? Or perhaps just obtain the raw, patient-level data from said company and analyze the data ourselves? I am just curious which you view as more efficacious?
    - jackmott 10y ago
      Better to replicate it. Science ain't easy.
      - jmde 10y ago
        Then you have to summarize those replications and decide whether or not the replication studies replicated findings... which means meta-analysis.
        ekianjo 10y ago
        No, you don't need to do meta-analysis. Without combining the results of multiple trials together, you can see if an observed effect in one trial is found in other trials through independent analysis.
        Fomite 10y ago
        But explaining whether or not "We didn't get the same answer" is the result of variation in the effect estimate, or because of a genuine failure to replicate, is one aspect of meta-analysis. The purpose of meta-analysis goes beyond pooling. Indeed, one of the first steps in meta-analysis is "Is pooling even a good idea?"
    - carsongross 10y ago
      If you want to claim to be a scientist, you replicate the work. If you want to be a statistician of increasingly obvious limited social utility, you rerun the statistics.
      - dfsegoat 10y ago
        Indeed - the Cochrane Collaboration is of quite limited utility: http://www.cochrane.org/evidence http://www.cochrane.org/evidence. They are only the gold-standard for systematic reviews of therapeutic interventions.
        carsongross 10y ago
        https://en.wikipedia.org/wiki/Replication_crisis https://en.wikipedia.org/wiki/Replication_crisis
      - epistasis 10y ago
        Question: why should we take your declarations of who is and isn't a scientist more seriously than the scientists themselves? Lets take a typical cancer drug. It goes through safty, dosing, and efficiacy clinical trials. In lucky cases, it will show efficacy in a multi-year clinical trial across many geographic locations with hundreds of patients. If the trial is successful, the FDA will eventually approve the drug, the drug can now be prescribed. The FDA mandates that follow up study is done continuously to learn more about the drug; better indications for use, contraindications for when it won't work, etc. Where does "replication" come into any of this? Why in the world would somebody replicate a dosing study and generate the same data? That would be unethical, dangerous, and counterproductive. At best, one would throw out bad data that was improperly collected. At worst, one would just abandon the drug and move on to a different candidate drug. When it comes to human studies that come at real cost to human life, not using all the best available information is unscientific, unethical, should result in civil penalties, and should probably result in criminal penalties as well. This is the situation that the parent was asking about; by making short blanket statements about what a scientist is and is not, without considering the real issues at hand, makes it seem like you're not engaging the issue. "Scientists replicate data" is a simple thing to say if you're looking at stars or running a particle collider or working on a new synthetic compound; taking that simple minded attitude is not appropriate for much of the most expensive research out there. The question of what to replicate and when is a difficult one; it's the tradeoff between new discovery and making sure you're on the right path. If you can make a new discovery that simultaneously proves or disproves that you're on the right path, that's a smarter move, but it's not "replication."
    - ekianjo 10y ago
      > Or perhaps just obtain the raw, patient-level data from said company and analyze the data ourselves? Data can be faked, manipulated, unless you can trace it back to its origin and ensure it was recorded properly. There are multiple cases (even recent ones) where investigators modified the results of their trials in order to make them look better than they were. Only replication can alleviate that concern.
      - Twisell 10y ago
        Though critical reading of raw data can reveal fake entry, witch is equally interesting. I agree that replication will always be better and should ideally be mandatory. But meta-review is still way better than nothing. And sometimes you just can't replicate for various reasons.
        dangerlibrary 10y ago
        I don't think it's obvious that meta-review is better than nothing. It can lend credence to incorrect/fraudulent results, making actual replication appear less important.
  - Angostura 10y ago
    It can be rigorous and useful. Let's not throw the baby out.
- tremon 10y ago
  One should note that meta-analysis is a form of rigorous independent verification. It most definitely is not. Meta-analysis performs aggregation of existing research data in an attempt to validate specific research results for a larger group than initially tested. As such, it is primarily an inductive proof. That it can also validate or invalidate prior research is more of an artifact than a goal. A rigorous independent verification can be performed deductively. No inference required.
  - Fomite 10y ago
    There are steps in properly performed meta-analysis that are very much about verification of a field of research as a whole, rather than a single study. For example, methods to detect likely publication bias, or heterogenity statistics that will indicate whether or not a given literature is in a state to be pooled at all. Both are methods to reveal "Something is wrong here..."
davidgerard 10y ago
Vox Day is literally a creationist (if an old-earth one), so might not be a great name to invoke for any aspect of thinking about this.
- Fomite 10y ago
  Vox Day is many, many things, few of which I'd want to be associated with.

Fomite 10y ago

One should note that meta-analysis is a form of rigorous independent verification. There's also some flaws in his analysis: https://www.ncbi.nlm.nih.gov/pubmed/27620683#cm27620683_26950 https://www.ncbi.nlm.nih.gov/pubmed/27620683#cm27620683_2695...

carsongross 10y ago
Meta-analysis is not a form of rigorous independent verification in the vast majority of cases. It is usually a statistical veneer placed, lazily, on top of a smear of vastly different experimental results, giving the appearance of rigor. Replication is a rigorous form of independent verification.
- dfsegoat 10y ago
  How should the average researcher go about replicating something like 3-5 clinical trials that a drug company may have run for a human drug? Should we replicate all that work? Or perhaps just obtain the raw, patient-level data from said company and analyze the data ourselves? I am just curious which you view as more efficacious?
  - jackmott 10y ago
    Better to replicate it. Science ain't easy.
    - jmde 10y ago
      Then you have to summarize those replications and decide whether or not the replication studies replicated findings... which means meta-analysis.
      - ekianjo 10y ago
        No, you don't need to do meta-analysis. Without combining the results of multiple trials together, you can see if an observed effect in one trial is found in other trials through independent analysis.
        Fomite 10y ago
        But explaining whether or not "We didn't get the same answer" is the result of variation in the effect estimate, or because of a genuine failure to replicate, is one aspect of meta-analysis. The purpose of meta-analysis goes beyond pooling. Indeed, one of the first steps in meta-analysis is "Is pooling even a good idea?"
  - carsongross 10y ago
    If you want to claim to be a scientist, you replicate the work. If you want to be a statistician of increasingly obvious limited social utility, you rerun the statistics.
    - dfsegoat 10y ago
      Indeed - the Cochrane Collaboration is of quite limited utility: http://www.cochrane.org/evidence http://www.cochrane.org/evidence. They are only the gold-standard for systematic reviews of therapeutic interventions.
      - carsongross 10y ago
        https://en.wikipedia.org/wiki/Replication_crisis https://en.wikipedia.org/wiki/Replication_crisis
    - epistasis 10y ago
      Question: why should we take your declarations of who is and isn't a scientist more seriously than the scientists themselves? Lets take a typical cancer drug. It goes through safty, dosing, and efficiacy clinical trials. In lucky cases, it will show efficacy in a multi-year clinical trial across many geographic locations with hundreds of patients. If the trial is successful, the FDA will eventually approve the drug, the drug can now be prescribed. The FDA mandates that follow up study is done continuously to learn more about the drug; better indications for use, contraindications for when it won't work, etc. Where does "replication" come into any of this? Why in the world would somebody replicate a dosing study and generate the same data? That would be unethical, dangerous, and counterproductive. At best, one would throw out bad data that was improperly collected. At worst, one would just abandon the drug and move on to a different candidate drug. When it comes to human studies that come at real cost to human life, not using all the best available information is unscientific, unethical, should result in civil penalties, and should probably result in criminal penalties as well. This is the situation that the parent was asking about; by making short blanket statements about what a scientist is and is not, without considering the real issues at hand, makes it seem like you're not engaging the issue. "Scientists replicate data" is a simple thing to say if you're looking at stars or running a particle collider or working on a new synthetic compound; taking that simple minded attitude is not appropriate for much of the most expensive research out there. The question of what to replicate and when is a difficult one; it's the tradeoff between new discovery and making sure you're on the right path. If you can make a new discovery that simultaneously proves or disproves that you're on the right path, that's a smarter move, but it's not "replication."
  - ekianjo 10y ago
    > Or perhaps just obtain the raw, patient-level data from said company and analyze the data ourselves? Data can be faked, manipulated, unless you can trace it back to its origin and ensure it was recorded properly. There are multiple cases (even recent ones) where investigators modified the results of their trials in order to make them look better than they were. Only replication can alleviate that concern.
    - Twisell 10y ago
      Though critical reading of raw data can reveal fake entry, witch is equally interesting. I agree that replication will always be better and should ideally be mandatory. But meta-review is still way better than nothing. And sometimes you just can't replicate for various reasons.
      - dangerlibrary 10y ago
        I don't think it's obvious that meta-review is better than nothing. It can lend credence to incorrect/fraudulent results, making actual replication appear less important.
- Angostura 10y ago
  It can be rigorous and useful. Let's not throw the baby out.
tremon 10y ago
One should note that meta-analysis is a form of rigorous independent verification. It most definitely is not. Meta-analysis performs aggregation of existing research data in an attempt to validate specific research results for a larger group than initially tested. As such, it is primarily an inductive proof. That it can also validate or invalidate prior research is more of an artifact than a goal. A rigorous independent verification can be performed deductively. No inference required.
- Fomite 10y ago
  There are steps in properly performed meta-analysis that are very much about verification of a field of research as a whole, rather than a single study. For example, methods to detect likely publication bias, or heterogenity statistics that will indicate whether or not a given literature is in a state to be pooled at all. Both are methods to reveal "Something is wrong here..."

davidgerard 10y ago

Vox Day is literally a creationist (if an old-earth one), so might not be a great name to invoke for any aspect of thinking about this.

Fomite 10y ago
Vox Day is many, many things, few of which I'd want to be associated with.