4 ms·

If I was a billionaire, I would set up some sort of screening lab for scientific/academic/research papers. There would be a statistics division for evaluating t

by Tycho 13y ago

If I was a billionaire, I would set up some sort of screening lab for scientific/academic/research papers. There would be a statistics division for evaluating the application of statistical methods being used; a replication division for checking that experiments do actually replicate; and a corruption division for investigating suspicious influences on the research. It would be tempting to then generate some sort of credibility rating for each institution based on the papers they're publishing, but that would probably invite too much trouble, so best just to publish the results and leave it at that. Arguably this would be a greater benefit to humanity than all the millions poured charitably into cancer research etc.

davmre 13y ago
Something like that idea has actually already been the inspiration for at least one startup: MetaMed (http://en.wikipedia.org/wiki/MetaMed http://en.wikipedia.org/wiki/MetaMed, http://nymag.com/health/bestdoctors/2013/metamed-personalized-health-care/ http://nymag.com/health/bestdoctors/2013/metamed-personalize...) does meta-level analysis of the medical literature to determine which treatments seem effective for rare conditions, taking into account the sample size, statistical methodology, funding sources, etc. of each study. Of course, medicine might be unique as a domain in which individuals are willing to pay vast sums of money to obtain slightly more trustworthy research conclusions, and the profit motive has obvious conflicts with "benefit to humanity" (if someone pays you to research a treatment for their disease, do you post the findings when done? Or hold them privately for the next person with the same problem?). But maybe there are other domains in which the market could support a (non-billionaire's) project for better-validated research.
- gwern 13y ago
  MetaMed, as far as I know, does basically just customized literature reviews; it isn't doing anything I'd recognize as 'meta-level analysis' like the work done by the Cochrane Collaboration or using meta-analytic techniques to directly estimate the reliability of existing medical treatments or beliefs.
merckx 13y ago
Something along those lines: Elizabeth Iorn's Reproducibility Initiative. [1] And an opinion piece which gives some context. [2] [1] https://www.scienceexchange.com/reproducibility https://www.scienceexchange.com/reproducibility [2] http://www.newscientist.com/article/mg21528826.000-is-medical-science-built-on-shaky-foundations.html http://www.newscientist.com/article/mg21528826.000-is-medica...
mathattack 13y ago
I think you'd get very depressed just by the statistics, let alone the reproduction. Especially if you included journals of econometrics.
- neoterics 13y ago
  You know what's more depressing when I read the Cochrane summaries about health studies...most of the medicines and treatments are no better than placebos!
anaphor 13y ago
Fuck yes, I would love to help do something like that. I'm not a statistician though, so I'm probably not very qualified.
neoterics 13y ago
Doesn't the Cochrane Institute already do something similar? They perform meta analysis of studies and talk about the validity of their statistical methods. http://summaries.cochrane.org/ http://summaries.cochrane.org/ Read about it in the book Bad Science by Ben Goldacre.
bjoernd 13y ago
SIGMOD (a database research conference) has set up a reproducibility committee [1]. Their goal is to ensure that the results can be reproduced by someone from the outside. If they succeed, you get an additional label for your graphs saying "Approved by the SIGMOD reproducibility committee." Notably, this is easier in computer science as you don't need to wait for hundreds of patients to turn up having a certain condition. [1] http://www.sigmod.org/2012/reproducibility.shtml http://www.sigmod.org/2012/reproducibility.shtml

by Tycho 13y ago

davmre 13y ago

Something like that idea has actually already been the inspiration for at least one startup: MetaMed (http://en.wikipedia.org/wiki/MetaMed http://en.wikipedia.org/wiki/MetaMed, http://nymag.com/health/bestdoctors/2013/metamed-personalized-health-care/ http://nymag.com/health/bestdoctors/2013/metamed-personalize...) does meta-level analysis of the medical literature to determine which treatments seem effective for rare conditions, taking into account the sample size, statistical methodology, funding sources, etc. of each study. Of course, medicine might be unique as a domain in which individuals are willing to pay vast sums of money to obtain slightly more trustworthy research conclusions, and the profit motive has obvious conflicts with "benefit to humanity" (if someone pays you to research a treatment for their disease, do you post the findings when done? Or hold them privately for the next person with the same problem?). But maybe there are other domains in which the market could support a (non-billionaire's) project for better-validated research.

gwern 13y ago
MetaMed, as far as I know, does basically just customized literature reviews; it isn't doing anything I'd recognize as 'meta-level analysis' like the work done by the Cochrane Collaboration or using meta-analytic techniques to directly estimate the reliability of existing medical treatments or beliefs.

merckx 13y ago

Something along those lines: Elizabeth Iorn's Reproducibility Initiative. [1] And an opinion piece which gives some context. [2] [1] https://www.scienceexchange.com/reproducibility https://www.scienceexchange.com/reproducibility [2] http://www.newscientist.com/article/mg21528826.000-is-medical-science-built-on-shaky-foundations.html http://www.newscientist.com/article/mg21528826.000-is-medica...

mathattack 13y ago

I think you'd get very depressed just by the statistics, let alone the reproduction. Especially if you included journals of econometrics.

neoterics 13y ago
You know what's more depressing when I read the Cochrane summaries about health studies...most of the medicines and treatments are no better than placebos!

anaphor 13y ago

Fuck yes, I would love to help do something like that. I'm not a statistician though, so I'm probably not very qualified.

neoterics 13y ago

Doesn't the Cochrane Institute already do something similar? They perform meta analysis of studies and talk about the validity of their statistical methods. http://summaries.cochrane.org/ http://summaries.cochrane.org/ Read about it in the book Bad Science by Ben Goldacre.

bjoernd 13y ago

SIGMOD (a database research conference) has set up a reproducibility committee [1]. Their goal is to ensure that the results can be reproduced by someone from the outside. If they succeed, you get an additional label for your graphs saying "Approved by the SIGMOD reproducibility committee." Notably, this is easier in computer science as you don't need to wait for hundreds of patients to turn up having a certain condition. [1] http://www.sigmod.org/2012/reproducibility.shtml http://www.sigmod.org/2012/reproducibility.shtml