6 ms·

Reproducibility Initiative gets $1.3M grant to validate 50 cancer studies

236 points by djkn0x 13y ago

dnautics 13y ago
this is really phenomenal. Congratulations, SE.
brianbreslin 13y ago
awesome congrats guys!
ypandit 13y ago
Congratulations SciEx !
irollboozers 13y ago
Science Exchange is leading and pushing ahead with this very important work. They are addressing what the public funders and private industry can't and won't do, but at scale this becomes really powerful. Great stuff.
- Create 13y ago
  They did it publicly, ie Why Most Published Research Findings Are False. PLoS Med 2(8): e124. doi:10.1371/journal.pmed.0020124 and privately: the matter of fact is, pharma companies are unhappy, because according to them, over 70% is irreproducible (euphemism for BS). This is why ie. BMJ will only accept papers from January on that publish evidence ie. data also. One wonders what happened before... BMJ 2013; 347 doi: http://dx.doi.org/10.1136/bmj.f5975 http://dx.doi.org/10.1136/bmj.f5975
gabemart 13y ago
I don't know very much about how reproducibility validation works. Is it the case that, if we assume p= ~0.05 and all 50 original studies are perfect, we would expect the first iteration of reproducibility validation to fail for ~2 of the 50 studies?
- mjn 13y ago
  Not necessarily 5%, no. The p-value is controlling the false-positive rate, i.e. it constrains the likelihood that a seemingly-significant result was really just by chance. Therefore, if we were now given the real ground truth, it's likely that ~5% of the original positive results (if the studies were properly conducted) would turn out to be unsubstantiated after all. But you seem to be positing the reverse: a hypothetical case where we assume that the results of the original studies were in fact correct (i.e. ground truth), and we want a sort of false-negative rate, the chance that a fact would not be confirmed by a replication study, despite actually being true. That depends on several things about the replication attempt, such as sample size and methodology, summarized into the concept of statistical power: http://en.wikipedia.org/wiki/Statistical_power http://en.wikipedia.org/wiki/Statistical_power The overall failure-to-replicate rate would involve both aspects, with four possible outcomes at varying likelihood: 1) a correct result that was confirmed; 2) a correct result that failed to be confirmed; 3) an incorrect result that was confirmed anyway; and 4) an incorrect result that failed to be confirmed. Obviously it would be ideal if #1 and #4 were much higher than #2 and #3.
  - gabemart 13y ago
    Thanks, very interesting. It was silly of me to assume that the p-value could be used in that way when, as you explain, it's not measuring or indicating that at all.
    - gwern 13y ago
      Don't feel too bad. It's a very common mistake; see some of the citations in https://en.wikipedia.org/wiki/P-value#Misunderstandings https://en.wikipedia.org/wiki/P-value#Misunderstandings and http://lesswrong.com/lw/g13/against_nhst/ http://lesswrong.com/lw/g13/against_nhst/
  - yummyfajitas 13y ago
    it's likely that ~5% of the original positive results (if the studies were properly conducted) would turn out to be unsubstantiated after all. Not quite. Assume P of the tests are truly positive, and N are truly negative. We expect to see approximately 0.05 x N + A x P positives in the entire sample, where A is the probability of a false negative. So the fraction of true positives not replicated is likely to be 0.05 x N / (0.05 x N + A x P).
Osmium 13y ago
Honestly, I'm impressed $1.3M is enough for 50 studies! Though, of course, verification should be cheaper than the original research since you know exactly what to look for and how to find it.
- mathattack 13y ago
  If $1.3M can cover 50, they should fund reproducing 500, or even 5000. I suspect that many pure science studies are relatively easier to reproduce since it's more about basic properties. If you're getting into medicine, it's more expensive because you may need to reproduce trials. Psychology even more so because the tests are so soft. But of course psych and medical studies are the ones most in need of replication.
  - Osmium 13y ago
    > I suspect that many pure science studies are relatively easier to reproduce since it's more about basic properties. Not necessarily true at all actually. To use an example, the field I work in requires that we grow our own semiconductor crystals. Different growth reactors in different universities producing nominally the exact same substance will result in sometimes vastly different properties of the end result when studied (usually due to minute trace impurities, or marginally different temperatures or flow rates during growth, or a host of other things–all of which need to be analysed and vetted–and this happens even if they're using growth reactors made by the same manufacturer!). It makes replication very difficult and often results in slow progress. And it's expensive too. So it's not just medicine that has that problem. Edit: I should add that obviously in the end it works out because these materials eventually make it into industry and into millions of devices. So it's ultimately reproducible. But that gap between first discovering a process or a mechanism or just something interesting and making it reproducible consistently enough and at a large enough scale for industry is just huge.
    - mathattack 13y ago
      Indeed - thank you for clarifying. This does sound expensive, though all the more necessary.
DaveWalk 13y ago
I'd be interested to know which studies they are targeting. Is SciEx testing a key figure from an expansive publication, or the entire methodology from discoveries with few tests? To me, this seems to be a conceptually difficult decision to make...most discoveries do not discuss the number of years (or failed attempts) that goes by before obtaining the quantifiable result. And where is the peer review in this process? I suppose as soon as something turns up unreproducible we will find out.
napoleoncomplex 13y ago
Reproducibility in science is something that badly needs this push. It's an incredibly difficult "sell" to anyone with funds for research, and I'm extremely happy that they've found capital for it. The foundations of our scientific knowledge need to be solidified, and from all the science news and developments, this one is the one that makes me by far the most excited for the future of science. Next on the list, open source repositories for protocols of experiments! Maybe someone surprises me with a link to an existing solution :).
- Alex3917 13y ago
  Assuming there are about 3 dozen common scientific mistakes in medical journal articles, each appearing in on average 10-20% of papers with a relatively high degree of independence, there isn't really much foundation for medical science to begin with. (And I think these assumptions are probably fairly accurate.)
  - tokenadult 13y ago
    You haven't validated any of those assumptions. Moreover a "common scientific mistake," while regrettable and something to reduce, may not have a lot of impact on actual clinical practice based on what physicians read in published medical journal articles. You'll have to connect a lot more dots to show that there is a high-impact problem here. Meanwhile, life expectancy at age 40, at age 60, and at even higher ages is still rising throughout the developed countries of the world,[1] so at least the trend lines look good for continued improvements in health as medical research is incrementally improved. Girls born since 2000 in the developed world are more likely than not to reach the age of 100, with boys likely to enjoy lifespans almost as long. The article "The Biodemography of Human Ageing"[2] by James Vaupel, originally published in the journal Nature in 2010, is a good current reference on the subject. Vaupel is one of the leading scholars on the demography of aging and how to adjust for time trends in life expectancy. His striking finding is "Humans are living longer than ever before. In fact, newborn children in high-income countries can expect to live to more than 100 years. Starting in the mid-1800s, human longevity has increased dramatically and life expectancy is increasing by an average of six hours a day."[3] An article in a series on Slate, "Why Are You Not Dead Yet? Life expectancy doubled in past 150 years. Here’s why"[4] makes clear that it's progress here, and progress there, and a lot of little things adding up that have resulted in the dramatic increases in healthy lifespan over the last few generations. [1] http://www.scientificamerican.com/article.cfm?id=longevity-why-we-die-global-life-expectancy http://www.scientificamerican.com/article.cfm?id=longevity-w... [2] http://www.demographic-challenge.com/files/downloads/2eb51e2860ef54d218ce5ce19abe6a59/dc_biodemography_of_human_ageing_nature_2010_vaupel.pdf http://www.demographic-challenge.com/files/downloads/2eb51e2... [3] http://www.prb.org/Journalists/Webcasts/2010/humanlongevity.aspx http://www.prb.org/Journalists/Webcasts/2010/humanlongevity.... [4] http://www.slate.com/articles/health_and_science/science_of_longevity/2013/09/life_expectancy_history_public_health_and_medical_advances_that_lead_to.html http://www.slate.com/articles/health_and_science/science_of_...
doctoboggan 13y ago
This is awesome. I met some of the people behind The Center For Open Science at SciPy this year. They seemed very passionate. I hope the idea of reproducing experiments as a matter of course becomes more common. Maybe in the future to be a reputable scientist you will have had to reproduce many of the current experiments of the time.

236 points by djkn0x 13y ago

dnautics 13y ago

this is really phenomenal. Congratulations, SE.

brianbreslin 13y ago

awesome congrats guys!

ypandit 13y ago

Congratulations SciEx !

irollboozers 13y ago

Science Exchange is leading and pushing ahead with this very important work. They are addressing what the public funders and private industry can't and won't do, but at scale this becomes really powerful. Great stuff.

Create 13y ago
They did it publicly, ie Why Most Published Research Findings Are False. PLoS Med 2(8): e124. doi:10.1371/journal.pmed.0020124 and privately: the matter of fact is, pharma companies are unhappy, because according to them, over 70% is irreproducible (euphemism for BS). This is why ie. BMJ will only accept papers from January on that publish evidence ie. data also. One wonders what happened before... BMJ 2013; 347 doi: http://dx.doi.org/10.1136/bmj.f5975 http://dx.doi.org/10.1136/bmj.f5975

gabemart 13y ago

I don't know very much about how reproducibility validation works. Is it the case that, if we assume p= ~0.05 and all 50 original studies are perfect, we would expect the first iteration of reproducibility validation to fail for ~2 of the 50 studies?

mjn 13y ago
Not necessarily 5%, no. The p-value is controlling the false-positive rate, i.e. it constrains the likelihood that a seemingly-significant result was really just by chance. Therefore, if we were now given the real ground truth, it's likely that ~5% of the original positive results (if the studies were properly conducted) would turn out to be unsubstantiated after all. But you seem to be positing the reverse: a hypothetical case where we assume that the results of the original studies were in fact correct (i.e. ground truth), and we want a sort of false-negative rate, the chance that a fact would not be confirmed by a replication study, despite actually being true. That depends on several things about the replication attempt, such as sample size and methodology, summarized into the concept of statistical power: http://en.wikipedia.org/wiki/Statistical_power http://en.wikipedia.org/wiki/Statistical_power The overall failure-to-replicate rate would involve both aspects, with four possible outcomes at varying likelihood: 1) a correct result that was confirmed; 2) a correct result that failed to be confirmed; 3) an incorrect result that was confirmed anyway; and 4) an incorrect result that failed to be confirmed. Obviously it would be ideal if #1 and #4 were much higher than #2 and #3.
- gabemart 13y ago
  Thanks, very interesting. It was silly of me to assume that the p-value could be used in that way when, as you explain, it's not measuring or indicating that at all.
  - gwern 13y ago
    Don't feel too bad. It's a very common mistake; see some of the citations in https://en.wikipedia.org/wiki/P-value#Misunderstandings https://en.wikipedia.org/wiki/P-value#Misunderstandings and http://lesswrong.com/lw/g13/against_nhst/ http://lesswrong.com/lw/g13/against_nhst/
- yummyfajitas 13y ago
  it's likely that ~5% of the original positive results (if the studies were properly conducted) would turn out to be unsubstantiated after all. Not quite. Assume P of the tests are truly positive, and N are truly negative. We expect to see approximately 0.05 x N + A x P positives in the entire sample, where A is the probability of a false negative. So the fraction of true positives not replicated is likely to be 0.05 x N / (0.05 x N + A x P).

Osmium 13y ago

Honestly, I'm impressed $1.3M is enough for 50 studies! Though, of course, verification should be cheaper than the original research since you know exactly what to look for and how to find it.

mathattack 13y ago
If $1.3M can cover 50, they should fund reproducing 500, or even 5000. I suspect that many pure science studies are relatively easier to reproduce since it's more about basic properties. If you're getting into medicine, it's more expensive because you may need to reproduce trials. Psychology even more so because the tests are so soft. But of course psych and medical studies are the ones most in need of replication.
- Osmium 13y ago
  > I suspect that many pure science studies are relatively easier to reproduce since it's more about basic properties. Not necessarily true at all actually. To use an example, the field I work in requires that we grow our own semiconductor crystals. Different growth reactors in different universities producing nominally the exact same substance will result in sometimes vastly different properties of the end result when studied (usually due to minute trace impurities, or marginally different temperatures or flow rates during growth, or a host of other things–all of which need to be analysed and vetted–and this happens even if they're using growth reactors made by the same manufacturer!). It makes replication very difficult and often results in slow progress. And it's expensive too. So it's not just medicine that has that problem. Edit: I should add that obviously in the end it works out because these materials eventually make it into industry and into millions of devices. So it's ultimately reproducible. But that gap between first discovering a process or a mechanism or just something interesting and making it reproducible consistently enough and at a large enough scale for industry is just huge.
  - mathattack 13y ago
    Indeed - thank you for clarifying. This does sound expensive, though all the more necessary.

DaveWalk 13y ago

I'd be interested to know which studies they are targeting. Is SciEx testing a key figure from an expansive publication, or the entire methodology from discoveries with few tests? To me, this seems to be a conceptually difficult decision to make...most discoveries do not discuss the number of years (or failed attempts) that goes by before obtaining the quantifiable result. And where is the peer review in this process? I suppose as soon as something turns up unreproducible we will find out.

napoleoncomplex 13y ago

Reproducibility in science is something that badly needs this push. It's an incredibly difficult "sell" to anyone with funds for research, and I'm extremely happy that they've found capital for it. The foundations of our scientific knowledge need to be solidified, and from all the science news and developments, this one is the one that makes me by far the most excited for the future of science. Next on the list, open source repositories for protocols of experiments! Maybe someone surprises me with a link to an existing solution :).

Alex3917 13y ago
Assuming there are about 3 dozen common scientific mistakes in medical journal articles, each appearing in on average 10-20% of papers with a relatively high degree of independence, there isn't really much foundation for medical science to begin with. (And I think these assumptions are probably fairly accurate.)
- tokenadult 13y ago
  You haven't validated any of those assumptions. Moreover a "common scientific mistake," while regrettable and something to reduce, may not have a lot of impact on actual clinical practice based on what physicians read in published medical journal articles. You'll have to connect a lot more dots to show that there is a high-impact problem here. Meanwhile, life expectancy at age 40, at age 60, and at even higher ages is still rising throughout the developed countries of the world,[1] so at least the trend lines look good for continued improvements in health as medical research is incrementally improved. Girls born since 2000 in the developed world are more likely than not to reach the age of 100, with boys likely to enjoy lifespans almost as long. The article "The Biodemography of Human Ageing"[2] by James Vaupel, originally published in the journal Nature in 2010, is a good current reference on the subject. Vaupel is one of the leading scholars on the demography of aging and how to adjust for time trends in life expectancy. His striking finding is "Humans are living longer than ever before. In fact, newborn children in high-income countries can expect to live to more than 100 years. Starting in the mid-1800s, human longevity has increased dramatically and life expectancy is increasing by an average of six hours a day."[3] An article in a series on Slate, "Why Are You Not Dead Yet? Life expectancy doubled in past 150 years. Here’s why"[4] makes clear that it's progress here, and progress there, and a lot of little things adding up that have resulted in the dramatic increases in healthy lifespan over the last few generations. [1] http://www.scientificamerican.com/article.cfm?id=longevity-why-we-die-global-life-expectancy http://www.scientificamerican.com/article.cfm?id=longevity-w... [2] http://www.demographic-challenge.com/files/downloads/2eb51e2860ef54d218ce5ce19abe6a59/dc_biodemography_of_human_ageing_nature_2010_vaupel.pdf http://www.demographic-challenge.com/files/downloads/2eb51e2... [3] http://www.prb.org/Journalists/Webcasts/2010/humanlongevity.aspx http://www.prb.org/Journalists/Webcasts/2010/humanlongevity.... [4] http://www.slate.com/articles/health_and_science/science_of_longevity/2013/09/life_expectancy_history_public_health_and_medical_advances_that_lead_to.html http://www.slate.com/articles/health_and_science/science_of_...

doctoboggan 13y ago

This is awesome. I met some of the people behind The Center For Open Science at SciPy this year. They seemed very passionate. I hope the idea of reproducing experiments as a matter of course becomes more common. Maybe in the future to be a reputable scientist you will have had to reproduce many of the current experiments of the time.