5 ms·

Preregistration of clinical trials causes medicines to stop working

84 points by stared 11y ago

Alex3917 11y ago
If clinical trials were required to be funded and conducted by the government then you'd see the same thing again.
chopin 11y ago
Would someone care to explain what the graphic shown means exactly? I have a hard time understanding what the vertical axis tells me.
- sp332 11y ago
  The studies on the chart are for treatment of cardiovascular disease. The lower they are on the chart, the more they reduced the risk of whatever it was they were trying to prevent (the "primary outcome" for each study).
- gech 11y ago
  When experimenters are required to state their targets beforehand, they fail to meet them. If they don't have to state the outcomes beforehand, then its much easier to spin what they do find as beneficial!
- jmount 11y ago
  (just skimming) I am guessing the y-axis is the observed expected value of the treatment (low being good) and the dots are re-colered to null if the sample size was too small to separate the (not shown) error bars around the dot from no effect. The punchline is: once the researchers had to declare their statistical procedure before turning in the results most of the dots get colored "null." Meaning the strongest effect in the past was the ability to argue your way into a more favorable analysis.
entee 11y ago
There's a huge amount of concern in the medical profession that lots of studies in the area are statistically compromised precisely because the authors gamed the statistics after the fact, or because improper metrics were used in the first place. See these (kind of old but still quite appropriate) links: http://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.0020124 http://journals.plos.org/plosmedicine/article?id=10.1371/jou... http://www.theatlantic.com/magazine/archive/2010/11/lies-damned-lies-and-medical-science/308269/ http://www.theatlantic.com/magazine/archive/2010/11/lies-dam... It's this kind of work that led to pressure to pre-declare statistical objectives, and now we see the results. Biology is hard, medicine is hard, we must be quite humble about what we understand in these fields and therefore how easy it is to be misled by early promising results.
JangoSteve 11y ago
In case it's not totally obvious, basically... * In 2000, a rule was passed that clinical trials must include their hypotheses (i.e. clinical outcomes) in their initial registration before the trial is conducted. * From the graph, the number and proportion of clinical trials through NLHBI with positive outcomes (i.e. an improvement in some health condition) is far lower after 2000 than before. It was previously easy for researchers to conduct a trial, and then find positive trends in the data to form positive outcomes by framing the data from the trial in a different manner, as if the trial were originally intended to test that positive outcome. This is a problem related to A Priori Probability, where you're essentially turning research, which could have been statistically applied to a larger population given some statistical confidence, into deductive reasoning which doesn't necessarily apply beyond the finite sample size [1] (also see @entee's comment and links for better explanation of why this is important). This article is stating that the graph implies that, because there are fewer positive outcomes after this new rule in 2000 than before, most of the positive outcomes prior to 2000 (which likely lead to new FDA-approved drugs on the market) must have been illegitimate. I would point out though, that at least from that graph, the last positive outcome shown prior to 2000 was in 1996, so it's not like it obviously continued right up until 2000 and then suddenly stopped. Even when that rule was imposed in 2000 (I'm not necessarily sure how the ruling was phased in), there was a 4-year gap since the last positive outcome before then, and I don't see another 4-year gap prior to that since before around 1979. I'm not saying I disagree with the article (obviously), but just that while the graphic used seems to imply a trend, it seems like more data would be needed to conclusively show that trend. [1] http://www.investopedia.com/terms/a/apriori.asp http://www.investopedia.com/terms/a/apriori.asp
- akiselev 11y ago
  The 90s is also around the time when we pretty much ran out of small molecule blockbuster drugs (the very profitable but low hanging fruit) and started shifting over to much more complex multi-drug therapies like those for HIV and cancer. There are still a few such small molecule candidates left but the major markets like blood pressure, heart disease, arthritis, etc. are very saturated. As the clinic trials start to target more complex diseases with more complicated drugs and therapies it adds a large confounding factor to the decrease in the number of successful trials.
NoMoreNicksLeft 11y ago
My god. Can we please stop requiring this? I need my placebos to work.
biot 11y ago
In other words, declaring your lottery number picks prior to the drawing results in fewer winners than being able to pick your numbers afterwards. Ergo, pre-declaring your picks causes lotteries to stop working. I really hope the article is satire.
- mizzao 11y ago
  It's not satire; that's just the sad state of scientific experimentation. A similar finding was widely publicized for psychology studies: http://www.nature.com/news/over-half-of-psychology-studies-fail-reproducibility-test-1.18248 http://www.nature.com/news/over-half-of-psychology-studies-f...

84 points by stared 11y ago

If clinical trials were required to be funded and conducted by the government then you'd see the same thing again.

Would someone care to explain what the graphic shown means exactly? I have a hard time understanding what the vertical axis tells me.

sp332 11y ago
The studies on the chart are for treatment of cardiovascular disease. The lower they are on the chart, the more they reduced the risk of whatever it was they were trying to prevent (the "primary outcome" for each study).
gech 11y ago
When experimenters are required to state their targets beforehand, they fail to meet them. If they don't have to state the outcomes beforehand, then its much easier to spin what they do find as beneficial!
jmount 11y ago
(just skimming) I am guessing the y-axis is the observed expected value of the treatment (low being good) and the dots are re-colered to null if the sample size was too small to separate the (not shown) error bars around the dot from no effect. The punchline is: once the researchers had to declare their statistical procedure before turning in the results most of the dots get colored "null." Meaning the strongest effect in the past was the ability to argue your way into a more favorable analysis.

entee 11y ago

There's a huge amount of concern in the medical profession that lots of studies in the area are statistically compromised precisely because the authors gamed the statistics after the fact, or because improper metrics were used in the first place. See these (kind of old but still quite appropriate) links: http://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.0020124 http://journals.plos.org/plosmedicine/article?id=10.1371/jou... http://www.theatlantic.com/magazine/archive/2010/11/lies-damned-lies-and-medical-science/308269/ http://www.theatlantic.com/magazine/archive/2010/11/lies-dam... It's this kind of work that led to pressure to pre-declare statistical objectives, and now we see the results. Biology is hard, medicine is hard, we must be quite humble about what we understand in these fields and therefore how easy it is to be misled by early promising results.

JangoSteve 11y ago

In case it's not totally obvious, basically... * In 2000, a rule was passed that clinical trials must include their hypotheses (i.e. clinical outcomes) in their initial registration before the trial is conducted. * From the graph, the number and proportion of clinical trials through NLHBI with positive outcomes (i.e. an improvement in some health condition) is far lower after 2000 than before. It was previously easy for researchers to conduct a trial, and then find positive trends in the data to form positive outcomes by framing the data from the trial in a different manner, as if the trial were originally intended to test that positive outcome. This is a problem related to A Priori Probability, where you're essentially turning research, which could have been statistically applied to a larger population given some statistical confidence, into deductive reasoning which doesn't necessarily apply beyond the finite sample size [1] (also see @entee's comment and links for better explanation of why this is important). This article is stating that the graph implies that, because there are fewer positive outcomes after this new rule in 2000 than before, most of the positive outcomes prior to 2000 (which likely lead to new FDA-approved drugs on the market) must have been illegitimate. I would point out though, that at least from that graph, the last positive outcome shown prior to 2000 was in 1996, so it's not like it obviously continued right up until 2000 and then suddenly stopped. Even when that rule was imposed in 2000 (I'm not necessarily sure how the ruling was phased in), there was a 4-year gap since the last positive outcome before then, and I don't see another 4-year gap prior to that since before around 1979. I'm not saying I disagree with the article (obviously), but just that while the graphic used seems to imply a trend, it seems like more data would be needed to conclusively show that trend. [1] http://www.investopedia.com/terms/a/apriori.asp http://www.investopedia.com/terms/a/apriori.asp

akiselev 11y ago
The 90s is also around the time when we pretty much ran out of small molecule blockbuster drugs (the very profitable but low hanging fruit) and started shifting over to much more complex multi-drug therapies like those for HIV and cancer. There are still a few such small molecule candidates left but the major markets like blood pressure, heart disease, arthritis, etc. are very saturated. As the clinic trials start to target more complex diseases with more complicated drugs and therapies it adds a large confounding factor to the decrease in the number of successful trials.

NoMoreNicksLeft 11y ago

My god. Can we please stop requiring this? I need my placebos to work.

biot 11y ago

In other words, declaring your lottery number picks prior to the drawing results in fewer winners than being able to pick your numbers afterwards. Ergo, pre-declaring your picks causes lotteries to stop working. I really hope the article is satire.

mizzao 11y ago
It's not satire; that's just the sad state of scientific experimentation. A similar finding was widely publicized for psychology studies: http://www.nature.com/news/over-half-of-psychology-studies-fail-reproducibility-test-1.18248 http://www.nature.com/news/over-half-of-psychology-studies-f...