5 ms·

In case it's not totally obvious, basically... * In 2000, a rule was passed that clinical trials must include their hypotheses (i.e. clinical outcomes) in thei

In case it's not totally obvious, basically... * In 2000, a rule was passed that clinical trials must include their hypotheses (i.e. clinical outcomes) in their initial registration before the trial is conducted. * From the graph, the number and proportion of clinical trials through NLHBI with positive outcomes (i.e. an improvement in some health condition) is far lower after 2000 than before. It was previously easy for researchers to conduct a trial, and then find positive trends in the data to form positive outcomes by framing the data from the trial in a different manner, as if the trial were originally intended to test that positive outcome. This is a problem related to A Priori Probability, where you're essentially turning research, which could have been statistically applied to a larger population given some statistical confidence, into deductive reasoning which doesn't necessarily apply beyond the finite sample size [1] (also see @entee's comment and links for better explanation of why this is important). This article is stating that the graph implies that, because there are fewer positive outcomes after this new rule in 2000 than before, most of the positive outcomes prior to 2000 (which likely lead to new FDA-approved drugs on the market) must have been illegitimate. I would point out though, that at least from that graph, the last positive outcome shown prior to 2000 was in 1996, so it's not like it obviously continued right up until 2000 and then suddenly stopped. Even when that rule was imposed in 2000 (I'm not necessarily sure how the ruling was phased in), there was a 4-year gap since the last positive outcome before then, and I don't see another 4-year gap prior to that since before around 1979. I'm not saying I disagree with the article (obviously), but just that while the graphic used seems to imply a trend, it seems like more data would be needed to conclusively show that trend. [1] http://www.investopedia.com/terms/a/apriori.asp http://www.investopedia.com/terms/a/apriori.asp

akiselev 11y ago
The 90s is also around the time when we pretty much ran out of small molecule blockbuster drugs (the very profitable but low hanging fruit) and started shifting over to much more complex multi-drug therapies like those for HIV and cancer. There are still a few such small molecule candidates left but the major markets like blood pressure, heart disease, arthritis, etc. are very saturated. As the clinic trials start to target more complex diseases with more complicated drugs and therapies it adds a large confounding factor to the decrease in the number of successful trials.
- acveilleux 11y ago
  I think you make a very salient point. From the abstract: > We identified all large NHLBI supported RCTs between 1970 and 2012 evaluating drugs or dietary supplements for the treatment or prevention of cardiovascular disease. So this is for a narrow field where there are effective drugs already and where all the low hangoing fruits are fully picked because the business opportunity is huge.
lutorm 11y ago
Was it possible before this change to run a trial and never publish the outcome? If so, it's also possible that people just kept running trials until they got a positive outcome.
- acveilleux 11y ago
  It's still possible now.
bjornsing 11y ago
> while the graphic used seems to imply a trend, it seems like more data would be needed to conclusively show that trend. And you can tell that by looking at the graphic...? There's also a reference to a X2-test with p=0.0005 (meaning that there's a 0.05% chance that the trend you see is accidental and would disappear with more data). I think you're right to be skeptical (correlation is not causation and so on...), but not that too little data is the problem here.
- JangoSteve 11y ago
  By too little data, I didn't mean that they should have collected more data, I meant that it would be nice to see a) other changes within the drug industry from the 90s and early 2000s that may or may not have contributed plotted on the same graph (not just a single line for this one rule as if that's the only event to affect clinical trials since the 70s), and b) more clinical trial outcome data into the future. More generally, I meant that it's hard to attribute historical events conclusively to specific cause-and-effect, because we can never rewind time in order to change one variable and observer its effects.
p4wnc6 11y ago
> This is a problem related to A Priori Probability, where you're essentially turning research, which could have been statistically applied to a larger population given some statistical confidence, into deductive reasoning which doesn't necessarily apply beyond the finite sample size [1] (also see @entee's comment and links for better explanation of why this is important). Maybe you are using these terms differently than they are used professionally, but this is not correct if it is meant as a criticism of Bayesian methods. There is nothing inherently better (and in fact there are several things potentially worse) about looking at the data with inductive reasoning about a potentially larger population with assumed baseline properties. The problem here is simply p-hacking, whether by accident or intentional. This can be done whether you are performing Bayesian analysis involving a prior distribution, or performing frequentist analysis with assumptions about null hypothesis and subjectively chosen rejection criteria. If you don't register your model validation procedures ahead of time, then after the fact you can data mine among many (or even all possible) model validation scores and only publish scores that show a positive effect. It's not about one framework of statistics or the other. It's about statistical rigor for whatever framework. It's about pursuing more than one study. And it's about attempting to use holistic model validation techniques that cover wide ranges of anticipated outcomes, to hopefully drive down the chances that you develop a conclusion from the study based only upon a narrow set of outcomes that could be biased.
- JangoSteve 11y ago
  I am definitely not using the terms as they are used professionally (and certainly not a criticism of Bayesian methods in any way). It was an attempt to explain in more lay terms how when p-hacking is performed on some dataset, the "findings" can rarely be applied to larger (or different) datasets due to the arbitrary criteria used to reject the parts of the data that aren't conducive to the selected outcome, since the rejection criteria were sort of custom-fit to that specific dataset. I agree with everything you said.

akiselev 11y ago

The 90s is also around the time when we pretty much ran out of small molecule blockbuster drugs (the very profitable but low hanging fruit) and started shifting over to much more complex multi-drug therapies like those for HIV and cancer. There are still a few such small molecule candidates left but the major markets like blood pressure, heart disease, arthritis, etc. are very saturated. As the clinic trials start to target more complex diseases with more complicated drugs and therapies it adds a large confounding factor to the decrease in the number of successful trials.

acveilleux 11y ago
I think you make a very salient point. From the abstract: > We identified all large NHLBI supported RCTs between 1970 and 2012 evaluating drugs or dietary supplements for the treatment or prevention of cardiovascular disease. So this is for a narrow field where there are effective drugs already and where all the low hangoing fruits are fully picked because the business opportunity is huge.

lutorm 11y ago

Was it possible before this change to run a trial and never publish the outcome? If so, it's also possible that people just kept running trials until they got a positive outcome.

acveilleux 11y ago
It's still possible now.

bjornsing 11y ago

> while the graphic used seems to imply a trend, it seems like more data would be needed to conclusively show that trend. And you can tell that by looking at the graphic...? There's also a reference to a X2-test with p=0.0005 (meaning that there's a 0.05% chance that the trend you see is accidental and would disappear with more data). I think you're right to be skeptical (correlation is not causation and so on...), but not that too little data is the problem here.

JangoSteve 11y ago
By too little data, I didn't mean that they should have collected more data, I meant that it would be nice to see a) other changes within the drug industry from the 90s and early 2000s that may or may not have contributed plotted on the same graph (not just a single line for this one rule as if that's the only event to affect clinical trials since the 70s), and b) more clinical trial outcome data into the future. More generally, I meant that it's hard to attribute historical events conclusively to specific cause-and-effect, because we can never rewind time in order to change one variable and observer its effects.

p4wnc6 11y ago

> This is a problem related to A Priori Probability, where you're essentially turning research, which could have been statistically applied to a larger population given some statistical confidence, into deductive reasoning which doesn't necessarily apply beyond the finite sample size [1] (also see @entee's comment and links for better explanation of why this is important). Maybe you are using these terms differently than they are used professionally, but this is not correct if it is meant as a criticism of Bayesian methods. There is nothing inherently better (and in fact there are several things potentially worse) about looking at the data with inductive reasoning about a potentially larger population with assumed baseline properties. The problem here is simply p-hacking, whether by accident or intentional. This can be done whether you are performing Bayesian analysis involving a prior distribution, or performing frequentist analysis with assumptions about null hypothesis and subjectively chosen rejection criteria. If you don't register your model validation procedures ahead of time, then after the fact you can data mine among many (or even all possible) model validation scores and only publish scores that show a positive effect. It's not about one framework of statistics or the other. It's about statistical rigor for whatever framework. It's about pursuing more than one study. And it's about attempting to use holistic model validation techniques that cover wide ranges of anticipated outcomes, to hopefully drive down the chances that you develop a conclusion from the study based only upon a narrow set of outcomes that could be biased.

JangoSteve 11y ago
I am definitely not using the terms as they are used professionally (and certainly not a criticism of Bayesian methods in any way). It was an attempt to explain in more lay terms how when p-hacking is performed on some dataset, the "findings" can rarely be applied to larger (or different) datasets due to the arbitrary criteria used to reject the parts of the data that aren't conducive to the selected outcome, since the rejection criteria were sort of custom-fit to that specific dataset. I agree with everything you said.