4 ms·

Yes, Clinical Trials Work

42 points by jtleek 13y ago

zekenie 13y ago
A great book RTCs is Ben Goldacre's Bad Science (http://www.badscience.net/ http://www.badscience.net/) He points out that some of the first clinical trials recorded were in the bible. I couldn't find his quote, but I found an NEJM piece that references the same thing, although I think its in a nutrition debate. "The first published report of a clinical trial has biblical origins. In the Book of Daniel,5 reference is made to the unwillingness of the Israelite Daniel to accept the diet offered by the Babylonian king Nebuchadnezzar. The king's official had put a steward in charge of Daniel and his three friends (Shadrach, Meshach, and Abednego):" "Daniel said to the steward . . . “Test your servants for ten days; let us be given vegetables to eat and water to drink. Then let our appearance and the appearance of the youths who eat the king's rich food be observed by you, and according to what you see deal with your servants.” So he hearkened to them in this matter, and tested them for ten days. At the end of ten days it was seen that they were better in appearance and fatter in flesh than all the youths who ate the king's rich food. So the steward took away their rich food and the wine they were to drink and gave them vegetables." (http://www.nejm.org/doi/full/10.1056/NEJM200301023480120 http://www.nejm.org/doi/full/10.1056/NEJM200301023480120)
- Renaud 13y ago
  Maybe I'm mistaken, but the historical bias fat=healthy seems to be visible there. It's doubtful that by eating only vegetables they could be 'fatter' than those who drank and ate rich foods. It's obvious the writers were trying to make a point, but I'm not persuaded by the accuracy of the findings.
  - zekenie 13y ago
    Yeah, I don't really care one way or the other about the context. I more care about the fact that there was a clinical trial thousands of years ago before concepts of statistics... I like that it can all be boiled down to: split people into two groups at random. Do something to one group, do nothing to the other, measure the results.
jpdoctor 13y ago
I think the blog post missed the most important point of the NYT piece: Too often, he says, trials are against “a straw-man comparator” like a placebo rather than a competing drug. So the studies don’t really help us understand which treatments for a disease work best. The standard is usually a placebo. This is done when you are not even sure the drug will have an effect. The standard should be a competing drugs, otherwise you have no measure of whether the drug under trial is better than current drugs. The financial incentives are not present for any of the parties (including the regulators) to use the higher bar.
- mikeash 13y ago
  Is it not possible to take the competing drug's trial against a placebo, the new drug's trial against a placebo, then compute the new drug's effectiveness compared to the competing drug? Is it just that it's too inaccurate to go through the indirection like that?
- potatolicious 13y ago
  I wonder if this will have unintended economic consequences. Clinical trials are astonishingly expensive - if we required all players to clinically test against competitors (or just the current leading competitor), it seems this would create a perverse incentive to be first to market. Not only would you rake in the early profits, but you'll also significant raise the bar for anyone trying to unseat you. I'm not convinced this will actually be the case, but it's a possible outcome worth thinking about. One thing I'm wondering: if all clinical studies are quantified via efficacy vs. a fixed placebo, shouldn't that make the results comparable? If Drug A is 200% more effective than a placebo, and Drug B is 300% more effective, does that not suggest that Drug B > Drug A?
  - pdpi 13y ago
    > One thing I'm wondering: if all clinical studies are quantified via efficacy vs. a fixed placebo, shouldn't that make the results comparable? If Drug A is 200% more effective than a placebo, and Drug B is 300% more effective, does that not suggest that Drug B > Drug A? I'm not sure that works. How do you account for different test conditions?
ronaldx 13y ago
This defensive blog post perhaps misses a crucial point that is made by the NYT through a haze of lay-person journalism. By placing the focus on an 'average' patient, there is a danger of missing drugs which work effectively but only in sub-populations while having costs in other sub-populations (as drugs generally do in practice). "Some who take Avastin significantly beat the average" is probably indeed worth noting because the naive RCT may be disguising significant effects of clinical importance.
- shoyer 13y ago
  The trouble is that it doesn't help anyone on average if you can't identify whether they're in the relevant subpopulation before you give people them the drug. That's why the FDA doesn't approve such treatments. Indeed, drug companies slice and dice their data from early clinical trials every which way to identify the best subpopulation to run their next trial on. Their interests are entirely aligned with figuring out some population for whom their drugs work.
- Tarrosion 13y ago
  I noticed this as well. In an extreme case, we may see very small variance among placebo or control groups and several clusters in the group using an experimental drug. If one or more of those clusters significantly beats the control group and we can accurately classify patients into clusters, that's important information which simple means and medians may obscure. This reminds me of the importance of looking at data and Anscombe's quartet [0]. [0] http://en.wikipedia.org/wiki/Anscombe%27s_quartet http://en.wikipedia.org/wiki/Anscombe%27s_quartet
- cowsandmilk 13y ago
  The article is completely misleading in this regard. Every modern clinical trial looks for these sub-populations. The article even states: Sixty percent of the new drugs in the works at Genentech/Roche are being developed with a companion diagnostic test to identify the patients who are most likely to benefit. That said, it is extremely frequent that clinical trials that follow up on these sub-population observances find them to be statistical anomalies. When you have 100 ways to subdivide your populations, you expect to have some anomalies at a typical 5% confidence level. The article points to I-SPY 2 as the future of clinical trials, making it sound like bayesian analysis is something novel to solely this clinical trial. As is alluded to in what you call the "defensive blog post", this is now standard practice in clinical trial design at most pharma companies. These companies don't like to throw 9 figures down the drain and they recognize that adaptive trial designs can get them statistical significance at lower cost while determining optimal treatment.
PaperclipTaken 13y ago
One of the problems with clinical trials is when industries have billions of dollars to spend creating drugs. If I flip a fair coin 100 times, and it comes up heads 65 times, we can reasonably conclude that it's not a fair coin. If I flip 100 coins 100 times and one of them comes up heads 65 times, the probability that it is the unfair coin is much lower. So it's important to take into account all of the failed drugs during clinical trials when you see one that has succeeded. Especially when a company has gone through dozens or hundreds of failed drugs before coming up with a successful one, you need to make sure that the trials have been numerous to overcome Hoeffding's bound. http://en.wikipedia.org/wiki/Hoeffding%27s_inequality http://en.wikipedia.org/wiki/Hoeffding%27s_inequality
szx 13y ago
What I found most worrisome about the NYT piece was the idea that "the payoff for a successful Phase 3 trial can be so enormous that drug makers will often roll the dice". Now, IANAStatistician but from what I understand there's an inherent (small) chance of any given trial showing a drug is effective. Wouldn't that mean that since there's such an incentive to test unpromising drugs, it's more likely some would get through by chance and get to the market?

42 points by jtleek 13y ago

A great book RTCs is Ben Goldacre's Bad Science (http://www.badscience.net/ http://www.badscience.net/) He points out that some of the first clinical trials recorded were in the bible. I couldn't find his quote, but I found an NEJM piece that references the same thing, although I think its in a nutrition debate. "The first published report of a clinical trial has biblical origins. In the Book of Daniel,5 reference is made to the unwillingness of the Israelite Daniel to accept the diet offered by the Babylonian king Nebuchadnezzar. The king's official had put a steward in charge of Daniel and his three friends (Shadrach, Meshach, and Abednego):" "Daniel said to the steward . . . “Test your servants for ten days; let us be given vegetables to eat and water to drink. Then let our appearance and the appearance of the youths who eat the king's rich food be observed by you, and according to what you see deal with your servants.” So he hearkened to them in this matter, and tested them for ten days. At the end of ten days it was seen that they were better in appearance and fatter in flesh than all the youths who ate the king's rich food. So the steward took away their rich food and the wine they were to drink and gave them vegetables." (http://www.nejm.org/doi/full/10.1056/NEJM200301023480120 http://www.nejm.org/doi/full/10.1056/NEJM200301023480120)

Renaud 13y ago
Maybe I'm mistaken, but the historical bias fat=healthy seems to be visible there. It's doubtful that by eating only vegetables they could be 'fatter' than those who drank and ate rich foods. It's obvious the writers were trying to make a point, but I'm not persuaded by the accuracy of the findings.
- zekenie 13y ago
  Yeah, I don't really care one way or the other about the context. I more care about the fact that there was a clinical trial thousands of years ago before concepts of statistics... I like that it can all be boiled down to: split people into two groups at random. Do something to one group, do nothing to the other, measure the results.

jpdoctor 13y ago

I think the blog post missed the most important point of the NYT piece: Too often, he says, trials are against “a straw-man comparator” like a placebo rather than a competing drug. So the studies don’t really help us understand which treatments for a disease work best. The standard is usually a placebo. This is done when you are not even sure the drug will have an effect. The standard should be a competing drugs, otherwise you have no measure of whether the drug under trial is better than current drugs. The financial incentives are not present for any of the parties (including the regulators) to use the higher bar.

mikeash 13y ago
Is it not possible to take the competing drug's trial against a placebo, the new drug's trial against a placebo, then compute the new drug's effectiveness compared to the competing drug? Is it just that it's too inaccurate to go through the indirection like that?
potatolicious 13y ago
I wonder if this will have unintended economic consequences. Clinical trials are astonishingly expensive - if we required all players to clinically test against competitors (or just the current leading competitor), it seems this would create a perverse incentive to be first to market. Not only would you rake in the early profits, but you'll also significant raise the bar for anyone trying to unseat you. I'm not convinced this will actually be the case, but it's a possible outcome worth thinking about. One thing I'm wondering: if all clinical studies are quantified via efficacy vs. a fixed placebo, shouldn't that make the results comparable? If Drug A is 200% more effective than a placebo, and Drug B is 300% more effective, does that not suggest that Drug B > Drug A?
- pdpi 13y ago
  > One thing I'm wondering: if all clinical studies are quantified via efficacy vs. a fixed placebo, shouldn't that make the results comparable? If Drug A is 200% more effective than a placebo, and Drug B is 300% more effective, does that not suggest that Drug B > Drug A? I'm not sure that works. How do you account for different test conditions?

ronaldx 13y ago

This defensive blog post perhaps misses a crucial point that is made by the NYT through a haze of lay-person journalism. By placing the focus on an 'average' patient, there is a danger of missing drugs which work effectively but only in sub-populations while having costs in other sub-populations (as drugs generally do in practice). "Some who take Avastin significantly beat the average" is probably indeed worth noting because the naive RCT may be disguising significant effects of clinical importance.

shoyer 13y ago
The trouble is that it doesn't help anyone on average if you can't identify whether they're in the relevant subpopulation before you give people them the drug. That's why the FDA doesn't approve such treatments. Indeed, drug companies slice and dice their data from early clinical trials every which way to identify the best subpopulation to run their next trial on. Their interests are entirely aligned with figuring out some population for whom their drugs work.
Tarrosion 13y ago
I noticed this as well. In an extreme case, we may see very small variance among placebo or control groups and several clusters in the group using an experimental drug. If one or more of those clusters significantly beats the control group and we can accurately classify patients into clusters, that's important information which simple means and medians may obscure. This reminds me of the importance of looking at data and Anscombe's quartet [0]. [0] http://en.wikipedia.org/wiki/Anscombe%27s_quartet http://en.wikipedia.org/wiki/Anscombe%27s_quartet
cowsandmilk 13y ago
The article is completely misleading in this regard. Every modern clinical trial looks for these sub-populations. The article even states: Sixty percent of the new drugs in the works at Genentech/Roche are being developed with a companion diagnostic test to identify the patients who are most likely to benefit. That said, it is extremely frequent that clinical trials that follow up on these sub-population observances find them to be statistical anomalies. When you have 100 ways to subdivide your populations, you expect to have some anomalies at a typical 5% confidence level. The article points to I-SPY 2 as the future of clinical trials, making it sound like bayesian analysis is something novel to solely this clinical trial. As is alluded to in what you call the "defensive blog post", this is now standard practice in clinical trial design at most pharma companies. These companies don't like to throw 9 figures down the drain and they recognize that adaptive trial designs can get them statistical significance at lower cost while determining optimal treatment.

PaperclipTaken 13y ago

One of the problems with clinical trials is when industries have billions of dollars to spend creating drugs. If I flip a fair coin 100 times, and it comes up heads 65 times, we can reasonably conclude that it's not a fair coin. If I flip 100 coins 100 times and one of them comes up heads 65 times, the probability that it is the unfair coin is much lower. So it's important to take into account all of the failed drugs during clinical trials when you see one that has succeeded. Especially when a company has gone through dozens or hundreds of failed drugs before coming up with a successful one, you need to make sure that the trials have been numerous to overcome Hoeffding's bound. http://en.wikipedia.org/wiki/Hoeffding%27s_inequality http://en.wikipedia.org/wiki/Hoeffding%27s_inequality

szx 13y ago

What I found most worrisome about the NYT piece was the idea that "the payoff for a successful Phase 3 trial can be so enormous that drug makers will often roll the dice". Now, IANAStatistician but from what I understand there's an inherent (small) chance of any given trial showing a drug is effective. Wouldn't that mean that since there's such an incentive to test unpromising drugs, it's more likely some would get through by chance and get to the market?