4 ms·

This defensive blog post perhaps misses a crucial point that is made by the NYT through a haze of lay-person journalism. By placing the focus on an 'average' p

by ronaldx 13y ago

This defensive blog post perhaps misses a crucial point that is made by the NYT through a haze of lay-person journalism. By placing the focus on an 'average' patient, there is a danger of missing drugs which work effectively but only in sub-populations while having costs in other sub-populations (as drugs generally do in practice). "Some who take Avastin significantly beat the average" is probably indeed worth noting because the naive RCT may be disguising significant effects of clinical importance.

shoyer 13y ago
The trouble is that it doesn't help anyone on average if you can't identify whether they're in the relevant subpopulation before you give people them the drug. That's why the FDA doesn't approve such treatments. Indeed, drug companies slice and dice their data from early clinical trials every which way to identify the best subpopulation to run their next trial on. Their interests are entirely aligned with figuring out some population for whom their drugs work.
Tarrosion 13y ago
I noticed this as well. In an extreme case, we may see very small variance among placebo or control groups and several clusters in the group using an experimental drug. If one or more of those clusters significantly beats the control group and we can accurately classify patients into clusters, that's important information which simple means and medians may obscure. This reminds me of the importance of looking at data and Anscombe's quartet [0]. [0] http://en.wikipedia.org/wiki/Anscombe%27s_quartet http://en.wikipedia.org/wiki/Anscombe%27s_quartet
cowsandmilk 13y ago
The article is completely misleading in this regard. Every modern clinical trial looks for these sub-populations. The article even states: Sixty percent of the new drugs in the works at Genentech/Roche are being developed with a companion diagnostic test to identify the patients who are most likely to benefit. That said, it is extremely frequent that clinical trials that follow up on these sub-population observances find them to be statistical anomalies. When you have 100 ways to subdivide your populations, you expect to have some anomalies at a typical 5% confidence level. The article points to I-SPY 2 as the future of clinical trials, making it sound like bayesian analysis is something novel to solely this clinical trial. As is alluded to in what you call the "defensive blog post", this is now standard practice in clinical trial design at most pharma companies. These companies don't like to throw 9 figures down the drain and they recognize that adaptive trial designs can get them statistical significance at lower cost while determining optimal treatment.
- ronaldx 13y ago
  I call it "defensive" because the answer is not simply "yes" and I think it's a poor position to view clinical trials uncritically. Clinical trials may indeed work, but it's absolutely not clear that they are optimally designed. Although few people on this thread are experts, there are a plethora of different comments which illustrate the impossibility of perfect clinical trial design - there are always improvements that could be made and subjective decisions that have to be taken. The point that you make about pharma companies being profit-making companies could equally be phrased as a very strong criticism: that any pharma clinical trial design is likely to be heavily biased by their desire to sell their most profitable ("optimal") products.

by ronaldx 13y ago

shoyer 13y ago

The trouble is that it doesn't help anyone on average if you can't identify whether they're in the relevant subpopulation before you give people them the drug. That's why the FDA doesn't approve such treatments. Indeed, drug companies slice and dice their data from early clinical trials every which way to identify the best subpopulation to run their next trial on. Their interests are entirely aligned with figuring out some population for whom their drugs work.

Tarrosion 13y ago

I noticed this as well. In an extreme case, we may see very small variance among placebo or control groups and several clusters in the group using an experimental drug. If one or more of those clusters significantly beats the control group and we can accurately classify patients into clusters, that's important information which simple means and medians may obscure. This reminds me of the importance of looking at data and Anscombe's quartet [0]. [0] http://en.wikipedia.org/wiki/Anscombe%27s_quartet http://en.wikipedia.org/wiki/Anscombe%27s_quartet

cowsandmilk 13y ago

The article is completely misleading in this regard. Every modern clinical trial looks for these sub-populations. The article even states: Sixty percent of the new drugs in the works at Genentech/Roche are being developed with a companion diagnostic test to identify the patients who are most likely to benefit. That said, it is extremely frequent that clinical trials that follow up on these sub-population observances find them to be statistical anomalies. When you have 100 ways to subdivide your populations, you expect to have some anomalies at a typical 5% confidence level. The article points to I-SPY 2 as the future of clinical trials, making it sound like bayesian analysis is something novel to solely this clinical trial. As is alluded to in what you call the "defensive blog post", this is now standard practice in clinical trial design at most pharma companies. These companies don't like to throw 9 figures down the drain and they recognize that adaptive trial designs can get them statistical significance at lower cost while determining optimal treatment.

ronaldx 13y ago
I call it "defensive" because the answer is not simply "yes" and I think it's a poor position to view clinical trials uncritically. Clinical trials may indeed work, but it's absolutely not clear that they are optimally designed. Although few people on this thread are experts, there are a plethora of different comments which illustrate the impossibility of perfect clinical trial design - there are always improvements that could be made and subjective decisions that have to be taken. The point that you make about pharma companies being profit-making companies could equally be phrased as a very strong criticism: that any pharma clinical trial design is likely to be heavily biased by their desire to sell their most profitable ("optimal") products.