3 ms·

Per the paper, for drug application to treat pancreatic cancer, the threshold should be lowered to 27.9%. In other words, if a company wants to make money, it

by ccw9302 11y ago

Per the paper, for drug application to treat pancreatic cancer, the threshold should be lowered to 27.9%. In other words, if a company wants to make money, it can randomly pick 4 non toxic compounds, and apply for FDA approval for treating of pancreatic cancer through click-trial, in average the company would get 4*0.279 ~=~ 1 compound approved and then market the medicine to the world that the medicine is effective. They give weight/cost to type I and type II error, and their whole analysis can be changed if the weights changed. I would treat the paper as a research-toy, self fulfillment. Try to change policy based on the paper would be absurd.

danieltillett 11y ago
You have just pointed out the problem with all effectiveness trials. If you used the current threshold you can still do the same thing, you just need to test more non-toxic drugs. A better approach would be to not have any effectiveness trials at all and just collect good data on all patients using the drug. If a drug really worked it would become obvious the more patients used the drug, while if it was worthless this would also become obvious. A new drug would start out unknown with only the most desperate trying it and overtime we would learn more and more about its effects and side-effects. This bayesian approach was not possible in the past, but with modern technology it is now. We would get better drugs with better known side-effects at far lower costs.
- schoen 11y ago
  There's a great xkcd about the p-value interpretation issue. https://xkcd.com/882/ https://xkcd.com/882/ So, how does the FDA deal with this issue today? Suppose the scientists wanted to market (some flavor) of jelly beans as a drug to cure acne (instead of to cause it), and they test 20 flavors and get evidence at p < 0.05 that white jelly beans cure acne. Or 1000 flavors and evidence at p < 0.001. Does the FDA look for a causal mechanism that would make the drug effective? Or are the companies afraid to market things that they genuinely suspect are ineffective because they'll eventually get sued? Or is the FDA's effectiveness standard so strong that you just can't cost-effectively game it like this today? (The last possibility seems a little bit unlikely.)
  - danieltillett 11y ago
    This is a good question and one that is hard to know. In theory you need to have some mechanism of action, but drugs do get approved without one, or more commonly with a wrong one. Quite a few of the drugs in the psychiatric field are little more than placebos-with-bad-side-effects that got lucky. As far as I know nobody has ever been sued over a drug that didn’t work. From a purely legal perspective you would better selling a sugar pill than a real drug that worked and caused 1 in 100,000 people to drop dead.
BurningFrog 11y ago
1. A false positive rate of 27.9% is not the same thing as approving 27.9% of all applications. 2. The full clinical trials still cost hundreds of millions of dollars.
- ccw9302 11y ago
  false positive rate 27.9% means: if the drug is not effective (at all), there is 27.9% probability that the drug would be approved by FDA.
  - gwern 11y ago
    > if the drug is not effective (at all), there is 27.9% probability that the drug would be approved by FDA. Which is only one of two possibilities, and hence, it cannot be 'the probability the drug is approved by the FDA', for the same reason that 'a p-value of 0.05' does not mean '95% probability the drug works'.

by ccw9302 11y ago

danieltillett 11y ago

You have just pointed out the problem with all effectiveness trials. If you used the current threshold you can still do the same thing, you just need to test more non-toxic drugs. A better approach would be to not have any effectiveness trials at all and just collect good data on all patients using the drug. If a drug really worked it would become obvious the more patients used the drug, while if it was worthless this would also become obvious. A new drug would start out unknown with only the most desperate trying it and overtime we would learn more and more about its effects and side-effects. This bayesian approach was not possible in the past, but with modern technology it is now. We would get better drugs with better known side-effects at far lower costs.

schoen 11y ago
There's a great xkcd about the p-value interpretation issue. https://xkcd.com/882/ https://xkcd.com/882/ So, how does the FDA deal with this issue today? Suppose the scientists wanted to market (some flavor) of jelly beans as a drug to cure acne (instead of to cause it), and they test 20 flavors and get evidence at p < 0.05 that white jelly beans cure acne. Or 1000 flavors and evidence at p < 0.001. Does the FDA look for a causal mechanism that would make the drug effective? Or are the companies afraid to market things that they genuinely suspect are ineffective because they'll eventually get sued? Or is the FDA's effectiveness standard so strong that you just can't cost-effectively game it like this today? (The last possibility seems a little bit unlikely.)
- danieltillett 11y ago
  This is a good question and one that is hard to know. In theory you need to have some mechanism of action, but drugs do get approved without one, or more commonly with a wrong one. Quite a few of the drugs in the psychiatric field are little more than placebos-with-bad-side-effects that got lucky. As far as I know nobody has ever been sued over a drug that didn’t work. From a purely legal perspective you would better selling a sugar pill than a real drug that worked and caused 1 in 100,000 people to drop dead.

BurningFrog 11y ago

1. A false positive rate of 27.9% is not the same thing as approving 27.9% of all applications. 2. The full clinical trials still cost hundreds of millions of dollars.

ccw9302 11y ago
false positive rate 27.9% means: if the drug is not effective (at all), there is 27.9% probability that the drug would be approved by FDA.
- gwern 11y ago
  > if the drug is not effective (at all), there is 27.9% probability that the drug would be approved by FDA. Which is only one of two possibilities, and hence, it cannot be 'the probability the drug is approved by the FDA', for the same reason that 'a p-value of 0.05' does not mean '95% probability the drug works'.