20 ms·

There is no news I enjoy reading more than that a clinical trial had to be ended early because the treatment was so effective that it would be unethical to cont

There is no news I enjoy reading more than that a clinical trial had to be ended early because the treatment was so effective that it would be unethical to continue with alternative treatments (or a placebo).

jcranmer 7y ago
Actually, it's quite terrifying when you remember that stopping a clinical trial early is a great way to create statistical fraud. https://blogs.sciencemag.org/pipeline/archives/2019/08/08/stopping-early https://blogs.sciencemag.org/pipeline/archives/2019/08/08/st... cites an example of a drug whose clinical trial was stopped early due to obviously effective success... only to discover later on that the "obviously effective" was a statistically spurious event, and the drug did not confer any benefits whatsoever.
- singingboyo 7y ago
  While I agree in general, the article indicates this was stopped after 681 of a planned 725 patients, which seems okay. Additionally, my understanding from a quick read is that the SPMS trials were aiming to delay progression in disability, rather than eliminate the effects altogether. That seems trickier to measure, and also more subject to just... not spending enough time looking at it. Ebola, on the other hand, has a binary outcome (dead vs cured) with a timeline of a couple weeks, IIRC.
  - justinclift 7y ago
    > Ebola, on the other hand, has a binary outcome (dead vs cured) with a timeline of a couple weeks, IIRC. Doesn't it leave (cured) people potentially permanently damaged as well? eg whatever organs were damaged, are still damaged
    - hanniabu 7y ago
      I think what the parent meant was that it's terminal, so if you're alive then you win, even if you're not in the condition you were before.
  - dahart 7y ago
    > While I agree in general, the article indicates this was stopped after 681 of a planned 725 patients, which seems okay. Are you talking about the European trial, referring to “358 patients with SP-MS were allocated placebo and 360 were allocated interferon beta-1b; 57 patients (31 placebo, 26 interferon beta-1b) were lost to follow-up.”? If so, your interpretation isn’t quite right. It’s not saying they stopped 57 patients short, it’s saying those 57 people didn’t participate at all, but that’s irrelevant to the early ending. The trial stopped short of it’s intended completion time, not short of a number of patients. The blog post indicates it was stopped 2 years early, out of a planned 2-3 year study period.
- eridius 7y ago
  That trial wasn't a "statistically spurious event", it was a flawed trial > the reason the first trial came to an exaggerated impression seemed to be the number of patients who might not have fully progressed to SPMS Stopping early can lead to statistical fraud, which is why the bar is so high on doing so. But it has to be balanced with the recognition that, if the interim results are correct, continuing the trial will lead to a significant number of avoidable deaths. And for what it's worth, given the believed cause of the flawed trial being cited, it sure sounds like if the trial had run to its conclusion it still would have produced flawed results.
  - tempguy9999 7y ago
    Having worked in clinical trialling, what I saw made me realise that the final stats may be worth much less because of mismanagement of data. I can give some hare-raising examples[0] but for obvious reasons... The one I can give is it was known that docs who prescribed this to multiple patients and saw an apparent improvement the attributed to the new drug, would switch the patients on the old drug to the new and not inform us. Obviously they couldn't or they'd invalidate the trial and they knew it. And it was done entirely with the patient's best interests at heart. Docs care about their lives. That may have been rare and a flaw in the trial protocol, but much worse stuff was done via utter incompetence. And I mean including at the top of these giant drug companies. Run by idiots, really. Wider lesson: just cos you hand over a process to a third party does not mean it's going to be done right. [0] tribute to the thread elsewhere
    - bigfudge 7y ago
      Switching extra patients to a new drug would make the trial more conservative though... so perhaps don’t worry so much. In general it’s worth remembering that RCTs are estimating the effect of an intention to treat ( the randomisation) not the treatment itself.
      - tempguy9999 7y ago
        > ...would make the trial more conservative though... If the doc correctly divines that the drug is improving things, yes, but there is noise in the signal so it may be just noise causing the few apparent improvements the doc sees. If so, doc's action may be smothering a less-then-obvious signal. There's too much at stake for that to be acceptable. > ...are estimating the effect of an intention to treat ( the randomisation) not the treatment itself. I don't understand - the protocol applies the drug (aka the treatment) and results are measured. I don't understand 'intention to treat'. What is 'intention' here, so tech term I am not familiar with?
- Aeolun 7y ago
  It seems that if this is the case for this new drug it would be fairly noticable that the death rate jumps back up to 50%
  - dmix 7y ago
    That works both ways too, the mortality rate went from 50-75% to 6% if they sought treatment immediately. Which is a significant indicator of success.
- IfOnlyYouKnew 7y ago
  It's only a "great way to create statistical fraud" if you assume that the statisticians, and the FDA, are either stupid or corrupt. And contrary to popular belief, they tend to be competent enough to at least cope with the sort of hair-brained schemes HN believes would fool them.[0] Stopping a trial early and it's statistical implications should, by the way, be somewhat familiar with web developers: it's commonly done with A/B tests, and the related problem is the "One-armed bandit" (https://en.wikipedia.org/wiki/Multi-armed_bandit#Empirical_motivation https://en.wikipedia.org/wiki/Multi-armed_bandit#Empirical_m...) [0]: Legend says they are even familiar with correlation != causation, and some have mastered the advanced skill of knowing that trial size matters, which is why they started, a few years back, to test drugs on more than one person.
  - Simon_says 7y ago
    The word is harebrained unless you think hair has a brain.
    - sjcsjc 7y ago
      Thanks - I didn't know this. http://www.word-detective.com/2012/08/harebrained/ http://www.word-detective.com/2012/08/harebrained/
    - the_seraphim 7y ago
      surely it means to have a brain made of hair?
  - WhitneyLand 7y ago
    >>the sort of hair-brained schemes HN believes would fool them Is there much value in an ad-hominem reply? Or in generalizing it to an entire community? The irony seems hard to avoid when such a generalization is made without mentioning any statistical foundation, while pontificating about statistics.
    - eitland 7y ago
      > Is there much value in an ad-hominem reply? > Or in generalizing it to an entire community? I don't like this style either but sometimes I think we deserve it for being a smug echo chamber. Basically, much the same thing could have been said in a different way and I would have upvoted it.
- hyper_reality 7y ago
  Statistics Done Wrong has a great chapter on this: https://www.statisticsdonewrong.com/regression.html https://www.statisticsdonewrong.com/regression.html
- ekianjo 7y ago
  no, when certain conditions lead to certain outcomes like cancer, infections and so on, you cannot really go wrong when results come in early and are so positive they are massively significantly different vs the usual course. effect size.
  - Fomite 7y ago
    Indeed you can go very wrong by not doing so. Medical ethics is a balance of risk and benefit, not "What is best for statistical power?" Equipoise is required for trials, and is often a precondition of being able to do studies in these settings. Heck, a lot of recent work on novel trial designs like stepped-wedge are specifically designed to balance equipoise and statistical needs.
- deleted 7y ago
  [deleted]
- yummybear 7y ago
  Doesn't giving the drug to everyone in the trial at least add some statistical information (i.e. what is the likelyhood on it working for everyone vs. it being a spurious event)?
- conjectures 7y ago
  FYI people design trials with breakpoints, bandits etc. The issue isn't early stopping, the issue is inappropriate analysis of early stopping.
- reilly3000 7y ago
  Right but its Ebola. Under the conditions of certain, painful death its difficult to imagine a scenario the risk of no benefits outweighs the cost of taking the drug.
- learnstats2 7y ago
  That's not statistical fraud. It can be important to ethically stop a trial based on the information available, even if the information available later turns out to be incorrect.
- WhompingWindows 7y ago
  It's not that terrifying, there are many drugs approved that are not even more effective than cheaper, older drugs on the market. It takes a wild-scale deployment with 1000's of patients to find rare side effects and to measure the true population-level effectiveness of a drug. This is contrary to the clinical-trial level "efficacy" of a drug, which is with a more limited in terms of N= and in terms of diversity, population. Really, the whole drug approval process is filled with uncertainty. Partly it's how the sausage is made in biotech, but also pharma companies love this method as it allows them to put forward non-inferior drugs with higher margins. We can reform the process, and should, but there will still be statistical uncertainty for 100's of new candidates, until they reach the post-market study phase.
- GuB-42 7y ago
  Just do stats all the way down. There are 6 possibilities: 1- stop early, effective 2- stop early, ineffective 3- passed all trials, effective 4- passed all trials, ineffective 5- failed trials, effective 6- failed trials, ineffective Assign a utility value (lives saved, risks, cost,...) and a probability to each situation and do the maths. P(1) = P(3) + P(5) P(2) = P(4) + P(6) So, if (P(3) + P(5))U(1) + (P(4) + P(6))U(2) > P(3)U(3) + P(4)U(4) + P(5)U(5) + P(6)U(6) it is better to stop early. The risk of fraud can be taken into account.
- bigfudge 7y ago
  There’s quite a large statistical literature on stopping rules for trials. Any trial that is stopped would have had to plan this in advance and an evaluation of his policy would be an important thing for the data monitoring and ethics committee to review, but shouldn’t be a cause for concern. OP is right to celebrate
- Fomite 7y ago
  These are, for the record, most often specified pre-trial (as are the stopping thresholds for harm)).

jcranmer 7y ago

Actually, it's quite terrifying when you remember that stopping a clinical trial early is a great way to create statistical fraud. https://blogs.sciencemag.org/pipeline/archives/2019/08/08/stopping-early https://blogs.sciencemag.org/pipeline/archives/2019/08/08/st... cites an example of a drug whose clinical trial was stopped early due to obviously effective success... only to discover later on that the "obviously effective" was a statistically spurious event, and the drug did not confer any benefits whatsoever.

singingboyo 7y ago
While I agree in general, the article indicates this was stopped after 681 of a planned 725 patients, which seems okay. Additionally, my understanding from a quick read is that the SPMS trials were aiming to delay progression in disability, rather than eliminate the effects altogether. That seems trickier to measure, and also more subject to just... not spending enough time looking at it. Ebola, on the other hand, has a binary outcome (dead vs cured) with a timeline of a couple weeks, IIRC.
- justinclift 7y ago
  > Ebola, on the other hand, has a binary outcome (dead vs cured) with a timeline of a couple weeks, IIRC. Doesn't it leave (cured) people potentially permanently damaged as well? eg whatever organs were damaged, are still damaged
  - hanniabu 7y ago
    I think what the parent meant was that it's terminal, so if you're alive then you win, even if you're not in the condition you were before.
- dahart 7y ago
  > While I agree in general, the article indicates this was stopped after 681 of a planned 725 patients, which seems okay. Are you talking about the European trial, referring to “358 patients with SP-MS were allocated placebo and 360 were allocated interferon beta-1b; 57 patients (31 placebo, 26 interferon beta-1b) were lost to follow-up.”? If so, your interpretation isn’t quite right. It’s not saying they stopped 57 patients short, it’s saying those 57 people didn’t participate at all, but that’s irrelevant to the early ending. The trial stopped short of it’s intended completion time, not short of a number of patients. The blog post indicates it was stopped 2 years early, out of a planned 2-3 year study period.
eridius 7y ago
That trial wasn't a "statistically spurious event", it was a flawed trial > the reason the first trial came to an exaggerated impression seemed to be the number of patients who might not have fully progressed to SPMS Stopping early can lead to statistical fraud, which is why the bar is so high on doing so. But it has to be balanced with the recognition that, if the interim results are correct, continuing the trial will lead to a significant number of avoidable deaths. And for what it's worth, given the believed cause of the flawed trial being cited, it sure sounds like if the trial had run to its conclusion it still would have produced flawed results.
- tempguy9999 7y ago
  Having worked in clinical trialling, what I saw made me realise that the final stats may be worth much less because of mismanagement of data. I can give some hare-raising examples[0] but for obvious reasons... The one I can give is it was known that docs who prescribed this to multiple patients and saw an apparent improvement the attributed to the new drug, would switch the patients on the old drug to the new and not inform us. Obviously they couldn't or they'd invalidate the trial and they knew it. And it was done entirely with the patient's best interests at heart. Docs care about their lives. That may have been rare and a flaw in the trial protocol, but much worse stuff was done via utter incompetence. And I mean including at the top of these giant drug companies. Run by idiots, really. Wider lesson: just cos you hand over a process to a third party does not mean it's going to be done right. [0] tribute to the thread elsewhere
  - bigfudge 7y ago
    Switching extra patients to a new drug would make the trial more conservative though... so perhaps don’t worry so much. In general it’s worth remembering that RCTs are estimating the effect of an intention to treat ( the randomisation) not the treatment itself.
    - tempguy9999 7y ago
      > ...would make the trial more conservative though... If the doc correctly divines that the drug is improving things, yes, but there is noise in the signal so it may be just noise causing the few apparent improvements the doc sees. If so, doc's action may be smothering a less-then-obvious signal. There's too much at stake for that to be acceptable. > ...are estimating the effect of an intention to treat ( the randomisation) not the treatment itself. I don't understand - the protocol applies the drug (aka the treatment) and results are measured. I don't understand 'intention to treat'. What is 'intention' here, so tech term I am not familiar with?
Aeolun 7y ago
It seems that if this is the case for this new drug it would be fairly noticable that the death rate jumps back up to 50%
- dmix 7y ago
  That works both ways too, the mortality rate went from 50-75% to 6% if they sought treatment immediately. Which is a significant indicator of success.
IfOnlyYouKnew 7y ago
It's only a "great way to create statistical fraud" if you assume that the statisticians, and the FDA, are either stupid or corrupt. And contrary to popular belief, they tend to be competent enough to at least cope with the sort of hair-brained schemes HN believes would fool them.[0] Stopping a trial early and it's statistical implications should, by the way, be somewhat familiar with web developers: it's commonly done with A/B tests, and the related problem is the "One-armed bandit" (https://en.wikipedia.org/wiki/Multi-armed_bandit#Empirical_motivation https://en.wikipedia.org/wiki/Multi-armed_bandit#Empirical_m...) [0]: Legend says they are even familiar with correlation != causation, and some have mastered the advanced skill of knowing that trial size matters, which is why they started, a few years back, to test drugs on more than one person.
- Simon_says 7y ago
  The word is harebrained unless you think hair has a brain.
  - sjcsjc 7y ago
    Thanks - I didn't know this. http://www.word-detective.com/2012/08/harebrained/ http://www.word-detective.com/2012/08/harebrained/
  - the_seraphim 7y ago
    surely it means to have a brain made of hair?
- WhitneyLand 7y ago
  >>the sort of hair-brained schemes HN believes would fool them Is there much value in an ad-hominem reply? Or in generalizing it to an entire community? The irony seems hard to avoid when such a generalization is made without mentioning any statistical foundation, while pontificating about statistics.
  - eitland 7y ago
    > Is there much value in an ad-hominem reply? > Or in generalizing it to an entire community? I don't like this style either but sometimes I think we deserve it for being a smug echo chamber. Basically, much the same thing could have been said in a different way and I would have upvoted it.
hyper_reality 7y ago
Statistics Done Wrong has a great chapter on this: https://www.statisticsdonewrong.com/regression.html https://www.statisticsdonewrong.com/regression.html
ekianjo 7y ago
no, when certain conditions lead to certain outcomes like cancer, infections and so on, you cannot really go wrong when results come in early and are so positive they are massively significantly different vs the usual course. effect size.
- Fomite 7y ago
  Indeed you can go very wrong by not doing so. Medical ethics is a balance of risk and benefit, not "What is best for statistical power?" Equipoise is required for trials, and is often a precondition of being able to do studies in these settings. Heck, a lot of recent work on novel trial designs like stepped-wedge are specifically designed to balance equipoise and statistical needs.
deleted 7y ago
[deleted]
yummybear 7y ago
Doesn't giving the drug to everyone in the trial at least add some statistical information (i.e. what is the likelyhood on it working for everyone vs. it being a spurious event)?
conjectures 7y ago
FYI people design trials with breakpoints, bandits etc. The issue isn't early stopping, the issue is inappropriate analysis of early stopping.
reilly3000 7y ago
Right but its Ebola. Under the conditions of certain, painful death its difficult to imagine a scenario the risk of no benefits outweighs the cost of taking the drug.
learnstats2 7y ago
That's not statistical fraud. It can be important to ethically stop a trial based on the information available, even if the information available later turns out to be incorrect.
WhompingWindows 7y ago
It's not that terrifying, there are many drugs approved that are not even more effective than cheaper, older drugs on the market. It takes a wild-scale deployment with 1000's of patients to find rare side effects and to measure the true population-level effectiveness of a drug. This is contrary to the clinical-trial level "efficacy" of a drug, which is with a more limited in terms of N= and in terms of diversity, population. Really, the whole drug approval process is filled with uncertainty. Partly it's how the sausage is made in biotech, but also pharma companies love this method as it allows them to put forward non-inferior drugs with higher margins. We can reform the process, and should, but there will still be statistical uncertainty for 100's of new candidates, until they reach the post-market study phase.
GuB-42 7y ago
Just do stats all the way down. There are 6 possibilities: 1- stop early, effective 2- stop early, ineffective 3- passed all trials, effective 4- passed all trials, ineffective 5- failed trials, effective 6- failed trials, ineffective Assign a utility value (lives saved, risks, cost,...) and a probability to each situation and do the maths. P(1) = P(3) + P(5) P(2) = P(4) + P(6) So, if (P(3) + P(5))U(1) + (P(4) + P(6))U(2) > P(3)U(3) + P(4)U(4) + P(5)U(5) + P(6)U(6) it is better to stop early. The risk of fraud can be taken into account.
bigfudge 7y ago
There’s quite a large statistical literature on stopping rules for trials. Any trial that is stopped would have had to plan this in advance and an evaluation of his policy would be an important thing for the data monitoring and ethics committee to review, but shouldn’t be a cause for concern. OP is right to celebrate
Fomite 7y ago
These are, for the record, most often specified pre-trial (as are the stopping thresholds for harm)).