6 ms·

> The original Remdesivir trial by the drug company was a high quality RCT (randomized control trial) - except that they stopped it early once they saw a statis

by cyphar 6y ago

> The original Remdesivir trial by the drug company was a high quality RCT (randomized control trial) - except that they stopped it early once they saw a statistically significant positive effect (even though the effect was somewhat underwhelming: just a couple fewer days of illness). To me this moving of the goal posts invalidates the quality of the RCT. Based on what I've seen discussed, ending trials early is a fairly common (though admittedly contentious) practice in medical RCTs when you're dealing with very serious illnesses, especially when there are no alternative effective treatments[1]. The issue is that there is a tradeoff between an abundance of caution to procure conclusive results and giving patients access to medicine which "clearly helps". In an ideal world you would run every RCT for several years to figure out any possible long term effects and accurately determine efficacy, but there are many circumstances where you could argue doing so (if it's clear the medicine works during the early stages of the trial) is more unethical than giving people medicine which shows some promise to work. But it should be noted that such trials are still solid science (the trial was an RCT), it just means you cannot take them as evidence of long-term efficacy. [1]: https://www.ahajournals.org/doi/full/10.1161/circheartfailure.111.965707 https://www.ahajournals.org/doi/full/10.1161/circheartfailur...

pen2l 6y ago
What is the _reason_ for ending it early? Is it the cost? I imagine it must be high enough because then they're gonna have to fight with this very criticism later on: RCT ended prematurely, therefore it is not as reliable. A question to you and grandparent post: what are the things that distinguish a high quality RCT from a poor one? The number of participants, the design study, etc.? Any links/resources that discuss this would be appreciated. edit: ok, I found a paper on this: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3429837/#:~:text=Randomized%20controlled%20trials%20(RCTs)%20have,of%20medical%20treatments%20(1) https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3429837/#:~:tex.... I am super curious now about any discussion that reveals why the WHO study was less-than-stellar, and why the original RCT was considered high-quality.
- shopwithpoints 6y ago
  Its typically an ethical decision. In the event of a deadly disease ( cancer, covid, hiv etc ) its considered harmful to continue giving a less effective treatment once you have shown some superiority/inferiority.
- arjunnarayan 6y ago
  The reasoning is that as you gain information, you also have a duty to the people in the control group to use the best available information to take care of their health. Once you gain "enough" information ("enough" being statistically defined) that the drug helps, each additional person you let languish in the control group (who is denied access to the drug) is a cost that must be weighed against the benefit of getting additional information. When the data is clear enough, the cost can exceed the benefit, and you stop early. Typically you'd register a "stopping rule" before you start your trial: a good drug often will trigger the stopping rule, as it helped so much that we learned about its efficacy on a smaller N than originally planned. There are many different stopping criteria, depending on the trial (in safety trials you'd typically stop because you've found evidence that the drug is unsafe and continuing would be unfair to the folks in the treatment group, whereas in efficacy trials after safety has been established, you'd stop because you've found evidence that the drug is effective and continuing would be unfair to the folks in the control group).
  - Siira 6y ago
    The reasoning is money, the unreasoning is ethical sophistry.
- paulmd 6y ago
  if you don't think the quality of the trial is so bad as to outright impair approval (and right now the political pressure for a tool, any tool, to fight COVID is so intense that there's zero chance it wouldn't be) then why continue? It costs money to do trials and it represents lost revenue where you could be the sole seller of the only known drug to treat a deadly pandemic. in fact there is a downside risk to continuing as well, further study might show that the early results were an outlier and the drug is not as effective as previously thought (or has some serious side effects in some situation), and then you lose all the money you've invested. But if you push it through, there is no risk of being sued since the US Government has waived any liability on the part of drug companies for these therapies. in a game-theoretic sense there is absolutely no benefit to continuing the trials and potential downsides to doing so. Remember, for-profit companies don't care about maximizing public health, they care about maximizing revenue. Especially when you've explicitly removed all financial liability for being risky with public health.
gregwebs 6y ago
The study you links to uses death or heart failure as an endpoint. If there had been lower mortality or ICU admission halting the trial early would make sense. With the only improvement being quicker recovery the value proposition of the treatment isn't that great. From a practical standpoint remdesivir is expensive and had supply issues so halting the trial early doesn't mean every American is going to start getting treated with it. I don't think the science is solid when a trial can pick from 2nd-tier endpoints and stop the trial once an improvement is found. It is quite possible that if the trial had gone on longer the one positive effect would be reduced or even go away entirely or that we would see a negative effect in a more important endpoint.
- wang_li 6y ago
  In urgent circumstances where the demand for hospital services are high, a twenty percent improvement in recovery times is the same as increased capacity.
  - mlyle 6y ago
    Nit: It's not quite the same. If need for hospitalization is growing quickly, cycling out people 20% quicker might not affect overload dramatically. Steady-state, sure, you can treat 25% more.
- Florin_Andrei 6y ago
  The fact remains that it is standard procedure to halt trials early when promising results appear, and people are dying waiting for a drug. This is not something that's controversial, and this is exactly what happened with the previous Remdesivir study. Of course it's disappointing to see it may not be as effective as we previously thought. And of course analysis may yet reveal issues with either one or the other or both studies. That's how science works - facts, not heartfelt desires.
  - gregwebs 6y ago
    Nobody is dying. The only endpoint that improved was recovery days.
contravariant 6y ago
The trade off between giving access to medicine that clearly helps and testing its effect carefully isn't done very well if you stop early with only a minor effect size. As at that point you're more than likely to just be giving people access to medicine that is far from clearly helping and possibly not even vaguely effective.
danepowell 6y ago
I disagree that ending trials early is solid science, or at least not _as_ solid as running them to completion. It makes it hard to know whether results are due to a treatment effect or just random fluctuations over time. Essentially, it allows for cherry-picking of significant results in a way that bypasses family-wise error correction.
- jrumbut 6y ago
  You can quantify, perhaps by simulation, the amount that it's less solid. What's balancing against that is continuing the trial means giving patients a placebo when you have evidence of a better treatment. It's a really complex issue both scientifically and ethically.
  - xiphias2 6y ago
    The solution to this is changing the stopping criterion in the trial protocol description, so that the real confidence interval can be calculated for the trial _with_ stopping (which is very different from running the trial until the end).
    - jrumbut 6y ago
      Good point, I should have added prospectively to that. But even then, more participants and more time for follow-ups is generally better all other things being equal (which they never are).
      - xiphias2 6y ago
        Things got extremely political around these experiments. When 2 days ago Orban announced from nothing that they will provide Remdesivir to everyone, I knew that again they did something stupid...I want the experimental drug that Trump got, not some leftover drug with lots of side effects. Now I know that it was not Orban who was screwed, but the whole EU.
- cyphar 6y ago
bernie_simon 6y ago
There's a well known pattern that studies conducted by a drug company are more likely to report positive results than studies conducted by a neutral party.

by cyphar 6y ago

pen2l 6y ago

What is the _reason_ for ending it early? Is it the cost? I imagine it must be high enough because then they're gonna have to fight with this very criticism later on: RCT ended prematurely, therefore it is not as reliable. A question to you and grandparent post: what are the things that distinguish a high quality RCT from a poor one? The number of participants, the design study, etc.? Any links/resources that discuss this would be appreciated. edit: ok, I found a paper on this: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3429837/#:~:text=Randomized%20controlled%20trials%20(RCTs)%20have,of%20medical%20treatments%20(1) https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3429837/#:~:tex.... I am super curious now about any discussion that reveals why the WHO study was less-than-stellar, and why the original RCT was considered high-quality.

shopwithpoints 6y ago
Its typically an ethical decision. In the event of a deadly disease ( cancer, covid, hiv etc ) its considered harmful to continue giving a less effective treatment once you have shown some superiority/inferiority.
arjunnarayan 6y ago
The reasoning is that as you gain information, you also have a duty to the people in the control group to use the best available information to take care of their health. Once you gain "enough" information ("enough" being statistically defined) that the drug helps, each additional person you let languish in the control group (who is denied access to the drug) is a cost that must be weighed against the benefit of getting additional information. When the data is clear enough, the cost can exceed the benefit, and you stop early. Typically you'd register a "stopping rule" before you start your trial: a good drug often will trigger the stopping rule, as it helped so much that we learned about its efficacy on a smaller N than originally planned. There are many different stopping criteria, depending on the trial (in safety trials you'd typically stop because you've found evidence that the drug is unsafe and continuing would be unfair to the folks in the treatment group, whereas in efficacy trials after safety has been established, you'd stop because you've found evidence that the drug is effective and continuing would be unfair to the folks in the control group).
- Siira 6y ago
  The reasoning is money, the unreasoning is ethical sophistry.
paulmd 6y ago
if you don't think the quality of the trial is so bad as to outright impair approval (and right now the political pressure for a tool, any tool, to fight COVID is so intense that there's zero chance it wouldn't be) then why continue? It costs money to do trials and it represents lost revenue where you could be the sole seller of the only known drug to treat a deadly pandemic. in fact there is a downside risk to continuing as well, further study might show that the early results were an outlier and the drug is not as effective as previously thought (or has some serious side effects in some situation), and then you lose all the money you've invested. But if you push it through, there is no risk of being sued since the US Government has waived any liability on the part of drug companies for these therapies. in a game-theoretic sense there is absolutely no benefit to continuing the trials and potential downsides to doing so. Remember, for-profit companies don't care about maximizing public health, they care about maximizing revenue. Especially when you've explicitly removed all financial liability for being risky with public health.

gregwebs 6y ago

The study you links to uses death or heart failure as an endpoint. If there had been lower mortality or ICU admission halting the trial early would make sense. With the only improvement being quicker recovery the value proposition of the treatment isn't that great. From a practical standpoint remdesivir is expensive and had supply issues so halting the trial early doesn't mean every American is going to start getting treated with it. I don't think the science is solid when a trial can pick from 2nd-tier endpoints and stop the trial once an improvement is found. It is quite possible that if the trial had gone on longer the one positive effect would be reduced or even go away entirely or that we would see a negative effect in a more important endpoint.

wang_li 6y ago
In urgent circumstances where the demand for hospital services are high, a twenty percent improvement in recovery times is the same as increased capacity.
- mlyle 6y ago
  Nit: It's not quite the same. If need for hospitalization is growing quickly, cycling out people 20% quicker might not affect overload dramatically. Steady-state, sure, you can treat 25% more.
Florin_Andrei 6y ago
The fact remains that it is standard procedure to halt trials early when promising results appear, and people are dying waiting for a drug. This is not something that's controversial, and this is exactly what happened with the previous Remdesivir study. Of course it's disappointing to see it may not be as effective as we previously thought. And of course analysis may yet reveal issues with either one or the other or both studies. That's how science works - facts, not heartfelt desires.
- gregwebs 6y ago
  Nobody is dying. The only endpoint that improved was recovery days.

contravariant 6y ago

The trade off between giving access to medicine that clearly helps and testing its effect carefully isn't done very well if you stop early with only a minor effect size. As at that point you're more than likely to just be giving people access to medicine that is far from clearly helping and possibly not even vaguely effective.

danepowell 6y ago

I disagree that ending trials early is solid science, or at least not _as_ solid as running them to completion. It makes it hard to know whether results are due to a treatment effect or just random fluctuations over time. Essentially, it allows for cherry-picking of significant results in a way that bypasses family-wise error correction.

jrumbut 6y ago
You can quantify, perhaps by simulation, the amount that it's less solid. What's balancing against that is continuing the trial means giving patients a placebo when you have evidence of a better treatment. It's a really complex issue both scientifically and ethically.
- xiphias2 6y ago
  The solution to this is changing the stopping criterion in the trial protocol description, so that the real confidence interval can be calculated for the trial _with_ stopping (which is very different from running the trial until the end).
  - jrumbut 6y ago
    Good point, I should have added prospectively to that. But even then, more participants and more time for follow-ups is generally better all other things being equal (which they never are).
    - xiphias2 6y ago
      Things got extremely political around these experiments. When 2 days ago Orban announced from nothing that they will provide Remdesivir to everyone, I knew that again they did something stupid...I want the experimental drug that Trump got, not some leftover drug with lots of side effects. Now I know that it was not Orban who was screwed, but the whole EU.
cyphar 6y ago

bernie_simon 6y ago

There's a well known pattern that studies conducted by a drug company are more likely to report positive results than studies conducted by a neutral party.