7 ms·

The original Remdesivir trial by the drug company was a high quality RCT (randomized control trial) - except that they stopped it early once they saw a statisti

The original Remdesivir trial by the drug company was a high quality RCT (randomized control trial) - except that they stopped it early once they saw a statistically significant positive effect (even though the effect was somewhat underwhelming: just fewer days of recovery). To me this moving of the goal posts invalidates the quality of the RCT. This study from an independent body without a moving of the goal posts is a more trustworthy result. With that being said, anti-virals often only work well when given very early (sometimes even as a preventative) and right now from looking at this study it doesn't seem like this was done. So this study may only be confirming what we already know about how anti-virals should be used (at a very early stage) but it will hopefully have the good effect of stopping improper usage.

cyphar 6y ago
> The original Remdesivir trial by the drug company was a high quality RCT (randomized control trial) - except that they stopped it early once they saw a statistically significant positive effect (even though the effect was somewhat underwhelming: just a couple fewer days of illness). To me this moving of the goal posts invalidates the quality of the RCT. Based on what I've seen discussed, ending trials early is a fairly common (though admittedly contentious) practice in medical RCTs when you're dealing with very serious illnesses, especially when there are no alternative effective treatments[1]. The issue is that there is a tradeoff between an abundance of caution to procure conclusive results and giving patients access to medicine which "clearly helps". In an ideal world you would run every RCT for several years to figure out any possible long term effects and accurately determine efficacy, but there are many circumstances where you could argue doing so (if it's clear the medicine works during the early stages of the trial) is more unethical than giving people medicine which shows some promise to work. But it should be noted that such trials are still solid science (the trial was an RCT), it just means you cannot take them as evidence of long-term efficacy. [1]: https://www.ahajournals.org/doi/full/10.1161/circheartfailure.111.965707 https://www.ahajournals.org/doi/full/10.1161/circheartfailur...
- pen2l 6y ago
  What is the _reason_ for ending it early? Is it the cost? I imagine it must be high enough because then they're gonna have to fight with this very criticism later on: RCT ended prematurely, therefore it is not as reliable. A question to you and grandparent post: what are the things that distinguish a high quality RCT from a poor one? The number of participants, the design study, etc.? Any links/resources that discuss this would be appreciated. edit: ok, I found a paper on this: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3429837/#:~:text=Randomized%20controlled%20trials%20(RCTs)%20have,of%20medical%20treatments%20(1) https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3429837/#:~:tex.... I am super curious now about any discussion that reveals why the WHO study was less-than-stellar, and why the original RCT was considered high-quality.
  - shopwithpoints 6y ago
    Its typically an ethical decision. In the event of a deadly disease ( cancer, covid, hiv etc ) its considered harmful to continue giving a less effective treatment once you have shown some superiority/inferiority.
  - arjunnarayan 6y ago
    The reasoning is that as you gain information, you also have a duty to the people in the control group to use the best available information to take care of their health. Once you gain "enough" information ("enough" being statistically defined) that the drug helps, each additional person you let languish in the control group (who is denied access to the drug) is a cost that must be weighed against the benefit of getting additional information. When the data is clear enough, the cost can exceed the benefit, and you stop early. Typically you'd register a "stopping rule" before you start your trial: a good drug often will trigger the stopping rule, as it helped so much that we learned about its efficacy on a smaller N than originally planned. There are many different stopping criteria, depending on the trial (in safety trials you'd typically stop because you've found evidence that the drug is unsafe and continuing would be unfair to the folks in the treatment group, whereas in efficacy trials after safety has been established, you'd stop because you've found evidence that the drug is effective and continuing would be unfair to the folks in the control group).
yread 6y ago
My statistics are a bit shaky but I understand they "pay" for taking an early look (interim analysis) by reducing the power of the trial by using something called alpha-spending function
cultus 6y ago
Wow, that's a textbook example of post-hoc hypothesizing, about the biggest statistical no-no in the book.
tptacek 6y ago
To continue an RCT once it reveals strong evidence for benefit in the treatment group, you have to deliberately inflict suffering on the control group. You can't do that ethically.
- gregwebs 6y ago
  Strong evidence for a benefit is often measured by death or something extremely severe. When using other clinical endpoints cherry-picking is to be avoided. [1] > For a trial to stop early for benefit, the observed effect on the composite end point ideally should be consistent in the individual components. [1] https://www.ahajournals.org/doi/full/10.1161/circheartfailure.111.965707 https://www.ahajournals.org/doi/full/10.1161/circheartfailur...
- beagle3 6y ago
  That's true, but that's not the case here. It showed no evidence for the strong benefit they were looking for (mortality), but a similar trial started in China before that trial, ended - and showed no change in mortality, but some shortening of the hospital stay (even though that wasn't its stated hypothesis). At that point, the western RCT remdesivir trial added the "hospital stay" target, and lo-and-behold, shortly afterwards, it turned out that it does result in no less mortality but shorter hospital stay as well. While it does give some credence to the "shorter hospital stay" result -- having reached it in to Gilead initiated RCTs -- It was statistically significant but not dramatic; And now another trial says even that's not so significant.
fspeech 6y ago
I wish antivirals are trialed with a different end point: reduction in viral load. This can likely be done definitively and a lot faster as well. Deaths caused by ARDS may be triggered by a virus but it doesn't follow that clearing the virus would cure one of ARDS. We really need a pill form of the antiviral and give that to people much earlier in the disease course. A pill is possible in this case as a Remdesivir analog https://www.statnews.com/pharmalot/2020/08/24/nih-gilead-remdesivir-covid19-coronavirus-pandemic/ https://www.statnews.com/pharmalot/2020/08/24/nih-gilead-rem...
- lbeltrame 6y ago
  Lilly and Regeneron's monoclonal antibodies use exactly that as primary endpoint. Unfortunately, most of them are e.v., with exception of one formulation by Regeneron, which is subcutaneous.

cyphar 6y ago

> The original Remdesivir trial by the drug company was a high quality RCT (randomized control trial) - except that they stopped it early once they saw a statistically significant positive effect (even though the effect was somewhat underwhelming: just a couple fewer days of illness). To me this moving of the goal posts invalidates the quality of the RCT. Based on what I've seen discussed, ending trials early is a fairly common (though admittedly contentious) practice in medical RCTs when you're dealing with very serious illnesses, especially when there are no alternative effective treatments[1]. The issue is that there is a tradeoff between an abundance of caution to procure conclusive results and giving patients access to medicine which "clearly helps". In an ideal world you would run every RCT for several years to figure out any possible long term effects and accurately determine efficacy, but there are many circumstances where you could argue doing so (if it's clear the medicine works during the early stages of the trial) is more unethical than giving people medicine which shows some promise to work. But it should be noted that such trials are still solid science (the trial was an RCT), it just means you cannot take them as evidence of long-term efficacy. [1]: https://www.ahajournals.org/doi/full/10.1161/circheartfailure.111.965707 https://www.ahajournals.org/doi/full/10.1161/circheartfailur...

pen2l 6y ago
What is the _reason_ for ending it early? Is it the cost? I imagine it must be high enough because then they're gonna have to fight with this very criticism later on: RCT ended prematurely, therefore it is not as reliable. A question to you and grandparent post: what are the things that distinguish a high quality RCT from a poor one? The number of participants, the design study, etc.? Any links/resources that discuss this would be appreciated. edit: ok, I found a paper on this: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3429837/#:~:text=Randomized%20controlled%20trials%20(RCTs)%20have,of%20medical%20treatments%20(1) https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3429837/#:~:tex.... I am super curious now about any discussion that reveals why the WHO study was less-than-stellar, and why the original RCT was considered high-quality.
- shopwithpoints 6y ago
  Its typically an ethical decision. In the event of a deadly disease ( cancer, covid, hiv etc ) its considered harmful to continue giving a less effective treatment once you have shown some superiority/inferiority.
- arjunnarayan 6y ago
  The reasoning is that as you gain information, you also have a duty to the people in the control group to use the best available information to take care of their health. Once you gain "enough" information ("enough" being statistically defined) that the drug helps, each additional person you let languish in the control group (who is denied access to the drug) is a cost that must be weighed against the benefit of getting additional information. When the data is clear enough, the cost can exceed the benefit, and you stop early. Typically you'd register a "stopping rule" before you start your trial: a good drug often will trigger the stopping rule, as it helped so much that we learned about its efficacy on a smaller N than originally planned. There are many different stopping criteria, depending on the trial (in safety trials you'd typically stop because you've found evidence that the drug is unsafe and continuing would be unfair to the folks in the treatment group, whereas in efficacy trials after safety has been established, you'd stop because you've found evidence that the drug is effective and continuing would be unfair to the folks in the control group).

yread 6y ago

My statistics are a bit shaky but I understand they "pay" for taking an early look (interim analysis) by reducing the power of the trial by using something called alpha-spending function

cultus 6y ago

Wow, that's a textbook example of post-hoc hypothesizing, about the biggest statistical no-no in the book.

tptacek 6y ago

To continue an RCT once it reveals strong evidence for benefit in the treatment group, you have to deliberately inflict suffering on the control group. You can't do that ethically.

gregwebs 6y ago
Strong evidence for a benefit is often measured by death or something extremely severe. When using other clinical endpoints cherry-picking is to be avoided. [1] > For a trial to stop early for benefit, the observed effect on the composite end point ideally should be consistent in the individual components. [1] https://www.ahajournals.org/doi/full/10.1161/circheartfailure.111.965707 https://www.ahajournals.org/doi/full/10.1161/circheartfailur...
beagle3 6y ago
That's true, but that's not the case here. It showed no evidence for the strong benefit they were looking for (mortality), but a similar trial started in China before that trial, ended - and showed no change in mortality, but some shortening of the hospital stay (even though that wasn't its stated hypothesis). At that point, the western RCT remdesivir trial added the "hospital stay" target, and lo-and-behold, shortly afterwards, it turned out that it does result in no less mortality but shorter hospital stay as well. While it does give some credence to the "shorter hospital stay" result -- having reached it in to Gilead initiated RCTs -- It was statistically significant but not dramatic; And now another trial says even that's not so significant.

fspeech 6y ago

I wish antivirals are trialed with a different end point: reduction in viral load. This can likely be done definitively and a lot faster as well. Deaths caused by ARDS may be triggered by a virus but it doesn't follow that clearing the virus would cure one of ARDS. We really need a pill form of the antiviral and give that to people much earlier in the disease course. A pill is possible in this case as a Remdesivir analog https://www.statnews.com/pharmalot/2020/08/24/nih-gilead-remdesivir-covid19-coronavirus-pandemic/ https://www.statnews.com/pharmalot/2020/08/24/nih-gilead-rem...

lbeltrame 6y ago
Lilly and Regeneron's monoclonal antibodies use exactly that as primary endpoint. Unfortunately, most of them are e.v., with exception of one formulation by Regeneron, which is subcutaneous.