7 ms·

This is not the bar that's typically applied for drug trials. The goal of a drug trial is not to prove beyond a shadow of a doubt that it DOESN'T work. The goal

This is not the bar that's typically applied for drug trials. The goal of a drug trial is not to prove beyond a shadow of a doubt that it DOESN'T work. The goal is to prove to a high level of confidence that it DOES work. For example, it's highly unlikely (roughly less than 2.5%) that HCQ reduces the onset of ARDS by half. If it ultimately turns out that HCQ reduces the rate of death by 5%, we'd have to do a tremendously large trial to distinguish that effect from the control with any confidence that it's not actually killing 2% more people. Additionally, you'd have to consider that vs. the fact that it will kill some fraction of people who take it from its cardiac side effects, even if doctors are vigilant about discontinuing therapy when arhythmias are detected.

radford-neal 7y ago
This is not a typical drug trial. Typical trials are designed to have a large enough sample size that they will be able to detect any effect big enough to be clinically important. Here, they just went with the sample size they had available (understandably), which was so small that only a very large effect could be clearly established. It is completely wrong to interpret the lack of a statistically significant effect with this small sample size as showing that the drug doesn't work. Consider the result regarding transfer to ICU or death: 20.2% patients in the HCQ group were transferred to the ICU or died within 7 days vs 22.1% in the no-HCQ group (16 vs 21 events, relative risk [RR] 0.91, 95% CI 0.47-1.80) The best estimate is for 9% less risk from using the drug, with the possibility of the drug cutting the risk by more than half not being excluded in the 95% confidence interval. Of course, the possibility that the drug actually makes things worse is also not excluded, but this can certainly not be characterized as proving that the drug doesn't work.
- URSpider94 7y ago
  I have no problems with the way this test was set up. It was a quick study that would have told us quickly if there was a huge benefit from HCQ. That wasn't the outcome. Larger tests with better statistical power will tell us if there is a smaller positive effect. My point is, the smaller the beneficial effect, the larger the test will have to be to find it, but also the lower the overall benefit to patients. Again, note also that more patients on HCQ were diagnosed with ARDS, so if you pick a different measure, you get the opposite result.
  - jamesblonde 7y ago
    My immediate reaction was that I would not expect to see an effect if you are giving HCQ to patients after they have been hospitalized by the virus. You are now into the inflamation/cytokine storm phase of COVID-19. The best chance of HCQ working is very early, ideally before or directly after exposure. The jury is still out if you take HCQ after the onset of sympthoms if that is also too late. Based on this in-vitro study - https://www.nature.com/articles/s41421-020-0156-0 https://www.nature.com/articles/s41421-020-0156-0 - if I hypothetically had both Chloroquine (CQ) and Hydroxychloroquine (HCQ), I would prefer HCQ for prophylaxis/pre-sympthoms, but if I was sympthomatic, I would choose CQ. The loading time or HCQ is longer and it appears less effective at higher MOIs "the data suggest that the anti-SARS-CoV-2 activity of HCQ seems to be less potent compared to CQ, at least at certain MOIs."
    - tptacek 7y ago
      For pre-symptomatic C19 patients, we're already doing a large-scale natural experiment, because HCQ is part of widespread protocols for common ailments like RA, and the dosages those patients get are comparable. As I understand it: we're admitting lots of RA patients with C19; there doesn't seem to be a huge effect with early dosages either?
      - Gibbon1 7y ago
        Problem with that is people with RA and Lupus are typically hyper careful about their health and most likely to be serious about self isolating.
        tptacek 7y ago
        I'll track down the source I read this from, but my claim was specifically that we are hospitalizing people with RA on HCQ already. So that doesn't really respond to what I said.
      - jamesblonde 7y ago
        Yeah, it's pretty clear there isn't a huge effect. But there may still be a big enough effect to move the needle. I have no knowledge of the RA/Lupus subgroup, so any anecdotes to share are welcome!
capnrefsmmat 7y ago
To be clear, this claim > For example, it's highly unlikely (roughly less than 2.5%) that HCQ reduces the onset of ARDS by half. is not one that can be justified by frequentist statistics such as those used in the article. We can't make probabilistic claims like that unless we go Bayesian. Well-designed drug trials typically plan in advance to have the sample size necessary to detect an effect of clinical significance. If, say, a 20% reduction in ARDS would be valuable, you would work out what sample size you'd need to reliably detect such a reduction. In this case, we simply don't have the sample size to conclude much of anything, because the confidence intervals include what I (as a non-physician, admittedly) see as a pretty wide range of clinically meaningful effect sizes, and we can't distinguish between them.
- URSpider94 7y ago
  Actually, I CAN make that claim based upon the data in the article. The relative risk [RR] of developing ARDS on HCQ vs. in the control group was 1.14, with 95% confidence interval of 0.65 - 2.00. That means that 95% of the time, the relative risk will fall between those two guardrails. If we take the single-sided probability, then the relative risk will be less 0.65 2.5% of the time. "Reducing the onset by half" means a relative risk of 0.5, which is less than the lower 95% confidence interval bound.
  - capnrefsmmat 7y ago
    No, this is a standard example of a claim you cannot make with frequentist statistics, and can only make with a Bayesian credible interval. In frequentist statistics, the true population parameter is fixed, not random, so the claim "the relative risk will be less 0.65 2.5% of the time" has no meaning. The 95% claim is that if we repeatedly sample new data, 95% of the time we will generate a confidence interval covering the true parameter; but that does not say anything about the individual confidence interval in this specific case, about which we have no probability claim. Indeed, it's not clear what a 95% chance would mean for a specific interval, since probability in frequentist statistics is defined in terms of long-run probabilities over repeated samples of data. But the impulse to interpret confidence intervals in a Bayesian way is common, since it's the question we want confidence intervals to answer. You can find a lot of academic papers arguing about this as a sign of the inadequacy of frequentist statistics.

radford-neal 7y ago

This is not a typical drug trial. Typical trials are designed to have a large enough sample size that they will be able to detect any effect big enough to be clinically important. Here, they just went with the sample size they had available (understandably), which was so small that only a very large effect could be clearly established. It is completely wrong to interpret the lack of a statistically significant effect with this small sample size as showing that the drug doesn't work. Consider the result regarding transfer to ICU or death: 20.2% patients in the HCQ group were transferred to the ICU or died within 7 days vs 22.1% in the no-HCQ group (16 vs 21 events, relative risk [RR] 0.91, 95% CI 0.47-1.80) The best estimate is for 9% less risk from using the drug, with the possibility of the drug cutting the risk by more than half not being excluded in the 95% confidence interval. Of course, the possibility that the drug actually makes things worse is also not excluded, but this can certainly not be characterized as proving that the drug doesn't work.

URSpider94 7y ago
I have no problems with the way this test was set up. It was a quick study that would have told us quickly if there was a huge benefit from HCQ. That wasn't the outcome. Larger tests with better statistical power will tell us if there is a smaller positive effect. My point is, the smaller the beneficial effect, the larger the test will have to be to find it, but also the lower the overall benefit to patients. Again, note also that more patients on HCQ were diagnosed with ARDS, so if you pick a different measure, you get the opposite result.
- jamesblonde 7y ago
  My immediate reaction was that I would not expect to see an effect if you are giving HCQ to patients after they have been hospitalized by the virus. You are now into the inflamation/cytokine storm phase of COVID-19. The best chance of HCQ working is very early, ideally before or directly after exposure. The jury is still out if you take HCQ after the onset of sympthoms if that is also too late. Based on this in-vitro study - https://www.nature.com/articles/s41421-020-0156-0 https://www.nature.com/articles/s41421-020-0156-0 - if I hypothetically had both Chloroquine (CQ) and Hydroxychloroquine (HCQ), I would prefer HCQ for prophylaxis/pre-sympthoms, but if I was sympthomatic, I would choose CQ. The loading time or HCQ is longer and it appears less effective at higher MOIs "the data suggest that the anti-SARS-CoV-2 activity of HCQ seems to be less potent compared to CQ, at least at certain MOIs."
  - tptacek 7y ago
    For pre-symptomatic C19 patients, we're already doing a large-scale natural experiment, because HCQ is part of widespread protocols for common ailments like RA, and the dosages those patients get are comparable. As I understand it: we're admitting lots of RA patients with C19; there doesn't seem to be a huge effect with early dosages either?
    - Gibbon1 7y ago
      Problem with that is people with RA and Lupus are typically hyper careful about their health and most likely to be serious about self isolating.
      - tptacek 7y ago
        I'll track down the source I read this from, but my claim was specifically that we are hospitalizing people with RA on HCQ already. So that doesn't really respond to what I said.
    - jamesblonde 7y ago
      Yeah, it's pretty clear there isn't a huge effect. But there may still be a big enough effect to move the needle. I have no knowledge of the RA/Lupus subgroup, so any anecdotes to share are welcome!

capnrefsmmat 7y ago

To be clear, this claim > For example, it's highly unlikely (roughly less than 2.5%) that HCQ reduces the onset of ARDS by half. is not one that can be justified by frequentist statistics such as those used in the article. We can't make probabilistic claims like that unless we go Bayesian. Well-designed drug trials typically plan in advance to have the sample size necessary to detect an effect of clinical significance. If, say, a 20% reduction in ARDS would be valuable, you would work out what sample size you'd need to reliably detect such a reduction. In this case, we simply don't have the sample size to conclude much of anything, because the confidence intervals include what I (as a non-physician, admittedly) see as a pretty wide range of clinically meaningful effect sizes, and we can't distinguish between them.

URSpider94 7y ago
Actually, I CAN make that claim based upon the data in the article. The relative risk [RR] of developing ARDS on HCQ vs. in the control group was 1.14, with 95% confidence interval of 0.65 - 2.00. That means that 95% of the time, the relative risk will fall between those two guardrails. If we take the single-sided probability, then the relative risk will be less 0.65 2.5% of the time. "Reducing the onset by half" means a relative risk of 0.5, which is less than the lower 95% confidence interval bound.
- capnrefsmmat 7y ago
  No, this is a standard example of a claim you cannot make with frequentist statistics, and can only make with a Bayesian credible interval. In frequentist statistics, the true population parameter is fixed, not random, so the claim "the relative risk will be less 0.65 2.5% of the time" has no meaning. The 95% claim is that if we repeatedly sample new data, 95% of the time we will generate a confidence interval covering the true parameter; but that does not say anything about the individual confidence interval in this specific case, about which we have no probability claim. Indeed, it's not clear what a 95% chance would mean for a specific interval, since probability in frequentist statistics is defined in terms of long-run probabilities over repeated samples of data. But the impulse to interpret confidence intervals in a Bayesian way is common, since it's the question we want confidence intervals to answer. You can find a lot of academic papers arguing about this as a sign of the inadequacy of frequentist statistics.