9 ms·
It's important to interpret these results carefully: "no evidence of efficacy" does not imply evidence of no efficacy, and in fact the uncertainty in this study
by capnrefsmmat 6y ago
It's important to interpret these results carefully: "no evidence of efficacy" does not imply evidence of no efficacy, and in fact the uncertainty in this study is quite large. The sample size is small enough that it would be very hard for this study to detect the effect of hydroxychloroquine unless that effect is very, very large.
From the abstract:
> In the HCQ group, 2.8% of the patients died within 7 days vs 4.6% in the no-HCQ group (3 vs 4 events, RR 0.61, 95% CI 0.13-2.89), and 27.4% and 24.1%, respectively, developed acute respiratory distress syndrome within 7 days (24 vs 23 events, RR 1.14, 95% CI 0.65-2.00).
The relative risks and confidence intervals are the important numbers here. For example, the relative risk for death is 0.61, meaning the observed risk of death was lower for patients treated with hydroxychloroquine -- but the confidence interval is 0.13 to 2.89, meaning the data is consistent with anything from the risk being much smaller to the risk being much larger. Since there were only 3 deaths in the treatment group and 4 deaths in the control group, it's very hard to draw precise conclusions about death rates.
I think we can interpret this result to mean that hydroxychloroquine doesn't have a miraculously large effect, but the evidence is weak. Other commenters are correct that large randomized trials will be more definitive.
(disclaimer: I am a statistician, not a doctor)
- blakesterz 6y agoWould it be the same thing to do 1,000 small studies like this one as doing one LARGE study with 1000x more people? Would the results be considered just as useful?
- lonelappde 6y agoSimilar. That's called a "meta analysis" and tends to happen after many independent uncoordinated small studies have published.
- emn13 6y agoAlso, it's extremely hard to account for all kinds of biases and systematic errors in a meta-analysis. I'd take most meta-analyses, even by renowned statiticians, very, very skeptically. Teasing out small effects from samples that need corrections for lots of confounding factors, and then trying do combine many such results is next to impossible. If your effect is small enough that you need a meta-analysis in the first place, that's a warning sign. Some background: https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0152719 https://journals.plos.org/plosone/article?id=10.1371/journal... (with perhaps more approachable explanation here https://www.talyarkoni.org/blog/2016/06/11/the-great-minds-journal-club-discusses-westfall-yarkoni-2016/ https://www.talyarkoni.org/blog/2016/06/11/the-great-minds-j... ) Obviously it's sometimes possible to do this right, but the reproducibiility crisis didn't appear out of nowhere; it's best to be wary of overly clever statistics; hard to know what you don't know.
- bluGill 6y agoThat isn't the same as the studies are likely to be different. If one study is old people, another young, another those with diabetes, and so on. Some may have data but it is clouded by a biased sample... Putting this all together is done in meta analysis which is complex and finds things in small studies that don't show up but it isn't trivial to do them.
- thedudeabides5 6y agoThis is a great comment thanks. Title of the paper could (with a tongue firmly in cheek) be called, "Weak evidence for effectiveness of hydroxychloroquine but our sample was likely too small for it to pass a T-test"
- briefcomment 6y agoWow, I would love titles like this.
- sadfklsjlkjwt 6y agoOr better "We did a study with too few examples and did not get meaningful data"
- A4ET8a8uTh0 6y agoI agree. I wonder if we could try to move click-baity articles under truth in advertising legislation of some sort. edit: clarified it a little
- marcosdumay 6y agoThis is the kind of study where it would be beneficial for it to be ignore. The correct headline is no headline at all.
- emn13 6y agoWell, given the time constraints, and the fact that the previous paper was had a much weaker setup yet claimed efficacy, I think this paper is a good idea. Incidentally, it's not the only such paper; a chinese analysis from the early days came to a comparable conclusion. Even based just on this evidence, the case for HCQ looks very, very weak. The only positive report was riddled with caveats that could easily explain a false-positive, and other reports see next to no effect. Note that it's perhaps not so interesting whether it has some effect; for this to really matter, it needs to be somewhat significant effect (in the real-world impact sense, not the statistically measurable sense). And even a small sample can reliably detect large effects; several studies have now not found such an effect; ergo: it doesn't exist. If HCQ has any benefit whatsoever as applied once very sick, the benefit is likely too small to matter much in terms of mortality. And it's unlikely to be relevant whether it works beforehand; there's no way to make enough to give it to everyone which is what you'd need to do if you wanted to significantly impact the pandemic if HCQ were prophylactic. It seems very unlikely at this point that HCQ is going to matter much in the course of the pandemic.
- mcguire 6y agoThe rest of the statistics: "In the weighted analysis, 20.2% patients in the HCQ group were transferred to the ICU or died within 7 days vs 22.1% in the no-HCQ group (16 vs 21 events, relative risk [RR] 0.91, 95% CI 0.47-1.80)." And... "Eight patients receiving HCQ (9.5%) experienced electrocardiogram modifications requiring HCQ discontinuation."
- shadowgovt 6y agoQuestion: are the induced arrhythmias permanent or do they discontinue when HCQ therapy discontinues? I'm curious as to the permanency of the damage.
- epmaybe 6y agothe qt prolongation of heart rhythm goes away once discontinued.
- jamesblonde 6y agoQt prolongation does not mean you have arrhythmias. You don't get tachycardia or atrial flutter or necessarily sympthoms. Clinically if it means there is a risk if the QT interval gets to long that you will have a fatal arrhythmia - torsades de pointes. So you need to know how bad the Qt prolongation is. In the paper 2 patients had serious side effects - one with QT prolongation over 500ms and the second with a first-degree atrioventricular block. Another maybe had a left bundle branch (not great, but not fatal). They resolved by stopping HCQ treatment.
- epmaybe 6y agoI didn't claim that the QT prolongation implied arrhythmia.
- DataWorker 6y agoWouldn’t ecg changes be plausibly the result of the virus itself? I haven’t read the paper so maybe they go into greater detail but it would be important to know whether such changes were seen in the control group as attrition/discontinuance is critically important to any study.
- DrNuke 6y agoMagic potions do not work, we all should know that in 2020 and wait for the scientific trials step by step... as for corona, today the most promising way before a vaccine, if any, is tailored therapy aimed at containing symptoms so that your body and immune system can fight off the virus at full strength... and we are starting to see a great number of cases that actually improve and can help similar cases (for age, sex, comorbidities, etc.) worldwide, so more hope today than one month ago even if infection numbers are still rising up.
- ajross 6y agoThis is absolutely true. But the reason that a bland non-result is newsworthy is precisely because various public entities decided to treat the previous result as much more significant than it was. And it absolutely does serve as a refutation of that rhetoric.
- esyir 6y agoAnd this we now have the opposite problem, where a potential option is treated like it has the legitimacy of crystal healing, depending on the political side you sit on.
- ajross 6y agoSorry, who exactly is likening chloroquines to crystal healing? I hear this all the time, but frankly it's a giant strawman. No one serious (i.e. a public figure with media coverage) is arguing against the efficacy of chloroquines. Boosting unproven drugs is dangerous and irresponsible, and arguing that fact is not the same thing as arguing that the drug is ineffective.
- asdfasgasdgasdg 6y ago> "no evidence of efficacy" does not imply evidence of no efficacy That's incorrect. Absence of evidence is evidence of absence. It's just not proof of absence. See https://www.lesswrong.com/posts/mnS2WYLCGJP2kQkRn/absence-of-evidence-is-evidence-of-absence https://www.lesswrong.com/posts/mnS2WYLCGJP2kQkRn/absence-of...
- Robotbeat 6y agoIt’s not if the statistics are too small. It’s only evidence against a strong effect (which the parent acknowledges). I think the parent post is making an important and accurate point.
- wigl 6y agoThe parent states: > "no evidence of efficacy" does not imply evidence of no efficacy, and in fact the uncertainty in this study is quite large. The sample size is small enough that it would be very hard for this study to detect the effect of hydroxychloroquine unless that effect is very, very large. What you're saying might be correct but misses the point of the parent (and perhaps statistics) completely.
- deleted 6y ago[deleted]
- 6gvONxR4sf7o 6y agoBy that logic, you could just not do the study at all and claim it as evidence of absence. Statistically, it's the same problem. You need data (and lots of it, relative to the noise) to provide evidence of anything.
- capnrefsmmat 6y agoI do not think this argument is relevant to estimating effect sizes, rather than binary outcomes as the Less Wrong post does. If the claim had been "no patients treated hydroxychloroquine recover", and the study found no patients that recovered, the argument would apply: there's no evidence that patients recover, which starts to suggest that indeed, patients do not recover. But here we are interested in estimating the size of the effect, which can take any continuous value. The statistical result is that the relative risk of death is, with 95% confidence, somewhere between 0.13 and 2.89. You can't interpret that as evidence relative risk is exactly 1 (equal risk with or without the drug) -- if you could, you could also take it as evidence the relative risk is 1.2, or 2.7, or 0.75.
- gameswithgo 6y agoIt is important to remember we also do not have evidence of no efficacy for chocolate milk. Yet we do not waste our time testing it.
- kerkeslager 6y agoOkay but there are actually good reasons to test hydroxychloroquine, as it is effective against viruses with relevant similarities to Covid-19. The same is not true for chocolate milk. To address the elephant in the room: yes, Trump did come out in favor of hydroxychloroquine, but his opinion on medical issues is literally meaningless noise in both the informational and auditory senses of the phrase. We certainly shouldn't be pursuing hydroxychloroquine because Trump said to, but there are other good reasons to at least investigate it.
- URSpider94 6y agoThis is not the bar that's typically applied for drug trials. The goal of a drug trial is not to prove beyond a shadow of a doubt that it DOESN'T work. The goal is to prove to a high level of confidence that it DOES work. For example, it's highly unlikely (roughly less than 2.5%) that HCQ reduces the onset of ARDS by half. If it ultimately turns out that HCQ reduces the rate of death by 5%, we'd have to do a tremendously large trial to distinguish that effect from the control with any confidence that it's not actually killing 2% more people. Additionally, you'd have to consider that vs. the fact that it will kill some fraction of people who take it from its cardiac side effects, even if doctors are vigilant about discontinuing therapy when arhythmias are detected.
- radford-neal 6y agoThis is not a typical drug trial. Typical trials are designed to have a large enough sample size that they will be able to detect any effect big enough to be clinically important. Here, they just went with the sample size they had available (understandably), which was so small that only a very large effect could be clearly established. It is completely wrong to interpret the lack of a statistically significant effect with this small sample size as showing that the drug doesn't work. Consider the result regarding transfer to ICU or death: 20.2% patients in the HCQ group were transferred to the ICU or died within 7 days vs 22.1% in the no-HCQ group (16 vs 21 events, relative risk [RR] 0.91, 95% CI 0.47-1.80) The best estimate is for 9% less risk from using the drug, with the possibility of the drug cutting the risk by more than half not being excluded in the 95% confidence interval. Of course, the possibility that the drug actually makes things worse is also not excluded, but this can certainly not be characterized as proving that the drug doesn't work.
- URSpider94 6y agoI have no problems with the way this test was set up. It was a quick study that would have told us quickly if there was a huge benefit from HCQ. That wasn't the outcome. Larger tests with better statistical power will tell us if there is a smaller positive effect. My point is, the smaller the beneficial effect, the larger the test will have to be to find it, but also the lower the overall benefit to patients. Again, note also that more patients on HCQ were diagnosed with ARDS, so if you pick a different measure, you get the opposite result.
- dfsegoat 6y agoAlso important to note: This article is a preprint and has not been certified by peer review
- deleted 6y ago[deleted]
- fspeech 6y agoThere are two potential benefits of hcq people are hoping to prove: first as an antiviral, secondly as an immuno modulator for the IL-6 pathway. As a nonspecific antiviral any benefit tends to be for the early stage of the disease, possibly as a prophylaxis or preventive. So to me this trial mostly shows that there is no discernible benefit for hcq as an immuno-modulator. That is disappointing but not too surprising. I hear people with autoimmune problems say that it takes a month or two for the drug's effect to kick in. That is about 12g of cumulative dose, too big to reach in a short time without acute toxicity. I wonder here if also people of the high risk group could benefit from prophylactic use. Is there any truth to the claim that lupus patients on hcq don't suffer from covid? This claim should be easy to check now that we have so many cases of covid around the world.
- jamesblonde 6y agoExcellent point - this study does add to our knowledge - the case is now pretty much closed on the efficacy of HCQ as an immuno modulator for COVID-19.
- acallaghan 6y ago> Other commenters are correct that large randomized trials will be more definitive. The reason they aren't doing large randomised controlled trials is because the hydroxychloroquine itself is killing people, and it's unethical to continue. People have said they'd rather have malaria than take it, and that should tell you something. The side-effects for hydroxychloroquine include death, which is not immediately obvious from reading HN. Doctors aren't just making this up and ignoring evidence, the drug is bad, and not effective for covid-19
- jfnixon 6y agoI have a friend who takes it for lupus, and has taken it for years. Clearly, he hasn't died, in part because it is a prescription drug and requires a doctor's script. He understands the risks and the benefits. He has taken it for years, as I mentioned, so your "people have said" is clearly not inclusive of a large number of people. The medicine is made by several manufacturers, and if there wasn't a market, that wouldn't happen. I call shenanigans.
- acallaghan 6y agoThere's a very strict set of people that prescriptions are given to - i.e. not a routine treatment for viruses like this. There's a reason it's not licensed for viruses, and why only a tiny number of people take it, and there's a reason why wide scale trials are being cancelled. There isn't a huge anti-hydroxychloroquine conspiracy here