8 ms·
I'm always confused about "effectiveness" numbers. Are these "effective at preventing severe disease" or "effective at preventing symptomatic disease" or agains
by throwaway894345 6y ago
I'm always confused about "effectiveness" numbers. Are these "effective at preventing severe disease" or "effective at preventing symptomatic disease" or against transmission period? The article seems to suggest it's the latter, but it seems somewhat ambiguous. Does this term have a specific definition in epidemiology that journalists and pharma marketers assume everyone understands what kind of "effectiveness" they're talking about?
- deleted 6y ago[deleted]
- ehsankia 6y agoIt's "effective at preventing symptomatic disease". All the vaccines so far are 100% effective against "preventing severe disease" The last one (non-symptomatic), we don't really know, I don't think any studies have been done which regularly test everyone, at scale. Someone correct me if I'm wrong.
- maxerickson 6y agoThere's some studies that have done regular testing, the vaccinations prevent infection. https://www.nejm.org/doi/full/10.1056/NEJMc2102153 https://www.nejm.org/doi/full/10.1056/NEJMc2102153 https://www.nejm.org/doi/full/10.1056/NEJMc2101927 https://www.nejm.org/doi/full/10.1056/NEJMc2101927 2 weeks after the second Pfizer dose, one of the medical centers had a positive test rate of 0.05% (not a typo, 0.05 percent).
- ehsankia 6y agoThose are great, though they are pretty early results. I think in both cases, they finished giving the 2nd dose around February, so we've only had around 1 month of data. I'm a little confused by the percentage though. Looking at the 2nd study, and ignoring 1-7 days after the second dose, there's still 8+7 people who got infected a week after the 2nd dose, out of ~5000 tested. How do they get to the 0.05% number? Also, if I understand correctly, the table says there's around 16,000 eligible, but only 4000-5000 were tested? Am I missing something?
- maxerickson 6y agoI misinterpreted them is what it is. The 0.05% is 7/14990, so it's the infections identified in the vaccinated group and isn't about regular testing, it's just that the result may include cases that were identified by testing rather than being symptomatic.
- ortusdux 6y agoThis is the key point that needs to be conveyed when the J&J 1-shot starts to be distributed. It has a 66-75% efficacy at preventing symptomatic disease, but 100% efficacy at preventing hospitalization. In situations where the 1-shot is ideal, patients being scared off by the lower efficacy rate could be an issue.
- ehsankia 6y agoNot only that, J&J is also doing a trial for 2nd shot, as well as some places such as UK ended up doing Pfizer/Moderna with one shot only with delayed 2nd shot (3-4 months later). So we'll see how that works out, there may be the possibility of getting a 2nd shot of J&J later down the line to increase your immunity. I believe Pfizer has even started doing 3rd shot boosters for very early phase 1-2 people from last year.
- arisAlexis 6y agothe 100% number probably doesn't include the P1,SA variants, no data on them. There is good probability that they become dominant so I'm waiting for the results on those.
- thepete2 6y agoThis is a good question that was picked up by a blog I read. IIRC it refers to the ratio of infections in the control group vs. the vaccinated group. Hence an effectiveness of 90% means 90% of cases occurred in the non-vaccinated group. It does not mean 10% of people end up with no protection at all, as was my first thought.
- m-ee 6y agoMinor nit. The ratio you refer to is called efficacy, not effectiveness. Effectiveness is the number you would get by observing people in the real world rather than the phase 3 trial. It's worth noting the difference because we're also starting to get data on effectiveness, like the CDC study that tracked healthcare workers and found the vaccine also prevented asymptomatic infection.
- clomond 6y agoIts a technical & medical term, rather than one that is meant to be useful for broader discussion. Efficacy in vaccines == people who were prevented from being confirmed COVID positive (symptoms sufficient to prompt a test, leading to a positive result) Therefore, a 91% effective metric for vaccines means that 91% of those who receive that vaccine are expected to NOT contract COVID 'at all'. Thing is though, many vaccines broadly do not prevent disease but shift the severity upon contraction. The metric that is actually useful for informing the public and policy is the answer to the question of 'does this prevent hospitalization, severe disease and death?'. Broadly speaking most approved vaccines globally have near 100% effectiveness in this.
- _Gyan_ 6y agoThe primary endpoint is symptomatic infection, not any infection per se. And it's a bit of a kludge, as all the Phase III protocols I've read define symptomatic infection as X no. of symptoms for Y+ no. of days i.e. there are people with fewer symptoms who aren't counted (or even tested to check for infection). The efficacy is the 1-relative risk of the primary endpoint.
- throwaway894345 6y ago> Therefore, a 91% effective metric for vaccines means that 91% of those who receive that vaccine are expected to NOT contract COVID 'at all'. Thing is though, many vaccines broadly do not prevent disease but shift the severity upon contraction. But I was just reading about another study of medical personnel who were taking periodic COVID tests so as to catch asymptomatic transmission, and I believe this study (or maybe merely the media about it) also used "effectiveness".
- deleted 6y ago[deleted]
- input_sh 6y agoThey're all 100% effective at stopping hospitalization/death (at least in phase 3, not necessarily out in the field). So 9% chance for any infection, but all those that test positive have mild to no symptoms. They basically split people at a certain area into two groups, give one the real thing, another one placebo, track infections, compare the results. Comparing vaccines by effectiveness is meaningless because they're not all tested at the same place at the same time. So J&J has lower effectiveness, but it was calculated at the peak of the outbreak, when each participant interacted with Covid more frequently in their everyday lives. For the comparisons to become meaningful, it'd need to be re-done at the same place at the same urban area, with 10s of thousands of participants for each vaccine. You'd also need to have approximately similar age spread, plus similar population distribution within an urban area. Since it's all volunteer-based, this kind of study won't happen. In conclusion: take any that's available to you.
- thepete2 6y agoI really don't know why this term is thrown around in the news so much without explanation. What I thought at first (and I think is the obvious interpretation) is that with an effectiveness of 90%, 10% of vaccinated people have no protection at all.
- maxerickson 6y agoThe companies have every incentive to publish the good results and the media doesn't care about explaining things clearly. I'm not sure I can envision a reasonable way to embargo the numbers.
- xeromal 6y agoYeah, J&J especially got the shit end of the stick on that. 72% effective, in the US, at no symptoms, but 100% effective at preventing death but every talks like it's a worthless vaccine.
- clarkevans 6y agoMoreover, it's only one shot. J&J is currently enrolling for a 2-shot trial (https://clinicaltrials.gov/ct2/show/NCT04614948 https://clinicaltrials.gov/ct2/show/NCT04614948), unfortunately, the control arm .. is no vaccination. I can imagine how this may hinder recruitment.
- deleted 6y ago[deleted]
- amichal 6y agoMy layman's understanding was that "91%" "effective at preventing disease" (from the TFA) goes something like this: - Take two identical[1] populations of people - Wait for some people (100 for easy math here) in the unvaccinated group to get the disease. Probably measured by a PCR test indicating presence of the virus and ignoring symptoms since they say "preventing disease" and not severe symptoms, hospitalization, death etc. - Count how many folks have the disease in the vaccinated group[2]. In the "91%" case we would expect 9 people to test positive, meaning 91 of 100 people we would expected to get the disease did not. I like to think of it as I am ~1/10th as likely to get the disease as I would be if i had not been vaccinated. Is this what they mean in this? [1] Doing your best to create identical groups by controlling for population differences such as gender, age, and other known risk factors, [2] We need also to control for time in the study etc. [Lots of editing for formatting etc]
- matwood 6y ago> I like to think of it as I am ~1/10th as likely to get the disease as I would be if i had not been vaccinated. Same, and it gets less likely if you are in a group of vaccinated people since the chances of being exposed to someone who is sick goes down. Herd immunity at that point.
- herrvogel- 6y agoThis short video[1] explains it quite well imo. They also make the interesting point that efficacy rates are not comparable because the studies were done at different times/locations with different spread of the virus in the population. [1]https://youtu.be/K3odScka55A https://youtu.be/K3odScka55A
- amichal 6y agoReading the more technically detailed comments I think really reinforces throwaway894345's point. I understood what "91%" means (much better than non-techies in my circle) but assumed a lot incorrectly about the controls and the actual "end point" they are measuring. It seems lots of folks are saying the different studies use different sets of symptoms etc and not routine testing at intervals to identify those who have the disease. I suspect most of the public doesn't even know what 91% means. Combine that with being bad at evaluating risk in general. It mostly represents a 10x improvement in cases per capita per time, but I think telling folks that their risk went from 1 in 10,000 to 1 in 100,000 (or whatever) probably doesn't mean a lot. Folks just want "safe" or "not-safe" :(
- breck 6y agoBe *very skeptical* of any future articles on the long run effectiveness of these vaccines. I was paid to work specifically on COVID, so was monitoring the vaccine data very closely. It was *very* promising, but there were a few things that puzzled me in where comparing results in placebo/control group vs vaccine groups, for both Pfizer and Moderna. So I would check near daily for new results. And then after Christmas break, when I went to check, found out both control groups had been effectively destroyed. (https://www.wsj.com/livecoverage/covid-2020-12-17/card/Pc6LVbfq91WyLWW1I71G https://www.wsj.com/livecoverage/covid-2020-12-17/card/Pc6LV...) I'm not kidding, there are no longer any control/placebo groups for these vaccines. These were studies that were approved to run for 2 years! I had to sit through hours of boring mandatory training from NIH on importance of control groups. In these cases they just threw that away. It reaks to high heaven to me. (Note: I've seen nothing to indicate these vaccines are dangerous. Effective? Looks very likely, at least in the short run. Long run effective? HIGHLY skeptical. Total contribution to ending the pandemic from the vaccines? I'm highly skeptical that it will actually be high, when an honest accounting comes out). Would I personally take the vaccine had I not had COVID already? Yes. Though not if I were a kid or teenager (very little risk from COVID). If you are in medical research, please don't look to these people as role models. Please do things openly, on git, and don't sweep uncomfortable truths under the rug.
- xxxtentachyon 6y agoCan you say more about what you think the implications of offering the control group vaccinations are? My prior assumption would be that protecting the control group would cause the study to underestimate the effectiveness of the vaccine since it now looks comparatively more similar to the control.
- breck 6y agoWe've lost any type of objective long term way to get to the truth on how dangerous COVID really was/is, and how long term effective the vaccines are. The numbers do not come close to supporting any ethical reason for doing this.
- jsnell 6y ago
- jsnell 6y agoThere is no standard definition. The clinical trial would have a set of predefined outcomes ("endpoints") that are used for the efficacy analysis. The endpoints will not necessarily be the same ones for each vaccine, even for the same disease. You'll need to read the actual trial protocol for a specific vaccine to find out just what they were measuring, and how they defined e.g. "serious" vs. "non-serious" cases.
- gregwebs 6y agoMost of the responses to your question that I have read are factually incorrect in some way. Efficacy/effectiveness is up to each study to define. Generally you should assume they are measuring symptoms of infections that go beyond the upper respiratory tract. This is because the vaccines provide little protection against an upper respiratory tract infection but instead are very effective at preventing it from progressing further into a more severe infection. This is referred to as non-sterilizing immunity: it is probably the most important concept for everyone to understand about the vaccines. I am seeing a lot of comments here claim that vaccines are 100% effective at preventing death or severe infection where severe is generally defined as hospitalization. Nothing is 100% effective, and particularly not vaccines for those whose immune system does not respond properly. Death in particular is generally not an endpoint that can be compared in a statistically significant way in these studies. [1] I will still be getting the first imperfect vaccine I am allowed to at the first chance I get. When I needed to get an appointment for my Mom I started writing a user script to help with that. [2] [1] https://dalewharrison.substack.com/p/vaccine-boosterism https://dalewharrison.substack.com/p/vaccine-boosterism [2] https://gist.github.com/gregwebs/265e0ef6b1a3051377cfc4d66acecf81 https://gist.github.com/gregwebs/265e0ef6b1a3051377cfc4d66ac...
- jjcon 6y ago> the vaccines provide little protection against an upper respiratory tract infection They prevent symptomatic covid infection which includes respiratory symptoms.
- fastball 6y agoI don't think "against transmission period" is really possible, no? The whole point of a vaccine is to prime your immune system, it does not provide a magical shield to your body. If you get exposed to COVID-19 after being vaxxed, you will still have COVID-19 in your system for some amount of time. It will just (hopefully) get killed very quickly when the memory B/T cells (that were created by your body when you had an immune response to the vaccine) ramp up production.
- quantgenius 6y agoIt’s a percentage reduction in risk. If a random unvaccinated person has a 10% chance of infection and a vaccinated person has a 1% chance of infection, the vaccine is 90% effective.