5 ms·

There seems to be no analysis of whether these are neutralizing antibodies. The idea of using serology for immunity certificates or "golden tickets" is never go

There seems to be no analysis of whether these are neutralizing antibodies. The idea of using serology for immunity certificates or "golden tickets" is never going to go well. Even with 99.9% specificity, if the population prevalence is 1%, 10% of positives will be false positives. If in real world testing, specificity is 99% and population prevalence is 1%, then 50% of positives are false positives.

wtvanhest 6y ago
Would you or someone else mind expanding on this thought a little? Why does 99.9% specificity mean that 10% will be false positives? {added: great answers below} well worth understanding this point. In short specificity measures the % of the population tested which had false positives, but doesn't give you the ratio of false positives to positives or the probability that a positive test means you actually have the anti-bodies.
- jcampbell1 6y ago
  Bayes theorem assuming 1% actual positive rate. If you test 1000 people, you will get roughly 10 positives and 1 false positive for a false positive rate of 10%.
- projektfu 6y ago
  https://en.wikipedia.org/wiki/Positive_and_negative_predictive_values https://en.wikipedia.org/wiki/Positive_and_negative_predicti... If the number of true positives is 10/1000, and the test gives you 11/1000 positive results, then 1/11 of your tested positive results are false positives. (Actually closer to 9% than 10%).
- gwd 6y ago
  Let me give it a try. Suppose we have 100,000 people in a statistically representative town. If 1% of people have had COVID-19, then that's 1000 people who have had it, and 99,000 people who haven't. The test has a sensitivity of 100%, which means all 1000 people who've had it will test positive. The test has a specificity of 99.9%, which means 98,901 of the 99,000 people who haven't had it will test negative; but that leaves 99 people who haven't had it, but test positive anyway. That gives us 1099 people who look like they have immunity; but only 91% of those people are actually immune: 9% of the people are false positives. If instead we have a specificity of 99%, then only 98,010 of the 99,000 people who haven't had it will test negative, leaving 990 people who haven't had it but test positive anyway. That gives us 1990 people who look like they have immunity; but only 50% of them actually do -- the other 50% are false positives.
  - InvaderFizz 6y ago
    So if I'm understanding this correctly, with this test. If you test negative, you are clear, guaranteed, no false negatives. If you test positive, there is a 10% chance it's a false positive. I guess my follow up question, does a retest of the positive population make that false positive rate drop to 0.1%, or is the reason for false positive significant to an individual and not random chance?
    - gwd 6y ago
      > If you test positive, there is a 10% chance it's a false positive. Well, don't misunderstand -- it's got nothing to do with the test per se, but with the probability that you had the disease in the first place. The test itself has two probabilities: 1. If you've had COVID-19, the probability that it will report positive (sensitivity) 2. If you haven't had COVID-19, the probability that it will report negative (selectivity) But those probabilities give you a mapping from reality -> test_result. What you want is the reverse of that -- and find the probability from a test_result -> reality. When you do that, you have to factor in the probability that you have the disease in the first place. If 50% of the population have had COVID-19, then a positive test means a 99.9% probability of having had the virus. If 1% of the population, a positive test means 91% likely you have it. If only 1 in a million people had COVID-19, then the number of false positives would completely overwhelm the number of true positives. This is sometimes called the "Base rate fallacy": forgetting to factor in the base rate when determining something like this. It's important for things like, say, systems which automatically detect terrorists at airports. How many travelers at an airport are actually terrorists planning to attack a plane? It's got to be one in hundreds of millions, if not billions. With that low of a base rate, even if you had a system that was 99.999% accurate, the vast majority of people it flagged up would be innocent.
    - jointpdf 6y ago
      I had the same question about retesting. Here’s a quote from Scott Gottlieb (former FDA commissioner): “While all of these tests can still generate false positives—a finding that you have the antibodies when you don’t—that risk can be sharply reduced by repeating the test if it comes back positive. The predictive value of two consecutive positive tests is high enough that you can be confident antibodies are present.” https://www.wsj.com/articles/antibody-knowledge-can-be-power-11589130864?redirect=amp#click=https://t.co/bvfqUSlklk https://www.wsj.com/articles/antibody-knowledge-can-be-power...
- Mvandenbergh 6y ago
  Let's say you have 1000 people, 1% or 10 have had the virus and are seropositive (have antibodies). Of the 900 people who do not have ABs, 99.9% or 899.1 are correctly identified as not having them, 0.9 is identified incorrectly as having them when they actually do not. Of the 10 who actually have antibodies, 100% are correctly identified. So 10.9 are identified as having antibodies, in 0.9 person's case incorrectly which is about 10%.
- deleted 6y ago
  [deleted]
- jointpdf 6y ago
  via Bayes Rule: “Assuming an underlying infection rate P(I), what’s the probability that a person is actually immune (=was infected), given that they test positive, i.e. P(I|+)?”: P(I|+) = P(+|I)*P(I) / P(+) = Sens*P(I) / [Sens*P(I) + (1-Spec)*(1-P(I))] = .01 / (.01 + .001*.99) This is exhibit A of the base rate fallacy (https://en.m.wikipedia.org/wiki/Base_rate_fallacy https://en.m.wikipedia.org/wiki/Base_rate_fallacy).
- StavrosK 6y ago
  I think it's easier to understand when you take it to the extremes: Assume nobody has the thing you're testing for. You test 100k people at 99.9% specificity, which means you get 1k positives just because of the rate. Since nobody hass the thing you're testing for, they're all false. When the thing you're testing for is very rare, it's just as rare that the people who tested positive will actually have it.
  - iso947 6y ago
    But we know there are more than 1% cases. We have good data that the IFR is in the 0.1-1% range, putting cases in say MA in the 7% range a couple of weeks ago (time from infection to death), which based on confirmed cases would put it well above 10% now That means you’d have 1k false positive and 10k true positive from a test.
Cerium 6y ago
If we want to ensure that the active population is always above herd immunity threshold, 10% or 20% false positives would be acceptable, since the population in contact would still be 80 or 90% immune.
- georgyo 6y ago
  If the reported immune rate is 80%, with 10-20% false positives, wouldn't the real immunity rate be 64-72%?
- Scipio_Afri 6y ago
  I think this is more about finding hotspots. Where could a seeding event lead to a large outbreak. Clearly 1 person entering into NYC with COVID today would not have the same effects as 4 months ago because supposedly 20% are immune. At least for the time being at least the assumption is. So there is that effect, but also how big could the next wave be based on that. Of course that all has to be proven that detectable immunity or levels above a certain threshold will prevent you from getting infected again or at least reduce the severity over some period of time. So how many antibodies, which type, how long they last, and how much do they have an effect on reducing reinfection or severity of the next infection - all yet to be determined.
  - vmception 6y ago
    > Of course that all has to be proven that detectable immunity or levels above a certain threshold will prevent you from getting infected again. South Korea is reporting that as of last week, and say their earlier "reinfection" results and subsequent scare were flawed.
- pjc50 6y ago
  Reminder that at the moment this cannot happen without 1% of them dying and many more suffering long lasting or even permanent debilitation. If we ever get to the point where 80% of the population has had the virus, that would be a massive failure.
  - 6y ago
klmadfejno 6y ago
What's wrong with 10% of them being false positives? (also I think it's 9%)
- binarymax 6y ago
  Because with a false positive antibody test then you can still contract and be a carrier. This is a problem for the immunity passport proposals (which I think are a terrible idea anyway, but that’s beside the point)
  - misun78 6y ago
    Why do you think immunity passports are a terrible idea? What else works better barring a long term vaccine?
    - klmadfejno 6y ago
      If my job prospects are shut down because I don't have immunity, I'm going to try and get infected.
      - Marsymars 6y ago
        This is especially bad if there's no government-sanctioned method for infection - then you end up with a mess of asymptomatic people trying to get infected while they're already infectious.
  - MertsA 6y ago
    It doesn't have to be perfect. If 90% of the at risk population where it's allowed to spread has immunity, then that's high enough that the virus would logarithmically decay. That's often referred to as "herd immunity". So long as on average cases result in less than one additional case you're fine. 1000 cases with an uninhibited net reproductive rate of 4 with a population that's 90% immune would result in the next "generation" having 400, then 160, then 64, then 26.... You get the point.
- service_bus 6y ago
projektfu 6y ago
If you want neutralizing antibodies you can do serum virus neutralization assay. It's more complicated and takes longer. Testing doesn't need to be 100% to be effective, but it does need to be better than random chance. A mixture of contact tracing, PCR testing, antibody testing and effective quarantines could be used to make the virus go away, but would require a coordinated strategy that the US has not attempted to implement, much to my dismay.
- lbeltrame 6y ago
  The people from the Duke-NUS Medical School have designed a "surrogate virus neutralization test"[1] which might be used to perform neutralization tests without the security requirements of using live virus. [1] https://www.researchsquare.com/article/rs-24574/v1 https://www.researchsquare.com/article/rs-24574/v1
  - projektfu 6y ago
    Cool info.
gridlockd 6y ago
What if prevalence is 5% or 10%? Your alarming false positive rates vanish. No measure is ever going to work perfectly.
- iso947 6y ago
  At an individual basis it can still be problematic as you can still get a super spreader who thinks they are immune, picks it up, then spreads it. On a population basis it’s more helpful.
mlyle 6y ago
> if the population prevalence is 1% But the population prevalence is much more than 1%: 80k deaths at a 1% infection fatality rate (and I believe this is high, but I'm being conservative) implies 8,000,000 infections so far. This is more like 2.5%. So far. 95% CI for specificity is 99.5%, so you can be reasonably confident that you're doing better than 85%. It may not be a perfect intervention, but you could really reduce risk. If there's an 85% chance that someone is immune, they do not share a household with a vulnerable person, and they are not in a high risk group themselves-- you've reduced the risk of death to basically nothing. I disagree with it for other reasons (it incents people to go get sick to be free/be able to work/etc).
- iso1631 6y ago
  > I disagree with it for other reasons (it incents people to go get sick to be free/be able to work/etc). On an individual basis yes, but doign large scale scientifically/statistically relevant antibody tests to see just how many people have actually had covid-19, would be very beneficial.
- lostmsu 6y ago
  This high prevalence is specific to a few regions though. I most states the registered total cases are still way less than 0.5%.

wtvanhest 6y ago

Would you or someone else mind expanding on this thought a little? Why does 99.9% specificity mean that 10% will be false positives? {added: great answers below} well worth understanding this point. In short specificity measures the % of the population tested which had false positives, but doesn't give you the ratio of false positives to positives or the probability that a positive test means you actually have the anti-bodies.

jcampbell1 6y ago
Bayes theorem assuming 1% actual positive rate. If you test 1000 people, you will get roughly 10 positives and 1 false positive for a false positive rate of 10%.
projektfu 6y ago
https://en.wikipedia.org/wiki/Positive_and_negative_predictive_values https://en.wikipedia.org/wiki/Positive_and_negative_predicti... If the number of true positives is 10/1000, and the test gives you 11/1000 positive results, then 1/11 of your tested positive results are false positives. (Actually closer to 9% than 10%).
gwd 6y ago
Let me give it a try. Suppose we have 100,000 people in a statistically representative town. If 1% of people have had COVID-19, then that's 1000 people who have had it, and 99,000 people who haven't. The test has a sensitivity of 100%, which means all 1000 people who've had it will test positive. The test has a specificity of 99.9%, which means 98,901 of the 99,000 people who haven't had it will test negative; but that leaves 99 people who haven't had it, but test positive anyway. That gives us 1099 people who look like they have immunity; but only 91% of those people are actually immune: 9% of the people are false positives. If instead we have a specificity of 99%, then only 98,010 of the 99,000 people who haven't had it will test negative, leaving 990 people who haven't had it but test positive anyway. That gives us 1990 people who look like they have immunity; but only 50% of them actually do -- the other 50% are false positives.
- InvaderFizz 6y ago
  So if I'm understanding this correctly, with this test. If you test negative, you are clear, guaranteed, no false negatives. If you test positive, there is a 10% chance it's a false positive. I guess my follow up question, does a retest of the positive population make that false positive rate drop to 0.1%, or is the reason for false positive significant to an individual and not random chance?
  - gwd 6y ago
    > If you test positive, there is a 10% chance it's a false positive. Well, don't misunderstand -- it's got nothing to do with the test per se, but with the probability that you had the disease in the first place. The test itself has two probabilities: 1. If you've had COVID-19, the probability that it will report positive (sensitivity) 2. If you haven't had COVID-19, the probability that it will report negative (selectivity) But those probabilities give you a mapping from reality -> test_result. What you want is the reverse of that -- and find the probability from a test_result -> reality. When you do that, you have to factor in the probability that you have the disease in the first place. If 50% of the population have had COVID-19, then a positive test means a 99.9% probability of having had the virus. If 1% of the population, a positive test means 91% likely you have it. If only 1 in a million people had COVID-19, then the number of false positives would completely overwhelm the number of true positives. This is sometimes called the "Base rate fallacy": forgetting to factor in the base rate when determining something like this. It's important for things like, say, systems which automatically detect terrorists at airports. How many travelers at an airport are actually terrorists planning to attack a plane? It's got to be one in hundreds of millions, if not billions. With that low of a base rate, even if you had a system that was 99.999% accurate, the vast majority of people it flagged up would be innocent.
  - jointpdf 6y ago
    I had the same question about retesting. Here’s a quote from Scott Gottlieb (former FDA commissioner): “While all of these tests can still generate false positives—a finding that you have the antibodies when you don’t—that risk can be sharply reduced by repeating the test if it comes back positive. The predictive value of two consecutive positive tests is high enough that you can be confident antibodies are present.” https://www.wsj.com/articles/antibody-knowledge-can-be-power-11589130864?redirect=amp#click=https://t.co/bvfqUSlklk https://www.wsj.com/articles/antibody-knowledge-can-be-power...
Mvandenbergh 6y ago
Let's say you have 1000 people, 1% or 10 have had the virus and are seropositive (have antibodies). Of the 900 people who do not have ABs, 99.9% or 899.1 are correctly identified as not having them, 0.9 is identified incorrectly as having them when they actually do not. Of the 10 who actually have antibodies, 100% are correctly identified. So 10.9 are identified as having antibodies, in 0.9 person's case incorrectly which is about 10%.
deleted 6y ago
[deleted]
jointpdf 6y ago
via Bayes Rule: “Assuming an underlying infection rate P(I), what’s the probability that a person is actually immune (=was infected), given that they test positive, i.e. P(I|+)?”: P(I|+) = P(+|I)*P(I) / P(+) = Sens*P(I) / [Sens*P(I) + (1-Spec)*(1-P(I))] = .01 / (.01 + .001*.99) This is exhibit A of the base rate fallacy (https://en.m.wikipedia.org/wiki/Base_rate_fallacy https://en.m.wikipedia.org/wiki/Base_rate_fallacy).
StavrosK 6y ago
I think it's easier to understand when you take it to the extremes: Assume nobody has the thing you're testing for. You test 100k people at 99.9% specificity, which means you get 1k positives just because of the rate. Since nobody hass the thing you're testing for, they're all false. When the thing you're testing for is very rare, it's just as rare that the people who tested positive will actually have it.
- iso947 6y ago
  But we know there are more than 1% cases. We have good data that the IFR is in the 0.1-1% range, putting cases in say MA in the 7% range a couple of weeks ago (time from infection to death), which based on confirmed cases would put it well above 10% now That means you’d have 1k false positive and 10k true positive from a test.

Cerium 6y ago

If we want to ensure that the active population is always above herd immunity threshold, 10% or 20% false positives would be acceptable, since the population in contact would still be 80 or 90% immune.

georgyo 6y ago
If the reported immune rate is 80%, with 10-20% false positives, wouldn't the real immunity rate be 64-72%?
Scipio_Afri 6y ago
I think this is more about finding hotspots. Where could a seeding event lead to a large outbreak. Clearly 1 person entering into NYC with COVID today would not have the same effects as 4 months ago because supposedly 20% are immune. At least for the time being at least the assumption is. So there is that effect, but also how big could the next wave be based on that. Of course that all has to be proven that detectable immunity or levels above a certain threshold will prevent you from getting infected again or at least reduce the severity over some period of time. So how many antibodies, which type, how long they last, and how much do they have an effect on reducing reinfection or severity of the next infection - all yet to be determined.
- vmception 6y ago
  > Of course that all has to be proven that detectable immunity or levels above a certain threshold will prevent you from getting infected again. South Korea is reporting that as of last week, and say their earlier "reinfection" results and subsequent scare were flawed.
pjc50 6y ago
Reminder that at the moment this cannot happen without 1% of them dying and many more suffering long lasting or even permanent debilitation. If we ever get to the point where 80% of the population has had the virus, that would be a massive failure.
- 6y ago

klmadfejno 6y ago

What's wrong with 10% of them being false positives? (also I think it's 9%)

binarymax 6y ago
Because with a false positive antibody test then you can still contract and be a carrier. This is a problem for the immunity passport proposals (which I think are a terrible idea anyway, but that’s beside the point)
- misun78 6y ago
  Why do you think immunity passports are a terrible idea? What else works better barring a long term vaccine?
  - klmadfejno 6y ago
    If my job prospects are shut down because I don't have immunity, I'm going to try and get infected.
    - Marsymars 6y ago
      This is especially bad if there's no government-sanctioned method for infection - then you end up with a mess of asymptomatic people trying to get infected while they're already infectious.
- MertsA 6y ago
  It doesn't have to be perfect. If 90% of the at risk population where it's allowed to spread has immunity, then that's high enough that the virus would logarithmically decay. That's often referred to as "herd immunity". So long as on average cases result in less than one additional case you're fine. 1000 cases with an uninhibited net reproductive rate of 4 with a population that's 90% immune would result in the next "generation" having 400, then 160, then 64, then 26.... You get the point.
service_bus 6y ago

projektfu 6y ago

If you want neutralizing antibodies you can do serum virus neutralization assay. It's more complicated and takes longer. Testing doesn't need to be 100% to be effective, but it does need to be better than random chance. A mixture of contact tracing, PCR testing, antibody testing and effective quarantines could be used to make the virus go away, but would require a coordinated strategy that the US has not attempted to implement, much to my dismay.

lbeltrame 6y ago
The people from the Duke-NUS Medical School have designed a "surrogate virus neutralization test"[1] which might be used to perform neutralization tests without the security requirements of using live virus. [1] https://www.researchsquare.com/article/rs-24574/v1 https://www.researchsquare.com/article/rs-24574/v1
- projektfu 6y ago
  Cool info.

gridlockd 6y ago

What if prevalence is 5% or 10%? Your alarming false positive rates vanish. No measure is ever going to work perfectly.

iso947 6y ago
At an individual basis it can still be problematic as you can still get a super spreader who thinks they are immune, picks it up, then spreads it. On a population basis it’s more helpful.

mlyle 6y ago

> if the population prevalence is 1% But the population prevalence is much more than 1%: 80k deaths at a 1% infection fatality rate (and I believe this is high, but I'm being conservative) implies 8,000,000 infections so far. This is more like 2.5%. So far. 95% CI for specificity is 99.5%, so you can be reasonably confident that you're doing better than 85%. It may not be a perfect intervention, but you could really reduce risk. If there's an 85% chance that someone is immune, they do not share a household with a vulnerable person, and they are not in a high risk group themselves-- you've reduced the risk of death to basically nothing. I disagree with it for other reasons (it incents people to go get sick to be free/be able to work/etc).

iso1631 6y ago
> I disagree with it for other reasons (it incents people to go get sick to be free/be able to work/etc). On an individual basis yes, but doign large scale scientifically/statistically relevant antibody tests to see just how many people have actually had covid-19, would be very beneficial.
lostmsu 6y ago
This high prevalence is specific to a few regions though. I most states the registered total cases are still way less than 0.5%.