9 ms·

I've been tracking the antibody study results in a spreadsheet, and they are suggesting a 10-20x undercount of cases in the official "confirmed" numbers. You ca

I've been tracking the antibody study results in a spreadsheet, and they are suggesting a 10-20x undercount of cases in the official "confirmed" numbers. You can see the data I've collected here: https://docs.google.com/spreadsheets/d/16onEUBWIV5IqN1RCvTlad1jnQZ9W7p_P_yS-0D6j1-8 https://docs.google.com/spreadsheets/d/16onEUBWIV5IqN1RCvTla...

bb2018 6y ago
I think antibody tests will soon become more useful for tracking disease progression in a population than the viral tests. The collection methods may skew things but they still are much more close to a random sample than the viral tests which have lots of issues with test shortages and people unable to get tested (or not wanting to go to the hospital with mild symptoms).
- garmaine 6y ago
  They have huge false positive rates though.
  - taeric 6y ago
    Do we know the rates already?
    - amluto 6y ago
      There’s preliminary data: https://covidtestingproject.org/ https://covidtestingproject.org/
  - deleted 6y ago
    [deleted]
  - danielfoster 6y ago
    And false negatives.
    - pedrosorio 6y ago
      If the rate of false positives and false negatives is the same but the true prevalence is << 0.5, then you will overestimate the number of positives.
  - brm 6y ago
    Only until the point that they don't. Unfortunately that requires a lot more people to have had covid...
  - ptrincr 6y ago
    Not sure why you are being downvoted: https://www.oxfordbiosystems.com/COVID-19-Rapid-test https://www.oxfordbiosystems.com/COVID-19-Rapid-test "In order to test the detection sensitivity and specificity of the COVID-19 IgG-IgM combined antibody test, blood samples were collected from COVID-19 patients from multiple hospitals and Chinese CDC laboratories. The tests were done separately at each site. A total of 525 cases were tested: 397 (positive) clinically confirmed (including PCR test) SARS-CoV-2-infected patients and 128 non- SARS-CoV-2-infected patients (128 negative). The testing results of vein blood without viral inactivation were summarized in the Table 1. Of the 397 blood samples from SARS-CoV-2-infected patients, 352 tested positive, resulting in a sensitivity of 88.66%. Twelve of the blood samples from the 128 non-SARS-CoV-2 infection patients tested positive, generating a specificity of 90.63%." That gives us 62% false positive ratio according to (where a study finds the prevalence to be 6% of subjects using the test): http://vassarstats.net/clin2.html http://vassarstats.net/clin2.html In some cases we have research being carried out with such low positive results that they can entirely be accounted for by the low specificity. So for example if you took samples from 100 people, based on 90% specificity, even if everyone had never had corona, 10 could be found positive. Credit to this post: https://old.reddit.com/r/COVID19/comments/g7f373/second_round_of_covid19_community_testing/foh3b3p/ https://old.reddit.com/r/COVID19/comments/g7f373/second_roun... However it should be noted the article in question for this submission does not mention the type of test used.
    - raverbashing 6y ago
      I wonder what's the process through which false positives happen in this case. Previous infection by milder Coronaviruses? Edit: I'm looking at the reddit post but I have a lot of reservations with the "prevalence 0.06", unless we'll use the test to test absolutely everybody and not only people who are suspect. Has that calculator been validated as well? If the test was 12 false positives in 128 negatives, how come they can claim the false positive rate is 60%?
      - ptrincr 6y ago
        Apologies for the way this was linked to. The 6% is from this study: https://www.miamidade.gov/releases/2020-04-24-sample-testing-results.asp https://www.miamidade.gov/releases/2020-04-24-sample-testing... "Our data from this week and last tell a very similar story. In both weeks, 6% of participants tested positive for COVID-19 antibodies, which equates to 165,000 Miami-Dade County residents" That is what the commentator is referring to in the linked post. So if you plug their own figures into the calculator: Sensitivity .8866 Specificity .9063 and a Prevalence of .06 based on the study, you get the 62% false positive rate. As the prevalence increases, as with the NYC study which found the positive rate to be 21% (prevalence), the false positive rate decreases, down to 28% of the NYC study.
  - deleted 6y ago
    [deleted]
  - timr 6y ago
    There are many different tests, from different manufacturers. Some of the tests have higher false-positive rates than others. Some have higher false-negative rates. Even a survey with an imperfect test can be designed to yield reliable data.
buboard 6y ago
i don't understand the value of this multiplier. Case statistics are not an official census, it's incidental , depending on the criteria with which each region makes tests. Case numbers are unimportant, it's the total infections and consequent deaths that matter
- SpicyLemonZest 6y ago
  You're probably right that they shouldn't be, but case statistics are regularly treated as an official census. I've seen many news articles in the vein that suchandsuch country is handling it better or worse because of their case numbers, or statistics like CFR that are computed from official case numbers.
  - buboard 6y ago
    while the absolute value doesn't matter, its time course is mostly representative , because countries rarely change strategies wrt testing.
    - SpicyLemonZest 6y ago
      That just doesn't seem true. In the US testing strategies are rapidly changing, since health officials indicate this will be essential to safely removing restrictions.
- analog31 6y ago
  One reason is that early on when the curves looked pretty much exponential, folks were trying to pin down how bad it could get, and when. This was when the worst case scenario was for nearly the entire population to get it. I'm not quite sure we're completely out of that woods yet.
skybrian 6y ago
I wonder which tests they used? A recent study is finding that some antibody tests are much better than others when it comes to false positives: https://tildes.net/~health.coronavirus/o6a/coronavirus_antibody_tests_can_you_trust_the_results https://tildes.net/~health.coronavirus/o6a/coronavirus_antib...
- marcell 6y ago
  I know that the California studies used the same test kit. It had 2 false positives out of 371 samples of pre-covid19 cases, and has a 10-20% false negative rate. Because the case numbers are so small, the false positives can skew things quite a bit. I took the midpoints of their 95% confidence intervals in the spreadsheet. I don't know what tests the other studies used.
DanBC 6y ago
What numbers are you using for sensitivity and specificity?
- marcell 6y ago
  It depends on the study, I've been using their reported confidence intervals. The two California studies (Santa Clara and Los Angeles) used the same kit, which has 2 false positives of 371 tests, and 10-20% false negative rate.
  - nodamage 6y ago
    It's worth noting that is the manufacturer claim but has not held up to independent validation. Specifically, the Premier Biotech/Hangzhou Biotest Biotech test was validated by a Chinese provincial CDC and found 4 false positives out of 150. [1] It was also validated by the COVID-19 Testing project and found 3 false positives out of 108. [2] The Biomedomics test used in the Miami Dade study was also validated by the COVID-19 Testing Project and found 14 false positives out of 107. [2] Hence I would recommend taking the results of the California and Florida studies with a huge grain of salt as the prevalence rates they found were within the false positive rates of the tests used. [1] https://imgcdn.mckesson.com/CumulusWeb/Click_and_learn/COVID19_CDC_Evaluation_Report.pdf https://imgcdn.mckesson.com/CumulusWeb/Click_and_learn/COVID... [2] https://covidtestingproject.org/ https://covidtestingproject.org/
cies 6y ago
I did similar calculations, and found the institutions in charge give us very unreliable data. The term "corona case" is very, very, ambiguous and cannot the understood as such without a detailed explanation on how the counting was done. Thanks for sharing. I found the peek in all-case mortality also very interesting, because that way counting is much more unambiguous: dead is dead. They showed a clear diversion from the "average" in recent weeks, but... they did not show the stdev for the averages. Finally I found a chart that shows that "outliers" are not uncommon. https://imgur.com/IPNiXRe https://imgur.com/IPNiXRe
- marcell 6y ago
  Yea, I agree that the case counts are very different between regions. I think the more interesting column is the IFR estimate based on the antibody study results, since dead is dead as you say. It is interesting though that the median undercount is converging to ~10-20x. Perhaps the protocols across regions are similar enough that the confirmed case counts are somewhat comparable.
  - pbhjpbhj 6y ago
    It seems possible to me that there are variations, mutations, in different regions that might account for some of the variance in apparent R value and such?
- bobongo 6y ago
  > because that way counting is much more unambiguous: dead is dead Unrealistically low death stats coming from Turkey compared to cases easily refute that argument. Dead is dead, unless the state finds a way to claim that it was not a COVID19 dead.
  - gaius_baltar 6y ago
    > Dead is dead, unless the state finds a way to claim that it was not a COVID19 dead. It's just a matter of demanding tests to declare as a COVID death and do not providing enough tests. Brazil, for example, has an artificially low count of cases due to the lack of tests and a similarly low number of deaths. However, cases of death by "pneumonia", generic types of SARS and "unexplained respiratory diseases" skyrocketed: https://oglobo.globo.com/sociedade/coronavirus/alem-da-covid-19-brasil-tem-outras-2771-mortes-por-problemas-respiratorios-sem-explicacao-24389276 https://oglobo.globo.com/sociedade/coronavirus/alem-da-covid...
joe_the_user 6y ago
Your sheet is interesting. Looking at it, the IFR varies from 1.66% down to .11% and the 0.11% is for the Santa Clara, which many considered rather suspect. The 1.66%, otoh, seems reasonably in line or at least compatible with what's been observed in Korea and elsewhere. Given age is going to skew things a good deal, it seems like a picture is emerging but not that new a picture. An IFR of even 1% is pretty bad, especially given these statistics show how infectious this virus is.
- gst 6y ago
  The numbers that are quoted for Austria (which has the listed IFR of 1.66% in the document) weren't obtained via antibody tests, but via PCR tests. Here's a better source than the one linked in the document: https://www.sora.at/uploads/media/Austria_COVID-19_Prevalence_BMBWF_SORA_20200410_EN_Version.pdf https://www.sora.at/uploads/media/Austria_COVID-19_Prevalenc... Most of those tests were done on April 4th and 5th which was 3 weeks after Austria started relatively strict lockdown measures, which also impacts that number, as this will result in the test to find an even lower number of positive people.
- newacct583 6y ago
  > An IFR of even 1% is pretty bad To be clear: it would be the most dangerous general epidemic disease since the advent of vaccination, and by a significant amount. You need to go back to measles and polio to find general population outbreaks that were more lethal.
  - rubber_duck 6y ago
    A very important thing is that % is concentrated among old and people with preexisting health conditions. Not every death is the same - a 80 year old with weak immune system could have lived 5 years longer without corona, but a healthy 20 year-old dying from cytokine storm caused by influenza has lost potentially 60 years of healthy life - the loss is much worse.
    - pbhjpbhj 6y ago
jMyles 6y ago
Here's a similar spreadsheet, with less detail but more studies: https://www.reddit.com/r/LockdownSkepticism/comments/g6eqtt/collection_of_pcr_and_serological_studies/ https://www.reddit.com/r/LockdownSkepticism/comments/g6eqtt/... It may be helpful to you. Thank you for tracking these metrics.
bagacrap 6y ago
you have the prevalence for NY state but the row is called NY city. The NYC prevalence was 21.2
analog31 6y ago
It's interesting that this jibes with conventional wisdom. Since nearly the very beginning of when people started looking at graphs of case counts, my colleagues and acquaintances (mostly scientists) assumed a ratio of roughly 10x. Folks would look at any graph and automatically bump it up by an order of magnitude.

bb2018 6y ago

I think antibody tests will soon become more useful for tracking disease progression in a population than the viral tests. The collection methods may skew things but they still are much more close to a random sample than the viral tests which have lots of issues with test shortages and people unable to get tested (or not wanting to go to the hospital with mild symptoms).

garmaine 6y ago
They have huge false positive rates though.
- taeric 6y ago
  Do we know the rates already?
  - amluto 6y ago
    There’s preliminary data: https://covidtestingproject.org/ https://covidtestingproject.org/
- deleted 6y ago
  [deleted]
- danielfoster 6y ago
  And false negatives.
  - pedrosorio 6y ago
    If the rate of false positives and false negatives is the same but the true prevalence is << 0.5, then you will overestimate the number of positives.
- brm 6y ago
  Only until the point that they don't. Unfortunately that requires a lot more people to have had covid...
- ptrincr 6y ago
  Not sure why you are being downvoted: https://www.oxfordbiosystems.com/COVID-19-Rapid-test https://www.oxfordbiosystems.com/COVID-19-Rapid-test "In order to test the detection sensitivity and specificity of the COVID-19 IgG-IgM combined antibody test, blood samples were collected from COVID-19 patients from multiple hospitals and Chinese CDC laboratories. The tests were done separately at each site. A total of 525 cases were tested: 397 (positive) clinically confirmed (including PCR test) SARS-CoV-2-infected patients and 128 non- SARS-CoV-2-infected patients (128 negative). The testing results of vein blood without viral inactivation were summarized in the Table 1. Of the 397 blood samples from SARS-CoV-2-infected patients, 352 tested positive, resulting in a sensitivity of 88.66%. Twelve of the blood samples from the 128 non-SARS-CoV-2 infection patients tested positive, generating a specificity of 90.63%." That gives us 62% false positive ratio according to (where a study finds the prevalence to be 6% of subjects using the test): http://vassarstats.net/clin2.html http://vassarstats.net/clin2.html In some cases we have research being carried out with such low positive results that they can entirely be accounted for by the low specificity. So for example if you took samples from 100 people, based on 90% specificity, even if everyone had never had corona, 10 could be found positive. Credit to this post: https://old.reddit.com/r/COVID19/comments/g7f373/second_round_of_covid19_community_testing/foh3b3p/ https://old.reddit.com/r/COVID19/comments/g7f373/second_roun... However it should be noted the article in question for this submission does not mention the type of test used.
  - raverbashing 6y ago
    I wonder what's the process through which false positives happen in this case. Previous infection by milder Coronaviruses? Edit: I'm looking at the reddit post but I have a lot of reservations with the "prevalence 0.06", unless we'll use the test to test absolutely everybody and not only people who are suspect. Has that calculator been validated as well? If the test was 12 false positives in 128 negatives, how come they can claim the false positive rate is 60%?
    - ptrincr 6y ago
      Apologies for the way this was linked to. The 6% is from this study: https://www.miamidade.gov/releases/2020-04-24-sample-testing-results.asp https://www.miamidade.gov/releases/2020-04-24-sample-testing... "Our data from this week and last tell a very similar story. In both weeks, 6% of participants tested positive for COVID-19 antibodies, which equates to 165,000 Miami-Dade County residents" That is what the commentator is referring to in the linked post. So if you plug their own figures into the calculator: Sensitivity .8866 Specificity .9063 and a Prevalence of .06 based on the study, you get the 62% false positive rate. As the prevalence increases, as with the NYC study which found the positive rate to be 21% (prevalence), the false positive rate decreases, down to 28% of the NYC study.
- deleted 6y ago
  [deleted]
- timr 6y ago
  There are many different tests, from different manufacturers. Some of the tests have higher false-positive rates than others. Some have higher false-negative rates. Even a survey with an imperfect test can be designed to yield reliable data.

buboard 6y ago

i don't understand the value of this multiplier. Case statistics are not an official census, it's incidental , depending on the criteria with which each region makes tests. Case numbers are unimportant, it's the total infections and consequent deaths that matter

SpicyLemonZest 6y ago
You're probably right that they shouldn't be, but case statistics are regularly treated as an official census. I've seen many news articles in the vein that suchandsuch country is handling it better or worse because of their case numbers, or statistics like CFR that are computed from official case numbers.
- buboard 6y ago
  while the absolute value doesn't matter, its time course is mostly representative , because countries rarely change strategies wrt testing.
  - SpicyLemonZest 6y ago
    That just doesn't seem true. In the US testing strategies are rapidly changing, since health officials indicate this will be essential to safely removing restrictions.
analog31 6y ago
One reason is that early on when the curves looked pretty much exponential, folks were trying to pin down how bad it could get, and when. This was when the worst case scenario was for nearly the entire population to get it. I'm not quite sure we're completely out of that woods yet.

skybrian 6y ago

I wonder which tests they used? A recent study is finding that some antibody tests are much better than others when it comes to false positives: https://tildes.net/~health.coronavirus/o6a/coronavirus_antibody_tests_can_you_trust_the_results https://tildes.net/~health.coronavirus/o6a/coronavirus_antib...

marcell 6y ago
I know that the California studies used the same test kit. It had 2 false positives out of 371 samples of pre-covid19 cases, and has a 10-20% false negative rate. Because the case numbers are so small, the false positives can skew things quite a bit. I took the midpoints of their 95% confidence intervals in the spreadsheet. I don't know what tests the other studies used.

DanBC 6y ago

What numbers are you using for sensitivity and specificity?

marcell 6y ago
It depends on the study, I've been using their reported confidence intervals. The two California studies (Santa Clara and Los Angeles) used the same kit, which has 2 false positives of 371 tests, and 10-20% false negative rate.
- nodamage 6y ago
  It's worth noting that is the manufacturer claim but has not held up to independent validation. Specifically, the Premier Biotech/Hangzhou Biotest Biotech test was validated by a Chinese provincial CDC and found 4 false positives out of 150. [1] It was also validated by the COVID-19 Testing project and found 3 false positives out of 108. [2] The Biomedomics test used in the Miami Dade study was also validated by the COVID-19 Testing Project and found 14 false positives out of 107. [2] Hence I would recommend taking the results of the California and Florida studies with a huge grain of salt as the prevalence rates they found were within the false positive rates of the tests used. [1] https://imgcdn.mckesson.com/CumulusWeb/Click_and_learn/COVID19_CDC_Evaluation_Report.pdf https://imgcdn.mckesson.com/CumulusWeb/Click_and_learn/COVID... [2] https://covidtestingproject.org/ https://covidtestingproject.org/

cies 6y ago

I did similar calculations, and found the institutions in charge give us very unreliable data. The term "corona case" is very, very, ambiguous and cannot the understood as such without a detailed explanation on how the counting was done. Thanks for sharing. I found the peek in all-case mortality also very interesting, because that way counting is much more unambiguous: dead is dead. They showed a clear diversion from the "average" in recent weeks, but... they did not show the stdev for the averages. Finally I found a chart that shows that "outliers" are not uncommon. https://imgur.com/IPNiXRe https://imgur.com/IPNiXRe

marcell 6y ago
Yea, I agree that the case counts are very different between regions. I think the more interesting column is the IFR estimate based on the antibody study results, since dead is dead as you say. It is interesting though that the median undercount is converging to ~10-20x. Perhaps the protocols across regions are similar enough that the confirmed case counts are somewhat comparable.
- pbhjpbhj 6y ago
  It seems possible to me that there are variations, mutations, in different regions that might account for some of the variance in apparent R value and such?
bobongo 6y ago
> because that way counting is much more unambiguous: dead is dead Unrealistically low death stats coming from Turkey compared to cases easily refute that argument. Dead is dead, unless the state finds a way to claim that it was not a COVID19 dead.
- gaius_baltar 6y ago
  > Dead is dead, unless the state finds a way to claim that it was not a COVID19 dead. It's just a matter of demanding tests to declare as a COVID death and do not providing enough tests. Brazil, for example, has an artificially low count of cases due to the lack of tests and a similarly low number of deaths. However, cases of death by "pneumonia", generic types of SARS and "unexplained respiratory diseases" skyrocketed: https://oglobo.globo.com/sociedade/coronavirus/alem-da-covid-19-brasil-tem-outras-2771-mortes-por-problemas-respiratorios-sem-explicacao-24389276 https://oglobo.globo.com/sociedade/coronavirus/alem-da-covid...

joe_the_user 6y ago

Your sheet is interesting. Looking at it, the IFR varies from 1.66% down to .11% and the 0.11% is for the Santa Clara, which many considered rather suspect. The 1.66%, otoh, seems reasonably in line or at least compatible with what's been observed in Korea and elsewhere. Given age is going to skew things a good deal, it seems like a picture is emerging but not that new a picture. An IFR of even 1% is pretty bad, especially given these statistics show how infectious this virus is.

gst 6y ago
The numbers that are quoted for Austria (which has the listed IFR of 1.66% in the document) weren't obtained via antibody tests, but via PCR tests. Here's a better source than the one linked in the document: https://www.sora.at/uploads/media/Austria_COVID-19_Prevalence_BMBWF_SORA_20200410_EN_Version.pdf https://www.sora.at/uploads/media/Austria_COVID-19_Prevalenc... Most of those tests were done on April 4th and 5th which was 3 weeks after Austria started relatively strict lockdown measures, which also impacts that number, as this will result in the test to find an even lower number of positive people.
newacct583 6y ago
> An IFR of even 1% is pretty bad To be clear: it would be the most dangerous general epidemic disease since the advent of vaccination, and by a significant amount. You need to go back to measles and polio to find general population outbreaks that were more lethal.
- rubber_duck 6y ago
  A very important thing is that % is concentrated among old and people with preexisting health conditions. Not every death is the same - a 80 year old with weak immune system could have lived 5 years longer without corona, but a healthy 20 year-old dying from cytokine storm caused by influenza has lost potentially 60 years of healthy life - the loss is much worse.
  - pbhjpbhj 6y ago

jMyles 6y ago

Here's a similar spreadsheet, with less detail but more studies: https://www.reddit.com/r/LockdownSkepticism/comments/g6eqtt/collection_of_pcr_and_serological_studies/ https://www.reddit.com/r/LockdownSkepticism/comments/g6eqtt/... It may be helpful to you. Thank you for tracking these metrics.

bagacrap 6y ago

you have the prevalence for NY state but the row is called NY city. The NYC prevalence was 21.2

analog31 6y ago

It's interesting that this jibes with conventional wisdom. Since nearly the very beginning of when people started looking at graphs of case counts, my colleagues and acquaintances (mostly scientists) assumed a ratio of roughly 10x. Folks would look at any graph and automatically bump it up by an order of magnitude.