13 ms·

Concerns with that Stanford antibody study of coronavirus prevalence

226 points by benchtobedside 6y ago

tptacek 6y ago
The original study was on the HN front page twice; biggest thread: https://news.ycombinator.com/item?id=22899272 https://news.ycombinator.com/item?id=22899272 This analysis is pretty damning, and is more credible in context (the topline findings of the original study constituted extraordinary claims, which, if extrapolated, could imply that a majority of all New Yorkers had C19 antibodies). Some of the underlying ideas here are pretty straightforward. For instance, the fact that even with 90+% specificity, if your rate of false positives exceeds the true positives in the population (as can happen even with good tests when the underlying condition is rare, as it is with C19), you're going to have problems.
- melling 6y ago
  Stanford has enough of a reputation that many in the general public will accept the results as proof.
  - tptacek 6y ago
    A fact discussed at some length in this article.
    - deleted 6y ago
      [deleted]
- SpicyLemonZest 6y ago
  Other antibody studies have consistently found similar results to the Stanford one. The Stanford numbers are very high - most have found closer to 20-30x reported cases than 50-85x - but it's virtually guaranteed that a huge fraction New Yorkers have C19 antibodies today. I'd be pretty surprised if it's less than 20%.
  - watwut 6y ago
    I would say that "20-30x reported cases" is not similar to "50-85x reported cases". Those are massively different numbers.
    - SpicyLemonZest 6y ago
      They are! I don't mean to deny that, there's a large discrepancy which Stanford and other researchers in Santa Clara County should work to resolve. But when newspapers and governments report numbers that are off by at least a factor of 20, a factor of 2-4 isn't the most pressing concern.
      - watwut 6y ago
        They report people who tested positive as cases. The testing is quite limited and almost never done for people without symptoms. It is typically done for people with strong symptoms. Existence of asymptomatic infected people is widely known at this point. So I would expect undereporting in cases. So the additional 2-4 factor from study that is supposed to figure out underreporting is actually big deal. Because reported cases where never assumed to be everyone sick (or infected) and I never seen anyone pretend they represent all infected people. Meanwhile, this study presents itself as estimate of immune people in population.
        SpicyLemonZest 6y ago
        I've never seen anyone assert the specific claim "reported cases capture everyone infected". But I've regularly seen people claim that we know it's a very deadly disease because the number of deaths is 1-2% the number of reported cases, or claim that there's some burden of proof which other studies have to give to justify deviating from the reported count.
        kgwgk 6y ago
        The number of deaths is 0.1-0.2% of the population of NYC. An infection fatality rate of 1% is plausible (corresponds to a 10-20% infection rate). The case fatality rate from confirmed cases is 7%, not 1-2%.
nabla9 6y ago
So far the best tests are from Iceland and Italian town Vo. In Iceland over 11% of population is tested, 4% positive. Results indicate that over 50% of infected are asymptomatic. Italian town Vo tested everyone in their village with similar results. Over 50% of those who test positive are completely asymptomatic.
- kgwgk 6y ago
  ... at the time of the test. Or there was a follow up?
  - DennisP 6y ago
    And what were the false positive rates?
    - andrewla 6y ago
      If I could upvote this a million times I would. If 50% of those tested positive are asymptomatic, then a good first-order approximation is that the false positive rate is 2x the actual prevalence, and the results say very little about prevalence that actual hospitalizations don't say. People, and to a large degree the media, are putting a huge emphasis on testing, not always in a way sensitive to the essential difficulty of evaluating test accuracy, especially in epidemiological settings.
      - ghastmaster 6y ago
        Would it be disingenuous to say that false positives in the current case fatality rate would bring the rate down because hospitals are using symptoms combined with PCR for deaths?
        ineedasername 6y ago
        From what I've heard, at least in some areas, a Covid test is often performed only after a (usually very fast) flu test is performed. So a death resulting from a person negative for flu but also false-positive for Covid would seem very unlikely. As would any death of this sort from a person who came up positive. Much more likely would be that false positives cluster among those with mild or no symptoms (with the mild symptoms instead coming from a more mundane infection) If false positives were pushing the published death rate up, it's important to remember there are also deaths among people infected but never tested that would be pushing the rate down. I don't know if they would cancel each other out though. I would however really like to see published data from the CDC about reported pneumonia deaths just prior to the "official" outbreak to see if there was an otherwise unexplained increase.
gnicholas 6y ago
I have wondered about the selection bias issue, and I was hopeful that this writeup would give a good look at this and other potential issues with the study. But when I read it, I was a bit turned off by the author's attitude — which seems to be that he or some of his colleagues should have been consulted by the study authors because they are "statistics experts". He refers to this apparent omission multiple times, and he also seems to think he's dunking on the authors when he references Theranos (and the fact that its advisors came from government/law/military). But this study is completely unrelated to Theranos (though they both involve blood and Stanford). Off-topic comments like these left me wondering if his analysis is a fair critique, or if he has an axe to grind.
- jxramos 6y ago
  could be the drop of blood intersection more than the generic subject of blood.
- paultopia 6y ago
  But he's right though---the authors should have consulted some survey data experts. And the other people he names are, in fact, well-known survey data experts who could have made the work better.
  - buboard 6y ago
    the authors list had plenty of epidemiology data experts
    - ineedasername 6y ago
      That is not the same as survey experts.
      - SpicyLemonZest 6y ago
        There's always going to be some subcategory of experts they didn't talk to. Do medical studies typically include survey experts signing off on the surveys?
gryson 6y ago
In regards to the issue of selection bias, this is what the ad looked like: https://twitter.com/foxjust/status/1251270848075440133 https://twitter.com/foxjust/status/1251270848075440133 Quote from ad: "We are looking for participants to get tested for antibodies to COVID-19." A quote from someone who participated in the study: "I participated in the study because I had been sick the week before and was very curious. In the intake questionnaire they asked if I had recent symptoms. I'm unpleasantly surprised that they seem not to have made an effort to use that data to unbias the study." https://twitter.com/mattmcnaughton/status/1251322235484168192 https://twitter.com/mattmcnaughton/status/125132223548416819... Another: "I was part of this study and that is totally why I signed up! People I talked to who tried to sign up had similar reasons. Lots of subjects at the testing site wearing masks, more than you see at the grocery, more evidence that a lot of us were more conscious about transmission" https://twitter.com/McSalter/status/1251511091294691328 https://twitter.com/McSalter/status/1251511091294691328 It's very hard to deny that selection bias may have played a part here.
- jdavis703 6y ago
  The symptoms of COVID range from a cold to a flu (for those who are ineligible for viral RNA test kits). If the selection is biased, it is biased toward people who felt cold or flu symptoms in the past few weeks, which is going to be a huge amount of people.
  - justicezyx 6y ago
    Nontheless, that's still a highly biased sample of the actual population.
    - bagacrap 6y ago
      "hypochondriacs who think they may have gotten covid" accounts for 100% of Californians
  - MiroF 6y ago
    That's just.. a basic misunderstanding of statistics.
sct202 6y ago
Also of note, Premier Biotech, the 'manufacturer' of the antibody tests in the Stanford study, has been accused of distributing non-FDA approved Chinese antibody tests. https://www.nbcnews.com/health/health-news/unapproved-chinese-coronavirus-antibody-tests-being-used-least-2-states-n1185131 https://www.nbcnews.com/health/health-news/unapproved-chines...
- acqq 6y ago
  More about how the antibody tests aren't doing what one would want to believe they do: https://www.theguardian.com/world/2020/apr/09/uk-government-urged-to-abandon-poor-finger-prick-antibody-tests-coronavirus https://www.theguardian.com/world/2020/apr/09/uk-government-... "None of 3.5m home tests ordered have so far been accurate enough to detect coronavirus immunity" UK got 3.5 million(!) unusable antibody tests. https://www.bmj.com/content/369/bmj.m1449 https://www.bmj.com/content/369/bmj.m1449 "John Newton, Public Health England’s director of health improvement, said:" "A number of companies were offering us these quick antibody tests, and we were hoping that they’d be fit for purpose, but when they got to test, they all worked but were just not good enough to rely on. “The judgment was made [that] it’s worth taking the time to develop a better antibody test before rolling it out, and that is what the current plan is.”" "Newton told the committee that the tests trialled so far had lacked sufficient sensitivity to identify people who had been infected. “We set a clear target for tests to achieve, and none of them frankly were close.”"
- nodamage 6y ago
  It looks like the test used in this study (and the LA study) was actually manufactured by Hangzhou Biotest Biotech Co., Ltd and is not FDA approved. Premier Biotech is simply the US distributor. [1] Given the news reports of Chinese companies shipping tons of faulty tests this does raise a serious question as to the reliability of this data. [2] The manual for the test also indicates it will produce a positive test result for other coronaviruses, which seems like a huge red flag. [3] > Positive results may be due to past or present infection with non-SARS-CoV-2 coronavirus strains, such as coronavirus HKU1, NL63, OC43, or 229E. [1] http://en.biotests.com.cn/newsitem/278470281 http://en.biotests.com.cn/newsitem/278470281 [2] https://www.nytimes.com/2020/04/19/us/coronavirus-antibody-tests.html https://www.nytimes.com/2020/04/19/us/coronavirus-antibody-t... [3] https://imgcdn.mckesson.com/CumulusWeb/Click_and_learn/Premier_Biotech_COVID19_Package_Insert.pdf https://imgcdn.mckesson.com/CumulusWeb/Click_and_learn/Premi...
deburo 6y ago
The comments below the article are also incredibly interesting. I'll now wait on feedback from the authors to the concerns expressed here. But also, the focus will be on many more serology studies in the coming months. Looking forward to their results.
mxcrossb 6y ago
> But we didn’t then write up a damn preprint and set the publicity machine into action. The latter part is crucial here. I don’t think you should have to apologize about mistakes in a preprint. Papers get improved by the review process. And most of us just upload them to get around journal paywalls, or to make it easier to share with our colleagues what we are working on. But when a University hears that you’ve got a result on a hot topic, dollar signs light up in their eyes, and they go to work. Scientist beware. I hope we can find a balance where scientists don’t rush something out just because it’s a hot topic, yet are also not paralyzed from working on something because of the dangers of the spotlight.
- sct202 6y ago
  The lead professor (Eran Bendavid) in charge of this study wrote an op-ed a month ago in the WSJ (with another contributor Jay Bhattacharya to this paper) about how we shouldn't shut down over coronavirus: https://www.wsj.com/articles/is-the-coronavirus-as-deadly-as-they-say-11585088464 https://www.wsj.com/articles/is-the-coronavirus-as-deadly-as... I would be very surprised if they weren't aware of what they were doing by releasing their pre-print.
  - mxcrossb 6y ago
    Thanks I was unaware of this. It looks like I was wrong.
strangeloops85 6y ago
Another odd thing: One of the co-authors of the study, Andrew Bogan seems to manage a hedge fund, and then published a WSJ op-ed on the study. He didn't note in the op-ed that he was involved in the study (and a co-author no less!). A lot of conflict of interest potential here. These authors seem to have an agenda and end-result in mind, no matter the data. Edit: here's the WSJ op-ed (paywalled): https://www.wsj.com/articles/new-data-suggest-the-coronavirus-isnt-as-deadly-as-we-thought-11587155298 https://www.wsj.com/articles/new-data-suggest-the-coronaviru... The more I think about this, the more outrageous I find this. It's a form of astroturfing to advance the beliefs of the authors of the study. (Note that the senior author Bhattacharya and of course Ionnadis were advancing this theory before data collection began. This means their analysis deserves even more scrutiny)
- pacala 6y ago
  They made a hypothesis and conducted a study to validate it. Perhaps they are wrong. Would be far from the first time studies are proven wrong. At least they did something. Make a better study. Make a hundred better studies. Prove them wrong!
  - fao_ 6y ago
    I think the user is referring to the conflict of interest and the lack of interest in proper ethical conduct by one of the co-authors. If they won't follow generally accepted ethical guidelines, what other rules will they ignore or bend?
    - pacala 6y ago
      The conflict of interest part is fair. The "Note that the senior author Bhattacharya and of course Ionnadis were advancing this theory before data collection began" is not. That's how science is done: you advance a theory, then you conduct experiments to validate / invalidate the theory.
      - strangeloops85 6y ago
HandsomeMD 6y ago
I get this study may be flawed, but there are several upcoming studies showing similar findings in respect of likely lower CFR than expected. The issue is, with what magnitude
pacala 6y ago
The OP is, for lack of better words, so academic. He wants an apology? OK, the study has flaws X, Y and Z. How about propose and conduct a better study? ASAP? Throw darts on a map if you have to. There are millions of people kicked out of a job. People defer medical procedures indefinitely. Kids skipping school for months on end. We will, sooner or later, run out of basic necessities as well. The world doesn't run on money or theories. It runs on us, real people, shuffling our hands and turning sun and soil into food and heat and clothing. Right now we are grounded at home. This can't go on forever. We are running against the clock. Do something about it!
- DangerousPie 6y ago
  The OP wants people to stop spreading misinformation.
  - pacala 6y ago
    We are already drowning in misinformation. We have no clue how deadly this disease is, and yet we are using what appears to be [gross?] overestimations to drive public policy to the tune of trillions of dollars printed with a flick of a pen. Produce and spread good information instead.
    - mcphage 6y ago
      > Produce and spread good information instead. Also: identify bad information and seek to stop it from spreading! That’s what this article is all about.
      - pacala 6y ago
        January. February. March. April. It's been almost 4 months since we know about Covid-19. From the rock I'm living under, it appears that the scientific community at large has produced remarkably little reliable data. Where are the epidemiological studies? Producing reliable data is the highest priority.
tbenst 6y ago
I think the criticism in this thread and elsewhere is a bit too harsh. It’s by no means a perfect study, nor the last word, but hopefully will motivate further studies. I volunteered on this study and talked with hundreds of the participants, at least 200 and possibly as many as 400. Two reported previous COVID symptoms, unprompted. The bigger problem was socioeconomic bias. Judging from number of Tesla’s, Audi’s, and Lamborghis, we also skewed affluent. Against the study instructions, several participants (driving the nicest cars I might add) registered both adults and tested two children. In general, these zip codes had a lower rate of infection. It’s very hard to understand which way this study is biased, and a recruiting strategy based on grocery stores might be more effective, but difficult to get zip code balance There has been additional validation since this preprint was posted and now there’s 118 known-negative samples that have been tested. Specificity remains at 100% for these samples. An updated version will be up soon on medrxiv.
- jariel 6y ago
  "The bigger problem was" ... conflict of interest.
  - tbenst 6y ago
    This strikes me as classic, Reddit-style conspiracy theory. I have no idea who Andrew Bogan is, but he didn’t play a major role in the execution of the study. I’m sure on the paper’s published the contribution section will specify what he did. Remember: this is a preprint!
    - tempsy 6y ago
      I hate that the word “conspiracy” is now lobbed at others to immediately invalidate what they have to say, even with evidence. Did you skim through Bogan’s op-ed? He references the study without ever mentioning once that he played a role in it (no matter how small it was). You don’t need a tin foil hat to ask yourself what incentives a hedge fund manager would have in downplaying the severity of the virus by participating in the creation and media distribution of a study that does exactly that.
seemslegit 6y ago
Why is "suggesting that nearly 2% of the population has been infected “under the radar” even remotely shocking ?
- vkou 6y ago
  Because when you claim that 2-3% of the population has been infected in an area where 1/30000 of the population died from the virus, it implies that the virus has a 1/600 infection/death rate. Since ~1/500 New Yorkers have died from this virus, that would imply that 120% of the population of NYC have been infected, and that new NYC cases will drop to zero in two weeks. Which is, obviously, nonsense. One of the numbers here doesn't fit the facts, and it's probably the one that claims that 2-3% of the population of Santa Clara was infected, with the overwhelming majority not having any symptoms. Data from the Diamond Princess points to something between 20% and 50% of infected people being asymptomatic - not 95%.
  - irq11 6y ago
    Try 1 in 600. There are also huge error bars on that ratio, because the denominator (population size of New York), is a rough estimate, at best. In any case, this is nit-picking: the obvious source of error here is trying to cross-apply the IFR from a limited sample in one city to another, completely different city. If you’re only off by a factor of 20%, that’s a pretty strong indication that you’re on to something real. New York is testing a lot more than California, so I expect their ratio of undiscovered cases to be lower. But it’s becoming clearer and clearer that the true IFR for this virus is substantially lower than previously estimated.
    - mcbits 6y ago
      Nobody is estimating that 100% of New York has already been (or even will be) infected, so something is off by much more than 20%. With an optimistic, but at least plausible, estimate that 40% of NYC have been infected, then the IFR is around 0.4% which actually is within 20% of the 0.5% conservative guess that people have been making.
snapetom 6y ago
As flawed as this is, it's in line with other studies around the world. You can nitpick and critique each one for something, but we now have a whole body of evidence using different techniques and different methods that are all stating the number of cases is vastly undercounted, and the IFR is under 1%. Scotland: https://www.medrxiv.org/content/10.1101/2020.04.13.20060467v1.full.pdf https://www.medrxiv.org/content/10.1101/2020.04.13.20060467v... NYC Pregnant Women: https://www.nejm.org/doi/full/10.1056/NEJMc2009316?query=C19&cid=DM90482_NEJM_COVID-19_Newsletter&bid=186123144 https://www.nejm.org/doi/full/10.1056/NEJMc2009316?query=C19... Finland: https://thl.fi/en/web/thlfi-en/-/number-of-people-with-coronavirus-infections-may-be-dozens-of-times-higher-than-the-number-of-confirmed-cases https://thl.fi/en/web/thlfi-en/-/number-of-people-with-coron... Germany: https://www.land.nrw/sites/default/files/asset/document/zwischenergebnis_covid19_case_study_gangelt_0.pdf https://www.land.nrw/sites/default/files/asset/document/zwis... Chelsea, Mass.: https://www.bostonglobe.com/2020/04/17/business/nearly-third-200-blood-samples-taken-chelsea-show-exposure-coronavirus/ https://www.bostonglobe.com/2020/04/17/business/nearly-third...
- nsrivast 6y ago
  The critique is the same for all papers and systematically impacts all estimates, so seeing more consistent results doesn't invalidate the criticism. That said, it's quite possible the number of cases is undercounted! Someone should do a meta-analysis to obtain a better estimate, given the testing uncertainty.
- duckMuppet 6y ago
  Please stop pointing out articles that question certain bias.. Anything that questions climate, overpopulation, our obvious pandemics must be accepted at full face value. Now that we've established what is not ok to question, please accept the initial study from imperial college London unquestioningly.. Also, you missed one from Los Angeles: https://www.latimes.com/california/story/2020-04-20/coronavirus-serology-testing-la-county https://www.latimes.com/california/story/2020-04-20/coronavi...
- wahern 6y ago
  No professional was ever under the delusion that the IFR was >= 1%. Perhaps you meant CFR[1], which by definition does not reflect total number of infections. See, e.g., https://en.wikipedia.org/wiki/Case_fatality_rate https://en.wikipedia.org/wiki/Case_fatality_rate [1] Where 'C' means "case" or "confirmed", which in the context of medical terminology and despite some ambiguity categorically is not synonymous with infected.
eanzenberg 6y ago
Its wishful thinking to overlook the mountain of evidence building for an IFR ~ 0.1-0.5% of this disease.
- guscost 6y ago
  "Wishful"? You must mean those of us who are still holding out for less than 0.1%...
Mikeb85 6y ago
Every study on coronavirus antibodies that's released gets panned here, yet every single one of them show strong enough evidence that infection is more widespread and death rates lower than widely assumed. Even if every one of them is flawed, all the information taken as an aggregate paints a picture. In my province (Alberta, Canada) the health authority has recently expanded tested and as a result there's more confirmed cases and a lower death rate. Other health authorities in the country have strongly suggested infection rates are much higher than confirmed cases (which lowers the death rate since every single death is being accounted for in our country). So there's concerns with this study, and maybe another, but there's no evidence to counter the conclusion we're seeing again and again.
- blago 6y ago
  I don't think anyone doubts that infections are much more common than the recorded rate - we all know that a significant number of people are asymptomatic, hence never tested and recorded. The problem is that those observations are typically used to conclude that "this is just a bad flu" and advance political demands such as "liberate X" and "immediately reopen the country". The question that truly matters is "which model is less wrong" and the overflowing ICUs in NYC and Europe (that's the bottom line) provide a clear answer - we should err on the side of caution.
  - Mikeb85 6y ago
    > The problem is that those observations are typically used to conclude that "this is just a bad flu" and advance political demands such as "immediately reopen of the country". And why not? Should studies and observations be dismissed just because the result isn't what some people want? >The question that truly matters is "which model is less wrong" and the overflowing ICUs in NYC and Europe No overflowing ICUs here. Emergency rooms are way under capacity. Where I am, we have a population of 4 million. Only 40 ICU visits during the whole pandemic. Only 59 deaths and the majority were individuals over the age of 80. Over half were in nursing homes. And yet there's a vocal group who don't want anything to reopen. Where is the line where we reopen? 25,000 Albertans die every year. 275,000 Canadians die per year. 2.8 million Americans die every year. It's not reasonable to wait until coronavirus is completely eliminated when economic hardship itself is correlated to a higher mortality rate. Edit - just looked at some closer stats for my region: only 3 deaths under the age of 60. Population 4 million. With 3k official cases and likely 30k or more total cases.
fapjacks 6y ago
Frankly I find Andrew Gelman at least as invested in a particular outcome of this political issue as a hedge fund manager.
deleted 6y ago
[deleted]
i_am_proteus 6y ago
Ioannidis warned us in advance that he'd publish bunk. "Why Most Published Research Findings Are False" [0] is his most famous work. [0] https://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.0020124 https://journals.plos.org/plosmedicine/article?id=10.1371/jo...
uoylj 6y ago
Chris Martenson goes through the major criticisms here: https://www.youtube.com/watch?v=R8Pv77R3g1E https://www.youtube.com/watch?v=R8Pv77R3g1E
Kiro 6y ago
In Sweden they did antibody tests on blood givers and found that 11% had antibodies. There was no ad or survey so there couldn't have been any selection bias.
- craydandy 6y ago
  Do you have a link for the study or news article?
  - Kiro 6y ago
    In Swedish: https://www.svt.se/nyheter/inrikes/11-procent-av-stockholmarna-har-antikroppar-mot-covid-19 https://www.svt.se/nyheter/inrikes/11-procent-av-stockholmar...
    - craydandy 6y ago
      Thanks!
- parineum 6y ago
  It's biased towards people who give blood. I have no idea what that bias is or if it's significant but it's definitely possible that it's still biased in some way.
_cs2017_ 6y ago
All these confidence interval discussions (both from the original study and from the critique) have no value besides entertaining professors who have more knowledge of math than common sense. Nothing good will ever come from buzzwords such as "Agresti-Coull 95% interval". Bayesian techniques are not much better since no one will ever agree on the prior. Just treat the study as some super rough point estimate. Adjust for biases such as selection bias if you can. Look at other studies too. Add your personal opinions (e.g., on whether conflicts of interest are relevant here). Complex statistical arguments won't buy you much more than that.
LarvaFX 6y ago
Take your generalisations back to reddit.
dr_faustus 6y ago
Well, all the antibody studies around the globe have several things in common: * relatively small numbers of participants (< 5000 and therefore only dozens of partifipants with postive results) * focusing on relatively small geographical areas * working with antibody test with a high uncertainty regarding the specifity One argument that strongly contradicts the narrative that a huge number of people already are/were SARS-COV-2-positive is the of positive PCR tests in Germany. Germany performs hundreds of thousands of PCR tests per week but still mainly tests people with some symptoms. If SARS-COV-2 were that prevalent, you would expect a large proportion of the tested to be positive but its only 4% by late March [1]. Every expert I heard admits that there is a significant number of undiagnosed cases. But 30x-60x seems to be quite unrealistic if even only 4% of people with symptoms are positive. [1] https://www.zeit.de/wissen/gesundheit/2020-03/coronatests-deutschland-coronavirus-covid-19-who-pandemie https://www.zeit.de/wissen/gesundheit/2020-03/coronatests-de...

The original study was on the HN front page twice; biggest thread: https://news.ycombinator.com/item?id=22899272 https://news.ycombinator.com/item?id=22899272 This analysis is pretty damning, and is more credible in context (the topline findings of the original study constituted extraordinary claims, which, if extrapolated, could imply that a majority of all New Yorkers had C19 antibodies). Some of the underlying ideas here are pretty straightforward. For instance, the fact that even with 90+% specificity, if your rate of false positives exceeds the true positives in the population (as can happen even with good tests when the underlying condition is rare, as it is with C19), you're going to have problems.

melling 6y ago
Stanford has enough of a reputation that many in the general public will accept the results as proof.
- tptacek 6y ago
  A fact discussed at some length in this article.
  - deleted 6y ago
    [deleted]
SpicyLemonZest 6y ago
Other antibody studies have consistently found similar results to the Stanford one. The Stanford numbers are very high - most have found closer to 20-30x reported cases than 50-85x - but it's virtually guaranteed that a huge fraction New Yorkers have C19 antibodies today. I'd be pretty surprised if it's less than 20%.
- watwut 6y ago
  I would say that "20-30x reported cases" is not similar to "50-85x reported cases". Those are massively different numbers.
  - SpicyLemonZest 6y ago
    They are! I don't mean to deny that, there's a large discrepancy which Stanford and other researchers in Santa Clara County should work to resolve. But when newspapers and governments report numbers that are off by at least a factor of 20, a factor of 2-4 isn't the most pressing concern.
    - watwut 6y ago
      They report people who tested positive as cases. The testing is quite limited and almost never done for people without symptoms. It is typically done for people with strong symptoms. Existence of asymptomatic infected people is widely known at this point. So I would expect undereporting in cases. So the additional 2-4 factor from study that is supposed to figure out underreporting is actually big deal. Because reported cases where never assumed to be everyone sick (or infected) and I never seen anyone pretend they represent all infected people. Meanwhile, this study presents itself as estimate of immune people in population.
      - SpicyLemonZest 6y ago
        I've never seen anyone assert the specific claim "reported cases capture everyone infected". But I've regularly seen people claim that we know it's a very deadly disease because the number of deaths is 1-2% the number of reported cases, or claim that there's some burden of proof which other studies have to give to justify deviating from the reported count.
        kgwgk 6y ago
        The number of deaths is 0.1-0.2% of the population of NYC. An infection fatality rate of 1% is plausible (corresponds to a 10-20% infection rate). The case fatality rate from confirmed cases is 7%, not 1-2%.

nabla9 6y ago

So far the best tests are from Iceland and Italian town Vo. In Iceland over 11% of population is tested, 4% positive. Results indicate that over 50% of infected are asymptomatic. Italian town Vo tested everyone in their village with similar results. Over 50% of those who test positive are completely asymptomatic.

kgwgk 6y ago
... at the time of the test. Or there was a follow up?
- DennisP 6y ago
  And what were the false positive rates?
  - andrewla 6y ago
    If I could upvote this a million times I would. If 50% of those tested positive are asymptomatic, then a good first-order approximation is that the false positive rate is 2x the actual prevalence, and the results say very little about prevalence that actual hospitalizations don't say. People, and to a large degree the media, are putting a huge emphasis on testing, not always in a way sensitive to the essential difficulty of evaluating test accuracy, especially in epidemiological settings.
    - ghastmaster 6y ago
      Would it be disingenuous to say that false positives in the current case fatality rate would bring the rate down because hospitals are using symptoms combined with PCR for deaths?
      - ineedasername 6y ago
        From what I've heard, at least in some areas, a Covid test is often performed only after a (usually very fast) flu test is performed. So a death resulting from a person negative for flu but also false-positive for Covid would seem very unlikely. As would any death of this sort from a person who came up positive. Much more likely would be that false positives cluster among those with mild or no symptoms (with the mild symptoms instead coming from a more mundane infection) If false positives were pushing the published death rate up, it's important to remember there are also deaths among people infected but never tested that would be pushing the rate down. I don't know if they would cancel each other out though. I would however really like to see published data from the CDC about reported pneumonia deaths just prior to the "official" outbreak to see if there was an otherwise unexplained increase.

gnicholas 6y ago

I have wondered about the selection bias issue, and I was hopeful that this writeup would give a good look at this and other potential issues with the study. But when I read it, I was a bit turned off by the author's attitude — which seems to be that he or some of his colleagues should have been consulted by the study authors because they are "statistics experts". He refers to this apparent omission multiple times, and he also seems to think he's dunking on the authors when he references Theranos (and the fact that its advisors came from government/law/military). But this study is completely unrelated to Theranos (though they both involve blood and Stanford). Off-topic comments like these left me wondering if his analysis is a fair critique, or if he has an axe to grind.

jxramos 6y ago
could be the drop of blood intersection more than the generic subject of blood.
paultopia 6y ago
But he's right though---the authors should have consulted some survey data experts. And the other people he names are, in fact, well-known survey data experts who could have made the work better.
- buboard 6y ago
  the authors list had plenty of epidemiology data experts
  - ineedasername 6y ago
    That is not the same as survey experts.
    - SpicyLemonZest 6y ago
      There's always going to be some subcategory of experts they didn't talk to. Do medical studies typically include survey experts signing off on the surveys?

gryson 6y ago

In regards to the issue of selection bias, this is what the ad looked like: https://twitter.com/foxjust/status/1251270848075440133 https://twitter.com/foxjust/status/1251270848075440133 Quote from ad: "We are looking for participants to get tested for antibodies to COVID-19." A quote from someone who participated in the study: "I participated in the study because I had been sick the week before and was very curious. In the intake questionnaire they asked if I had recent symptoms. I'm unpleasantly surprised that they seem not to have made an effort to use that data to unbias the study." https://twitter.com/mattmcnaughton/status/1251322235484168192 https://twitter.com/mattmcnaughton/status/125132223548416819... Another: "I was part of this study and that is totally why I signed up! People I talked to who tried to sign up had similar reasons. Lots of subjects at the testing site wearing masks, more than you see at the grocery, more evidence that a lot of us were more conscious about transmission" https://twitter.com/McSalter/status/1251511091294691328 https://twitter.com/McSalter/status/1251511091294691328 It's very hard to deny that selection bias may have played a part here.

jdavis703 6y ago
The symptoms of COVID range from a cold to a flu (for those who are ineligible for viral RNA test kits). If the selection is biased, it is biased toward people who felt cold or flu symptoms in the past few weeks, which is going to be a huge amount of people.
- justicezyx 6y ago
  Nontheless, that's still a highly biased sample of the actual population.
  - bagacrap 6y ago
    "hypochondriacs who think they may have gotten covid" accounts for 100% of Californians
- MiroF 6y ago
  That's just.. a basic misunderstanding of statistics.

sct202 6y ago

Also of note, Premier Biotech, the 'manufacturer' of the antibody tests in the Stanford study, has been accused of distributing non-FDA approved Chinese antibody tests. https://www.nbcnews.com/health/health-news/unapproved-chinese-coronavirus-antibody-tests-being-used-least-2-states-n1185131 https://www.nbcnews.com/health/health-news/unapproved-chines...

acqq 6y ago
More about how the antibody tests aren't doing what one would want to believe they do: https://www.theguardian.com/world/2020/apr/09/uk-government-urged-to-abandon-poor-finger-prick-antibody-tests-coronavirus https://www.theguardian.com/world/2020/apr/09/uk-government-... "None of 3.5m home tests ordered have so far been accurate enough to detect coronavirus immunity" UK got 3.5 million(!) unusable antibody tests. https://www.bmj.com/content/369/bmj.m1449 https://www.bmj.com/content/369/bmj.m1449 "John Newton, Public Health England’s director of health improvement, said:" "A number of companies were offering us these quick antibody tests, and we were hoping that they’d be fit for purpose, but when they got to test, they all worked but were just not good enough to rely on. “The judgment was made [that] it’s worth taking the time to develop a better antibody test before rolling it out, and that is what the current plan is.”" "Newton told the committee that the tests trialled so far had lacked sufficient sensitivity to identify people who had been infected. “We set a clear target for tests to achieve, and none of them frankly were close.”"
nodamage 6y ago
It looks like the test used in this study (and the LA study) was actually manufactured by Hangzhou Biotest Biotech Co., Ltd and is not FDA approved. Premier Biotech is simply the US distributor. [1] Given the news reports of Chinese companies shipping tons of faulty tests this does raise a serious question as to the reliability of this data. [2] The manual for the test also indicates it will produce a positive test result for other coronaviruses, which seems like a huge red flag. [3] > Positive results may be due to past or present infection with non-SARS-CoV-2 coronavirus strains, such as coronavirus HKU1, NL63, OC43, or 229E. [1] http://en.biotests.com.cn/newsitem/278470281 http://en.biotests.com.cn/newsitem/278470281 [2] https://www.nytimes.com/2020/04/19/us/coronavirus-antibody-tests.html https://www.nytimes.com/2020/04/19/us/coronavirus-antibody-t... [3] https://imgcdn.mckesson.com/CumulusWeb/Click_and_learn/Premier_Biotech_COVID19_Package_Insert.pdf https://imgcdn.mckesson.com/CumulusWeb/Click_and_learn/Premi...

deburo 6y ago

The comments below the article are also incredibly interesting. I'll now wait on feedback from the authors to the concerns expressed here. But also, the focus will be on many more serology studies in the coming months. Looking forward to their results.

mxcrossb 6y ago

> But we didn’t then write up a damn preprint and set the publicity machine into action. The latter part is crucial here. I don’t think you should have to apologize about mistakes in a preprint. Papers get improved by the review process. And most of us just upload them to get around journal paywalls, or to make it easier to share with our colleagues what we are working on. But when a University hears that you’ve got a result on a hot topic, dollar signs light up in their eyes, and they go to work. Scientist beware. I hope we can find a balance where scientists don’t rush something out just because it’s a hot topic, yet are also not paralyzed from working on something because of the dangers of the spotlight.

sct202 6y ago
The lead professor (Eran Bendavid) in charge of this study wrote an op-ed a month ago in the WSJ (with another contributor Jay Bhattacharya to this paper) about how we shouldn't shut down over coronavirus: https://www.wsj.com/articles/is-the-coronavirus-as-deadly-as-they-say-11585088464 https://www.wsj.com/articles/is-the-coronavirus-as-deadly-as... I would be very surprised if they weren't aware of what they were doing by releasing their pre-print.
- mxcrossb 6y ago
  Thanks I was unaware of this. It looks like I was wrong.

strangeloops85 6y ago

Another odd thing: One of the co-authors of the study, Andrew Bogan seems to manage a hedge fund, and then published a WSJ op-ed on the study. He didn't note in the op-ed that he was involved in the study (and a co-author no less!). A lot of conflict of interest potential here. These authors seem to have an agenda and end-result in mind, no matter the data. Edit: here's the WSJ op-ed (paywalled): https://www.wsj.com/articles/new-data-suggest-the-coronavirus-isnt-as-deadly-as-we-thought-11587155298 https://www.wsj.com/articles/new-data-suggest-the-coronaviru... The more I think about this, the more outrageous I find this. It's a form of astroturfing to advance the beliefs of the authors of the study. (Note that the senior author Bhattacharya and of course Ionnadis were advancing this theory before data collection began. This means their analysis deserves even more scrutiny)

pacala 6y ago
They made a hypothesis and conducted a study to validate it. Perhaps they are wrong. Would be far from the first time studies are proven wrong. At least they did something. Make a better study. Make a hundred better studies. Prove them wrong!
- fao_ 6y ago
  I think the user is referring to the conflict of interest and the lack of interest in proper ethical conduct by one of the co-authors. If they won't follow generally accepted ethical guidelines, what other rules will they ignore or bend?
  - pacala 6y ago
    The conflict of interest part is fair. The "Note that the senior author Bhattacharya and of course Ionnadis were advancing this theory before data collection began" is not. That's how science is done: you advance a theory, then you conduct experiments to validate / invalidate the theory.
    - strangeloops85 6y ago

HandsomeMD 6y ago

I get this study may be flawed, but there are several upcoming studies showing similar findings in respect of likely lower CFR than expected. The issue is, with what magnitude

pacala 6y ago

The OP is, for lack of better words, so academic. He wants an apology? OK, the study has flaws X, Y and Z. How about propose and conduct a better study? ASAP? Throw darts on a map if you have to. There are millions of people kicked out of a job. People defer medical procedures indefinitely. Kids skipping school for months on end. We will, sooner or later, run out of basic necessities as well. The world doesn't run on money or theories. It runs on us, real people, shuffling our hands and turning sun and soil into food and heat and clothing. Right now we are grounded at home. This can't go on forever. We are running against the clock. Do something about it!

DangerousPie 6y ago
The OP wants people to stop spreading misinformation.
- pacala 6y ago
  We are already drowning in misinformation. We have no clue how deadly this disease is, and yet we are using what appears to be [gross?] overestimations to drive public policy to the tune of trillions of dollars printed with a flick of a pen. Produce and spread good information instead.
  - mcphage 6y ago
    > Produce and spread good information instead. Also: identify bad information and seek to stop it from spreading! That’s what this article is all about.
    - pacala 6y ago
      January. February. March. April. It's been almost 4 months since we know about Covid-19. From the rock I'm living under, it appears that the scientific community at large has produced remarkably little reliable data. Where are the epidemiological studies? Producing reliable data is the highest priority.

tbenst 6y ago

I think the criticism in this thread and elsewhere is a bit too harsh. It’s by no means a perfect study, nor the last word, but hopefully will motivate further studies. I volunteered on this study and talked with hundreds of the participants, at least 200 and possibly as many as 400. Two reported previous COVID symptoms, unprompted. The bigger problem was socioeconomic bias. Judging from number of Tesla’s, Audi’s, and Lamborghis, we also skewed affluent. Against the study instructions, several participants (driving the nicest cars I might add) registered both adults and tested two children. In general, these zip codes had a lower rate of infection. It’s very hard to understand which way this study is biased, and a recruiting strategy based on grocery stores might be more effective, but difficult to get zip code balance There has been additional validation since this preprint was posted and now there’s 118 known-negative samples that have been tested. Specificity remains at 100% for these samples. An updated version will be up soon on medrxiv.

jariel 6y ago
"The bigger problem was" ... conflict of interest.
- tbenst 6y ago
  This strikes me as classic, Reddit-style conspiracy theory. I have no idea who Andrew Bogan is, but he didn’t play a major role in the execution of the study. I’m sure on the paper’s published the contribution section will specify what he did. Remember: this is a preprint!
  - tempsy 6y ago
    I hate that the word “conspiracy” is now lobbed at others to immediately invalidate what they have to say, even with evidence. Did you skim through Bogan’s op-ed? He references the study without ever mentioning once that he played a role in it (no matter how small it was). You don’t need a tin foil hat to ask yourself what incentives a hedge fund manager would have in downplaying the severity of the virus by participating in the creation and media distribution of a study that does exactly that.

seemslegit 6y ago

Why is "suggesting that nearly 2% of the population has been infected “under the radar” even remotely shocking ?

vkou 6y ago
Because when you claim that 2-3% of the population has been infected in an area where 1/30000 of the population died from the virus, it implies that the virus has a 1/600 infection/death rate. Since ~1/500 New Yorkers have died from this virus, that would imply that 120% of the population of NYC have been infected, and that new NYC cases will drop to zero in two weeks. Which is, obviously, nonsense. One of the numbers here doesn't fit the facts, and it's probably the one that claims that 2-3% of the population of Santa Clara was infected, with the overwhelming majority not having any symptoms. Data from the Diamond Princess points to something between 20% and 50% of infected people being asymptomatic - not 95%.
- irq11 6y ago
  Try 1 in 600. There are also huge error bars on that ratio, because the denominator (population size of New York), is a rough estimate, at best. In any case, this is nit-picking: the obvious source of error here is trying to cross-apply the IFR from a limited sample in one city to another, completely different city. If you’re only off by a factor of 20%, that’s a pretty strong indication that you’re on to something real. New York is testing a lot more than California, so I expect their ratio of undiscovered cases to be lower. But it’s becoming clearer and clearer that the true IFR for this virus is substantially lower than previously estimated.
  - mcbits 6y ago
    Nobody is estimating that 100% of New York has already been (or even will be) infected, so something is off by much more than 20%. With an optimistic, but at least plausible, estimate that 40% of NYC have been infected, then the IFR is around 0.4% which actually is within 20% of the 0.5% conservative guess that people have been making.

snapetom 6y ago

As flawed as this is, it's in line with other studies around the world. You can nitpick and critique each one for something, but we now have a whole body of evidence using different techniques and different methods that are all stating the number of cases is vastly undercounted, and the IFR is under 1%. Scotland: https://www.medrxiv.org/content/10.1101/2020.04.13.20060467v1.full.pdf https://www.medrxiv.org/content/10.1101/2020.04.13.20060467v... NYC Pregnant Women: https://www.nejm.org/doi/full/10.1056/NEJMc2009316?query=C19&cid=DM90482_NEJM_COVID-19_Newsletter&bid=186123144 https://www.nejm.org/doi/full/10.1056/NEJMc2009316?query=C19... Finland: https://thl.fi/en/web/thlfi-en/-/number-of-people-with-coronavirus-infections-may-be-dozens-of-times-higher-than-the-number-of-confirmed-cases https://thl.fi/en/web/thlfi-en/-/number-of-people-with-coron... Germany: https://www.land.nrw/sites/default/files/asset/document/zwischenergebnis_covid19_case_study_gangelt_0.pdf https://www.land.nrw/sites/default/files/asset/document/zwis... Chelsea, Mass.: https://www.bostonglobe.com/2020/04/17/business/nearly-third-200-blood-samples-taken-chelsea-show-exposure-coronavirus/ https://www.bostonglobe.com/2020/04/17/business/nearly-third...

nsrivast 6y ago
The critique is the same for all papers and systematically impacts all estimates, so seeing more consistent results doesn't invalidate the criticism. That said, it's quite possible the number of cases is undercounted! Someone should do a meta-analysis to obtain a better estimate, given the testing uncertainty.
duckMuppet 6y ago
Please stop pointing out articles that question certain bias.. Anything that questions climate, overpopulation, our obvious pandemics must be accepted at full face value. Now that we've established what is not ok to question, please accept the initial study from imperial college London unquestioningly.. Also, you missed one from Los Angeles: https://www.latimes.com/california/story/2020-04-20/coronavirus-serology-testing-la-county https://www.latimes.com/california/story/2020-04-20/coronavi...
wahern 6y ago
No professional was ever under the delusion that the IFR was >= 1%. Perhaps you meant CFR[1], which by definition does not reflect total number of infections. See, e.g., https://en.wikipedia.org/wiki/Case_fatality_rate https://en.wikipedia.org/wiki/Case_fatality_rate [1] Where 'C' means "case" or "confirmed", which in the context of medical terminology and despite some ambiguity categorically is not synonymous with infected.

eanzenberg 6y ago

Its wishful thinking to overlook the mountain of evidence building for an IFR ~ 0.1-0.5% of this disease.

guscost 6y ago
"Wishful"? You must mean those of us who are still holding out for less than 0.1%...

Mikeb85 6y ago

Every study on coronavirus antibodies that's released gets panned here, yet every single one of them show strong enough evidence that infection is more widespread and death rates lower than widely assumed. Even if every one of them is flawed, all the information taken as an aggregate paints a picture. In my province (Alberta, Canada) the health authority has recently expanded tested and as a result there's more confirmed cases and a lower death rate. Other health authorities in the country have strongly suggested infection rates are much higher than confirmed cases (which lowers the death rate since every single death is being accounted for in our country). So there's concerns with this study, and maybe another, but there's no evidence to counter the conclusion we're seeing again and again.

blago 6y ago
I don't think anyone doubts that infections are much more common than the recorded rate - we all know that a significant number of people are asymptomatic, hence never tested and recorded. The problem is that those observations are typically used to conclude that "this is just a bad flu" and advance political demands such as "liberate X" and "immediately reopen the country". The question that truly matters is "which model is less wrong" and the overflowing ICUs in NYC and Europe (that's the bottom line) provide a clear answer - we should err on the side of caution.
- Mikeb85 6y ago
  > The problem is that those observations are typically used to conclude that "this is just a bad flu" and advance political demands such as "immediately reopen of the country". And why not? Should studies and observations be dismissed just because the result isn't what some people want? >The question that truly matters is "which model is less wrong" and the overflowing ICUs in NYC and Europe No overflowing ICUs here. Emergency rooms are way under capacity. Where I am, we have a population of 4 million. Only 40 ICU visits during the whole pandemic. Only 59 deaths and the majority were individuals over the age of 80. Over half were in nursing homes. And yet there's a vocal group who don't want anything to reopen. Where is the line where we reopen? 25,000 Albertans die every year. 275,000 Canadians die per year. 2.8 million Americans die every year. It's not reasonable to wait until coronavirus is completely eliminated when economic hardship itself is correlated to a higher mortality rate. Edit - just looked at some closer stats for my region: only 3 deaths under the age of 60. Population 4 million. With 3k official cases and likely 30k or more total cases.

fapjacks 6y ago

Frankly I find Andrew Gelman at least as invested in a particular outcome of this political issue as a hedge fund manager.

deleted 6y ago

[deleted]

i_am_proteus 6y ago

Ioannidis warned us in advance that he'd publish bunk. "Why Most Published Research Findings Are False" [0] is his most famous work. [0] https://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.0020124 https://journals.plos.org/plosmedicine/article?id=10.1371/jo...

uoylj 6y ago

Chris Martenson goes through the major criticisms here: https://www.youtube.com/watch?v=R8Pv77R3g1E https://www.youtube.com/watch?v=R8Pv77R3g1E

Kiro 6y ago

In Sweden they did antibody tests on blood givers and found that 11% had antibodies. There was no ad or survey so there couldn't have been any selection bias.

craydandy 6y ago
Do you have a link for the study or news article?
- Kiro 6y ago
  In Swedish: https://www.svt.se/nyheter/inrikes/11-procent-av-stockholmarna-har-antikroppar-mot-covid-19 https://www.svt.se/nyheter/inrikes/11-procent-av-stockholmar...
  - craydandy 6y ago
    Thanks!
parineum 6y ago
It's biased towards people who give blood. I have no idea what that bias is or if it's significant but it's definitely possible that it's still biased in some way.

_cs2017_ 6y ago

All these confidence interval discussions (both from the original study and from the critique) have no value besides entertaining professors who have more knowledge of math than common sense. Nothing good will ever come from buzzwords such as "Agresti-Coull 95% interval". Bayesian techniques are not much better since no one will ever agree on the prior. Just treat the study as some super rough point estimate. Adjust for biases such as selection bias if you can. Look at other studies too. Add your personal opinions (e.g., on whether conflicts of interest are relevant here). Complex statistical arguments won't buy you much more than that.

LarvaFX 6y ago

Take your generalisations back to reddit.

dr_faustus 6y ago

Well, all the antibody studies around the globe have several things in common: * relatively small numbers of participants (< 5000 and therefore only dozens of partifipants with postive results) * focusing on relatively small geographical areas * working with antibody test with a high uncertainty regarding the specifity One argument that strongly contradicts the narrative that a huge number of people already are/were SARS-COV-2-positive is the of positive PCR tests in Germany. Germany performs hundreds of thousands of PCR tests per week but still mainly tests people with some symptoms. If SARS-COV-2 were that prevalent, you would expect a large proportion of the tested to be positive but its only 4% by late March [1]. Every expert I heard admits that there is a significant number of undiagnosed cases. But 30x-60x seems to be quite unrealistic if even only 4% of people with symptoms are positive. [1] https://www.zeit.de/wissen/gesundheit/2020-03/coronatests-deutschland-coronavirus-covid-19-who-pandemie https://www.zeit.de/wissen/gesundheit/2020-03/coronatests-de...