4 ms·

Do you have any links handy discussing the issues? I have not come across anything like that in my reading and would like to read more on the critiques.

mikeyouse 6y ago
A lot of the discussion is happening on twitter. One such thread: https://twitter.com/wfithian/status/1252692357788479488 https://twitter.com/wfithian/status/1252692357788479488 > I have been corresponding with the authors of the well-known Santa Clara County COVID-19 preprint, and I am alarmed at their sloppy behavior. The confidence interval calculation in their preprint made demonstrable math errors - 'not' just questionable methodological choices. .. > The errors are not debatable and can be seen in these two screenshots of the supplement: 0.0034, the standard error meant to measure uncertainty about prevalence pi, is not the square root of 0.039, and the variance of a binomial estimate of proportion depends on the sample size. Another critique: https://twitter.com/jjcherian/status/1251272333177880576 https://twitter.com/jjcherian/status/1251272333177880576 > Ok, so what's wrong with the confidence intervals in this preprint? Well they publish a confidence interval on the specificity of the test that runs between 98.3% and 99.9%, but only 1.5% of all the tests came back positive! > That means that if the true specificity of the test lies somewhere close to 98.3%, nearly all of the positive results can be explained away as false positives (and we know next to nothing about the true prevalence of COVID-19 in Santa Clara County) > They report a 95% confidence interval for the prevalence of COVID-19 in Santa Clara County that runs from 2.01% to 3.49% though! That seems oddly narrow, given that they have already shown that it is within the realm of possibility that the data collected are all false positives!
rallison 6y ago
Sure! Andrew Gelman (Stats at Columbia) had a commonly shared piece: https://statmodeling.stat.columbia.edu/2020/04/19/fatal-flaws-in-stanford-study-of-coronavirus-prevalence/ https://statmodeling.stat.columbia.edu/2020/04/19/fatal-flaw... Also a good dive into the issues: https://medium.com/@balajis/peer-review-of-covid-19-antibody-seroprevalence-in-santa-clara-county-california-1f6382258c25 https://medium.com/@balajis/peer-review-of-covid-19-antibody... Mercury News also had a good article covering a lot of this: https://www.mercurynews.com/2020/04/20/feud-over-stanford-coronavirus-study-the-authors-owe-us-all-an-apology/ https://www.mercurynews.com/2020/04/20/feud-over-stanford-co... And yes, lots of twitter discussions from folks in the field, e.g. Natalie Dean of University of Florida https://twitter.com/nataliexdean/status/1251309217215942656 https://twitter.com/nataliexdean/status/1251309217215942656 and Trevor Bedford (Fred Hutchinson) https://twitter.com/trvrb/status/1251332447691628545 https://twitter.com/trvrb/status/1251332447691628545 and others.
- timr 6y ago
  The most interesting thing (to me) about the Gelman page is that by the PPPS, he's hedging all of his most significant criticisms: "The data as reported are also consistent with infection rates of 2% or 4%. Indeed, as I wrote above, 3% seems like a plausible number. As I wrote above, “I’m not saying that the claims in the above-linked paper are wrong,” and I’m certainly not saying we should take our skepticism in their specific claims and use that as evidence in favor of a null hypothesis. I think we just need to accept some uncertainty here. The Bendavid et al. study is problematic if it is taken as strong evidence for those particular estimates, but it’s valuable if it’s considered as one piece of information that’s part of a big picture that remains uncertain. When I wrote that the authors of the article owe us all an apology, I didn’t mean they owed us an apology for doing the study, I meant they owed us an apology for avoidable errors in the statistical analysis that led to overconfident claims. But, again, let’s not make the opposite mistake of using uncertainty as a way to affirm a null hypothesis." The twitterthink reaction to this study has been vicious, mostly based on amateur re-hashes of the Gelman critique, which even Gelman himself doesn't really believe.
  - Karrot_Kream 6y ago
    The study pre-print is published and some of the numbers are publicly available, we don't need to play a game of revelations here between one person and another, or incorporate Twitter users into the mix. (I didn't even realize this was being criticized over Twitter, as I don't really use the service.) Gelman's critique is quite substantive, and commenters on Gelman's post have created Bayesian analyses which incorporate the uncertainty from test sensitivity and specificity. When I made one in PyMC3 (which lined up with a commenter's approach with PyStan), the 97% CI for the prevalence based on the non-poststratified data I got had the prevalence between (-0.3%, 1.7%). What does that mean? The test just isn't certain enough to allow us to make any conclusions, not that the null hypothesis is correct or that we can reject the null hypothesis. There's nothing wrong with performing the study. Indeed, the publishing of the study allows us to have these vigorous debates about methods and informs future trials from being more exact and not suffering from the same problems as previous studies. But trying to extrapolate a conclusion for something as important as COVID based on studies with extremely high uncertainty is highly irresponsible. Sometimes we have to accept that coming up with statistically significant conclusions is difficult.
dboreham 6y ago
Mainstream media even has articles e.g. https://www.theguardian.com/world/2020/apr/23/coronavirus-antibody-studies-california-stanford https://www.theguardian.com/world/2020/apr/23/coronavirus-an...

mikeyouse 6y ago

A lot of the discussion is happening on twitter. One such thread: https://twitter.com/wfithian/status/1252692357788479488 https://twitter.com/wfithian/status/1252692357788479488 > I have been corresponding with the authors of the well-known Santa Clara County COVID-19 preprint, and I am alarmed at their sloppy behavior. The confidence interval calculation in their preprint made demonstrable math errors - 'not' just questionable methodological choices. .. > The errors are not debatable and can be seen in these two screenshots of the supplement: 0.0034, the standard error meant to measure uncertainty about prevalence pi, is not the square root of 0.039, and the variance of a binomial estimate of proportion depends on the sample size. Another critique: https://twitter.com/jjcherian/status/1251272333177880576 https://twitter.com/jjcherian/status/1251272333177880576 > Ok, so what's wrong with the confidence intervals in this preprint? Well they publish a confidence interval on the specificity of the test that runs between 98.3% and 99.9%, but only 1.5% of all the tests came back positive! > That means that if the true specificity of the test lies somewhere close to 98.3%, nearly all of the positive results can be explained away as false positives (and we know next to nothing about the true prevalence of COVID-19 in Santa Clara County) > They report a 95% confidence interval for the prevalence of COVID-19 in Santa Clara County that runs from 2.01% to 3.49% though! That seems oddly narrow, given that they have already shown that it is within the realm of possibility that the data collected are all false positives!

rallison 6y ago

Sure! Andrew Gelman (Stats at Columbia) had a commonly shared piece: https://statmodeling.stat.columbia.edu/2020/04/19/fatal-flaws-in-stanford-study-of-coronavirus-prevalence/ https://statmodeling.stat.columbia.edu/2020/04/19/fatal-flaw... Also a good dive into the issues: https://medium.com/@balajis/peer-review-of-covid-19-antibody-seroprevalence-in-santa-clara-county-california-1f6382258c25 https://medium.com/@balajis/peer-review-of-covid-19-antibody... Mercury News also had a good article covering a lot of this: https://www.mercurynews.com/2020/04/20/feud-over-stanford-coronavirus-study-the-authors-owe-us-all-an-apology/ https://www.mercurynews.com/2020/04/20/feud-over-stanford-co... And yes, lots of twitter discussions from folks in the field, e.g. Natalie Dean of University of Florida https://twitter.com/nataliexdean/status/1251309217215942656 https://twitter.com/nataliexdean/status/1251309217215942656 and Trevor Bedford (Fred Hutchinson) https://twitter.com/trvrb/status/1251332447691628545 https://twitter.com/trvrb/status/1251332447691628545 and others.

timr 6y ago
The most interesting thing (to me) about the Gelman page is that by the PPPS, he's hedging all of his most significant criticisms: "The data as reported are also consistent with infection rates of 2% or 4%. Indeed, as I wrote above, 3% seems like a plausible number. As I wrote above, “I’m not saying that the claims in the above-linked paper are wrong,” and I’m certainly not saying we should take our skepticism in their specific claims and use that as evidence in favor of a null hypothesis. I think we just need to accept some uncertainty here. The Bendavid et al. study is problematic if it is taken as strong evidence for those particular estimates, but it’s valuable if it’s considered as one piece of information that’s part of a big picture that remains uncertain. When I wrote that the authors of the article owe us all an apology, I didn’t mean they owed us an apology for doing the study, I meant they owed us an apology for avoidable errors in the statistical analysis that led to overconfident claims. But, again, let’s not make the opposite mistake of using uncertainty as a way to affirm a null hypothesis." The twitterthink reaction to this study has been vicious, mostly based on amateur re-hashes of the Gelman critique, which even Gelman himself doesn't really believe.
- Karrot_Kream 6y ago
  The study pre-print is published and some of the numbers are publicly available, we don't need to play a game of revelations here between one person and another, or incorporate Twitter users into the mix. (I didn't even realize this was being criticized over Twitter, as I don't really use the service.) Gelman's critique is quite substantive, and commenters on Gelman's post have created Bayesian analyses which incorporate the uncertainty from test sensitivity and specificity. When I made one in PyMC3 (which lined up with a commenter's approach with PyStan), the 97% CI for the prevalence based on the non-poststratified data I got had the prevalence between (-0.3%, 1.7%). What does that mean? The test just isn't certain enough to allow us to make any conclusions, not that the null hypothesis is correct or that we can reject the null hypothesis. There's nothing wrong with performing the study. Indeed, the publishing of the study allows us to have these vigorous debates about methods and informs future trials from being more exact and not suffering from the same problems as previous studies. But trying to extrapolate a conclusion for something as important as COVID based on studies with extremely high uncertainty is highly irresponsible. Sometimes we have to accept that coming up with statistically significant conclusions is difficult.

dboreham 6y ago

Mainstream media even has articles e.g. https://www.theguardian.com/world/2020/apr/23/coronavirus-antibody-studies-california-stanford https://www.theguardian.com/world/2020/apr/23/coronavirus-an...