5 ms·

You've quoted the conclusions, but in the spirit of the STAT article, can you also quote the relevant scientific debate in your article, and what specifically l

You've quoted the conclusions, but in the spirit of the STAT article, can you also quote the relevant scientific debate in your article, and what specifically leads to such bold conclusions (e.g. "owe an apology", "wasted time and effort", "statistical error", "screwed up", etc.)? I skimmed through the article and, because I'm not a statistician, I could not follow what was so blatantly wrong, if we were to charitably assume that the two sets of authors may have different premises or interpretations. Edit: In the spirit of the STAT article (edit: removed extraneous text), I searched for "Ioannidis Gelman" (the latter being the author of your piece) and found: https://www.mercurynews.com/2020/04/20/feud-over-stanford-coronavirus-study-the-authors-owe-us-all-an-apology/ https://www.mercurynews.com/2020/04/20/feud-over-stanford-co... > In response, on Sunday, the Stanford study’s authors said they are planning to soon release a detailed appendix that addresses many of the “constructive comments and suggestions” the team has received. > “This is exactly the way peer-review should work in scientific work, and we are looking forward to engaging with other scholars as we proceed in this important work,” said Dr. Jayanta Bhattacharya, professor of medicine at Stanford University, who along with colleague Dr. Eran Bendavid, also assisted with the USC study. Edit2: I'm not sure if this is the appendix referred to (probably not because the file has an older date in it), but there is a statistical appendix in the cited article: https://www.medrxiv.org/content/medrxiv/suppl/2020/04/17/2020.04.14.20062463.DC1/2020.04.14.20062463-1.pdf https://www.medrxiv.org/content/medrxiv/suppl/2020/04/17/202...

knzhou 6y ago
> what specifically leads to such bold conclusions (e.g. "owe an apology", "wasted time and effort", "statistical error", "screwed up", etc.)? They used a test with a false positive rate of about 2%, and found a positive rate of about 2%. It is impossible to use this information to establish a lower bound on the number infected. However, by screwing up the computation of the confidence interval, they claimed a huge lower bound.
- statquontrarian 6y ago
  Thanks. By the way, the STAT article links to https://undark.org/2020/04/24/john-ioannidis-covid-19-death-rate-critics/ https://undark.org/2020/04/24/john-ioannidis-covid-19-death-... in which Ioannidis is quoted: > “We have been working very hard to address all the comments raised by many colleagues and I am always grateful to receive constructive criticism,” he said. “While we received thousands of congratulations for this work, I cherish far more the comments of the scientists who criticized our work and I thank them deeply for their comments.”
  - knzhou 6y ago
    That's an admirable sentiment, Ionannidis must be such a wonderful guy. However, the damage has already been done. He's already done his media blitz, so no matter what happens next, effectively zero people will hear the correction (or more likely, subsequent retraction). It'll just become yet another piece of zombie information, which will appear in dishonest talking points or gish gallops. The quality of public discourse and the reputation of science will continue its decline, and we'll be talking about the next example of this tomorrow.
    - statquontrarian 6y ago
      If we try to be charitable, Ioannidis was too confident in his research team who made mistakes; for example: > The authors said by email that they used a built-in Stata function and aren't sure themselves how the software used the input weights. I suspect they misapplied that function (too complicated to tweet why) but I don't know Stata well enough to be sure; it seems neither do they. https://twitter.com/wfithian/status/1252692357788479488 https://twitter.com/wfithian/status/1252692357788479488 Then, Ioannidis probably didn't double check the work in depth because of confirmation bias and over-confidence in his protégés.
- statquontrarian 6y ago
  I checked with a friend who has a PhD in statistics specifically about this Twitter thread (which was referenced in some of the articles about this): https://twitter.com/wfithian/status/1252692357788479488 https://twitter.com/wfithian/status/1252692357788479488 One claim Will Fithian makes is: > The errors are not debatable and can be seen in these two screenshots of the supplement: 0.0034, the standard error meant to measure uncertainty about prevalence pi, is not the square root of 0.039, and the variance of a binomial estimate of proportion depends on the sample size. My statistician friend says: > In fact the math checks out in favor of the authors. Square root of .039, then divide by square root of 3330. .= 0034 just like reported. > So the error is debatable. > Now, sometimes you should be using the standard deviation, not the standard error, but that isn't his claim. The .0034 is absolutely the correct standard error, unless there is a difference in terminology for the field or something.
MengerSponge 6y ago
There was also an absurd amount of selection bias in their population. They recruited people for a COVID-19 antibody screening, and there were emails encouraging people who suspected they had recovered from the disease to get screened. If the researchers knew about the email, it's actionable research misconduct. If they didn't know (a coauthor's wife sent the email), then it's just very very bad work.
- briandear 6y ago
  However what about the USC and the NYC studies? Are they wrong too?
  - ekidd 6y ago
    The NYC studies actually look halfway decent. The big problem with serological tests is that even good ones may have, say, a 2% false positive rate. So if you test a population where nobody is infected, the test will say that 2% have been infected. To get good numbers, you need a population where lots of people have been infected. That will allow true positives to swamp the false positives, which is what happened in the NYC study that claimed 21% of people had already been infected. Even if it's really only 19%, that doesn't change much. You also need a good random sampling procedure. IIRC, the NYC study tested people shopping for essentials at mid-day. This underrepresents essential workers, people who are really good at self-isolation, and people who are currently sick. But still, it's not too bad. The New York City numbers suggest that very roughly 1% of people infected with the virus will die, depending on how you count probable but unconfirmed COVID deaths reported by NYC. This could be off by a factor of two in either direction, depending on how good the random sampling was, and the characteristics of a given population.
    - MengerSponge 6y ago
      Nicely put. With a case fatality of ~1%, we need orders of magnitude reduction, not factors of 2. If anyone wants to quibble over a factor of 2 you've missed the point. Now my nightly rant: I did this kind of false positive calculation in high school statistics, and I hated it. Still do. I hate that we're dealing with this BS. I hate that stupid people put us in this position, and that smart people aren't smart enough to not be absolute knobs.

knzhou 6y ago

> what specifically leads to such bold conclusions (e.g. "owe an apology", "wasted time and effort", "statistical error", "screwed up", etc.)? They used a test with a false positive rate of about 2%, and found a positive rate of about 2%. It is impossible to use this information to establish a lower bound on the number infected. However, by screwing up the computation of the confidence interval, they claimed a huge lower bound.

statquontrarian 6y ago
Thanks. By the way, the STAT article links to https://undark.org/2020/04/24/john-ioannidis-covid-19-death-rate-critics/ https://undark.org/2020/04/24/john-ioannidis-covid-19-death-... in which Ioannidis is quoted: > “We have been working very hard to address all the comments raised by many colleagues and I am always grateful to receive constructive criticism,” he said. “While we received thousands of congratulations for this work, I cherish far more the comments of the scientists who criticized our work and I thank them deeply for their comments.”
- knzhou 6y ago
  That's an admirable sentiment, Ionannidis must be such a wonderful guy. However, the damage has already been done. He's already done his media blitz, so no matter what happens next, effectively zero people will hear the correction (or more likely, subsequent retraction). It'll just become yet another piece of zombie information, which will appear in dishonest talking points or gish gallops. The quality of public discourse and the reputation of science will continue its decline, and we'll be talking about the next example of this tomorrow.
  - statquontrarian 6y ago
    If we try to be charitable, Ioannidis was too confident in his research team who made mistakes; for example: > The authors said by email that they used a built-in Stata function and aren't sure themselves how the software used the input weights. I suspect they misapplied that function (too complicated to tweet why) but I don't know Stata well enough to be sure; it seems neither do they. https://twitter.com/wfithian/status/1252692357788479488 https://twitter.com/wfithian/status/1252692357788479488 Then, Ioannidis probably didn't double check the work in depth because of confirmation bias and over-confidence in his protégés.
statquontrarian 6y ago
I checked with a friend who has a PhD in statistics specifically about this Twitter thread (which was referenced in some of the articles about this): https://twitter.com/wfithian/status/1252692357788479488 https://twitter.com/wfithian/status/1252692357788479488 One claim Will Fithian makes is: > The errors are not debatable and can be seen in these two screenshots of the supplement: 0.0034, the standard error meant to measure uncertainty about prevalence pi, is not the square root of 0.039, and the variance of a binomial estimate of proportion depends on the sample size. My statistician friend says: > In fact the math checks out in favor of the authors. Square root of .039, then divide by square root of 3330. .= 0034 just like reported. > So the error is debatable. > Now, sometimes you should be using the standard deviation, not the standard error, but that isn't his claim. The .0034 is absolutely the correct standard error, unless there is a difference in terminology for the field or something.

MengerSponge 6y ago

There was also an absurd amount of selection bias in their population. They recruited people for a COVID-19 antibody screening, and there were emails encouraging people who suspected they had recovered from the disease to get screened. If the researchers knew about the email, it's actionable research misconduct. If they didn't know (a coauthor's wife sent the email), then it's just very very bad work.

briandear 6y ago
However what about the USC and the NYC studies? Are they wrong too?
- ekidd 6y ago
  The NYC studies actually look halfway decent. The big problem with serological tests is that even good ones may have, say, a 2% false positive rate. So if you test a population where nobody is infected, the test will say that 2% have been infected. To get good numbers, you need a population where lots of people have been infected. That will allow true positives to swamp the false positives, which is what happened in the NYC study that claimed 21% of people had already been infected. Even if it's really only 19%, that doesn't change much. You also need a good random sampling procedure. IIRC, the NYC study tested people shopping for essentials at mid-day. This underrepresents essential workers, people who are really good at self-isolation, and people who are currently sick. But still, it's not too bad. The New York City numbers suggest that very roughly 1% of people infected with the virus will die, depending on how you count probable but unconfirmed COVID deaths reported by NYC. This could be off by a factor of two in either direction, depending on how good the random sampling was, and the characteristics of a given population.
  - MengerSponge 6y ago
    Nicely put. With a case fatality of ~1%, we need orders of magnitude reduction, not factors of 2. If anyone wants to quibble over a factor of 2 you've missed the point. Now my nightly rant: I did this kind of false positive calculation in high school statistics, and I hated it. Still do. I hate that we're dealing with this BS. I hate that stupid people put us in this position, and that smart people aren't smart enough to not be absolute knobs.