5 ms·

All three studies created their sample groups differently. Are all three methods flawed in some way? Sure. Is it more reasonable to completely ignore the pictu

All three studies created their sample groups differently. Are all three methods flawed in some way? Sure. Is it more reasonable to completely ignore the picture all three are painting or to consider there might be something to the trend?

zamfi 6y ago
> Are all three methods flawed in some way? Sure. They're not just all flawed in some way -- along at least one dimension they are flawed in the same way, that is, they all have a biased sample of the population that seems more likely to be infected than a true random sample. So, sure, they're "created differently", i.e., they're not all facebook ads, or grocery stores, but all those methods bias towards higher infection rates, don't they? Given that, you'd expect high precision, low accuracy -- these tests all paint a trend for sure, but we don't know how close that trend is to reality.
- timr 6y ago
  These studies have completely different methodologies, and the only way you can plausibly call them similar is by arguing that anyone willing to get serological testing is substantially more likely to be infected than those who are not. That's little more than a fancy way of disregarding any testing result you don't happen to like. We've been systematically biasing toward testing the old and infected for months now. Why should rates derived from PCR testing with known severity bias considered a strong prior, yet rates derived from serological surveys with hypothetical severity bias be disregarded for the same reasons? At this point, we have many different studies from different parts of the world -- each with its own methodological flaws and biases -- and yet all are pointing in the same direction: a systematic under-count of cases, and an IFR somewhere between 0.5% and 1%, with a bias toward the lower value.
  - zamfi 6y ago
    > arguing that anyone willing to get serological testing is substantially more likely to be infected than those who are not. Yes, which is clearly true, no? > That's little more than a fancy way of disregarding any testing result you don't happen to like. We all have biases. I'm not sure how pointing out that adverse selection is a thing, though, is a bias. Any stats can be quibbled with, and I have no personal skin in this game -- if anything, I would love (like almost everyone) for the IFR to be <0.1% and for hard immunity to already be a thing. But I'm also aware of (some of!) my own biases. > and yet all are pointing in the same direction: a systematic under-count of cases We already know we're undercounting, because we're only explicitly testing people with symptoms, and not even all people with symptoms! We didn't need these studies for that. These studies would ideally be true random samples so that we can know what the true infection rate is. Some locales are doing that kind of testing, and I'll be much more interested in those reports than studies in which respondents can opt into participation.
    - timr 6y ago
      "Yes, which is clearly true, no?" No. As far as I can tell, approximately 100% of people in New York want to be tested, for many different reasons. You're assuming something and projecting it as truth. "These studies would ideally be true random samples so that we can know what the true infection rate is." But by your own logic, there can be no "random sample"...we can't tie people down and force them to be tested, so we have to ask them. This means we're back to the self-selection problem that apparently ruins the sample. The truth is, the New York study did pretty much what we always do to get a random sample: asked a bunch of people at random to participate in the program.
      - zamfi 6y ago
        > No. As far as I can tell, approximately 100% of people in New York want to be tested, for many different reasons. You're assuming something and projecting it as truth. In NY, the issue was not about wanting to be tested, it was about "are you there when they're picking people to test"; quoting from the article, which is very thin on info, people were selected "at 40 locations that included grocery and big-box stores". (In Santa Clara, with the Facebook ads, people "wanting to be tested" by going to a testing center is not anywhere close to 100%.) The assumption that people who are at grocery and big-box stores are representative, infection-wise, of the population of a whole is not obviously true to me. Many other posters in this thread point out reasons why they might be more likely to be infected. > But by your own logic, there can be no "random sample"...we can't tie people down and force them to be tested This approach may go against American sensibilities, but other states (SK, DE, IT) have had better luck with the approach of sampling everyone in a town/region/area. Or at least pick people truly at random, not just a random sample of people who happen to show up in a public place in the middle of a pandemic with a shelter-in-place order in effect. (Also, I'm not sure how "my own logic" makes any claims about the impossibility of a random sample, could you elaborate?) Even in America, you can at least try to account for the ways in which your sample is likely to be different from a truly random sample. The study by Stanford in Santa Clara county I know at least tried to do this, but of course you can't account for "thinks is infected" as a demographic attribute because that's the dependent variable; I don't know if the NYC study did this type of accounting, but I assume they tried to too. > The truth is, the New York study did pretty much what we always do to get a random sample: asked a bunch of people at random to participate in the program. You clearly know about sample bias, so I'm not sure why you think going to a single location (or even 40 single locations) and randomly asking people to be sampled is a good way to get a random sample. Think about going to every Costco and Whole Foods in the state and asking "people at random" for their political views. You're going to get a biased sample, even though you're nominally asking people at random. You can try to account for bias, based on demographic factors you observe in your sample that correlate with political views, right? But we don't know enough about infection rate to be able to effectively account for infection rate given that people who are showing up at grocery and big-box stores on a random day are more likely to be exposed (they're at a store!) and thus more likely to have been infected than the population as a whole? Literally the only info you have is that they're more likely to be infected -- there aren't well-established demographic correlates with infection.
  - qqqwerty 6y ago
    Anyone worth arguing with acknowledges the limitations of the PCR testing. You are essentially suggesting that we ignore the limitations and biases of these serological surveys. If a handful of internet denizens can quickly point out methodological flaws, maybe the researches should try to address or account for those. I fully understand the urgency of the matter, and that urgency can lead to mistakes. But these studies are going to be used in policy making and could potentially impact millions of lives. The difference between 1% and 0.5% is huge, that is over 1.5 million in the US. And powerful people are trying to use these studies to make a case for re-opening the economy. I think it is completely reasonable to push back on these flaws and biases in light of that.
    - timr 6y ago
      "You are essentially suggesting that we ignore the limitations and biases of these serological surveys." I am not. I am suggesting that you are preferring a rate based on an even more biased method, because the method yields a bigger number. "If a handful of internet denizens can quickly point out methodological flaws, maybe the researches should try to address or account for those." If a handful of internet denizens can quickly point out "methodological flaws", it usually means that the "methodological flaws" they have discovered are well-known and accounted for. In this particular case, sample bias is not a new statistical phenomenon, and we've been doing serological surveys for a long time.
      - qqqwerty 6y ago
        > I am suggesting that you are preferring a rate based on an even more biased method, because the method yields a bigger number. What method? I merely stated that PCR has limitations, I never said I was using it to calculate IFR. For the record, my assumptions are that IFR is around 1%. That is largely based on the Diamond Princess data. The original paper suggested it was 0.5%[1], but at the time there were only 7 fatalities. The current number is 13. The raw IFR is now up to 1.8%. Crudely adjusting for demographics based on the ratio in the paper puts the IFR around 1%. This is the only population that was both comprehensively tested and occurred long enough ago for most of the cases to resolve (although some cases are still active/critical[2]). And I fully acknowledge that this approach has limitations. It is a small population, it happened earlier in the crisis when we knew less about treatment, etc... But I sill think it is the best data point we have. > If a handful of internet denizens can quickly point out "methodological flaws", it usually means that the "methodological flaws" they have discovered are well-known and accounted for. Except the Santa Clara study did not do that. They acknowledged potential bias, but did nothing to adjust for it. From the source: "Other biases, such as bias favoring individuals in good health capable of attending our testing sites, or bias favoring those with prior COVID-like illnesses seeking antibody confirmation are also possible. The overall effect of such biases is hard to ascertain." The LA and NY papers have not been published yet. [1] https://cmmid.github.io/topics/covid19/diamond_cruise_cfr_estimates.html https://cmmid.github.io/topics/covid19/diamond_cruise_cfr_es... [2] https://www.worldometers.info/coronavirus/ https://www.worldometers.info/coronavirus/
monadic2 6y ago
> Is it more reasonable to completely ignore the picture all three are painting or to consider there might be something to the trend? The reasonable conclusion would be no conclusion. The comforting conclusion is to extrapolate a trend from noise.
koheripbal 6y ago
They are all flawed in the SAME way - they sampled people who they found outside their homes. People 65+ are NOT leaving their homes.

zamfi 6y ago

> Are all three methods flawed in some way? Sure. They're not just all flawed in some way -- along at least one dimension they are flawed in the same way, that is, they all have a biased sample of the population that seems more likely to be infected than a true random sample. So, sure, they're "created differently", i.e., they're not all facebook ads, or grocery stores, but all those methods bias towards higher infection rates, don't they? Given that, you'd expect high precision, low accuracy -- these tests all paint a trend for sure, but we don't know how close that trend is to reality.

timr 6y ago
These studies have completely different methodologies, and the only way you can plausibly call them similar is by arguing that anyone willing to get serological testing is substantially more likely to be infected than those who are not. That's little more than a fancy way of disregarding any testing result you don't happen to like. We've been systematically biasing toward testing the old and infected for months now. Why should rates derived from PCR testing with known severity bias considered a strong prior, yet rates derived from serological surveys with hypothetical severity bias be disregarded for the same reasons? At this point, we have many different studies from different parts of the world -- each with its own methodological flaws and biases -- and yet all are pointing in the same direction: a systematic under-count of cases, and an IFR somewhere between 0.5% and 1%, with a bias toward the lower value.
- zamfi 6y ago
  > arguing that anyone willing to get serological testing is substantially more likely to be infected than those who are not. Yes, which is clearly true, no? > That's little more than a fancy way of disregarding any testing result you don't happen to like. We all have biases. I'm not sure how pointing out that adverse selection is a thing, though, is a bias. Any stats can be quibbled with, and I have no personal skin in this game -- if anything, I would love (like almost everyone) for the IFR to be <0.1% and for hard immunity to already be a thing. But I'm also aware of (some of!) my own biases. > and yet all are pointing in the same direction: a systematic under-count of cases We already know we're undercounting, because we're only explicitly testing people with symptoms, and not even all people with symptoms! We didn't need these studies for that. These studies would ideally be true random samples so that we can know what the true infection rate is. Some locales are doing that kind of testing, and I'll be much more interested in those reports than studies in which respondents can opt into participation.
  - timr 6y ago
    "Yes, which is clearly true, no?" No. As far as I can tell, approximately 100% of people in New York want to be tested, for many different reasons. You're assuming something and projecting it as truth. "These studies would ideally be true random samples so that we can know what the true infection rate is." But by your own logic, there can be no "random sample"...we can't tie people down and force them to be tested, so we have to ask them. This means we're back to the self-selection problem that apparently ruins the sample. The truth is, the New York study did pretty much what we always do to get a random sample: asked a bunch of people at random to participate in the program.
    - zamfi 6y ago
      > No. As far as I can tell, approximately 100% of people in New York want to be tested, for many different reasons. You're assuming something and projecting it as truth. In NY, the issue was not about wanting to be tested, it was about "are you there when they're picking people to test"; quoting from the article, which is very thin on info, people were selected "at 40 locations that included grocery and big-box stores". (In Santa Clara, with the Facebook ads, people "wanting to be tested" by going to a testing center is not anywhere close to 100%.) The assumption that people who are at grocery and big-box stores are representative, infection-wise, of the population of a whole is not obviously true to me. Many other posters in this thread point out reasons why they might be more likely to be infected. > But by your own logic, there can be no "random sample"...we can't tie people down and force them to be tested This approach may go against American sensibilities, but other states (SK, DE, IT) have had better luck with the approach of sampling everyone in a town/region/area. Or at least pick people truly at random, not just a random sample of people who happen to show up in a public place in the middle of a pandemic with a shelter-in-place order in effect. (Also, I'm not sure how "my own logic" makes any claims about the impossibility of a random sample, could you elaborate?) Even in America, you can at least try to account for the ways in which your sample is likely to be different from a truly random sample. The study by Stanford in Santa Clara county I know at least tried to do this, but of course you can't account for "thinks is infected" as a demographic attribute because that's the dependent variable; I don't know if the NYC study did this type of accounting, but I assume they tried to too. > The truth is, the New York study did pretty much what we always do to get a random sample: asked a bunch of people at random to participate in the program. You clearly know about sample bias, so I'm not sure why you think going to a single location (or even 40 single locations) and randomly asking people to be sampled is a good way to get a random sample. Think about going to every Costco and Whole Foods in the state and asking "people at random" for their political views. You're going to get a biased sample, even though you're nominally asking people at random. You can try to account for bias, based on demographic factors you observe in your sample that correlate with political views, right? But we don't know enough about infection rate to be able to effectively account for infection rate given that people who are showing up at grocery and big-box stores on a random day are more likely to be exposed (they're at a store!) and thus more likely to have been infected than the population as a whole? Literally the only info you have is that they're more likely to be infected -- there aren't well-established demographic correlates with infection.
- qqqwerty 6y ago
  Anyone worth arguing with acknowledges the limitations of the PCR testing. You are essentially suggesting that we ignore the limitations and biases of these serological surveys. If a handful of internet denizens can quickly point out methodological flaws, maybe the researches should try to address or account for those. I fully understand the urgency of the matter, and that urgency can lead to mistakes. But these studies are going to be used in policy making and could potentially impact millions of lives. The difference between 1% and 0.5% is huge, that is over 1.5 million in the US. And powerful people are trying to use these studies to make a case for re-opening the economy. I think it is completely reasonable to push back on these flaws and biases in light of that.
  - timr 6y ago
    "You are essentially suggesting that we ignore the limitations and biases of these serological surveys." I am not. I am suggesting that you are preferring a rate based on an even more biased method, because the method yields a bigger number. "If a handful of internet denizens can quickly point out methodological flaws, maybe the researches should try to address or account for those." If a handful of internet denizens can quickly point out "methodological flaws", it usually means that the "methodological flaws" they have discovered are well-known and accounted for. In this particular case, sample bias is not a new statistical phenomenon, and we've been doing serological surveys for a long time.
    - qqqwerty 6y ago
      > I am suggesting that you are preferring a rate based on an even more biased method, because the method yields a bigger number. What method? I merely stated that PCR has limitations, I never said I was using it to calculate IFR. For the record, my assumptions are that IFR is around 1%. That is largely based on the Diamond Princess data. The original paper suggested it was 0.5%[1], but at the time there were only 7 fatalities. The current number is 13. The raw IFR is now up to 1.8%. Crudely adjusting for demographics based on the ratio in the paper puts the IFR around 1%. This is the only population that was both comprehensively tested and occurred long enough ago for most of the cases to resolve (although some cases are still active/critical[2]). And I fully acknowledge that this approach has limitations. It is a small population, it happened earlier in the crisis when we knew less about treatment, etc... But I sill think it is the best data point we have. > If a handful of internet denizens can quickly point out "methodological flaws", it usually means that the "methodological flaws" they have discovered are well-known and accounted for. Except the Santa Clara study did not do that. They acknowledged potential bias, but did nothing to adjust for it. From the source: "Other biases, such as bias favoring individuals in good health capable of attending our testing sites, or bias favoring those with prior COVID-like illnesses seeking antibody confirmation are also possible. The overall effect of such biases is hard to ascertain." The LA and NY papers have not been published yet. [1] https://cmmid.github.io/topics/covid19/diamond_cruise_cfr_estimates.html https://cmmid.github.io/topics/covid19/diamond_cruise_cfr_es... [2] https://www.worldometers.info/coronavirus/ https://www.worldometers.info/coronavirus/

monadic2 6y ago

> Is it more reasonable to completely ignore the picture all three are painting or to consider there might be something to the trend? The reasonable conclusion would be no conclusion. The comforting conclusion is to extrapolate a trend from noise.

koheripbal 6y ago

They are all flawed in the SAME way - they sampled people who they found outside their homes. People 65+ are NOT leaving their homes.