5 ms·

> Sure all of these studies are potentially flawed but they're all generally pointing in the same direction. How on earth could you differentiate this from the

> Sure all of these studies are potentially flawed but they're all generally pointing in the same direction. How on earth could you differentiate this from the sample bias?

TheBlight 6y ago
All three studies created their sample groups differently. Are all three methods flawed in some way? Sure. Is it more reasonable to completely ignore the picture all three are painting or to consider there might be something to the trend?
- zamfi 6y ago
  > Are all three methods flawed in some way? Sure. They're not just all flawed in some way -- along at least one dimension they are flawed in the same way, that is, they all have a biased sample of the population that seems more likely to be infected than a true random sample. So, sure, they're "created differently", i.e., they're not all facebook ads, or grocery stores, but all those methods bias towards higher infection rates, don't they? Given that, you'd expect high precision, low accuracy -- these tests all paint a trend for sure, but we don't know how close that trend is to reality.
  - timr 6y ago
    These studies have completely different methodologies, and the only way you can plausibly call them similar is by arguing that anyone willing to get serological testing is substantially more likely to be infected than those who are not. That's little more than a fancy way of disregarding any testing result you don't happen to like. We've been systematically biasing toward testing the old and infected for months now. Why should rates derived from PCR testing with known severity bias considered a strong prior, yet rates derived from serological surveys with hypothetical severity bias be disregarded for the same reasons? At this point, we have many different studies from different parts of the world -- each with its own methodological flaws and biases -- and yet all are pointing in the same direction: a systematic under-count of cases, and an IFR somewhere between 0.5% and 1%, with a bias toward the lower value.
    - zamfi 6y ago
      > arguing that anyone willing to get serological testing is substantially more likely to be infected than those who are not. Yes, which is clearly true, no? > That's little more than a fancy way of disregarding any testing result you don't happen to like. We all have biases. I'm not sure how pointing out that adverse selection is a thing, though, is a bias. Any stats can be quibbled with, and I have no personal skin in this game -- if anything, I would love (like almost everyone) for the IFR to be <0.1% and for hard immunity to already be a thing. But I'm also aware of (some of!) my own biases. > and yet all are pointing in the same direction: a systematic under-count of cases We already know we're undercounting, because we're only explicitly testing people with symptoms, and not even all people with symptoms! We didn't need these studies for that. These studies would ideally be true random samples so that we can know what the true infection rate is. Some locales are doing that kind of testing, and I'll be much more interested in those reports than studies in which respondents can opt into participation.
      - timr 6y ago
        "Yes, which is clearly true, no?" No. As far as I can tell, approximately 100% of people in New York want to be tested, for many different reasons. You're assuming something and projecting it as truth. "These studies would ideally be true random samples so that we can know what the true infection rate is." But by your own logic, there can be no "random sample"...we can't tie people down and force them to be tested, so we have to ask them. This means we're back to the self-selection problem that apparently ruins the sample. The truth is, the New York study did pretty much what we always do to get a random sample: asked a bunch of people at random to participate in the program.
        zamfi 6y ago
        > No. As far as I can tell, approximately 100% of people in New York want to be tested, for many different reasons. You're assuming something and projecting it as truth. In NY, the issue was not about wanting to be tested, it was about "are you there when they're picking people to test"; quoting from the article, which is very thin on info, people were selected "at 40 locations that included grocery and big-box stores". (In Santa Clara, with the Facebook ads, people "wanting to be tested" by going to a testing center is not anywhere close to 100%.) The assumption that people who are at grocery and big-box stores are representative, infection-wise, of the population of a whole is not obviously true to me. Many other posters in this thread point out reasons why they might be more likely to be infected. > But by your own logic, there can be no "random sample"...we can't tie people down and force them to be tested This approach may go against American sensibilities, but other states (SK, DE, IT) have had better luck with the approach of sampling everyone in a town/region/area. Or at least pick people truly at random, not just a random sample of people who happen to show up in a public place in the middle of a pandemic with a shelter-in-place order in effect. (Also, I'm not sure how "my own logic" makes any claims about the impossibility of a random sample, could you elaborate?) Even in America, you can at least try to account for the ways in which your sample is likely to be different from a truly random sample. The study by Stanford in Santa Clara county I know at least tried to do this, but of course you can't account for "thinks is infected" as a demographic attribute because that's the dependent variable; I don't know if the NYC study did this type of accounting, but I assume they tried to too. > The truth is, the New York study did pretty much what we always do to get a random sample: asked a bunch of people at random to participate in the program. You clearly know about sample bias, so I'm not sure why you think going to a single location (or even 40 single locations) and randomly asking people to be sampled is a good way to get a random sample. Think about going to every Costco and Whole Foods in the state and asking "people at random" for their political views. You're going to get a biased sample, even though you're nominally asking people at random. You can try to account for bias, based on demographic factors you observe in your sample that correlate with political views, right? But we don't know enough about infection rate to be able to effectively account for infection rate given that people who are showing up at grocery and big-box stores on a random day are more likely to be exposed (they're at a store!) and thus more likely to have been infected than the population as a whole? Literally the only info you have is that they're more likely to be infected -- there aren't well-established demographic correlates with infection.
- monadic2 6y ago
  > Is it more reasonable to completely ignore the picture all three are painting or to consider there might be something to the trend? The reasonable conclusion would be no conclusion. The comforting conclusion is to extrapolate a trend from noise.
- koheripbal 6y ago
  They are all flawed in the SAME way - they sampled people who they found outside their homes. People 65+ are NOT leaving their homes.

> Sure all of these studies are potentially flawed but they're all generally pointing in the same direction. How on earth could you differentiate this from the sample bias?

TheBlight 6y ago

All three studies created their sample groups differently. Are all three methods flawed in some way? Sure. Is it more reasonable to completely ignore the picture all three are painting or to consider there might be something to the trend?

zamfi 6y ago
> Are all three methods flawed in some way? Sure. They're not just all flawed in some way -- along at least one dimension they are flawed in the same way, that is, they all have a biased sample of the population that seems more likely to be infected than a true random sample. So, sure, they're "created differently", i.e., they're not all facebook ads, or grocery stores, but all those methods bias towards higher infection rates, don't they? Given that, you'd expect high precision, low accuracy -- these tests all paint a trend for sure, but we don't know how close that trend is to reality.
- timr 6y ago
  These studies have completely different methodologies, and the only way you can plausibly call them similar is by arguing that anyone willing to get serological testing is substantially more likely to be infected than those who are not. That's little more than a fancy way of disregarding any testing result you don't happen to like. We've been systematically biasing toward testing the old and infected for months now. Why should rates derived from PCR testing with known severity bias considered a strong prior, yet rates derived from serological surveys with hypothetical severity bias be disregarded for the same reasons? At this point, we have many different studies from different parts of the world -- each with its own methodological flaws and biases -- and yet all are pointing in the same direction: a systematic under-count of cases, and an IFR somewhere between 0.5% and 1%, with a bias toward the lower value.
  - zamfi 6y ago
    > arguing that anyone willing to get serological testing is substantially more likely to be infected than those who are not. Yes, which is clearly true, no? > That's little more than a fancy way of disregarding any testing result you don't happen to like. We all have biases. I'm not sure how pointing out that adverse selection is a thing, though, is a bias. Any stats can be quibbled with, and I have no personal skin in this game -- if anything, I would love (like almost everyone) for the IFR to be <0.1% and for hard immunity to already be a thing. But I'm also aware of (some of!) my own biases. > and yet all are pointing in the same direction: a systematic under-count of cases We already know we're undercounting, because we're only explicitly testing people with symptoms, and not even all people with symptoms! We didn't need these studies for that. These studies would ideally be true random samples so that we can know what the true infection rate is. Some locales are doing that kind of testing, and I'll be much more interested in those reports than studies in which respondents can opt into participation.
    - timr 6y ago
      "Yes, which is clearly true, no?" No. As far as I can tell, approximately 100% of people in New York want to be tested, for many different reasons. You're assuming something and projecting it as truth. "These studies would ideally be true random samples so that we can know what the true infection rate is." But by your own logic, there can be no "random sample"...we can't tie people down and force them to be tested, so we have to ask them. This means we're back to the self-selection problem that apparently ruins the sample. The truth is, the New York study did pretty much what we always do to get a random sample: asked a bunch of people at random to participate in the program.
      - zamfi 6y ago
        > No. As far as I can tell, approximately 100% of people in New York want to be tested, for many different reasons. You're assuming something and projecting it as truth. In NY, the issue was not about wanting to be tested, it was about "are you there when they're picking people to test"; quoting from the article, which is very thin on info, people were selected "at 40 locations that included grocery and big-box stores". (In Santa Clara, with the Facebook ads, people "wanting to be tested" by going to a testing center is not anywhere close to 100%.) The assumption that people who are at grocery and big-box stores are representative, infection-wise, of the population of a whole is not obviously true to me. Many other posters in this thread point out reasons why they might be more likely to be infected. > But by your own logic, there can be no "random sample"...we can't tie people down and force them to be tested This approach may go against American sensibilities, but other states (SK, DE, IT) have had better luck with the approach of sampling everyone in a town/region/area. Or at least pick people truly at random, not just a random sample of people who happen to show up in a public place in the middle of a pandemic with a shelter-in-place order in effect. (Also, I'm not sure how "my own logic" makes any claims about the impossibility of a random sample, could you elaborate?) Even in America, you can at least try to account for the ways in which your sample is likely to be different from a truly random sample. The study by Stanford in Santa Clara county I know at least tried to do this, but of course you can't account for "thinks is infected" as a demographic attribute because that's the dependent variable; I don't know if the NYC study did this type of accounting, but I assume they tried to too. > The truth is, the New York study did pretty much what we always do to get a random sample: asked a bunch of people at random to participate in the program. You clearly know about sample bias, so I'm not sure why you think going to a single location (or even 40 single locations) and randomly asking people to be sampled is a good way to get a random sample. Think about going to every Costco and Whole Foods in the state and asking "people at random" for their political views. You're going to get a biased sample, even though you're nominally asking people at random. You can try to account for bias, based on demographic factors you observe in your sample that correlate with political views, right? But we don't know enough about infection rate to be able to effectively account for infection rate given that people who are showing up at grocery and big-box stores on a random day are more likely to be exposed (they're at a store!) and thus more likely to have been infected than the population as a whole? Literally the only info you have is that they're more likely to be infected -- there aren't well-established demographic correlates with infection.
monadic2 6y ago
> Is it more reasonable to completely ignore the picture all three are painting or to consider there might be something to the trend? The reasonable conclusion would be no conclusion. The comforting conclusion is to extrapolate a trend from noise.
koheripbal 6y ago
They are all flawed in the SAME way - they sampled people who they found outside their homes. People 65+ are NOT leaving their homes.