5 ms·

> Sure all of these studies are potentially flawed but they're all generally pointing in the same direction That's unfortunately not the way statistics works.

> Sure all of these studies are potentially flawed but they're all generally pointing in the same direction That's unfortunately not the way statistics works. Combining multiple bad tests just makes the results incorrect or highly uncertain; the devil often is in the details. The Stanford/Santa Clara County study is a good example of how the details can really undermine a study. Things do indeed seem to be converging to the idea that the CFR is not near the IFR, but none of this is new news, and the IFR remains very close to what most epidemiological predictions indicated early last month. If anything, it confirms that COVID is a difficult beast to tame.

TheBlight 7y ago
We don't have perfect studies and perfect data but calling these tests "bad" seems slightly unfair. They give us an imperfect but useful snapshot of what's going on. But thanks for the condescension. I'd expect nothing less on HN.
- CydeWeys 7y ago
  The study is dramatically over-sampling exactly the people who have the most potential exposure to COVID-19, and dramatically over-sampling the people who don't. This could easily be way off. It's not testing the people who aren't home at all, and it has a low chance of only testing the people who leave home rarely, only when strictly necessary. It's mainly finding the people who leave home a lot.
  - TheBlight 7y ago
    How do you know how often these people are leaving their homes? Everyone has to leave their home to go to the grocery store eventually or someone in their family does. Yes there are delivery services but the vast majority of people aren't using those.
    - kgwgk 7y ago
      The more often they go, the more likely you encounter them.
      - FlyMoreRockets 7y ago
        True, but if you encounter one daily shopper, two weekly shoppers and twenty monthly shoppers, it will skew the results. Same if the numbers are reversed.
        kgwgk 7y ago
        I don’t get your point. Maybe we agree. If in one city there are 1000 30-year-old daily-shopers, 1000 50-year-old weekly-shoppers and 1000 70-year-old monthly-shoppers it will not be a good idea to estimate the age of the population by sampling people who enters the store. The average age is 50, not 34.
        jdavis703 7y ago
        Most people in NYC are almost certainly not shopping monthly because of storage constraints. Even bi-weekly shopping is likely to exhaust storage space for city-dwellers in small apartments.
        CydeWeys 7y ago
        But you won't, the opposite will happen; you'll proportionally encounter way more daily shoppers than infrequent shoppers, for the simple tautological reason that the daily shoppers are going more often. This is a sampling bias, and it's the exact same problem as if you ran a survey trying to ascertain how many people have landlines ... by calling random phone numbers.
  - varenc 7y ago
    I think the overwhelming majority of people are visiting grocery stores to buy food. (Or have someone in their household doing it) If the studied presume this then it makes more sense.
    - CydeWeys 7y ago
      You can't assume that. I know plenty of people who haven't left their apartments in a month plus, because they're having groceries ordered in. And you completely missed the point that people who are going outside a lot more often are both (a) more likely to have been exposed to COVID-19, and (b) more likely to be encountered by your survey. If there are e.g. 700 daily shoppers and 700 weekly shoppers going to a given store, then if you go on one day and sample every shopper you're not going to get half and half by population as you'd need to for a true random sample; instead, you're going to get 700 daily shoppers and only 100 weekly shoppers! This is a hugely biased sample!
      - leereeves 7y ago
        Are you describing the actual study or the study as you're imagining it? Because I imagine the statisticians have thought of these concerns and attempted to correct for them. I find it interesting that this study corresponds to an IFR around 0.5%, similar to the IFR arrived at by other studies using very different methods. Perhaps the statisticians who conduct these studies actually know what they're doing better than Hacker News commenters. And perhaps it's time to accept the politically inconvenient scientific truth these studies are revealing.
        twomoretime 7y ago
        >similar to the IFR arrived at by other studies using very different methods With the wild range of IFR from so many studies, you could trivially cherry pick supporting studies for a bunch of different numbers, especially if you throw in am error range.
      - deleted 7y ago
        [deleted]
  - deleted 7y ago
    [deleted]
  - akiselev 7y ago
    Couldn't you get a bad study that reverses the sampling bias to get a decent picture of social distancing strategies using the two bad studies? That alone seems far more useful than knowing the IFR because its actionable information.
    - qqqwerty 7y ago
      Or better yet, do a simple survey and ask the participants how frequently they go to the grocery store. And if you see a correlation between frequency and infection rate, then try and adjust your weights accordingly. These bias issues are well known among survey designers. There are ways that you can try and account for them. It seems like maybe the medical community is too used to having control groups in their studies?
  - SubiculumCode 7y ago
    Most people go to the grocery store. There seems to be this attitude on HN of immediately dismissing research. No human research is ever perfect, flaws and limitations will always be present, and in the present case, study designs are being expedited in order to be timely. Each study is a piece, and together can provide convergent evidence.
capkutay 7y ago
OP's whole point is it's not a good sample because people were recruited while grocery shopping...I can't think of a more tech industry-oriented view of the world; most people work from home on zoom and use instacart right?? No. I'd expect people who order deliveries online and can do their entire job on Zoom/slack all day to be a minority to the point where including them in the study would be covered in a sampling error.
- mlthoughts2018 7y ago
  You’re completely missing the point. The selection effects would be things like able-bodied / non-immunocompromised / already-recovered people being selected way higher, which would inflate the apparent infected population size estimate and underestimate the death rate among those infected. This is not some snooty tech point of view - this kind of flaw in statistical studies matters a lot. The more important the policy implications are, the more important the validity of statistical methods. It’s anything but parochial.
  - mlyle 7y ago
    I don't think anyone disagrees that the sample has bias. On the other hand, since the majority of people have no choice but to grocery shop in person, I do not think the sample is as tilted as you imply. And even if it were--- it's notable that it's this easy to find a subgroup with cleared infections this far above what was anticipated to be found across NY/NYC. And-- it's remarkably consistent with statistical estimates made through a variety of other means, e.g. https://www.medrxiv.org/content/10.1101/2020.04.18.20070821v1.full.pdf https://www.medrxiv.org/content/10.1101/2020.04.18.20070821v...
    - mlthoughts2018 7y ago
      > “ since the majority of people have no choice but to grocery shop in person” This is deeply flawed. I live in NYC and have been making donations to have food delivered to elderly people since this started. Just the other day, de Blasio announced that taxi and rideshare drivers would be paid by the city to deliver food - and city agencies would be serving as overflow dispatchers and tech support for it. Many, many people do not have to go to the grocery store, or they have their healthy younger family member go in their place, etc. To boot, as Gelman has written on his blog, these studies do not agree with a wide range of other studies, and in most cases the test specificity is on par with the incidence rate itself, making incidence rate estimates from _all_ of the studies very unreliable for correlated reasons, so that pooling the studies does nothing to overcome the huge uncertainty they suffer in incidence rates.
np_tedious 7y ago
> Combining multiple bad tests just makes the results incorrect or highly uncertain But it's not just, say, taking a bunch of faulty studies and average them and saying voila. They are in agreement. They'd have to be bad in similar ways in order to converge like that.
Mikeb85 7y ago
> That's unfortunately not the way statistics works. It absolutely is. That's why we have things like confidence intervals.

TheBlight 7y ago

We don't have perfect studies and perfect data but calling these tests "bad" seems slightly unfair. They give us an imperfect but useful snapshot of what's going on. But thanks for the condescension. I'd expect nothing less on HN.

CydeWeys 7y ago
The study is dramatically over-sampling exactly the people who have the most potential exposure to COVID-19, and dramatically over-sampling the people who don't. This could easily be way off. It's not testing the people who aren't home at all, and it has a low chance of only testing the people who leave home rarely, only when strictly necessary. It's mainly finding the people who leave home a lot.
- TheBlight 7y ago
  How do you know how often these people are leaving their homes? Everyone has to leave their home to go to the grocery store eventually or someone in their family does. Yes there are delivery services but the vast majority of people aren't using those.
  - kgwgk 7y ago
    The more often they go, the more likely you encounter them.
    - FlyMoreRockets 7y ago
      True, but if you encounter one daily shopper, two weekly shoppers and twenty monthly shoppers, it will skew the results. Same if the numbers are reversed.
      - kgwgk 7y ago
        I don’t get your point. Maybe we agree. If in one city there are 1000 30-year-old daily-shopers, 1000 50-year-old weekly-shoppers and 1000 70-year-old monthly-shoppers it will not be a good idea to estimate the age of the population by sampling people who enters the store. The average age is 50, not 34.
      - jdavis703 7y ago
        Most people in NYC are almost certainly not shopping monthly because of storage constraints. Even bi-weekly shopping is likely to exhaust storage space for city-dwellers in small apartments.
      - CydeWeys 7y ago
        But you won't, the opposite will happen; you'll proportionally encounter way more daily shoppers than infrequent shoppers, for the simple tautological reason that the daily shoppers are going more often. This is a sampling bias, and it's the exact same problem as if you ran a survey trying to ascertain how many people have landlines ... by calling random phone numbers.
- varenc 7y ago
  I think the overwhelming majority of people are visiting grocery stores to buy food. (Or have someone in their household doing it) If the studied presume this then it makes more sense.
  - CydeWeys 7y ago
    You can't assume that. I know plenty of people who haven't left their apartments in a month plus, because they're having groceries ordered in. And you completely missed the point that people who are going outside a lot more often are both (a) more likely to have been exposed to COVID-19, and (b) more likely to be encountered by your survey. If there are e.g. 700 daily shoppers and 700 weekly shoppers going to a given store, then if you go on one day and sample every shopper you're not going to get half and half by population as you'd need to for a true random sample; instead, you're going to get 700 daily shoppers and only 100 weekly shoppers! This is a hugely biased sample!
    - leereeves 7y ago
      Are you describing the actual study or the study as you're imagining it? Because I imagine the statisticians have thought of these concerns and attempted to correct for them. I find it interesting that this study corresponds to an IFR around 0.5%, similar to the IFR arrived at by other studies using very different methods. Perhaps the statisticians who conduct these studies actually know what they're doing better than Hacker News commenters. And perhaps it's time to accept the politically inconvenient scientific truth these studies are revealing.
      - twomoretime 7y ago
        >similar to the IFR arrived at by other studies using very different methods With the wild range of IFR from so many studies, you could trivially cherry pick supporting studies for a bunch of different numbers, especially if you throw in am error range.
    - deleted 7y ago
      [deleted]
- deleted 7y ago
  [deleted]
- akiselev 7y ago
  Couldn't you get a bad study that reverses the sampling bias to get a decent picture of social distancing strategies using the two bad studies? That alone seems far more useful than knowing the IFR because its actionable information.
  - qqqwerty 7y ago
    Or better yet, do a simple survey and ask the participants how frequently they go to the grocery store. And if you see a correlation between frequency and infection rate, then try and adjust your weights accordingly. These bias issues are well known among survey designers. There are ways that you can try and account for them. It seems like maybe the medical community is too used to having control groups in their studies?
- SubiculumCode 7y ago
  Most people go to the grocery store. There seems to be this attitude on HN of immediately dismissing research. No human research is ever perfect, flaws and limitations will always be present, and in the present case, study designs are being expedited in order to be timely. Each study is a piece, and together can provide convergent evidence.

capkutay 7y ago

OP's whole point is it's not a good sample because people were recruited while grocery shopping...I can't think of a more tech industry-oriented view of the world; most people work from home on zoom and use instacart right?? No. I'd expect people who order deliveries online and can do their entire job on Zoom/slack all day to be a minority to the point where including them in the study would be covered in a sampling error.

mlthoughts2018 7y ago
You’re completely missing the point. The selection effects would be things like able-bodied / non-immunocompromised / already-recovered people being selected way higher, which would inflate the apparent infected population size estimate and underestimate the death rate among those infected. This is not some snooty tech point of view - this kind of flaw in statistical studies matters a lot. The more important the policy implications are, the more important the validity of statistical methods. It’s anything but parochial.
- mlyle 7y ago
  I don't think anyone disagrees that the sample has bias. On the other hand, since the majority of people have no choice but to grocery shop in person, I do not think the sample is as tilted as you imply. And even if it were--- it's notable that it's this easy to find a subgroup with cleared infections this far above what was anticipated to be found across NY/NYC. And-- it's remarkably consistent with statistical estimates made through a variety of other means, e.g. https://www.medrxiv.org/content/10.1101/2020.04.18.20070821v1.full.pdf https://www.medrxiv.org/content/10.1101/2020.04.18.20070821v...
  - mlthoughts2018 7y ago
    > “ since the majority of people have no choice but to grocery shop in person” This is deeply flawed. I live in NYC and have been making donations to have food delivered to elderly people since this started. Just the other day, de Blasio announced that taxi and rideshare drivers would be paid by the city to deliver food - and city agencies would be serving as overflow dispatchers and tech support for it. Many, many people do not have to go to the grocery store, or they have their healthy younger family member go in their place, etc. To boot, as Gelman has written on his blog, these studies do not agree with a wide range of other studies, and in most cases the test specificity is on par with the incidence rate itself, making incidence rate estimates from _all_ of the studies very unreliable for correlated reasons, so that pooling the studies does nothing to overcome the huge uncertainty they suffer in incidence rates.

np_tedious 7y ago

> Combining multiple bad tests just makes the results incorrect or highly uncertain But it's not just, say, taking a bunch of faulty studies and average them and saying voila. They are in agreement. They'd have to be bad in similar ways in order to converge like that.

Mikeb85 7y ago

> That's unfortunately not the way statistics works. It absolutely is. That's why we have things like confidence intervals.