7 ms·

"We recruited participants by placing targeted advertisements on Facebook aimed at residents of Santa Clara County..." I wonder if these ads mentioned the purp

by gryson 6y ago

"We recruited participants by placing targeted advertisements on Facebook aimed at residents of Santa Clara County..." I wonder if these ads mentioned the purpose of the study, or if that information was only given out after initial contact (I don't know enough about IRB requirements here). A concern is that this could cause a selection bias for people who suspected they had the virus. People may respond to such an ad out of curiosity ("I think I had the virus, so it would be good to know for sure") or obligation ("I'm pretty sure I had the virus, so I should help out with this study"). It wouldn't have to be a large selection bias, either. Of the 3,330 people they tested, they found only 50 who tested positive. I would like to see a bit of a better method of sample selection before drawing any conclusions.

tshadley 6y ago
Why would this matter as long as they got a good geographical sampling? "We used Facebook to quickly reach a large number of county residents and because it allows for granular targeting by zip code and sociodemographic characteristics. We used a combination of two targeting strategies: ads aimed at a representative population of the county by zip code, and specially targeted ads to balance our sample for under-represented zip codes. In addition, we capped registrations from overrepresented areas." Update: Okay, "Other biases, such as bias favoring individuals in good health capable of attending our testing sites, or bias favoring those with prior COVID-like illnesses seeking antibody confirmation are also possible."
- rhema 6y ago
  If part of the value for participants was to know whether they had antibodies, then those that had gone through a flu (or covid) would be more likely to respond. This means that the overall sample likely includes more people who had COVID than a true random sampling would.
  - tshadley 6y ago
    It looks like there's an argument to be made it could overcount or undercount depending on importance of each potential bias. "This study had several limitations. First, our sampling strategy selected for members of Santa Clara County with access to Facebook and a car to attend drive-through testing sites. This resulted in an overrepresentation of white women between the ages of 19 and 64, and an under-representation of Hispanic and Asian populations, relative to our community. Those imbalances were partly addressed by weighting our sample population by zip code, race, and sex to match the county. We did not account for age imbalance in our sample, and could not ascertain representativeness of SARS-CoV-2 antibodies in homeless populations. Other biases, such as bias favoring individuals in good health capable of attending our testing sites, or bias favoring those with prior COVID-like illnesses seeking antibody confirmation are also possible. The overall effect of such biases is hard to ascertain."
not2b 6y ago
Yes, the ads accurately described the study. Several friends of mine participated (I live in Santa Clara County). Of course there is selection bias, which is why they make adjustments to make the sample better represent the population (see the abstract for details on this, or read the paper for all the details, there's well-established science behind these adjustments and it's up to the reviewers to be sure that they did it correctly). The point is that "they only found 50" is quite a large number compared to the number of confirmed cases.
- gryson 6y ago
  What adjustments were made to account for selection bias of the sort described? It's not covered in the paper, and "well-established science" certainly doesn't answer the question.
  - compiler-guy 6y ago
    These are very common statistical methods when dealing with surveys and populations. It isn't this paper's job to describe the background on something like that, just like it isn't this papers job to explain blood testing. They describe their limitations and adjustments exactly like most other studies of this sort do. For example: "Those imbalances were partly addressed by weighting our sample population by zip code, race, and sex to match the county. "
    - tomerico 6y ago
      They said that they collected prior clinical symptoms in the survey, but have not mentioned that they adjusted for those. Given that, I think it's fair to say that they have not adjusted for the selection bias of people with prior clinical symptoms being more likely to click the ad.
    - gryson 6y ago
      On the contrary - detecting selection bias of this sort can be very difficult. Do you measure it using a questionnaire? You might ask your subjects "Do you think you previously had COVID-19?" But what do you do with that response data? Exclude everyone that says yes? Then you're skewing one way. There's obviously no population data to match that response with like there is for general demographics such as race and age. Many studies are only as good as their sampling methods. A good sampling method will alleviate concerns of selection bias.
networkimprov 6y ago
In the data from Italy, ~14% of the >1M tests done for the virus (not antibodies) were positive. That prob includes ppl tested multiple times, or after recovery. I wonder why this is so vastly different from the study above. https://en.wikipedia.org/wiki/2020_coronavirus_pandemic_in_Italy https://en.wikipedia.org/wiki/2020_coronavirus_pandemic_in_I...
lern_too_spel 6y ago
There is also selection bias in whom Facebook will show the ads to due to Facebook's CTR models, other bidders' bids, and who is browsing Facebook more to see more impressions in general. They should have worked with an ad platform to run the ad as a PSA so they could use the ad platform's own randomization instead of relying on having the auctions they win result in anywhere close to a usable sample.

by gryson 6y ago

tshadley 6y ago

Why would this matter as long as they got a good geographical sampling? "We used Facebook to quickly reach a large number of county residents and because it allows for granular targeting by zip code and sociodemographic characteristics. We used a combination of two targeting strategies: ads aimed at a representative population of the county by zip code, and specially targeted ads to balance our sample for under-represented zip codes. In addition, we capped registrations from overrepresented areas." Update: Okay, "Other biases, such as bias favoring individuals in good health capable of attending our testing sites, or bias favoring those with prior COVID-like illnesses seeking antibody confirmation are also possible."

rhema 6y ago
If part of the value for participants was to know whether they had antibodies, then those that had gone through a flu (or covid) would be more likely to respond. This means that the overall sample likely includes more people who had COVID than a true random sampling would.
- tshadley 6y ago
  It looks like there's an argument to be made it could overcount or undercount depending on importance of each potential bias. "This study had several limitations. First, our sampling strategy selected for members of Santa Clara County with access to Facebook and a car to attend drive-through testing sites. This resulted in an overrepresentation of white women between the ages of 19 and 64, and an under-representation of Hispanic and Asian populations, relative to our community. Those imbalances were partly addressed by weighting our sample population by zip code, race, and sex to match the county. We did not account for age imbalance in our sample, and could not ascertain representativeness of SARS-CoV-2 antibodies in homeless populations. Other biases, such as bias favoring individuals in good health capable of attending our testing sites, or bias favoring those with prior COVID-like illnesses seeking antibody confirmation are also possible. The overall effect of such biases is hard to ascertain."

not2b 6y ago

Yes, the ads accurately described the study. Several friends of mine participated (I live in Santa Clara County). Of course there is selection bias, which is why they make adjustments to make the sample better represent the population (see the abstract for details on this, or read the paper for all the details, there's well-established science behind these adjustments and it's up to the reviewers to be sure that they did it correctly). The point is that "they only found 50" is quite a large number compared to the number of confirmed cases.

gryson 6y ago
What adjustments were made to account for selection bias of the sort described? It's not covered in the paper, and "well-established science" certainly doesn't answer the question.
- compiler-guy 6y ago
  These are very common statistical methods when dealing with surveys and populations. It isn't this paper's job to describe the background on something like that, just like it isn't this papers job to explain blood testing. They describe their limitations and adjustments exactly like most other studies of this sort do. For example: "Those imbalances were partly addressed by weighting our sample population by zip code, race, and sex to match the county. "
  - tomerico 6y ago
    They said that they collected prior clinical symptoms in the survey, but have not mentioned that they adjusted for those. Given that, I think it's fair to say that they have not adjusted for the selection bias of people with prior clinical symptoms being more likely to click the ad.
  - gryson 6y ago
    On the contrary - detecting selection bias of this sort can be very difficult. Do you measure it using a questionnaire? You might ask your subjects "Do you think you previously had COVID-19?" But what do you do with that response data? Exclude everyone that says yes? Then you're skewing one way. There's obviously no population data to match that response with like there is for general demographics such as race and age. Many studies are only as good as their sampling methods. A good sampling method will alleviate concerns of selection bias.

networkimprov 6y ago

In the data from Italy, ~14% of the >1M tests done for the virus (not antibodies) were positive. That prob includes ppl tested multiple times, or after recovery. I wonder why this is so vastly different from the study above. https://en.wikipedia.org/wiki/2020_coronavirus_pandemic_in_Italy https://en.wikipedia.org/wiki/2020_coronavirus_pandemic_in_I...

lern_too_spel 6y ago

There is also selection bias in whom Facebook will show the ads to due to Facebook's CTR models, other bidders' bids, and who is browsing Facebook more to see more impressions in general. They should have worked with an ad platform to run the ad as a PSA so they could use the ad platform's own randomization instead of relying on having the auctions they win result in anywhere close to a usable sample.