4 ms·

The article addresses your point in some detail.

by d0mine 3y ago

derbOac 3y ago
I do survey research, and there's ways of addressing the concerns the author raises with survey techniques. It's still interesting and they're right that these types of things (rare, taboo) are sometimes actually best studied with biased samples sometimes in some ways, with lots of anonymity. However, there are ways of addressing these concerns in "normal" samples (list randomization is an example; https://www.svri.org/blog/measuring-taboo-topics-list-randomization-gender-based-violence https://www.svri.org/blog/measuring-taboo-topics-list-random...), especially if they're large enough, and with trying to weight things. There are anonymous surveys that are carefully sampled; I'm not sure people would trust those less than something like the linked survey. If you used the right questions, you could also probably try to reweight the sample for other behaviors. This comes across as a criticism and maybe it is but I tend to think the more information like this the better, so I still think of it as a net good thing. It's just difficult to make much of things like percentiles in the absence of a very strong rationale for what you're comparing against, either in terms of a comparison population or in terms of what is being compared. Is an overall kink score meaningful? What about an overall sadomasochism score? Relative to who? You never really know your population, but sometimes you can get enough information to get a good enough sense of where your sample is in context.
- d0mine 3y ago
  Is list randomization applicable to a survey with ~100 sensitive questions? From your link "In basic terms, list randomization aggregates a response to a sensitive question alongside responses to non-sensitive questions, thereby masking the respondent’s specific answer to the sensitive question. By randomizing lists with and without the sensitive question, researchers can identify prevalence or incidence of the sensitive item within the population or differences between groups (for example treatment and control), but not attribute the sensitive response on an individual basis. If respondents believe that their sensitive answer is not disclosed to the interviewer, they may be more likely to report private behaviors, such as experience of violence."
  - derbOac 3y ago
    100 questions would be a lot, yes, unless you had very large samples, which isn't unheard of in today's age. I suspect it would work better to combine the items somehow. Alternatively, you might be able to somehow use a subset of the 100 as a kind of anchor, to see how distorted a differently designed sample was, and possibly reweight things. For example, if you were really serious, you might be able to identify, say, 5-10 items with a lot of variance in the population, and ask about them using list randomization, and also maybe directly to determine how people endorse the items differently if asked indirectly versus directly. Depending on what you did with that, you could then use those items as weighting variables for the other 90-95 items in another sample. In the very least you'd be able to compare the samples to see how different the distributions are on the anchor items.

The article addresses your point in some detail.

by d0mine 3y ago

The article addresses your point in some detail.

I do survey research, and there's ways of addressing the concerns the author raises with survey techniques. It's still interesting and they're right that these types of things (rare, taboo) are sometimes actually best studied with biased samples sometimes in some ways, with lots of anonymity. However, there are ways of addressing these concerns in "normal" samples (list randomization is an example; https://www.svri.org/blog/measuring-taboo-topics-list-randomization-gender-based-violence https://www.svri.org/blog/measuring-taboo-topics-list-random...), especially if they're large enough, and with trying to weight things. There are anonymous surveys that are carefully sampled; I'm not sure people would trust those less than something like the linked survey. If you used the right questions, you could also probably try to reweight the sample for other behaviors. This comes across as a criticism and maybe it is but I tend to think the more information like this the better, so I still think of it as a net good thing. It's just difficult to make much of things like percentiles in the absence of a very strong rationale for what you're comparing against, either in terms of a comparison population or in terms of what is being compared. Is an overall kink score meaningful? What about an overall sadomasochism score? Relative to who? You never really know your population, but sometimes you can get enough information to get a good enough sense of where your sample is in context.

d0mine 3y ago
Is list randomization applicable to a survey with ~100 sensitive questions? From your link "In basic terms, list randomization aggregates a response to a sensitive question alongside responses to non-sensitive questions, thereby masking the respondent’s specific answer to the sensitive question. By randomizing lists with and without the sensitive question, researchers can identify prevalence or incidence of the sensitive item within the population or differences between groups (for example treatment and control), but not attribute the sensitive response on an individual basis. If respondents believe that their sensitive answer is not disclosed to the interviewer, they may be more likely to report private behaviors, such as experience of violence."
- derbOac 3y ago
  100 questions would be a lot, yes, unless you had very large samples, which isn't unheard of in today's age. I suspect it would work better to combine the items somehow. Alternatively, you might be able to somehow use a subset of the 100 as a kind of anchor, to see how distorted a differently designed sample was, and possibly reweight things. For example, if you were really serious, you might be able to identify, say, 5-10 items with a lot of variance in the population, and ask about them using list randomization, and also maybe directly to determine how people endorse the items differently if asked indirectly versus directly. Depending on what you did with that, you could then use those items as weighting variables for the other 90-95 items in another sample. In the very least you'd be able to compare the samples to see how different the distributions are on the anchor items.