6 ms·

The sample selection / non-response bias highlighted in this write-up is a _Big Idea_ problem I've been thinking about recently: Limitations...*Trust in surv

The sample selection / non-response bias highlighted in this write-up is a _Big Idea_ problem I've been thinking about recently: Limitations...*Trust in surveys and political leanings:* About 95 percent of people contacted for the panel chose not to participate because of lack of trust in having a third-party application installed on their computer or other concerns for privacy. Think about that - a reputable, privacy-first organization asked people to opt-in to fully consented, voluntary, compensated research and ~95% declined! I can't even imagine what hidden skews are present in the 5% that agreed. This issue is systemic in consumer research and impacts both public (e.g. election polling, U.S. census) and private (pharmaceutical trials, media/advertising research, voluntary AI/ML training daat) polling. Governments and businesses make biased, potentially discriminatory decisions if a non-random segment of the population chooses to never be counted. The ad industry attempts to circumvent this through non-voluntary passive tracking, which trades off non-response bias with bulldozing user privacy. The headwinds are only growing too, as consumer awareness of privacy lapses and the politicization of polling continues to reduce who participates in opt-in research. Finding a solution to this that doesn't resort to privacy-eroding tactics is a moonshot level problem in terms of the size-of-the-prize if solved.

Nextgrid 6y ago
Is it a problem that should be solved? Shouldn't people have the right to live in peace and not be forced to participate in some survey? I also don't really see the problem with 95% of people declining; they made the smart choice. If I was a regular Facebook user I would also decline because the account would contain tons of very sensitive data such as DMs and running an untrusted, unknown application on my main computer is also a major dealbreaker.
- typeiierror 6y ago
  Try applying the problem to other issues to see the impact: * An advertiser wants to place ads on sites / tv networks that have an audience that is more likely to buy their product upon seeing their ads. If they don't want to violate privacy, they run a survey. What if the response rate among a historical disenfranchised group (e.g. African Americans) is terrible? The modern "data driven" marketer would see little reason to advertise on Black media properties. This isn't a fictious example - it's a current problem in the media planning / agency industry. * A local government has to decide between investing in more ESL resources in public education vs. other competing budget needs. They look at census / community survey data (which some Hispanic and immigrant populations are fearful of responding to d/t politicization) and decide to prioritize other asks due to undercounted demand. The data could also be skewed in other ways that warp their decision, like allocating budget to school zones that only represent specific immigrant communities that haven't historically been disenfranchised. The big picture issue here is governments/businesses making decisions with bias information leading to incorrect conclusions, and the only know recourse currently is to scrap privacy.
  - TeMPOraL 6y ago
    Look at it from the point of view of regular folks: * An advertiser - a malicious being intent on tricking me out of my money - wants to make a survey to determine how to make it easier to trick people into parting with their money. Why would I help someone make my life, and life of other people like me, worse? The answer to that is to beat advertising down until it isn't so blatantly customer-hostile. Then people may be more willing to help. * I'm in a politically precarious situation and the government is asking questions - ostensibly for purposes that could benefit me, but if my honest answers were seen by a different government agency, it would cause me a world of hurt. I hide away. Or lie. The answer to that is ideally to fix the politically precarious situation of a subset of your population - but at the very least, to foster the trust in information separation between government agencies, so that I can e.g. afford to be honest with the census bureau without worrying about the IRS or the police. That level of trust is not the default.
    - indymike 6y ago
      You've really summed up the state of the world right now: we're in a crisis of trust. We don't trust each other, we don't trust institutions and the result is anxiety, fear and anger.
      - ryukafalz 6y ago
        I broadly agree, but I would frame it a bit differently: we have a severe lack of trustworthiness in our modern world; or, at least, the trustworthy voices are lost in the noise. This is a big big part of why I primarily use FOSS as much as possible. Generally speaking, FOSS developers and distributors seem to act with the user's interests in mind more often than proprietary software vendors. (Certainly the distributions do, probably out of necessity - there's no shortage of competitive distro options, so a distro being shady is practically a death sentence. Individual developers still deserve more scrutiny.) The advertisers certainly do not have my best interests in mind.
        danShumway 6y ago
nitwit005 6y ago
An organization people had never heard of asked them to install a browser plugin, and they declined like they should. Even if it's an organization you recognize, verifying it's not someone using their name for some sort of scam isn't always straightforward.
- danShumway 6y ago
  Right. An alternate take on this passage is, "95% of contacts exercised basic security measures rather than blindly install unvetted code on their machine." It doesn't take that much of a shift in perspective to turn this into a very optimistic statistic indicating that normal users aren't always quite as security-unconscious as we normally think -- that for whatever reason (security, apathy, paranoia, whatever) sometimes they do actually make the right choices when interacting with sensitive information like their browsing habits. You could not pay me to install an unvetted Electron application where I can't even see the source code, that is designed to MITM my browsing activity. Even if I trust the author's intention, who wrote the app? Who tested it? How do I know that the automatic redactors are going to actually work? It's not like it's hard to have security leaks in Electron.
- kleiba 6y ago
  But that's exactly the point of the OP: 95% "declined like they should" -- but what does that say about the 5% that didn't? What general conclusions can you draw from data elicited through people that are clearly unlike the mainstream?
pjc50 6y ago
The key thing here is trust. Trust is a resource like a rainforest: you can exploit it sustainably, or you can get a far greater profit by destroying it. Trusting people are a resource that can be exploited for fraud, which the internet is great at producing. It's not really surprising that random organizations find low trust. It is however very unfortunate, as historically being a "high trust society" has been a great advantage of the west. And it's going to take a lot of repairing.
ForHackernews 6y ago
Just do what Nielson does: Pay people for their data. I don't think it's that difficult; if pay-for-survey skews results toward overvaluing the opinions of the poor and/desperate-for-money, well, then it would be the first time in history.
OneGuy123 6y ago
> a reputable, privacy-first organization... Are you a shill for them? "Reputable" as mainstream media? "Reputable" as in the fake news is reputable? Whenever someone claims "I am reputable" you should run away as fast as you can. This "I am reputable" is purely subjective based on your own biases and incentives. No one can be reputable in the news space and people must understand this as soon as possible: everyone lies, even natural science which is supposed to be the gold standard gets so many studies wrong. The only way news organizations stay alive is either by: a) clickbait articles which eventualy devolves into lying or exaggeration at best. b) News orgs that are financed by private people/corporations who have their own agendas. There is no such thing as a "reputable" news source. Perhaps one in ten thousand jounralist is still legit, so that he is an actual investigative journalists. But 99.999 of "journalists" are actually just script readers and clickbait writers. Do you not see where "I am reputable" leads to? Soon the "I am reputable" organization will get political power and then they will make laws based around "my reputable reporting" and this will lead to censortship. "I am reputable" always leads to censorship down the line since it implies that "my opponent is not reputable and is lying and MUST BE SHUT DOWN IN THE NAME OF DEMOCRACY".
- skinnymuch 6y ago
  Look at your response. Why would OP be a shill when you’ve responded this way? It’s not hard to see people will have strong opinions on things even if they believe their opinion isn’t wild, like yourself.
- pjc50 6y ago
  Per my answer above: the situation where nobody can be trusted is horribly unstable, because evidence-based or impartial policymaking or even justice becomes impossible. This tends to result in replacing trust relationships with force relationships, and the society devolves into warlordism or dictatorship in order to restore order and control. You can't expect people to trust an election where all candidates are disreputable, so they vote in a dictator. This is why the unreliability of news organizations is such a serious problem.
nxpnsv 6y ago
Meanwhile fb has all of this info from the 100%...
AlexandrB 6y ago
In my opinion, this is a symptom of weak/ineffective regulation in the personal information space. The consequences for data breaches to the guilty parties have been minimal at best. Meanwhile responsibility for fraud has been pushed onto individuals via concepts like "identity theft". Even if the company in question was indeed reputable and well-known, most people don't have the technical expertise to evaluate any claims about security or privacy. Who would take that risk knowing that at the end of the day most of the consequences will fall on them personally?

Nextgrid 6y ago

Is it a problem that should be solved? Shouldn't people have the right to live in peace and not be forced to participate in some survey? I also don't really see the problem with 95% of people declining; they made the smart choice. If I was a regular Facebook user I would also decline because the account would contain tons of very sensitive data such as DMs and running an untrusted, unknown application on my main computer is also a major dealbreaker.

typeiierror 6y ago
Try applying the problem to other issues to see the impact: * An advertiser wants to place ads on sites / tv networks that have an audience that is more likely to buy their product upon seeing their ads. If they don't want to violate privacy, they run a survey. What if the response rate among a historical disenfranchised group (e.g. African Americans) is terrible? The modern "data driven" marketer would see little reason to advertise on Black media properties. This isn't a fictious example - it's a current problem in the media planning / agency industry. * A local government has to decide between investing in more ESL resources in public education vs. other competing budget needs. They look at census / community survey data (which some Hispanic and immigrant populations are fearful of responding to d/t politicization) and decide to prioritize other asks due to undercounted demand. The data could also be skewed in other ways that warp their decision, like allocating budget to school zones that only represent specific immigrant communities that haven't historically been disenfranchised. The big picture issue here is governments/businesses making decisions with bias information leading to incorrect conclusions, and the only know recourse currently is to scrap privacy.
- TeMPOraL 6y ago
  Look at it from the point of view of regular folks: * An advertiser - a malicious being intent on tricking me out of my money - wants to make a survey to determine how to make it easier to trick people into parting with their money. Why would I help someone make my life, and life of other people like me, worse? The answer to that is to beat advertising down until it isn't so blatantly customer-hostile. Then people may be more willing to help. * I'm in a politically precarious situation and the government is asking questions - ostensibly for purposes that could benefit me, but if my honest answers were seen by a different government agency, it would cause me a world of hurt. I hide away. Or lie. The answer to that is ideally to fix the politically precarious situation of a subset of your population - but at the very least, to foster the trust in information separation between government agencies, so that I can e.g. afford to be honest with the census bureau without worrying about the IRS or the police. That level of trust is not the default.
  - indymike 6y ago
    You've really summed up the state of the world right now: we're in a crisis of trust. We don't trust each other, we don't trust institutions and the result is anxiety, fear and anger.
    - ryukafalz 6y ago
      I broadly agree, but I would frame it a bit differently: we have a severe lack of trustworthiness in our modern world; or, at least, the trustworthy voices are lost in the noise. This is a big big part of why I primarily use FOSS as much as possible. Generally speaking, FOSS developers and distributors seem to act with the user's interests in mind more often than proprietary software vendors. (Certainly the distributions do, probably out of necessity - there's no shortage of competitive distro options, so a distro being shady is practically a death sentence. Individual developers still deserve more scrutiny.) The advertisers certainly do not have my best interests in mind.
      - danShumway 6y ago

nitwit005 6y ago

An organization people had never heard of asked them to install a browser plugin, and they declined like they should. Even if it's an organization you recognize, verifying it's not someone using their name for some sort of scam isn't always straightforward.

danShumway 6y ago
Right. An alternate take on this passage is, "95% of contacts exercised basic security measures rather than blindly install unvetted code on their machine." It doesn't take that much of a shift in perspective to turn this into a very optimistic statistic indicating that normal users aren't always quite as security-unconscious as we normally think -- that for whatever reason (security, apathy, paranoia, whatever) sometimes they do actually make the right choices when interacting with sensitive information like their browsing habits. You could not pay me to install an unvetted Electron application where I can't even see the source code, that is designed to MITM my browsing activity. Even if I trust the author's intention, who wrote the app? Who tested it? How do I know that the automatic redactors are going to actually work? It's not like it's hard to have security leaks in Electron.
kleiba 6y ago
But that's exactly the point of the OP: 95% "declined like they should" -- but what does that say about the 5% that didn't? What general conclusions can you draw from data elicited through people that are clearly unlike the mainstream?

pjc50 6y ago

The key thing here is trust. Trust is a resource like a rainforest: you can exploit it sustainably, or you can get a far greater profit by destroying it. Trusting people are a resource that can be exploited for fraud, which the internet is great at producing. It's not really surprising that random organizations find low trust. It is however very unfortunate, as historically being a "high trust society" has been a great advantage of the west. And it's going to take a lot of repairing.

ForHackernews 6y ago

Just do what Nielson does: Pay people for their data. I don't think it's that difficult; if pay-for-survey skews results toward overvaluing the opinions of the poor and/desperate-for-money, well, then it would be the first time in history.

OneGuy123 6y ago

> a reputable, privacy-first organization... Are you a shill for them? "Reputable" as mainstream media? "Reputable" as in the fake news is reputable? Whenever someone claims "I am reputable" you should run away as fast as you can. This "I am reputable" is purely subjective based on your own biases and incentives. No one can be reputable in the news space and people must understand this as soon as possible: everyone lies, even natural science which is supposed to be the gold standard gets so many studies wrong. The only way news organizations stay alive is either by: a) clickbait articles which eventualy devolves into lying or exaggeration at best. b) News orgs that are financed by private people/corporations who have their own agendas. There is no such thing as a "reputable" news source. Perhaps one in ten thousand jounralist is still legit, so that he is an actual investigative journalists. But 99.999 of "journalists" are actually just script readers and clickbait writers. Do you not see where "I am reputable" leads to? Soon the "I am reputable" organization will get political power and then they will make laws based around "my reputable reporting" and this will lead to censortship. "I am reputable" always leads to censorship down the line since it implies that "my opponent is not reputable and is lying and MUST BE SHUT DOWN IN THE NAME OF DEMOCRACY".

skinnymuch 6y ago
Look at your response. Why would OP be a shill when you’ve responded this way? It’s not hard to see people will have strong opinions on things even if they believe their opinion isn’t wild, like yourself.
pjc50 6y ago
Per my answer above: the situation where nobody can be trusted is horribly unstable, because evidence-based or impartial policymaking or even justice becomes impossible. This tends to result in replacing trust relationships with force relationships, and the society devolves into warlordism or dictatorship in order to restore order and control. You can't expect people to trust an election where all candidates are disreputable, so they vote in a dictator. This is why the unreliability of news organizations is such a serious problem.

nxpnsv 6y ago

Meanwhile fb has all of this info from the 100%...

AlexandrB 6y ago

In my opinion, this is a symptom of weak/ineffective regulation in the personal information space. The consequences for data breaches to the guilty parties have been minimal at best. Meanwhile responsibility for fraud has been pushed onto individuals via concepts like "identity theft". Even if the company in question was indeed reputable and well-known, most people don't have the technical expertise to evaluate any claims about security or privacy. Who would take that risk knowing that at the end of the day most of the consequences will fall on them personally?