7 ms·

Most important paragraph, IMO: > Conducted Oct. 8 to 10, the online poll surveyed 1,545 Canadians and cannot be assigned a margin of error because internet-bas

Most important paragraph, IMO: > Conducted Oct. 8 to 10, the online poll surveyed 1,545 Canadians and cannot be assigned a margin of error because internet-based polls are not considered random samples. It's a pretty good sample-size, but we don't get error bars. I don't remember if there was anything FB-critical happening that week (when did the study about teen mental health come out?), but the timeframe could also affect the survey.

cblconfederate 5y ago
it 's not just missing error bars, this is not a sample, period. so it's not something reliable.
- xdavidliu 5y ago
  it is a sample. It's just not an unbiased sample [1]. [1] https://en.wikipedia.org/wiki/Sample_(statistics)#Kinds_of_samples https://en.wikipedia.org/wiki/Sample_(statistics)#Kinds_of_s...
  - hcrisp 5y ago
    Very unbiased. Only sampled people who were online and ostensibly only those who used Facebook. Since maybe ~60% of people use Facebook, I don't think the results speak for _all_ Canadians. If it did that's saying a lot for secondhand effects of social media platforms.
    - karmanyaahm 5y ago
      Not totally sure, but aren't people with extreme feelings more likely to answer a survey? Voluntary or nonresponse biases
      - itsoktocry 5y ago
        >Not totally sure, but... This kind of discourse is annoying. You don't get "points" for making some hypothetical claim. If you think there is something wrong with an analysis, demonstrate it. Isn't that what "pro-science" people do?
        jaybz000 5y ago
        And therein lies the problem with online polling. The randomness, or lack thereof, is not demonstrable without polling people who did not respond to the poll, via a different medium. Hence the standard disclaimer.
    - stormbrew 5y ago
      It's not a facebook poll, not sure why you thought that. It was a poll taken from a random (presumably population-reflecting) sample of Leger's online polling platform. It's not technically a random poll like wardialing in the 40s got you but it's not just an opt-in facebook poll either. https://leger360.com/surveys/legers-north-american-tracker-october-13-2021/ https://leger360.com/surveys/legers-north-american-tracker-o...
      - hcrisp 5y ago
        Without access to the questions I don't know exactly who would have a tendency to respond. But would you answer a poll about how a service is affecting you if you didn't use it? And if not, that's my point -- that both the article's title and contents seem to speak for Canadians en masse but is likely not to, because it likely only sampled the 60% who use Facebook.
        stormbrew 5y ago
        I really don't know why you'd assume people wouldn't tbh. Yes, I am (approximately, at least) a non-user of facebook and if I were being polled about it I would answer a question about my views on its effects on society. Also, the most pertinent question to this discussion is in the post by the polling company I linked to above.
not-elite 5y ago
The formula for error bars under the traditional Binomial assumption is: +/- sqrt(p * (1-p) / N) So the errors are around +/- 1% for most values of p in this article. The article (rightly) points out that the binomial assumption is not reasonable given the survey method. https://stats.stackexchange.com/questions/29641/standard-error-for-the-mean-of-a-sample-of-binomial-random-variables https://stats.stackexchange.com/questions/29641/standard-err...
- klyrs 5y ago
  The traditional binomial assumption assumes independence of the samples, as the accepted answer states. The authors are acknowledging that this assumption does not hold.
  - notafraudster 5y ago
    It's an odd disclaimer. If it's a quota-based sample then the performance in terms of nominal coverage is going to look similar to a probabilistic sample (especially given the current environment for probabilistic samples, which is quite poor). Probabilistic samples don't report design effects or adjust MOEs to reflect them either, the norm is just to report a classical normal approximate binomial confidence interval (e.g. +/- 1/sqrt(n)) even when a real design effect exists. My guess is the reason for this disclaimer is that it's not a quota sample, it's just literally a completely undirected opt-in survey and there's no reason to believe this is anything resembling a representative sample, probabilistic or not.
    - stormbrew 5y ago
      This disclaimer is applied to pretty much all polling done online, even if the samples are weighted to match the population. If you go look at election polls, for eg., the non-IVR ones will all have something like this, or a "If this were a traditional phone poll, the margin of error would be..." Online polls are usually done by letting people opt-in and then sorting and weighting to sample, just like phone polls. The idea though is that because they aren't reached "randomly" in the first place (as they are by war dialing phone numbers for eg.) there's additional sampling biases at play that a margin of error doesn't account for. Leger is a legitimate polling company in Canada, and I doubt they did it any different from how they do it for election polls. I'm not sure why people are assuming this was just an unweighted facebook poll or something. But the reality is that "traditional polls" are probably no better because of the extremely high non-response rate these days. Inertia is a thing though. An article about this subject: https://www.huffpost.com/entry/margin-of-error-debate_n_6565788 https://www.huffpost.com/entry/margin-of-error-debate_n_6565...
passivate 5y ago
Its this sort of sensational/half-true reporting that makes people lose faith in media brands.
- unglaublich 5y ago
  prisoner's dilemma: gain income while lying or let others drain your income by lying?
- itsoktocry 5y ago
  >Its this sort of sensational/half-true reporting that makes people lose faith in media brands. If you think not reporting error bars is what makes people "lose faith in media brands", you are so far away from reality I'm not sure what to tell you. But how about you put aside the statistical nuance of this for 10 seconds and acknowledge the fact that real people, when asked, actually consider Facebook -- a piece of software-- harmful. They are saying that. What does that mean?
  - sigstoat 5y ago
    > But how about you put aside the statistical nuance of this for 10 seconds and acknowledge the fact that real people, when asked, actually consider Facebook -- a piece of software-- harmful. They are saying that. What does that mean? we can probably find 1,545 people who think everything you love and hold dear is ruining the world. what does that mean?
  - passivate 5y ago
    >If you think not reporting error bars is what makes people "lose faith in media brands", you are so far away from reality I'm not sure what to tell you. "If you think.." - Why not ask me what I think instead of assuming and putting up a strawman? >when asked, actually consider Facebook -- a piece of software-- harmful. They are saying that. What does that mean? I don't consider opinion polls as pathways to truths. They're interesting data points, but not some kind of absolute truth. Human opinions change, perspectives change, often multiple times when provided new/different data. Opinions can also be in conflict with each other. We're not automations or pinnacles of logical consistency. You can hand-wave it away as "nuance", but it isn't. Its fundamental. It's messy.
lr4444lr 5y ago
Do you really think non-internet users would make thr "FB is harmful" opinion LESS chosen?
- zdragnar 5y ago
  Plenty aren't aware of what harms it might cause due to lack of exposure to it. Plenty of others might be completely biased based only on anecdotes and bad press. There's no way to tell what the balance is without actually asking them

cblconfederate 5y ago

it 's not just missing error bars, this is not a sample, period. so it's not something reliable.

xdavidliu 5y ago
it is a sample. It's just not an unbiased sample [1]. [1] https://en.wikipedia.org/wiki/Sample_(statistics)#Kinds_of_samples https://en.wikipedia.org/wiki/Sample_(statistics)#Kinds_of_s...
- hcrisp 5y ago
  Very unbiased. Only sampled people who were online and ostensibly only those who used Facebook. Since maybe ~60% of people use Facebook, I don't think the results speak for _all_ Canadians. If it did that's saying a lot for secondhand effects of social media platforms.
  - karmanyaahm 5y ago
    Not totally sure, but aren't people with extreme feelings more likely to answer a survey? Voluntary or nonresponse biases
    - itsoktocry 5y ago
      >Not totally sure, but... This kind of discourse is annoying. You don't get "points" for making some hypothetical claim. If you think there is something wrong with an analysis, demonstrate it. Isn't that what "pro-science" people do?
      - jaybz000 5y ago
        And therein lies the problem with online polling. The randomness, or lack thereof, is not demonstrable without polling people who did not respond to the poll, via a different medium. Hence the standard disclaimer.
  - stormbrew 5y ago
    It's not a facebook poll, not sure why you thought that. It was a poll taken from a random (presumably population-reflecting) sample of Leger's online polling platform. It's not technically a random poll like wardialing in the 40s got you but it's not just an opt-in facebook poll either. https://leger360.com/surveys/legers-north-american-tracker-october-13-2021/ https://leger360.com/surveys/legers-north-american-tracker-o...
    - hcrisp 5y ago
      Without access to the questions I don't know exactly who would have a tendency to respond. But would you answer a poll about how a service is affecting you if you didn't use it? And if not, that's my point -- that both the article's title and contents seem to speak for Canadians en masse but is likely not to, because it likely only sampled the 60% who use Facebook.
      - stormbrew 5y ago
        I really don't know why you'd assume people wouldn't tbh. Yes, I am (approximately, at least) a non-user of facebook and if I were being polled about it I would answer a question about my views on its effects on society. Also, the most pertinent question to this discussion is in the post by the polling company I linked to above.

not-elite 5y ago

The formula for error bars under the traditional Binomial assumption is: +/- sqrt(p * (1-p) / N) So the errors are around +/- 1% for most values of p in this article. The article (rightly) points out that the binomial assumption is not reasonable given the survey method. https://stats.stackexchange.com/questions/29641/standard-error-for-the-mean-of-a-sample-of-binomial-random-variables https://stats.stackexchange.com/questions/29641/standard-err...

klyrs 5y ago
The traditional binomial assumption assumes independence of the samples, as the accepted answer states. The authors are acknowledging that this assumption does not hold.
- notafraudster 5y ago
  It's an odd disclaimer. If it's a quota-based sample then the performance in terms of nominal coverage is going to look similar to a probabilistic sample (especially given the current environment for probabilistic samples, which is quite poor). Probabilistic samples don't report design effects or adjust MOEs to reflect them either, the norm is just to report a classical normal approximate binomial confidence interval (e.g. +/- 1/sqrt(n)) even when a real design effect exists. My guess is the reason for this disclaimer is that it's not a quota sample, it's just literally a completely undirected opt-in survey and there's no reason to believe this is anything resembling a representative sample, probabilistic or not.
  - stormbrew 5y ago
    This disclaimer is applied to pretty much all polling done online, even if the samples are weighted to match the population. If you go look at election polls, for eg., the non-IVR ones will all have something like this, or a "If this were a traditional phone poll, the margin of error would be..." Online polls are usually done by letting people opt-in and then sorting and weighting to sample, just like phone polls. The idea though is that because they aren't reached "randomly" in the first place (as they are by war dialing phone numbers for eg.) there's additional sampling biases at play that a margin of error doesn't account for. Leger is a legitimate polling company in Canada, and I doubt they did it any different from how they do it for election polls. I'm not sure why people are assuming this was just an unweighted facebook poll or something. But the reality is that "traditional polls" are probably no better because of the extremely high non-response rate these days. Inertia is a thing though. An article about this subject: https://www.huffpost.com/entry/margin-of-error-debate_n_6565788 https://www.huffpost.com/entry/margin-of-error-debate_n_6565...

passivate 5y ago

Its this sort of sensational/half-true reporting that makes people lose faith in media brands.

unglaublich 5y ago
prisoner's dilemma: gain income while lying or let others drain your income by lying?
itsoktocry 5y ago
>Its this sort of sensational/half-true reporting that makes people lose faith in media brands. If you think not reporting error bars is what makes people "lose faith in media brands", you are so far away from reality I'm not sure what to tell you. But how about you put aside the statistical nuance of this for 10 seconds and acknowledge the fact that real people, when asked, actually consider Facebook -- a piece of software-- harmful. They are saying that. What does that mean?
- sigstoat 5y ago
  > But how about you put aside the statistical nuance of this for 10 seconds and acknowledge the fact that real people, when asked, actually consider Facebook -- a piece of software-- harmful. They are saying that. What does that mean? we can probably find 1,545 people who think everything you love and hold dear is ruining the world. what does that mean?
- passivate 5y ago
  >If you think not reporting error bars is what makes people "lose faith in media brands", you are so far away from reality I'm not sure what to tell you. "If you think.." - Why not ask me what I think instead of assuming and putting up a strawman? >when asked, actually consider Facebook -- a piece of software-- harmful. They are saying that. What does that mean? I don't consider opinion polls as pathways to truths. They're interesting data points, but not some kind of absolute truth. Human opinions change, perspectives change, often multiple times when provided new/different data. Opinions can also be in conflict with each other. We're not automations or pinnacles of logical consistency. You can hand-wave it away as "nuance", but it isn't. Its fundamental. It's messy.

lr4444lr 5y ago

Do you really think non-internet users would make thr "FB is harmful" opinion LESS chosen?

zdragnar 5y ago
Plenty aren't aware of what harms it might cause due to lack of exposure to it. Plenty of others might be completely biased based only on anecdotes and bad press. There's no way to tell what the balance is without actually asking them