7 ms·

Yes! There are unavoidable, expected, sampling errors which are, by definition, random. That's why valid, trusted polls calculate a confidence interval instead

Yes! There are unavoidable, expected, sampling errors which are, by definition, random. That's why valid, trusted polls calculate a confidence interval instead of a single discrete result. Other types of "errors" -- election results that repeatedly fall outside the confidence interval, or are consistently on only one side of the mean -- only arise when the poll is flawed for some reason. Maybe you relied on landlines only, maybe you spoke with too many men, or too many young people, asked bad questions, miscalculated "likely voter," whatever. Accurate, valid, trusted polls don't have these flaws, the ONLY errors are small, random, expected sampling errors.

Karrot_Kream 6y ago
> That's why valid, trusted polls calculate a confidence interval instead of a single discrete result. That is what each of these statistical models did, yes. And the actual outcomes fell into these confidence intervals. > Other types of "errors" -- election results that repeatedly fall outside the confidence interval, or are consistently on only one side of the mean -- only arise when the poll is flawed for some reason. Or the model was inaccurate. Perhaps the priors were too specific. Perhaps the data was missing, misrecorded, not tabulated properly, who knows. Again, the results fell within the CI of most models, the problem was simply that the result fell too close to the mean for most statisticians' comfort.
- listenallyall 6y ago
  Landed within the confidence interval? Are you kidding? CI is generally 2-4 points in these election polls. - Texas: 538 said Trump +1.0, actually won by 6 - Ohio: Trump +0.4, won by 8 - Iowa: Trump +1.4, won by 8 - Florida: Trump -2.5, won by 3 - Penn.: Trump -4.6, lost by 0.5 - Nevada: Trump -4.9, lost by 2 - Wisconsin: Trump -7.9, lost by 0.6 https://en.wikipedia.org/wiki/Statewide_opinion_polling_for_the_2020_United_States_presidential_election https://en.wikipedia.org/wiki/Statewide_opinion_polling_for_... https://www.nytimes.com/interactive/2020/11/03/us/elections/results-president.html https://www.nytimes.com/interactive/2020/11/03/us/elections/...
  - Karrot_Kream 6y ago
    > Landed within the confidence interval? Are you kidding? CI is generally 2-4 points in these election polls. The models and their data are public. The 538 model predicted an 80%CI of electoral votes for Biden as: 267-419, with the CI centered around 348.49 EVs. That means that Biden had an 80% chance of landing in the above confidence interval. Things seem to be shaking out to Biden winning with 297 EVs. Notice that this falls squarely within the CI of the model, but much further from the median of the CI than expected. So yes, the results fell within the CI. Drilling into Florida specifically (simply because I've been playing around with Florida's data), the 538 model predicts an 80%CI of Biden winning 47.55%-54.19% of the vote. Biden lost Florida, and received 47.8% of the vote. Again, note that this is on the left side of this CI but still within it. The 538 model was correct, the actual results just resided in its left tail.
    - listenallyall 6y ago
      Dude, you're gaslighting by using the national results as evidence instead of the individual states, which is what this has always been about since my original comment. Nearly every consequential state fell at, or beyond, the tail end of 538's confidence interval (BTW, who uses 80%? and not 90-95%?), on the same side. A bit closer to the mean in AZ and GA but same side, over-estimating Biden's margin of victory. Deny it all you want, gaslight, cover your eyes, whatever -- but clear, convincing, overwhelming evidence of a systematic flaw or bias in the underlying polls is right there in front of you. Many political handicappers had predicted that the Democrats would pick up three to 15 seats, growing their 232-to-197 majority https://www.washingtonpost.com/politics/house-races/2020/11/02/9823d974-1d64-11eb-b532-05c751cd5dc2_story.html https://www.washingtonpost.com/politics/house-races/2020/11/... Entering Election Day, forecasters projected Democrats would gain House seats and challenge for the Senate majority. https://www.cnbc.com/2020/11/05/2020-election-results-democrats-to-lose-house-seats-but-keep-majority.html https://www.cnbc.com/2020/11/05/2020-election-results-democr... Most nonpartisan handicappers had long since predicted that Democrats were very likely to win the majority on November 3. "Democrats remain the clear favorites to take back the Senate with just days to go until Election Day," wrote the Cook Political Report's Senate editor Jessica Taylor on October 29. https://www.cnn.com/2020/11/04/politics/2020-election-senate-majority-gop/index.html https://www.cnn.com/2020/11/04/politics/2020-election-senate...
- hackinthebochs 6y ago
  >That is what each of these statistical models did, yes. And the actual outcomes fell into these confidence intervals. The CI is due to sampling error, not model error. If the error of the estimate is due to sampling error, the estimate should be randomly distributed about true value. When the estimate is consistently biased in one direction, that's modelling error, which the CI does not capture.
  - Karrot_Kream 6y ago
    > If the error of the estimate is due to sampling error What does "estimate" mean here? Gelman's model is a Bayesian one, and 538 uses a Markov Chain model. In these instances, what would the "estimate" be? In a frequentist model, yes, you come up with an ML (or MAP or such) estimate, and if the ML estimate is incorrect, then there probably is an issue with the model, but neither of these models use a single estimate. Bayesian methods are all about modelling a posterior, and so the CI is "just" finding which parts of the posterior centered around the median contain the area of your CI. I'm not saying that there isn't model error or sampling error or both. I'm just saying we don't know what caused it yet.
dragonwriter 6y ago
> Accurate, valid, trusted polls don't have these flaw Yes, they do. Because (among many other reasons) humans have a choice whether or not to respond, you can't do an ideal random sample subject to only sampling error for a poll. All polls have non-sampling error on top of sampling error, it is impossible not to.
- listenallyall 6y ago
  when polls don't match up with reality, as they didn't in 2016, the pollsters have a responsibility to re-calibrate the way they conduct the poll. Ask different questions, find new ways of obtaining respondents from all demographics, adjust raw data, etc. A professional pollster doesn't just get to say, hey, some people didn't want to talk to me ¯\_(ツ)_/¯
  - dragonwriter 6y ago
    > when polls don't match up with reality, as they didn't in 2016, the pollsters have a responsibility to re-calibrate the way they conduct the poll. Pollsters do that for continuously, and there were definite recalibrations in the wake of 2016. OTOH, the conditions which produce non-sampling errors aren't static, and it's impossible to reliably even measure the aggregate of non-sampling error in any particular event (because sampling error exists, and while it's statistical distribution can be computed the actual error attributable to it in a by particular event can't be, so you never no how much actual error is due to non-sampling error much less any particular source of non-sampling error.)
JamisonM 6y ago
"There are unavoidable, expected, sampling errors which are, by definition, random." This is false, if you think sampling errors are "by definition random" you don't understand polling. Which is fine, just accept that you don't and dig into the literature or move on.

Karrot_Kream 6y ago

> That's why valid, trusted polls calculate a confidence interval instead of a single discrete result. That is what each of these statistical models did, yes. And the actual outcomes fell into these confidence intervals. > Other types of "errors" -- election results that repeatedly fall outside the confidence interval, or are consistently on only one side of the mean -- only arise when the poll is flawed for some reason. Or the model was inaccurate. Perhaps the priors were too specific. Perhaps the data was missing, misrecorded, not tabulated properly, who knows. Again, the results fell within the CI of most models, the problem was simply that the result fell too close to the mean for most statisticians' comfort.

listenallyall 6y ago
Landed within the confidence interval? Are you kidding? CI is generally 2-4 points in these election polls. - Texas: 538 said Trump +1.0, actually won by 6 - Ohio: Trump +0.4, won by 8 - Iowa: Trump +1.4, won by 8 - Florida: Trump -2.5, won by 3 - Penn.: Trump -4.6, lost by 0.5 - Nevada: Trump -4.9, lost by 2 - Wisconsin: Trump -7.9, lost by 0.6 https://en.wikipedia.org/wiki/Statewide_opinion_polling_for_the_2020_United_States_presidential_election https://en.wikipedia.org/wiki/Statewide_opinion_polling_for_... https://www.nytimes.com/interactive/2020/11/03/us/elections/results-president.html https://www.nytimes.com/interactive/2020/11/03/us/elections/...
- Karrot_Kream 6y ago
  > Landed within the confidence interval? Are you kidding? CI is generally 2-4 points in these election polls. The models and their data are public. The 538 model predicted an 80%CI of electoral votes for Biden as: 267-419, with the CI centered around 348.49 EVs. That means that Biden had an 80% chance of landing in the above confidence interval. Things seem to be shaking out to Biden winning with 297 EVs. Notice that this falls squarely within the CI of the model, but much further from the median of the CI than expected. So yes, the results fell within the CI. Drilling into Florida specifically (simply because I've been playing around with Florida's data), the 538 model predicts an 80%CI of Biden winning 47.55%-54.19% of the vote. Biden lost Florida, and received 47.8% of the vote. Again, note that this is on the left side of this CI but still within it. The 538 model was correct, the actual results just resided in its left tail.
  - listenallyall 6y ago
    Dude, you're gaslighting by using the national results as evidence instead of the individual states, which is what this has always been about since my original comment. Nearly every consequential state fell at, or beyond, the tail end of 538's confidence interval (BTW, who uses 80%? and not 90-95%?), on the same side. A bit closer to the mean in AZ and GA but same side, over-estimating Biden's margin of victory. Deny it all you want, gaslight, cover your eyes, whatever -- but clear, convincing, overwhelming evidence of a systematic flaw or bias in the underlying polls is right there in front of you. Many political handicappers had predicted that the Democrats would pick up three to 15 seats, growing their 232-to-197 majority https://www.washingtonpost.com/politics/house-races/2020/11/02/9823d974-1d64-11eb-b532-05c751cd5dc2_story.html https://www.washingtonpost.com/politics/house-races/2020/11/... Entering Election Day, forecasters projected Democrats would gain House seats and challenge for the Senate majority. https://www.cnbc.com/2020/11/05/2020-election-results-democrats-to-lose-house-seats-but-keep-majority.html https://www.cnbc.com/2020/11/05/2020-election-results-democr... Most nonpartisan handicappers had long since predicted that Democrats were very likely to win the majority on November 3. "Democrats remain the clear favorites to take back the Senate with just days to go until Election Day," wrote the Cook Political Report's Senate editor Jessica Taylor on October 29. https://www.cnn.com/2020/11/04/politics/2020-election-senate-majority-gop/index.html https://www.cnn.com/2020/11/04/politics/2020-election-senate...
hackinthebochs 6y ago
>That is what each of these statistical models did, yes. And the actual outcomes fell into these confidence intervals. The CI is due to sampling error, not model error. If the error of the estimate is due to sampling error, the estimate should be randomly distributed about true value. When the estimate is consistently biased in one direction, that's modelling error, which the CI does not capture.
- Karrot_Kream 6y ago
  > If the error of the estimate is due to sampling error What does "estimate" mean here? Gelman's model is a Bayesian one, and 538 uses a Markov Chain model. In these instances, what would the "estimate" be? In a frequentist model, yes, you come up with an ML (or MAP or such) estimate, and if the ML estimate is incorrect, then there probably is an issue with the model, but neither of these models use a single estimate. Bayesian methods are all about modelling a posterior, and so the CI is "just" finding which parts of the posterior centered around the median contain the area of your CI. I'm not saying that there isn't model error or sampling error or both. I'm just saying we don't know what caused it yet.

dragonwriter 6y ago

> Accurate, valid, trusted polls don't have these flaw Yes, they do. Because (among many other reasons) humans have a choice whether or not to respond, you can't do an ideal random sample subject to only sampling error for a poll. All polls have non-sampling error on top of sampling error, it is impossible not to.

listenallyall 6y ago
when polls don't match up with reality, as they didn't in 2016, the pollsters have a responsibility to re-calibrate the way they conduct the poll. Ask different questions, find new ways of obtaining respondents from all demographics, adjust raw data, etc. A professional pollster doesn't just get to say, hey, some people didn't want to talk to me ¯\_(ツ)_/¯
- dragonwriter 6y ago
  > when polls don't match up with reality, as they didn't in 2016, the pollsters have a responsibility to re-calibrate the way they conduct the poll. Pollsters do that for continuously, and there were definite recalibrations in the wake of 2016. OTOH, the conditions which produce non-sampling errors aren't static, and it's impossible to reliably even measure the aggregate of non-sampling error in any particular event (because sampling error exists, and while it's statistical distribution can be computed the actual error attributable to it in a by particular event can't be, so you never no how much actual error is due to non-sampling error much less any particular source of non-sampling error.)

JamisonM 6y ago

"There are unavoidable, expected, sampling errors which are, by definition, random." This is false, if you think sampling errors are "by definition random" you don't understand polling. Which is fine, just accept that you don't and dig into the literature or move on.