5 ms·

538 analyzed the various polls--and corrected for their methodologies and predictive power--just fine! A sibling comment brings up a batter, and I think that's

538 analyzed the various polls--and corrected for their methodologies and predictive power--just fine! A sibling comment brings up a batter, and I think that's a great analogy. A good batter has a batting average of, say 0.273. Nobody is shocked when they hit the ball! 538 gave a similar chance to Trump. He hit the ball.

rayiner 6y ago
Yes and no. 538 published an article after the election explaining that there were in fact systemic errors in the models; the result wasn't just random probability within models that were otherwise correct: https://fivethirtyeight.com/features/the-polls-missed-trump-we-asked-pollsters-why/ https://fivethirtyeight.com/features/the-polls-missed-trump-... > While the errors were nationwide, they were spread unevenly. The more whites without college degrees were in a state, the more Trump outperformed his FiveThirtyEight polls-only adjusted polling average,1 suggesting the polls underestimated his support with that group. And the bigger the lead we forecast for Trump, the more he outperformed his polls.2 In the average state won by Trump, the polls missed by an average of 7.4 percentage points (in either direction); in Clinton states, they missed by an average of 3.7 points. It’s typical for polls to miss in states that aren’t close, though. The most important concentration of polling errors was regional: Polls understated Trump’s margin by 4 points or more in a group of Midwestern states that he was expected to mostly lose but mostly won: Iowa, Ohio, Pennsylvania, Michigan, Wisconsin and Minnesota. If the errors had been random rather than systematic, it would have been statistically quite unlikely for Trump to have swept all of those midwestern states.
- bryanlarsen 6y ago
  AFAICT the polls had a systemic error AND the 538 models included the possibility of a systemic error. It's really the only way that they could have given Trump a 28% chance in the first place. Of course, the polls themselves also acknowledge the chance of a systemic error. They're reported similar to +/- 2% 19 times out of 20. They make no claim that the 19/20 of multiple polls are not correlated.
  - rayiner 6y ago
    The +/- in a given poll is measurement error derived from the sample size. They don’t account for systemic error.
    - bryanlarsen 6y ago
      19/20 does, though.
      - rayiner 6y ago
        What is 19/20?
        deleted 6y ago
        [deleted]
        bryanlarsen 6y ago
        Polls are reported with fine print similar to "+/- 2%, 19 times out of 20".
    - function_seven 6y ago
      Right, but 538 did account for such an error in coming up with their ~75/25% outcome chances. Had polling errors just been random, the overall prediction would have been more like 95/5%. But their key insight is that polling errors are often correlated, and not just statistical sampling errors that can randomly offset in either direction.
    - dragonwriter 6y ago
      > The +/- in a given poll is measurement error derived from the sample size. They don’t account for systemic error. Sure, but even the worst poll-based prediction outfit isn't just aggregating poll results and using the uncertainty based on sample size to determine the probabilities of different outcomes. Pretty much all of them are using some model derived from the past relationships of polls to election results, which will, to a greater or lesser extent, capture systemic polling error that is not unique to the current cycle in the model.

rayiner 6y ago

Yes and no. 538 published an article after the election explaining that there were in fact systemic errors in the models; the result wasn't just random probability within models that were otherwise correct: https://fivethirtyeight.com/features/the-polls-missed-trump-we-asked-pollsters-why/ https://fivethirtyeight.com/features/the-polls-missed-trump-... > While the errors were nationwide, they were spread unevenly. The more whites without college degrees were in a state, the more Trump outperformed his FiveThirtyEight polls-only adjusted polling average,1 suggesting the polls underestimated his support with that group. And the bigger the lead we forecast for Trump, the more he outperformed his polls.2 In the average state won by Trump, the polls missed by an average of 7.4 percentage points (in either direction); in Clinton states, they missed by an average of 3.7 points. It’s typical for polls to miss in states that aren’t close, though. The most important concentration of polling errors was regional: Polls understated Trump’s margin by 4 points or more in a group of Midwestern states that he was expected to mostly lose but mostly won: Iowa, Ohio, Pennsylvania, Michigan, Wisconsin and Minnesota. If the errors had been random rather than systematic, it would have been statistically quite unlikely for Trump to have swept all of those midwestern states.

bryanlarsen 6y ago
AFAICT the polls had a systemic error AND the 538 models included the possibility of a systemic error. It's really the only way that they could have given Trump a 28% chance in the first place. Of course, the polls themselves also acknowledge the chance of a systemic error. They're reported similar to +/- 2% 19 times out of 20. They make no claim that the 19/20 of multiple polls are not correlated.
- rayiner 6y ago
  The +/- in a given poll is measurement error derived from the sample size. They don’t account for systemic error.
  - bryanlarsen 6y ago
    19/20 does, though.
    - rayiner 6y ago
      What is 19/20?
      - deleted 6y ago
        [deleted]
      - bryanlarsen 6y ago
        Polls are reported with fine print similar to "+/- 2%, 19 times out of 20".
  - function_seven 6y ago
    Right, but 538 did account for such an error in coming up with their ~75/25% outcome chances. Had polling errors just been random, the overall prediction would have been more like 95/5%. But their key insight is that polling errors are often correlated, and not just statistical sampling errors that can randomly offset in either direction.
  - dragonwriter 6y ago
    > The +/- in a given poll is measurement error derived from the sample size. They don’t account for systemic error. Sure, but even the worst poll-based prediction outfit isn't just aggregating poll results and using the uncertainty based on sample size to determine the probabilities of different outcomes. Pretty much all of them are using some model derived from the past relationships of polls to election results, which will, to a greater or lesser extent, capture systemic polling error that is not unique to the current cycle in the model.