2 ms·

Even state polling in 2016 was pretty good, historically. The reason to think otherwise is that there was a specific error (non-college white voters swinging t

Even state polling in 2016 was pretty good, historically. The reason to think otherwise is that there was a specific error (non-college white voters swinging to R) that happened not to be tracked well (those voters don't respond as well to polling) nor accounted for (most pollsters didn't weight for education). And it happened to be most concentrated in a few states in the upper midwest that were close to the tipping point. AND it happened to be JUST ENOUGH to have predicted a very narrow (sub 1% in PA/WI/MI) win incorrectly. To wit: 2016 was a perfect storm. Swap any of those factors above and 538 would have been predicting a 50:50 race (instead of 70:30) at election day and Trump's very narrow victory would have been "correctly predicted". Assuming that's going to happen again seems a bit like magical thinking to me. Remember that in 2012 the polls were "wrong" in the opposite direction (undercounting Obama votes by a bit) and no one cared.

lern_too_spel 6y ago
If that's the case, the pollsters have an easy way to regain trust in their methodologies. They can simply reweight their 2016 data to ensure education sampling matches the population and show that they now predict 2016 correctly. Have any of them done so?
- newacct583 6y ago
  People have done this analysis repeatedly, you just aren't reading it. And polling in 2018 was generally quite good. But again, there's nothing wrong with their "methodologies". Polling errors in 2016 were in fact not far from historical norms. Numerate consensus (c.f. 538) picked a likely but not certain Clinton victory and the result was an extremely narrow loss instead. That stuff happens. Unfortunately a bunch of people whose political desires hinge on this kind of error being present in all elections and always to the advantage of "the other side" never want to hear this analysis and so repeat the nonsense that "polls are always wrong". But they're not. They're just not as precise as we want. If the polls are exactly as wrong this year, and in the same direction, as they were in 2016 then Biden wins a solid victory. If they're as wrong as they were in 2012 then he wins a landslide with Texas and Georgia going blue.
  - lern_too_spel 6y ago
    You're arguing against a straw man. I never said that polls are always wrong. My political desires don't hinge on the current polls being wrong. I'm simply asking a question about whether the pollsters have corrected their methodologies. There was clearly something wrong in 2016 when they repeatedly failed to predict the GOP primaries and failed to predict swing states and even some states that weren't considered swing states in the general election. You're right that I haven't read the analysis that you say exists. That's why I'm asking for it. Which pollsters have done the reweighting on their 2016 data and shown that they then align with reality?
    - newacct583 6y ago
      Uh... polls reliably predicted Trump's win in the primaries long before anyone in the pundit class believed it. For months and months it was explained away as simple name recognition. But he was ahead the whole time. Honestly, you're misremembering this. Polls were not wrong in 2016 like you think they were. Just browse through the article history at 538, they really are the best source for this stuff.
      - lern_too_spel 6y ago
        > Just browse through the article history at 538 I did. They say the polls were wrong. https://fivethirtyeight.com/features/the-polls-missed-trump-we-asked-pollsters-why/ https://fivethirtyeight.com/features/the-polls-missed-trump-... You yourself said the polls were wrong because they didn't weight for education. Now you say they weren't. Are you walking back your original statement? That statement is what this whole discussion is about.
    - dragonwriter 6y ago
      > I'm simply asking a question about whether the pollsters have corrected their methodologies. There was clearly something wrong in 2016 when they repeatedly failed to predict the GOP primaries Polls don't predict anything. Poll-based forecasts are different than polls, and done by different people. The major polls in 2016 conducted close to the election were well within the MoE of the actual results, so as far as polls go there is very little evidence that anything was wrong. Some of the poll-based forecasts made errors like assuming independence of state-level deviations from polling averages woere independent rather than linked, which gave Trump a very small chance of victory; that doesn't accord with history, was called out by 538 before the election, and I doubt anyone is making that mistake again.
nl 6y ago
> Even state polling in 2016 was pretty good, historically. The reason to think otherwise is that there was a specific error (non-college white voters swinging to R) that happened not to be tracked well (those voters don't respond as well to polling) nor accounted for (most pollsters didn't weight for education). And it happened to be most concentrated in a few states in the upper midwest that were close to the tipping point. AND it happened to be JUST ENOUGH to have predicted a very narrow (sub 1% in PA/WI/MI) win incorrectly. Yes, this is a good summary, and I'm well aware of what went wrong. I ran a prediction project for a few years so I realise how hard this job is. But just because it's hard doesn't mean we shouldn't recognize the issues. My argument is that historically state level polls have always had issues but have rarely mattered. > Remember that in 2012 the polls were "wrong" in the opposite direction (undercounting Obama votes by a bit) and no one cared. Exactly my point! If you look at the data on this, the state level polls had even worse accuracy. The average error was 1.6%, compared to 0.8% for national polls. As you note, no one really cared. https://www.pewresearch.org/2009/06/25/perils-of-polling-in-election-08/ https://www.pewresearch.org/2009/06/25/perils-of-polling-in-...

lern_too_spel 6y ago

If that's the case, the pollsters have an easy way to regain trust in their methodologies. They can simply reweight their 2016 data to ensure education sampling matches the population and show that they now predict 2016 correctly. Have any of them done so?

newacct583 6y ago
People have done this analysis repeatedly, you just aren't reading it. And polling in 2018 was generally quite good. But again, there's nothing wrong with their "methodologies". Polling errors in 2016 were in fact not far from historical norms. Numerate consensus (c.f. 538) picked a likely but not certain Clinton victory and the result was an extremely narrow loss instead. That stuff happens. Unfortunately a bunch of people whose political desires hinge on this kind of error being present in all elections and always to the advantage of "the other side" never want to hear this analysis and so repeat the nonsense that "polls are always wrong". But they're not. They're just not as precise as we want. If the polls are exactly as wrong this year, and in the same direction, as they were in 2016 then Biden wins a solid victory. If they're as wrong as they were in 2012 then he wins a landslide with Texas and Georgia going blue.
- lern_too_spel 6y ago
  You're arguing against a straw man. I never said that polls are always wrong. My political desires don't hinge on the current polls being wrong. I'm simply asking a question about whether the pollsters have corrected their methodologies. There was clearly something wrong in 2016 when they repeatedly failed to predict the GOP primaries and failed to predict swing states and even some states that weren't considered swing states in the general election. You're right that I haven't read the analysis that you say exists. That's why I'm asking for it. Which pollsters have done the reweighting on their 2016 data and shown that they then align with reality?
  - newacct583 6y ago
    Uh... polls reliably predicted Trump's win in the primaries long before anyone in the pundit class believed it. For months and months it was explained away as simple name recognition. But he was ahead the whole time. Honestly, you're misremembering this. Polls were not wrong in 2016 like you think they were. Just browse through the article history at 538, they really are the best source for this stuff.
    - lern_too_spel 6y ago
      > Just browse through the article history at 538 I did. They say the polls were wrong. https://fivethirtyeight.com/features/the-polls-missed-trump-we-asked-pollsters-why/ https://fivethirtyeight.com/features/the-polls-missed-trump-... You yourself said the polls were wrong because they didn't weight for education. Now you say they weren't. Are you walking back your original statement? That statement is what this whole discussion is about.
  - dragonwriter 6y ago
    > I'm simply asking a question about whether the pollsters have corrected their methodologies. There was clearly something wrong in 2016 when they repeatedly failed to predict the GOP primaries Polls don't predict anything. Poll-based forecasts are different than polls, and done by different people. The major polls in 2016 conducted close to the election were well within the MoE of the actual results, so as far as polls go there is very little evidence that anything was wrong. Some of the poll-based forecasts made errors like assuming independence of state-level deviations from polling averages woere independent rather than linked, which gave Trump a very small chance of victory; that doesn't accord with history, was called out by 538 before the election, and I doubt anyone is making that mistake again.

nl 6y ago

> Even state polling in 2016 was pretty good, historically. The reason to think otherwise is that there was a specific error (non-college white voters swinging to R) that happened not to be tracked well (those voters don't respond as well to polling) nor accounted for (most pollsters didn't weight for education). And it happened to be most concentrated in a few states in the upper midwest that were close to the tipping point. AND it happened to be JUST ENOUGH to have predicted a very narrow (sub 1% in PA/WI/MI) win incorrectly. Yes, this is a good summary, and I'm well aware of what went wrong. I ran a prediction project for a few years so I realise how hard this job is. But just because it's hard doesn't mean we shouldn't recognize the issues. My argument is that historically state level polls have always had issues but have rarely mattered. > Remember that in 2012 the polls were "wrong" in the opposite direction (undercounting Obama votes by a bit) and no one cared. Exactly my point! If you look at the data on this, the state level polls had even worse accuracy. The average error was 1.6%, compared to 0.8% for national polls. As you note, no one really cared. https://www.pewresearch.org/2009/06/25/perils-of-polling-in-election-08/ https://www.pewresearch.org/2009/06/25/perils-of-polling-in-...