5 ms·

People have done this analysis repeatedly, you just aren't reading it. And polling in 2018 was generally quite good. But again, there's nothing wrong with the

People have done this analysis repeatedly, you just aren't reading it. And polling in 2018 was generally quite good. But again, there's nothing wrong with their "methodologies". Polling errors in 2016 were in fact not far from historical norms. Numerate consensus (c.f. 538) picked a likely but not certain Clinton victory and the result was an extremely narrow loss instead. That stuff happens. Unfortunately a bunch of people whose political desires hinge on this kind of error being present in all elections and always to the advantage of "the other side" never want to hear this analysis and so repeat the nonsense that "polls are always wrong". But they're not. They're just not as precise as we want. If the polls are exactly as wrong this year, and in the same direction, as they were in 2016 then Biden wins a solid victory. If they're as wrong as they were in 2012 then he wins a landslide with Texas and Georgia going blue.

lern_too_spel 6y ago
You're arguing against a straw man. I never said that polls are always wrong. My political desires don't hinge on the current polls being wrong. I'm simply asking a question about whether the pollsters have corrected their methodologies. There was clearly something wrong in 2016 when they repeatedly failed to predict the GOP primaries and failed to predict swing states and even some states that weren't considered swing states in the general election. You're right that I haven't read the analysis that you say exists. That's why I'm asking for it. Which pollsters have done the reweighting on their 2016 data and shown that they then align with reality?
- newacct583 6y ago
  Uh... polls reliably predicted Trump's win in the primaries long before anyone in the pundit class believed it. For months and months it was explained away as simple name recognition. But he was ahead the whole time. Honestly, you're misremembering this. Polls were not wrong in 2016 like you think they were. Just browse through the article history at 538, they really are the best source for this stuff.
  - lern_too_spel 6y ago
    > Just browse through the article history at 538 I did. They say the polls were wrong. https://fivethirtyeight.com/features/the-polls-missed-trump-we-asked-pollsters-why/ https://fivethirtyeight.com/features/the-polls-missed-trump-... You yourself said the polls were wrong because they didn't weight for education. Now you say they weren't. Are you walking back your original statement? That statement is what this whole discussion is about.
- dragonwriter 6y ago
  > I'm simply asking a question about whether the pollsters have corrected their methodologies. There was clearly something wrong in 2016 when they repeatedly failed to predict the GOP primaries Polls don't predict anything. Poll-based forecasts are different than polls, and done by different people. The major polls in 2016 conducted close to the election were well within the MoE of the actual results, so as far as polls go there is very little evidence that anything was wrong. Some of the poll-based forecasts made errors like assuming independence of state-level deviations from polling averages woere independent rather than linked, which gave Trump a very small chance of victory; that doesn't accord with history, was called out by 538 before the election, and I doubt anyone is making that mistake again.
  - lern_too_spel 6y ago
    > Polls don't predict anything. Semantic nitpicking. Clearly, my comment was about how well the polls reflected reality. > Some of the poll-based forecasts made errors like.... You're making a different statement than the person I'm responding to, who said that the polls didn't align with reality because they did not weight for education. That is something that is easy to verify. I was simply asking if it was verified by showing that weighting by education made the 2016 polls align with reality.
    - dragonwriter 6y ago
      > You're making a different statement than the person I'm responding to, who said that the polls didn't align with reality because they did not weight for education. Yes, I'm disagreeing; the polls in 2016 reflected the actual outcome very well (they were off, but not unusually far; the entire idea that there was an unusual error is based on overemphasis of fairly new poll-based predictors, especially one at the NYTimes, that painted an extremely high probability of a Clinton win based on a flawed prediction methodology which was separate from any problem with the polls.) > I was simply asking if it was verified by showing that weighting by education made the 2016 polls align with reality. If you want to fit a particular single result you can change how you handle any factor and get a perfect match. You'd actually need to test against a different set of real data than you used to determine the adjustment to validate it, not the same single result.
- nl 6y ago
  Here's the analysis.[1] It points out that (to quote): As this report documents, the national polls in 2016 were quite accurate, while polls in key battleground states showed some large, problematic errors. (which is my claim: national polling is ok, state polls aren't great). The education weighing issue is commonly accepted as the reason for the inaccuracy. The evidence for that isn't as strong as people assume. Section 3.4 talks about the weighing issue, and shows some where reweighing worked: Following the election, two different state-level pollsters acknowledged that they had not adjusted for education and conducted their own post-hoc analysis to examine what difference that would have made in their estimates. Both pollsters found that adjusting for education would have meaningfully improved their poll’s accuracy by reducing over-statement of Clinton support. but also: Despite this, it is not clear that adjusting to a more detailed education variable would have universally improved polls in 2016. Analysis of the effect from weighting by five education categories rather than three categories in four national polls (Appendix A.H) yielded an average change of less than 0.4 percentage points in the vote estimates and no systematic improvement. It's worth noting that in previous elections this had never been the case before (which was why it wasn't done). [1] https://www.aapor.org/Education-Resources/Reports/An-Evaluation-of-2016-Election-Polls-in-the-U-S.aspx https://www.aapor.org/Education-Resources/Reports/An-Evaluat...
  - lern_too_spel 6y ago
    Thank you. This is exactly what I was looking for.

lern_too_spel 6y ago

You're arguing against a straw man. I never said that polls are always wrong. My political desires don't hinge on the current polls being wrong. I'm simply asking a question about whether the pollsters have corrected their methodologies. There was clearly something wrong in 2016 when they repeatedly failed to predict the GOP primaries and failed to predict swing states and even some states that weren't considered swing states in the general election. You're right that I haven't read the analysis that you say exists. That's why I'm asking for it. Which pollsters have done the reweighting on their 2016 data and shown that they then align with reality?

newacct583 6y ago
Uh... polls reliably predicted Trump's win in the primaries long before anyone in the pundit class believed it. For months and months it was explained away as simple name recognition. But he was ahead the whole time. Honestly, you're misremembering this. Polls were not wrong in 2016 like you think they were. Just browse through the article history at 538, they really are the best source for this stuff.
- lern_too_spel 6y ago
  > Just browse through the article history at 538 I did. They say the polls were wrong. https://fivethirtyeight.com/features/the-polls-missed-trump-we-asked-pollsters-why/ https://fivethirtyeight.com/features/the-polls-missed-trump-... You yourself said the polls were wrong because they didn't weight for education. Now you say they weren't. Are you walking back your original statement? That statement is what this whole discussion is about.
dragonwriter 6y ago
> I'm simply asking a question about whether the pollsters have corrected their methodologies. There was clearly something wrong in 2016 when they repeatedly failed to predict the GOP primaries Polls don't predict anything. Poll-based forecasts are different than polls, and done by different people. The major polls in 2016 conducted close to the election were well within the MoE of the actual results, so as far as polls go there is very little evidence that anything was wrong. Some of the poll-based forecasts made errors like assuming independence of state-level deviations from polling averages woere independent rather than linked, which gave Trump a very small chance of victory; that doesn't accord with history, was called out by 538 before the election, and I doubt anyone is making that mistake again.
- lern_too_spel 6y ago
  > Polls don't predict anything. Semantic nitpicking. Clearly, my comment was about how well the polls reflected reality. > Some of the poll-based forecasts made errors like.... You're making a different statement than the person I'm responding to, who said that the polls didn't align with reality because they did not weight for education. That is something that is easy to verify. I was simply asking if it was verified by showing that weighting by education made the 2016 polls align with reality.
  - dragonwriter 6y ago
    > You're making a different statement than the person I'm responding to, who said that the polls didn't align with reality because they did not weight for education. Yes, I'm disagreeing; the polls in 2016 reflected the actual outcome very well (they were off, but not unusually far; the entire idea that there was an unusual error is based on overemphasis of fairly new poll-based predictors, especially one at the NYTimes, that painted an extremely high probability of a Clinton win based on a flawed prediction methodology which was separate from any problem with the polls.) > I was simply asking if it was verified by showing that weighting by education made the 2016 polls align with reality. If you want to fit a particular single result you can change how you handle any factor and get a perfect match. You'd actually need to test against a different set of real data than you used to determine the adjustment to validate it, not the same single result.
nl 6y ago
Here's the analysis.[1] It points out that (to quote): As this report documents, the national polls in 2016 were quite accurate, while polls in key battleground states showed some large, problematic errors. (which is my claim: national polling is ok, state polls aren't great). The education weighing issue is commonly accepted as the reason for the inaccuracy. The evidence for that isn't as strong as people assume. Section 3.4 talks about the weighing issue, and shows some where reweighing worked: Following the election, two different state-level pollsters acknowledged that they had not adjusted for education and conducted their own post-hoc analysis to examine what difference that would have made in their estimates. Both pollsters found that adjusting for education would have meaningfully improved their poll’s accuracy by reducing over-statement of Clinton support. but also: Despite this, it is not clear that adjusting to a more detailed education variable would have universally improved polls in 2016. Analysis of the effect from weighting by five education categories rather than three categories in four national polls (Appendix A.H) yielded an average change of less than 0.4 percentage points in the vote estimates and no systematic improvement. It's worth noting that in previous elections this had never been the case before (which was why it wasn't done). [1] https://www.aapor.org/Education-Resources/Reports/An-Evaluation-of-2016-Election-Polls-in-the-U-S.aspx https://www.aapor.org/Education-Resources/Reports/An-Evaluat...
- lern_too_spel 6y ago
  Thank you. This is exactly what I was looking for.