9 ms·

If you dig into the the margins they aren't looking very good, even if they got a fair number of eventual winners correct. Predicted vs Actual (FiveThirtyEight

If you dig into the the margins they aren't looking very good, even if they got a fair number of eventual winners correct. Predicted vs Actual (FiveThirtyEight's averages vs NYTimes' current tally; Biden's margin is positive) PA: +4.7% vs +.5% (-4.2%) FL: +2.5 vs -3.4 (-5.9) TX: -1.5 vs -5.9 (-4.4) OH: -0.6 vs -8.1 (-7.5) PA: +4.7 vs +0.5 (-4.2) IA: -1.5 vs -8.2 (-6.7) NC: +1.7 vs -1.4 (-3.1) WI: +8.3 vs +0.6 (-7.7) GA: +0.9 vs +0.1 (-.8) MI: +8.0 vs +2.6 (-5.4) AZ: +2.6 vs +0.6 (-2.0) NV: +6.2 vs +2.0 (-4.2) They pretty consistently overpredicted Biden's margin by about 4-7% in almost all of the swing states. Even if that's within, or close to, the margin of error; there's a systemic issue if it's happening in almost every state. Edit: Added AZ and NV; and the difference

paulddraper 6y ago
^ Exactly. The polls were at least as wrong in 2016, even if that error was less significant this time (because the polled margin started out bigger). Pollsters and poll-watchers downplaying this is not a good sign for future improvement. EDIT: Overall though, I'm not sure that's a loss. People pay way more attention to polls than they should anyway.
OskarS 6y ago
This is such a damning list. At some point, reporting on polls when they have this bad of a track-record serves no journalistic purpose and just confuses the public. Like, the discussions people were having in the days before the election about Biden's strength barely resembles reality. I respect FiveThirtyEight and their work, and I do think they're intellectually honest and generally speaking good at their jobs. But they can only be as good as their sources, and when the sources are this terrible, they shouldn't be reported on this, and not given the statistical and scientific sheen of authenticity 538 gives them. Like, that site is starting to have a net negative influence on the world, and when that is the case with a journalistic institution, what are you even doing?
- paulddraper 6y ago
  > I respect FiveThirtyEight and their work, and I do think they're intellectually honest and generally speaking good at their jobs. But 538 chooses how to weight the polls. For example, they only gave Rasmussen (which was far closer on these) a C rating, preferring less accurate polls. FWIW, their articles clearly lean left [1], IDK if that affects their analysis/forecasts. [1] https://www.adfontesmedia.com/interactive-media-bias-chart-2/ https://www.adfontesmedia.com/interactive-media-bias-chart-2... similar to WaPo
  - mewse 6y ago
    Boy, that rear view mirror makes criticism a whole bunch easier, doesn’t it? :)
- systemvoltage 6y ago
  One of the things that's annoying about 538 is that they don't admit they're wrong. The excuse they keep repeating is that "1 in 10 chance Trump can win. That's about the same chance as rain in LA, but it does rain in LA!". 538 seems to have forgotten in their condenscencion is that we are not look at you to explain the odds. We are looking at you to provide an accurate probability estimate. Their defense is completely ridiculous. I follow betting odds which had Biden/Trump at 60/40 split before the election day - what's the point of polling when you can look at people putting money on the line, let the incentives work for you.
  - evgen 6y ago
    They provided the probability distribution. The fact that you can’t handle math and need some sort of absolute certainty for a future event is not 538’s problem.
    - systemvoltage 6y ago
      I am questioning - why even bother? It's not like we do elections every month to test out their probability distribution against empirical data. The distribution collapses into a binary outcome at the end. I have a dice. I claim the distribution is of equal outcome for each side. Well...we don't get to test the dice more than once. 1 sample size does not prove that the 538's predictions were right (or wrong). Thanks for assuming I can't do math, no way to argue with someone but I am actually pretty bad at it. :-)
      - evgen 6y ago
        Everyone is bad at probability and statistical distributions, not just you. The problem with modeling elections is that there are so few of them and the data is very noisy and until quite recently rather suspect. Let's not pretend that this was a normal election, either in the candidates running or in the manner in which the campaign and election was conducted. As to the question of why bother, it is because bad polling is better than no polling at all. Campaigns are now multi billion dollar enterprises managing tens of thousands of temporary employees for the creation of a product that will only be sold once and in 18+ months from when they start the process. Any data is better than nothing. The fact that the public has become obsessed with polls is probably due to the ongoing nationalization of politics.
- chkaloon 6y ago
  538 does make adjustments based on what they think are biases in individual polls. So they can't blame it all on their sources. They've taken on some responsibility to evaluate those sources and adjust their model accordingly. The polls had significant bias, and 538 failed to fully adjust for it.
- ianai 6y ago
  I am reminded of all the times in the last four years (really my entire adult life) where during policy discussions of any magnitude the ultimate “kill” for an idea is “but it’s only popular with $small_number of the vote” or “but poll after poll shows it as a losing position.” It’s used as a cudgel to push out things that would actually make people feel their government is doing things with impunity. Donald Trump’s approval rating rarely if ever dipped below ~43% (with the usual and huge error bars) and consequently republicans reported they couldn’t go against him ever because of the polls. “Spending political capital” is a concept where a ruling party takes unpopular actions/enacts unpopular law because it’ll only push their approvals toward their long run averages. I think this needs to be a serious topic of discussion. After this administration, “governing by polling firm” seems disingenuous at best and outright detrimental to all involved.
ummonk 6y ago
There definitely seems to have been systemic error. Note though that many of these states are likely to rise by 0.5-1.0% as they finish counting ballots.
Misdicorl 6y ago
And its actually much worse in most of the states that voted for Trump. IIRC MO was +8ish Trump in the polling and ended up +16 in the election
cptnapalm 6y ago
Obviously, the focus is on the presidential polling, but I wonder if polling is off by the same amount when there are polls regarding what Americans want or think about any given topic.
alkonaut 6y ago
There are tons of sources of systematic error in a "strange year" election plus a "nontraditional incumbent". One of the most plausible ones mentioned by Nate was the fact that people who wfh weré more likely to be reached by pollsters this year, and may skew D.
zumu 6y ago
I wonder if fear of being outed as a Trump supporter prevents pro-Trump people from responding to these polls, contributing to the skew towards Biden. There's definitely a demographic who voted for Trump, but aren't open about it for social reasons.
- mmaurizi 6y ago
  That wouldn't explain why the polls were off in the Maine Senate race by 10 points though...
radioactivist 6y ago
One thing I found interesting about the final polling averages was that they were much more correct about Biden's share of the vote, even when they were very wrong about the margin itself (since these polls don't add to 100%, that's possible). See below for the 538 polling average vs. the current NYT tally (% for Biden): PA: 50.2% vs 49.7% FL: 49.1% vs 47.8% TX: 47.4% vs 46.3% OH: 46.8% vs 45.2% IA: 46.3% vs 44.9% NC: 48.9% vs 48.6% WI: 52.1% vs 49.5% MI: 51.2% vs 50.5% GA: 48.5% vs 49.3% AZ: 48.7% vs 48.9% NV: 49.7% vs 49.9% Most are within ~1% or so (some for Biden, some for Trump) with some outliers being Wisconsin (-2.6%), Ohio (-1.6%) and Iowa (-1.4%) in Trump's favor still not being so far off (relative to margins of error). I'm not sure what that "missing" bit in the polls really means (undecided?) but it seems the issue was that that bit of the electorate ended up going entirely for Trump in many places. (edit: had the PA 538 # wrong)
- scrooched_moose 6y ago
  That's really interesting, I hadn't sliced it that way yet. I know FiveThirtyEight allocates undecideds evenly - if 8% respond undecided they assume 4% will break Biden and 4% will break Trump. In one of the podcasts they discussed this assumption and it's what they have found to be the most accurate historically. We might be seeing a shift towards that no longer being true - maybe 75% break red and 25% break blue now. Edit: Also, It looks like Jorgensen (Libertarian) significantly underperformed her polling (~3% > ~1%) in the first few states I spot checked. If those voters broke for Trump, that makes up about 2% of the error. That'd take a lot more in depth checking though.
  - evgen 6y ago
    Probably worth noting that the asymmetric break of the undecideds in 2016 was what won the election for Trump. There were a lot of undecided voters and a lot of them decided for Trump late in the process so models that assumed a traditional 50-50 split were wrong.
    - cloudwizard 6y ago
      It breaks against the least popular. Clinton was a known unpopular. Now that everyone knows Trump, he was the least popular.
r-w 6y ago
But you didn't actually show the margin of error. In any case, I think I'd take this argument more seriously in an article of its own, rather than on a back-and-forth forum like this. Not enough opportunities to request and provide context.
- ehsankia 6y ago
  This exactly. If there's one thing all my physics and math teachers burned into my head, it's that a number without an error is meaningless. Florida for examples may be off by 8 points, but 538 gave Trump a 1/3 chance of winning, so it was well within the error. What people don't realize is that the error on polls are pretty damn big, and elections in modern times have been extremely close. Like most of the swing states come down to under 100k votes / 1%. It's insane how close these rates are, and no polls will ever have any chance at predicting things like that.
Dumblydorr 6y ago
You wasted your efforts by comparing a point estimate of a probability distribution with the actual outcome of the single election day.
- ehsankia 6y ago
  Not only that, 538 gave Trump a 1/3 chance of winning Florida, so it's not a surprise at all, and all things considered now that the votes are almost final, 1/10 chance for Trump to win seems reasonable. But as you mention, there's no way to figure out what the true probability was. The bigger mistake here though is looking at the mean and ignoring the error on those values. 538 is fairly conservative in their calculations and their error bars are pretty big, so almost all of these results are well within their error bars.
mensetmanusman 6y ago
At least in the circles I’m familiar with, there is open discussion about faking answers to pollsters because of how much the media is distrusted.
softwaredoug 6y ago
The problem is even these have narrowed as votes have counted. PA might be ~1 point victory. Many of these have shrunk. We can't really say what the 'actual' is until all the votes are counted.

paulddraper 6y ago

^ Exactly. The polls were at least as wrong in 2016, even if that error was less significant this time (because the polled margin started out bigger). Pollsters and poll-watchers downplaying this is not a good sign for future improvement. EDIT: Overall though, I'm not sure that's a loss. People pay way more attention to polls than they should anyway.

OskarS 6y ago

This is such a damning list. At some point, reporting on polls when they have this bad of a track-record serves no journalistic purpose and just confuses the public. Like, the discussions people were having in the days before the election about Biden's strength barely resembles reality. I respect FiveThirtyEight and their work, and I do think they're intellectually honest and generally speaking good at their jobs. But they can only be as good as their sources, and when the sources are this terrible, they shouldn't be reported on this, and not given the statistical and scientific sheen of authenticity 538 gives them. Like, that site is starting to have a net negative influence on the world, and when that is the case with a journalistic institution, what are you even doing?

paulddraper 6y ago
> I respect FiveThirtyEight and their work, and I do think they're intellectually honest and generally speaking good at their jobs. But 538 chooses how to weight the polls. For example, they only gave Rasmussen (which was far closer on these) a C rating, preferring less accurate polls. FWIW, their articles clearly lean left [1], IDK if that affects their analysis/forecasts. [1] https://www.adfontesmedia.com/interactive-media-bias-chart-2/ https://www.adfontesmedia.com/interactive-media-bias-chart-2... similar to WaPo
- mewse 6y ago
  Boy, that rear view mirror makes criticism a whole bunch easier, doesn’t it? :)
systemvoltage 6y ago
One of the things that's annoying about 538 is that they don't admit they're wrong. The excuse they keep repeating is that "1 in 10 chance Trump can win. That's about the same chance as rain in LA, but it does rain in LA!". 538 seems to have forgotten in their condenscencion is that we are not look at you to explain the odds. We are looking at you to provide an accurate probability estimate. Their defense is completely ridiculous. I follow betting odds which had Biden/Trump at 60/40 split before the election day - what's the point of polling when you can look at people putting money on the line, let the incentives work for you.
- evgen 6y ago
  They provided the probability distribution. The fact that you can’t handle math and need some sort of absolute certainty for a future event is not 538’s problem.
  - systemvoltage 6y ago
    I am questioning - why even bother? It's not like we do elections every month to test out their probability distribution against empirical data. The distribution collapses into a binary outcome at the end. I have a dice. I claim the distribution is of equal outcome for each side. Well...we don't get to test the dice more than once. 1 sample size does not prove that the 538's predictions were right (or wrong). Thanks for assuming I can't do math, no way to argue with someone but I am actually pretty bad at it. :-)
    - evgen 6y ago
      Everyone is bad at probability and statistical distributions, not just you. The problem with modeling elections is that there are so few of them and the data is very noisy and until quite recently rather suspect. Let's not pretend that this was a normal election, either in the candidates running or in the manner in which the campaign and election was conducted. As to the question of why bother, it is because bad polling is better than no polling at all. Campaigns are now multi billion dollar enterprises managing tens of thousands of temporary employees for the creation of a product that will only be sold once and in 18+ months from when they start the process. Any data is better than nothing. The fact that the public has become obsessed with polls is probably due to the ongoing nationalization of politics.
chkaloon 6y ago
538 does make adjustments based on what they think are biases in individual polls. So they can't blame it all on their sources. They've taken on some responsibility to evaluate those sources and adjust their model accordingly. The polls had significant bias, and 538 failed to fully adjust for it.
ianai 6y ago
I am reminded of all the times in the last four years (really my entire adult life) where during policy discussions of any magnitude the ultimate “kill” for an idea is “but it’s only popular with $small_number of the vote” or “but poll after poll shows it as a losing position.” It’s used as a cudgel to push out things that would actually make people feel their government is doing things with impunity. Donald Trump’s approval rating rarely if ever dipped below ~43% (with the usual and huge error bars) and consequently republicans reported they couldn’t go against him ever because of the polls. “Spending political capital” is a concept where a ruling party takes unpopular actions/enacts unpopular law because it’ll only push their approvals toward their long run averages. I think this needs to be a serious topic of discussion. After this administration, “governing by polling firm” seems disingenuous at best and outright detrimental to all involved.

ummonk 6y ago

There definitely seems to have been systemic error. Note though that many of these states are likely to rise by 0.5-1.0% as they finish counting ballots.

Misdicorl 6y ago

And its actually much worse in most of the states that voted for Trump. IIRC MO was +8ish Trump in the polling and ended up +16 in the election

cptnapalm 6y ago

Obviously, the focus is on the presidential polling, but I wonder if polling is off by the same amount when there are polls regarding what Americans want or think about any given topic.

alkonaut 6y ago

There are tons of sources of systematic error in a "strange year" election plus a "nontraditional incumbent". One of the most plausible ones mentioned by Nate was the fact that people who wfh weré more likely to be reached by pollsters this year, and may skew D.

zumu 6y ago

I wonder if fear of being outed as a Trump supporter prevents pro-Trump people from responding to these polls, contributing to the skew towards Biden. There's definitely a demographic who voted for Trump, but aren't open about it for social reasons.

mmaurizi 6y ago
That wouldn't explain why the polls were off in the Maine Senate race by 10 points though...

radioactivist 6y ago

One thing I found interesting about the final polling averages was that they were much more correct about Biden's share of the vote, even when they were very wrong about the margin itself (since these polls don't add to 100%, that's possible). See below for the 538 polling average vs. the current NYT tally (% for Biden): PA: 50.2% vs 49.7% FL: 49.1% vs 47.8% TX: 47.4% vs 46.3% OH: 46.8% vs 45.2% IA: 46.3% vs 44.9% NC: 48.9% vs 48.6% WI: 52.1% vs 49.5% MI: 51.2% vs 50.5% GA: 48.5% vs 49.3% AZ: 48.7% vs 48.9% NV: 49.7% vs 49.9% Most are within ~1% or so (some for Biden, some for Trump) with some outliers being Wisconsin (-2.6%), Ohio (-1.6%) and Iowa (-1.4%) in Trump's favor still not being so far off (relative to margins of error). I'm not sure what that "missing" bit in the polls really means (undecided?) but it seems the issue was that that bit of the electorate ended up going entirely for Trump in many places. (edit: had the PA 538 # wrong)

scrooched_moose 6y ago
That's really interesting, I hadn't sliced it that way yet. I know FiveThirtyEight allocates undecideds evenly - if 8% respond undecided they assume 4% will break Biden and 4% will break Trump. In one of the podcasts they discussed this assumption and it's what they have found to be the most accurate historically. We might be seeing a shift towards that no longer being true - maybe 75% break red and 25% break blue now. Edit: Also, It looks like Jorgensen (Libertarian) significantly underperformed her polling (~3% > ~1%) in the first few states I spot checked. If those voters broke for Trump, that makes up about 2% of the error. That'd take a lot more in depth checking though.
- evgen 6y ago
  Probably worth noting that the asymmetric break of the undecideds in 2016 was what won the election for Trump. There were a lot of undecided voters and a lot of them decided for Trump late in the process so models that assumed a traditional 50-50 split were wrong.
  - cloudwizard 6y ago
    It breaks against the least popular. Clinton was a known unpopular. Now that everyone knows Trump, he was the least popular.

r-w 6y ago

But you didn't actually show the margin of error. In any case, I think I'd take this argument more seriously in an article of its own, rather than on a back-and-forth forum like this. Not enough opportunities to request and provide context.

ehsankia 6y ago
This exactly. If there's one thing all my physics and math teachers burned into my head, it's that a number without an error is meaningless. Florida for examples may be off by 8 points, but 538 gave Trump a 1/3 chance of winning, so it was well within the error. What people don't realize is that the error on polls are pretty damn big, and elections in modern times have been extremely close. Like most of the swing states come down to under 100k votes / 1%. It's insane how close these rates are, and no polls will ever have any chance at predicting things like that.

Dumblydorr 6y ago

You wasted your efforts by comparing a point estimate of a probability distribution with the actual outcome of the single election day.

ehsankia 6y ago
Not only that, 538 gave Trump a 1/3 chance of winning Florida, so it's not a surprise at all, and all things considered now that the votes are almost final, 1/10 chance for Trump to win seems reasonable. But as you mention, there's no way to figure out what the true probability was. The bigger mistake here though is looking at the mean and ignoring the error on those values. 538 is fairly conservative in their calculations and their error bars are pretty big, so almost all of these results are well within their error bars.

mensetmanusman 6y ago

At least in the circles I’m familiar with, there is open discussion about faking answers to pollsters because of how much the media is distrusted.

softwaredoug 6y ago

The problem is even these have narrowed as votes have counted. PA might be ~1 point victory. Many of these have shrunk. We can't really say what the 'actual' is until all the votes are counted.