7 ms·

For context, this blog post is by Andrew Gelman, a professor of statistics and political science at Columbia University, and one of the people developing the pr

For context, this blog post is by Andrew Gelman, a professor of statistics and political science at Columbia University, and one of the people developing the presidential forecast model for The Economist. So he's not a pollster, but he's relying heavily on poll data for making predictions, which is why his particular focus in this post is not just about the ways the polls were off, but how the polls were incorporated into The Economist prediction model. I would argue 538 made a similar error[1], but unlike Gelman, Nate Silver is a less likely to admit their model was off, and consequently examine/discuss the underlying factors that lead to this issue. [1] I don't know if error is the right term here, but basically the fact that the actual election results seem to be landing near the lower end of the predicted 95% intervals for both the 538 and the Economist models.

ehsankia 6y ago
Well for one, 538 gave Trump 10% at the lowest, which is quite a bit better than 4% by CU. Also, putting Florida aside, everything else so far has been well within the error margin, even if the overall mean is 3-4 points shifted towards Trump. I'm not an expert, but that's the whole point of confidence intervals. Polls will never be accurate, but it's about using the right uncertainty, and also mixing uncertainties properly. EDIT: Let's look at a few of the tipping point results from 538 [1] Florida: 50.9/48.4 -> 47.8/51.2 (2.8 diff) [99% reporting] Arizona: 50.7/48.1 -> 50.7/47.9 (0.1 diff) [86% reporting] Texas: 48.8/50.3 -> 46.4/52.2 (1.9 diff) [85% reporting] Georgia: 50.1/49.2 -> 49.1/49.7 (0.7 diff) [98% reporting] Minnesota: 53.7/44.6 -> 52.6/45.4 (0.9 diff) [99% reporting] It's also worth noting that a lot of these are still at 85-95 percent reporting, so it's not over, but yes I don't see anything that's very off, even Florida that was so off was only 3 points off. I do consider 3 points to be well within the toss-up category. [1] https://projects.fivethirtyeight.com/2020-election-forecast/ https://projects.fivethirtyeight.com/2020-election-forecast/ EDIT2: I think the main thing that surprised people is how in some places, mail-in votes came first, whereas in some other places, mail-in votes are coming much later. This really skews the early data but as things settle, everything is coming much closer to the predictions. But this is something 538 and others have been warning about since May or earlier.
- ars 6y ago
  > but that's the whole point of confidence intervals. That would make sense if some parts of the country had errors one way, and others another way. i.e. random error. But if every part of the country has an error, all in the same direction, then there's something larger going on.
  - dragonwriter 6y ago
    > That would make sense if some parts of the country had errors one way, and others another way. i.e. random error. As Silver pointed out in 2016 (and why his model then gave Trump a much better chance than the NYTimes model), historically, when election outcomes differ from polls in either direction, the polling error between states is correlated, not independent. That is, it is, to all evidence, a historical fact that poll-based predictions must account for that within election cycles, there is systematic polling error, even if the direction and magnitude of that error is not consistent between cycles.
  - akersten 6y ago
    > i.e. random error. Did they say that their confidence interval was confidence against random error? I don't think that's a requirement of a confidence interval, I think it's certainly valid to encode e.g. the chance of "shy Trump" respondents in a confidence interval.
  - slg 6y ago
    The results of state elections aren't independent. The problems that cause most errors are going to exist in multiple states and cause the errors to be correlated.
    - thaumasiotes 6y ago
      But something that causes the results everywhere to be off in the same direction isn't the kind of error you say "well, that's why you don't make point predictions" about. It's a factor that has a significant, predictable influence on the phenomenon you're studying; if it's not in your model, that means you screwed up when making the model. The whole point of the model is to account explicitly for such factors.
slg 6y ago
> I would argue 538 made a similar error[1], but unlike Gelman, Nate Silver is a less likely to admit their model was off, and consequently examine/discuss the underlying factors that lead to this issue. Why do you say this? He did plenty of writing on what he, the site, and the whole industry got wrong in 2016 for Trump's win in both the primary[1] and general[2][3]. I really don't understand the hate Nate Silver gets from basically all sides. [1] - https://fivethirtyeight.com/features/how-i-acted-like-a-pundit-and-screwed-up-on-donald-trump/ https://fivethirtyeight.com/features/how-i-acted-like-a-pund... [2] - https://fivethirtyeight.com/features/the-real-story-of-2016/ https://fivethirtyeight.com/features/the-real-story-of-2016/ [3] - https://fivethirtyeight.com/tag/the-real-story-of-2016/ https://fivethirtyeight.com/tag/the-real-story-of-2016/
- saeranv 6y ago
  In a lot of small ways, Gelman (and the Economist team) engage in more transparency, and self-critique about forecasting practices relative to the 538 team. I don't mean to come off as a Nate Silver hater. My criticism is made with a lot of affection, and awareness of how much he discusses the role of uncertainty in his work. But the Economist team has made some legitimate critiques of some wacky tail behavior in the 538 model[1] which Nate Silver has basically dismissed without engaging, whereas the Economist team has written plenty critiquing their own model[2]. I mean this is Gelman's whole schtick about the scientific process: he thinks that errors are to be expected and should be transparently discussed and fixed and iteratively improved over time. That's partly why they've released all their code on Github, and Gelman has sort of poked at 538 for not doing the same with their code. [1] https://statmodeling.stat.columbia.edu/2020/10/24/reverse-engineering-the-problematic-tail-behavior-of-the-fivethirtyeight-presidential-election-forecast/ https://statmodeling.stat.columbia.edu/2020/10/24/reverse-en... [2] https://statmodeling.stat.columbia.edu/2020/10/24/reverse-engineering-the-problematic-tail-behavior-of-the-fivethirtyeight-presidential-election-forecast/ https://statmodeling.stat.columbia.edu/2020/10/24/reverse-en...
- peteretep 6y ago
  People hate Nate because he’s pretty ruthless in calling out bullshit and there’s a great deal of bullshit produced by amateurs and professionals on both sides
eitau_1 6y ago
Chart polls vs results: https://twitter.com/thomasjwood/status/1324171829007757316 https://twitter.com/thomasjwood/status/1324171829007757316
tehjoker 6y ago
Nate Silver is clearly just doing electioneering for the Democratic party at this point, trying to bolster turnout by exciting the democratic base and discouraging republicans. Clearly this scientific fiction has failed, twice now and by an ever greater margin, and until there is a full and unabashed accounting and rectification of this in public these pollsters can never be trusted again by anyone.
- johncolanduoni 6y ago
  What percentage chance must a forecast assign to an outcome before you would be able to accept it happening not being due to dishonesty? 538 gave Trump a ~30% chance of winning on Election Day 2016. If you roll a die twice and it comes up less than 4 both times, would you start looking for criminal intent? It’s also worth noting that 538 doesn’t conduct any of the polls that feed into their forecast, and so aren’t pollsters in a relevant sense here. Their entire reason for existing is that they agree that there is systematic bias in polls so they attempt to model it.
  - tehjoker 6y ago
    I feel the fact that statistics are uncertain gives them incredible cover for their deceptions. They aren't just slightly wrong, they are extremely wrong in not just the overall picture, but in state after state and race after race, in multiple election cycles to the degree that if it happened in any other country, the UN would intervene. The magnitude of their errors are shocking, stunning, and increasing over time. I can't understand how anyone can ever look at these again without a canister of salt. If Trump won in a landslide after they predicted Biden up by a landslide but assigned it some kind of probability you're going to give them a pass? Of course not. The fact that there are so many polls and they are all wrong means that there is something foul at play at the heart of this entire enterprise. The most charitable thing I could say is that they are completely useless, a toy metric for amusements.
    - johncolanduoni 6y ago
      Could you quantify the extremely wrong part? You can see their forecast map here[1]. Even if we just look at whether the probability of Trump winning a particular state was > 50% (and ignore the magnitude), the only state we have some semblance of results for that they got wrong is Florida, plus one of the Maine congressional districts. For over 50 discrete predictions that definitely doesn’t look extremely wrong to me. Given how negative you are about them “over time”, I wouldn’t have expected you to take a detailed look at their model this time around. Out of curiosity, what drove you to take a look this time around so you could claim their errors are “shocking, stunning, and increasing”? [1]: https://projects.fivethirtyeight.com/trump-biden-election-map/ https://projects.fivethirtyeight.com/trump-biden-election-ma...
tyre 6y ago
My issue with 538’s models is how wide their margins are. I believe they use “results are within the 80% bounds of our model” to gauge whether or not it reasonably “worked.” But that range is so huge as to be useless. Yes, it only predicts probabilities and can only be as accurate as the underlying data (which seems to be Bad.) But they don’t show/market it as being “anything could happen, this model is wide open and should only be used to rule out crazy outliers”.
- dragonwriter 6y ago
  > My issue with 538’s models is how wide their margins are. I believe they use “results are within the 80% bounds of our model” to gauge whether or not it reasonably “worked.” They also assess their models in aggregate, but that obviously isn't applicable to the Presidential model in isolation, since it has so few runs; OTOH, it does validate their broad approach while leaving open the possibility that there is something especially wacky with the Presidential model. But without lots of runs of that model, you can't really clearly single that out from chance errors, and if there are isolated instances of unusual effects that show up only in the Presidential election and only, say, for one or two cycles (e.g., if there is an effect around Trump which is atypical but not new normal) that aren't accounted for but also aren't long-term features, you probably can't even reliably detect them once there are large numbers of runs of the Presidential model. Unfortunately, validating a statistical predictive model isn't itself a a probabilistic exercise subject to small-N problems.
- divbzero 6y ago
  I don’t see how wide margins are a fault. To me that’s just being honest about the uncertainty. I also don’t think they hide this uncertainty. The model results I’ve seen on FiveThirtyEight always feature a distribution that shows the electoral margin could end up being anything in a wide –X to +Y range.
  - jonstewart 6y ago
    The problem’s that 538 is selling certainty. That’s where their pixels go, with the implication that they’re big math nerds who can divine the future based on combining lots of polls and other data sources. If it all amounts to “eh, we don’t really know,” then 538 doesn’t have much of a business model.
interestica 6y ago
There was some really great insight tonight on The Agenda (Current issues/politics show on TVOntario Canada) from the president of Advanced Symbolics. She went over what they got wrong and why. https://www.tvo.org/video/what-the-polls-got-wrong https://www.tvo.org/video/what-the-polls-got-wrong
gpm 6y ago
> In the end, polling averages will probably "call" the winners of all but 1-3 states correctly, along with the winner of the popular vote, which should wind up at Biden +4/+5. That's not great. It's better to look at margins and some of the margins were off. - Nate Silver https://twitter.com/NateSilver538/status/1324149741463228416 https://twitter.com/NateSilver538/status/1324149741463228416
- rayiner 6y ago
  The problem with ignoring that they missed many states by 5-7 points is that it completely scrambled the view of down-ballot races. Democrats poured $300 million into Senate races the lost by large margins. Not only is it unlikely that Democrats gain the Senate, they lost seats in the House. That’s an insane result when national polls were predicting 7+ for Biden.
  - gpm 6y ago
    > The problem with ignoring that they missed many states by 5-7 points That's literally the opposite of what he's doing in the post you are replying to...
    - rayiner 6y ago
      I’m adding to the point about why margins matter not contradicting Silver. (Though I’m convinced that Silver understands statistics better than politics.)
kgwgk 6y ago
> unlike Gelman, Nate Silver is a less likely to admit their model was off The irony is that Gelman has spent quite some time lately criticizing Silver et al. for forcing too much uncertainty into their model.
- mcphage 6y ago
  > The irony is that Gelman has spent quite some time lately criticizing Silver et al. for forcing too much uncertainty into their model. IIRC Silver got the same criticisms just before the 2016 election, too—that the only reason 538 didn't have Clinton's odds in the upper 90s, and had Clinton losing Florida, were to drum up interest and attention at the end of the race.
  - saeranv 6y ago
    Gelman's criticism is a little different. It's not that 538 has too much uncertainty, but the way they created the uncertainty. The way I understand it (and I'm an amateur so I could be wrong), the criticism is that the 538 team basically fudged state-level uncertainties to get the presidential forecast uncertainty that looked right to them, with the result you get a lot of unrealisitic state-by-state breakdowns (i.e Trump wins New Jersey but can't win Alaska). Gelman's critique is not that the uncertainty is too big, but that it's not integrated correctly.

ehsankia 6y ago

Well for one, 538 gave Trump 10% at the lowest, which is quite a bit better than 4% by CU. Also, putting Florida aside, everything else so far has been well within the error margin, even if the overall mean is 3-4 points shifted towards Trump. I'm not an expert, but that's the whole point of confidence intervals. Polls will never be accurate, but it's about using the right uncertainty, and also mixing uncertainties properly. EDIT: Let's look at a few of the tipping point results from 538 [1] Florida: 50.9/48.4 -> 47.8/51.2 (2.8 diff) [99% reporting] Arizona: 50.7/48.1 -> 50.7/47.9 (0.1 diff) [86% reporting] Texas: 48.8/50.3 -> 46.4/52.2 (1.9 diff) [85% reporting] Georgia: 50.1/49.2 -> 49.1/49.7 (0.7 diff) [98% reporting] Minnesota: 53.7/44.6 -> 52.6/45.4 (0.9 diff) [99% reporting] It's also worth noting that a lot of these are still at 85-95 percent reporting, so it's not over, but yes I don't see anything that's very off, even Florida that was so off was only 3 points off. I do consider 3 points to be well within the toss-up category. [1] https://projects.fivethirtyeight.com/2020-election-forecast/ https://projects.fivethirtyeight.com/2020-election-forecast/ EDIT2: I think the main thing that surprised people is how in some places, mail-in votes came first, whereas in some other places, mail-in votes are coming much later. This really skews the early data but as things settle, everything is coming much closer to the predictions. But this is something 538 and others have been warning about since May or earlier.

ars 6y ago
> but that's the whole point of confidence intervals. That would make sense if some parts of the country had errors one way, and others another way. i.e. random error. But if every part of the country has an error, all in the same direction, then there's something larger going on.
- dragonwriter 6y ago
  > That would make sense if some parts of the country had errors one way, and others another way. i.e. random error. As Silver pointed out in 2016 (and why his model then gave Trump a much better chance than the NYTimes model), historically, when election outcomes differ from polls in either direction, the polling error between states is correlated, not independent. That is, it is, to all evidence, a historical fact that poll-based predictions must account for that within election cycles, there is systematic polling error, even if the direction and magnitude of that error is not consistent between cycles.
- akersten 6y ago
  > i.e. random error. Did they say that their confidence interval was confidence against random error? I don't think that's a requirement of a confidence interval, I think it's certainly valid to encode e.g. the chance of "shy Trump" respondents in a confidence interval.
- slg 6y ago
  The results of state elections aren't independent. The problems that cause most errors are going to exist in multiple states and cause the errors to be correlated.
  - thaumasiotes 6y ago
    But something that causes the results everywhere to be off in the same direction isn't the kind of error you say "well, that's why you don't make point predictions" about. It's a factor that has a significant, predictable influence on the phenomenon you're studying; if it's not in your model, that means you screwed up when making the model. The whole point of the model is to account explicitly for such factors.

slg 6y ago

> I would argue 538 made a similar error[1], but unlike Gelman, Nate Silver is a less likely to admit their model was off, and consequently examine/discuss the underlying factors that lead to this issue. Why do you say this? He did plenty of writing on what he, the site, and the whole industry got wrong in 2016 for Trump's win in both the primary[1] and general[2][3]. I really don't understand the hate Nate Silver gets from basically all sides. [1] - https://fivethirtyeight.com/features/how-i-acted-like-a-pundit-and-screwed-up-on-donald-trump/ https://fivethirtyeight.com/features/how-i-acted-like-a-pund... [2] - https://fivethirtyeight.com/features/the-real-story-of-2016/ https://fivethirtyeight.com/features/the-real-story-of-2016/ [3] - https://fivethirtyeight.com/tag/the-real-story-of-2016/ https://fivethirtyeight.com/tag/the-real-story-of-2016/

saeranv 6y ago
In a lot of small ways, Gelman (and the Economist team) engage in more transparency, and self-critique about forecasting practices relative to the 538 team. I don't mean to come off as a Nate Silver hater. My criticism is made with a lot of affection, and awareness of how much he discusses the role of uncertainty in his work. But the Economist team has made some legitimate critiques of some wacky tail behavior in the 538 model[1] which Nate Silver has basically dismissed without engaging, whereas the Economist team has written plenty critiquing their own model[2]. I mean this is Gelman's whole schtick about the scientific process: he thinks that errors are to be expected and should be transparently discussed and fixed and iteratively improved over time. That's partly why they've released all their code on Github, and Gelman has sort of poked at 538 for not doing the same with their code. [1] https://statmodeling.stat.columbia.edu/2020/10/24/reverse-engineering-the-problematic-tail-behavior-of-the-fivethirtyeight-presidential-election-forecast/ https://statmodeling.stat.columbia.edu/2020/10/24/reverse-en... [2] https://statmodeling.stat.columbia.edu/2020/10/24/reverse-engineering-the-problematic-tail-behavior-of-the-fivethirtyeight-presidential-election-forecast/ https://statmodeling.stat.columbia.edu/2020/10/24/reverse-en...
peteretep 6y ago
People hate Nate because he’s pretty ruthless in calling out bullshit and there’s a great deal of bullshit produced by amateurs and professionals on both sides

eitau_1 6y ago

Chart polls vs results: https://twitter.com/thomasjwood/status/1324171829007757316 https://twitter.com/thomasjwood/status/1324171829007757316

tehjoker 6y ago

Nate Silver is clearly just doing electioneering for the Democratic party at this point, trying to bolster turnout by exciting the democratic base and discouraging republicans. Clearly this scientific fiction has failed, twice now and by an ever greater margin, and until there is a full and unabashed accounting and rectification of this in public these pollsters can never be trusted again by anyone.

johncolanduoni 6y ago
What percentage chance must a forecast assign to an outcome before you would be able to accept it happening not being due to dishonesty? 538 gave Trump a ~30% chance of winning on Election Day 2016. If you roll a die twice and it comes up less than 4 both times, would you start looking for criminal intent? It’s also worth noting that 538 doesn’t conduct any of the polls that feed into their forecast, and so aren’t pollsters in a relevant sense here. Their entire reason for existing is that they agree that there is systematic bias in polls so they attempt to model it.
- tehjoker 6y ago
  I feel the fact that statistics are uncertain gives them incredible cover for their deceptions. They aren't just slightly wrong, they are extremely wrong in not just the overall picture, but in state after state and race after race, in multiple election cycles to the degree that if it happened in any other country, the UN would intervene. The magnitude of their errors are shocking, stunning, and increasing over time. I can't understand how anyone can ever look at these again without a canister of salt. If Trump won in a landslide after they predicted Biden up by a landslide but assigned it some kind of probability you're going to give them a pass? Of course not. The fact that there are so many polls and they are all wrong means that there is something foul at play at the heart of this entire enterprise. The most charitable thing I could say is that they are completely useless, a toy metric for amusements.
  - johncolanduoni 6y ago
    Could you quantify the extremely wrong part? You can see their forecast map here[1]. Even if we just look at whether the probability of Trump winning a particular state was > 50% (and ignore the magnitude), the only state we have some semblance of results for that they got wrong is Florida, plus one of the Maine congressional districts. For over 50 discrete predictions that definitely doesn’t look extremely wrong to me. Given how negative you are about them “over time”, I wouldn’t have expected you to take a detailed look at their model this time around. Out of curiosity, what drove you to take a look this time around so you could claim their errors are “shocking, stunning, and increasing”? [1]: https://projects.fivethirtyeight.com/trump-biden-election-map/ https://projects.fivethirtyeight.com/trump-biden-election-ma...

tyre 6y ago

My issue with 538’s models is how wide their margins are. I believe they use “results are within the 80% bounds of our model” to gauge whether or not it reasonably “worked.” But that range is so huge as to be useless. Yes, it only predicts probabilities and can only be as accurate as the underlying data (which seems to be Bad.) But they don’t show/market it as being “anything could happen, this model is wide open and should only be used to rule out crazy outliers”.

dragonwriter 6y ago
> My issue with 538’s models is how wide their margins are. I believe they use “results are within the 80% bounds of our model” to gauge whether or not it reasonably “worked.” They also assess their models in aggregate, but that obviously isn't applicable to the Presidential model in isolation, since it has so few runs; OTOH, it does validate their broad approach while leaving open the possibility that there is something especially wacky with the Presidential model. But without lots of runs of that model, you can't really clearly single that out from chance errors, and if there are isolated instances of unusual effects that show up only in the Presidential election and only, say, for one or two cycles (e.g., if there is an effect around Trump which is atypical but not new normal) that aren't accounted for but also aren't long-term features, you probably can't even reliably detect them once there are large numbers of runs of the Presidential model. Unfortunately, validating a statistical predictive model isn't itself a a probabilistic exercise subject to small-N problems.
divbzero 6y ago
I don’t see how wide margins are a fault. To me that’s just being honest about the uncertainty. I also don’t think they hide this uncertainty. The model results I’ve seen on FiveThirtyEight always feature a distribution that shows the electoral margin could end up being anything in a wide –X to +Y range.
- jonstewart 6y ago
  The problem’s that 538 is selling certainty. That’s where their pixels go, with the implication that they’re big math nerds who can divine the future based on combining lots of polls and other data sources. If it all amounts to “eh, we don’t really know,” then 538 doesn’t have much of a business model.

interestica 6y ago

There was some really great insight tonight on The Agenda (Current issues/politics show on TVOntario Canada) from the president of Advanced Symbolics. She went over what they got wrong and why. https://www.tvo.org/video/what-the-polls-got-wrong https://www.tvo.org/video/what-the-polls-got-wrong

gpm 6y ago

> In the end, polling averages will probably "call" the winners of all but 1-3 states correctly, along with the winner of the popular vote, which should wind up at Biden +4/+5. That's not great. It's better to look at margins and some of the margins were off. - Nate Silver https://twitter.com/NateSilver538/status/1324149741463228416 https://twitter.com/NateSilver538/status/1324149741463228416

rayiner 6y ago
The problem with ignoring that they missed many states by 5-7 points is that it completely scrambled the view of down-ballot races. Democrats poured $300 million into Senate races the lost by large margins. Not only is it unlikely that Democrats gain the Senate, they lost seats in the House. That’s an insane result when national polls were predicting 7+ for Biden.
- gpm 6y ago
  > The problem with ignoring that they missed many states by 5-7 points That's literally the opposite of what he's doing in the post you are replying to...
  - rayiner 6y ago
    I’m adding to the point about why margins matter not contradicting Silver. (Though I’m convinced that Silver understands statistics better than politics.)

kgwgk 6y ago

> unlike Gelman, Nate Silver is a less likely to admit their model was off The irony is that Gelman has spent quite some time lately criticizing Silver et al. for forcing too much uncertainty into their model.

mcphage 6y ago
> The irony is that Gelman has spent quite some time lately criticizing Silver et al. for forcing too much uncertainty into their model. IIRC Silver got the same criticisms just before the 2016 election, too—that the only reason 538 didn't have Clinton's odds in the upper 90s, and had Clinton losing Florida, were to drum up interest and attention at the end of the race.
- saeranv 6y ago
  Gelman's criticism is a little different. It's not that 538 has too much uncertainty, but the way they created the uncertainty. The way I understand it (and I'm an amateur so I could be wrong), the criticism is that the 538 team basically fudged state-level uncertainties to get the presidential forecast uncertainty that looked right to them, with the result you get a lot of unrealisitic state-by-state breakdowns (i.e Trump wins New Jersey but can't win Alaska). Gelman's critique is not that the uncertainty is too big, but that it's not integrated correctly.