11 ms·

Put not thy trust in Nate Silver

13 points by seinundzeit 6y ago

xhkkffbf 6y ago
I think Nate is pretty sharp -- but I don't trust the people who answer polls. Who has time to pick up the phone? Most of the time it's some spammer. And if you do answer, why tell the truth about your vote?
- scarmig 6y ago
  That's also true of every election in recent history, and yet polls still historically have had predictive power. Even shy Tory stories are perennial fixtures in the poll criticism genre; perhaps it could exist as a stronger effect this year, but as yet there's no evidence that it is.
- saeranv 6y ago
  538 does rank pollsters based on "historical accuracy and methodology"[1], which are then weighted accordingly in their prediction model. And Nate and Elliot Morris (from the economist) have talked about how polling error tends to be normally distributed, so using multiple pollsters with different biases could correct for polling error. Pollsters are picking representative samples and weighting it to reflect the population, so it's not like they're just counting on who picks up their phone (and phone calls aren't the only way to poll anymore anyway). As I recall, in 2016 the pollsters did oversample college-educated white voters, which is what gave Clinton her misleading lead. This apparently has been fixed. There's also the fact that any demographic under or oversampling should be accounted for in the uncertainty range produced by the prediction. I haven't been able to find decent write-ups of this particular part of model forecasting, but from what I've been able to piece together : I believe they quantify the uncertainty of their presidential predictions by building logistic regressions of the output based on highly-correlated predictor data (i.e demographic information, GDP) and then simulating thousands of possible scenarios where demographics trend differently. I don't exactly know how these simulations are set up (I would love to know though). And presumably this is updated in a bayesian matter everytime they get more input data, so the prior probabilities will have an impact on the posterior. If anyone has a different or more detailed idea of how this is done, let me know. In fact, here's the actual code for Elliot Morris' forecasting model: https://github.com/TheEconomist/us-potus-model https://github.com/TheEconomist/us-potus-model if anyone can figure it out better then me. [1] https://projects.fivethirtyeight.com/pollster-ratings/ https://projects.fivethirtyeight.com/pollster-ratings/ [2] https://www.vox.com/21538156/biden-polls-lead-election-trump-2020-2016 https://www.vox.com/21538156/biden-polls-lead-election-trump...
- cameldrv 6y ago
  I think that the point is that both the pollsters and Nate are looking at how various polling methodologies correlate to actual past election results. The raw numbers from a poll are no good though, because you have a biased sample. The people who answer the survey are not representative of the people who show up to vote. They use the actual election results to calibrate the correction for this. As long as these relationships stay stable, polls should be pretty accurate. In my understanding, the biggest problem in 2016 was that the polls were undercorrecting for the difference between college educated voters vs. not college educated voters. Historically, this made less of a difference than other factors in voting behavior, so it had less impact in the weighting model. In 2016, it had a lot of impact, and college educated voters were more likely to answer the survey, so their tendency to vote Hillary was overrepresented in the polls. That said, the polls were off by less than 2% nationally. Perhaps that's not a great result, but it's not as if they were wildly wrong, it was just a close election. Unfortunately this election is also somewhat close this year, and voting behavior is going to be different with the pandemic, mail in voting, etc. I think it's very hard to reliably call Pennsylvania.
- smitty1e 6y ago
  Trump in 2016, and again in 2020 is running a disintermediation campaign and going straight to the voters with the rallies. If again effective, perhaps the teachers will return to school and learn something.
- dave333 6y ago
  This what scares me about the polls - that they don't account for Trump supporters being less truthful than the rest of us. Just look at Trump - he hasn't said anything true for years.
aidenn0 6y ago
I haven't finished this article, but what I've read so far just reads like anti-empiricism fluff, so I don't plan on finishing unless someone here suggests otherwise.
ravitation 6y ago
To be honest, this reads as quite the Luddite article. Not to mention the focus on Simulmatics is akin to using the 19th century use of bleeding as the basis for a criticism of modern medicine. Also, the modern example of Cambridge Analytica obviously has essentially nothing to do with what 538 actually does... Towards the end, the criticism of if a predictive model is falsifiable is an accurate one given that its output is probabilistic, and given the small number, and relative infrequency of, American elections (especially presidential ones) - which impacts the model's calibration. But, Nate actually brings this up quite often (which I assume the author would attribute to him "covering his bases").
xiainx 6y ago
Never heard of this news source before. Anyone know how reputable it is?
- deleted 6y ago
  [deleted]
- timbit42 6y ago
  Here is one opinion: https://mediabiasfactcheck.com/the-new-atlantis/ https://mediabiasfactcheck.com/the-new-atlantis/
scarmig 6y ago
It's pretty lame to paste an image of the NY Time's 82% chance of Clinton winning the election in an article that calls out Nate by name. Nate gave Clinton like a 70% chance. Can model building predict human behavior? It's fuzzier than the hard sciences, but not that much fuzzier. And if you do believe it's impossible to model human behavior, you have to throw out vast swaths of knowledge and areas of study: there goes sociology, anthropology, economics, psychology, epidemiology...
- deleted 6y ago
  [deleted]
- xlm1717 6y ago
  Economics is being thrown out in favor of Modern Monetary Theory. Psychology has a reproducibility crisis. Epidemiology... isn't doing so well. And I have no problem throwing out sociology and anthropology.
  - scarmig 6y ago
    All of these wildly miss the point. > Economics is being thrown out in favor of Modern Monetary Theory. MMT is many things, but it's most definitely not a rejection of the idea that human behavior can be modeled.
  - dwaltrip 6y ago
    Don’t throw out the baby with the bathwater.
mabbo 6y ago
Nate Silver and 538 gave Trump a 28% chance of winning on Election night in 2016. Nobody else gave him even close to that kind of probability. Sam Wang, considered another very good polling expert said he would eat a bug if Trump won, giving him less than 1% of a chance (and he DID[0]). And here is the New Atlantic saying not to trust Nate Silver? Give me a break. Statistical models give probabilities, not certainties. [0]https://www.cnn.com/2016/11/12/politics/pollster-eats-bug-after-donald-trump-win/index.html https://www.cnn.com/2016/11/12/politics/pollster-eats-bug-af...
- bassman9000 6y ago
  https://projects.fivethirtyeight.com/2020-election-forecast/?cid=rrpromo https://projects.fivethirtyeight.com/2020-election-forecast/... 89% chance to Biden. We'll see in a few days.
bleah1000 6y ago
The article is way too long, and doesn't really talk about Nate Silver much. However, one interesting item is the idea that there is no way to verify these predictions. Because they are forecasting one time events, there is no way to validate if their models are in the least bit accurate. If you have a X% chance of event happening and it doesn't, were you wrong? No because it wasn't 100%, and even if you are right, it's not terribly meaningful. It would be interesting to see if a site like five thirty-eight modeled all elections results, and then see how accurate they were across all of their predictions. Maybe you would have to look at all predictions with > 70% chance of happening and see if they got 70% of those correct. However, in the end those predictions seem like they could be done by throwing darts at a dart board and we wouldn't really know.
- scarmig 6y ago
  It's not as dire as you suggest. To simplify a bit, you can take the set of ~75% predictions created by a model (which can be an actual model, or it can be just the meta model of whatever model Nate Silver wants us to trust right now), and then see what percentage of them were right. If that comes to 75%, then the model is well-calibrated, and it's a solid enough basis to assign a 75% probability to future 75% predictions, which is often as good as you can get when trying to make decisions and plans.
- borramakot 6y ago
  Boy, have I got the incredibly specific article for that exact question: https://fivethirtyeight.com/features/when-we-say-70-percent-it-really-means-70-percent/ https://fivethirtyeight.com/features/when-we-say-70-percent-...
  - borramakot 6y ago
    I don't think this article provides strong evidence that they are well calibrated on the presidential election specifically (sample size N=3), or that they are correctly accounting for rare black swan events, but it does seem to imply that the criticisms about "538 claims victory no matter what because they always have non-zero probabilities" are oversimplified.

13 points by seinundzeit 6y ago

I think Nate is pretty sharp -- but I don't trust the people who answer polls. Who has time to pick up the phone? Most of the time it's some spammer. And if you do answer, why tell the truth about your vote?

scarmig 6y ago
That's also true of every election in recent history, and yet polls still historically have had predictive power. Even shy Tory stories are perennial fixtures in the poll criticism genre; perhaps it could exist as a stronger effect this year, but as yet there's no evidence that it is.
saeranv 6y ago
538 does rank pollsters based on "historical accuracy and methodology"[1], which are then weighted accordingly in their prediction model. And Nate and Elliot Morris (from the economist) have talked about how polling error tends to be normally distributed, so using multiple pollsters with different biases could correct for polling error. Pollsters are picking representative samples and weighting it to reflect the population, so it's not like they're just counting on who picks up their phone (and phone calls aren't the only way to poll anymore anyway). As I recall, in 2016 the pollsters did oversample college-educated white voters, which is what gave Clinton her misleading lead. This apparently has been fixed. There's also the fact that any demographic under or oversampling should be accounted for in the uncertainty range produced by the prediction. I haven't been able to find decent write-ups of this particular part of model forecasting, but from what I've been able to piece together : I believe they quantify the uncertainty of their presidential predictions by building logistic regressions of the output based on highly-correlated predictor data (i.e demographic information, GDP) and then simulating thousands of possible scenarios where demographics trend differently. I don't exactly know how these simulations are set up (I would love to know though). And presumably this is updated in a bayesian matter everytime they get more input data, so the prior probabilities will have an impact on the posterior. If anyone has a different or more detailed idea of how this is done, let me know. In fact, here's the actual code for Elliot Morris' forecasting model: https://github.com/TheEconomist/us-potus-model https://github.com/TheEconomist/us-potus-model if anyone can figure it out better then me. [1] https://projects.fivethirtyeight.com/pollster-ratings/ https://projects.fivethirtyeight.com/pollster-ratings/ [2] https://www.vox.com/21538156/biden-polls-lead-election-trump-2020-2016 https://www.vox.com/21538156/biden-polls-lead-election-trump...
cameldrv 6y ago
I think that the point is that both the pollsters and Nate are looking at how various polling methodologies correlate to actual past election results. The raw numbers from a poll are no good though, because you have a biased sample. The people who answer the survey are not representative of the people who show up to vote. They use the actual election results to calibrate the correction for this. As long as these relationships stay stable, polls should be pretty accurate. In my understanding, the biggest problem in 2016 was that the polls were undercorrecting for the difference between college educated voters vs. not college educated voters. Historically, this made less of a difference than other factors in voting behavior, so it had less impact in the weighting model. In 2016, it had a lot of impact, and college educated voters were more likely to answer the survey, so their tendency to vote Hillary was overrepresented in the polls. That said, the polls were off by less than 2% nationally. Perhaps that's not a great result, but it's not as if they were wildly wrong, it was just a close election. Unfortunately this election is also somewhat close this year, and voting behavior is going to be different with the pandemic, mail in voting, etc. I think it's very hard to reliably call Pennsylvania.
smitty1e 6y ago
Trump in 2016, and again in 2020 is running a disintermediation campaign and going straight to the voters with the rallies. If again effective, perhaps the teachers will return to school and learn something.
dave333 6y ago
This what scares me about the polls - that they don't account for Trump supporters being less truthful than the rest of us. Just look at Trump - he hasn't said anything true for years.

aidenn0 6y ago

I haven't finished this article, but what I've read so far just reads like anti-empiricism fluff, so I don't plan on finishing unless someone here suggests otherwise.

ravitation 6y ago

To be honest, this reads as quite the Luddite article. Not to mention the focus on Simulmatics is akin to using the 19th century use of bleeding as the basis for a criticism of modern medicine. Also, the modern example of Cambridge Analytica obviously has essentially nothing to do with what 538 actually does... Towards the end, the criticism of if a predictive model is falsifiable is an accurate one given that its output is probabilistic, and given the small number, and relative infrequency of, American elections (especially presidential ones) - which impacts the model's calibration. But, Nate actually brings this up quite often (which I assume the author would attribute to him "covering his bases").

xiainx 6y ago

Never heard of this news source before. Anyone know how reputable it is?

deleted 6y ago
[deleted]
timbit42 6y ago
Here is one opinion: https://mediabiasfactcheck.com/the-new-atlantis/ https://mediabiasfactcheck.com/the-new-atlantis/

scarmig 6y ago

It's pretty lame to paste an image of the NY Time's 82% chance of Clinton winning the election in an article that calls out Nate by name. Nate gave Clinton like a 70% chance. Can model building predict human behavior? It's fuzzier than the hard sciences, but not that much fuzzier. And if you do believe it's impossible to model human behavior, you have to throw out vast swaths of knowledge and areas of study: there goes sociology, anthropology, economics, psychology, epidemiology...

deleted 6y ago
[deleted]
xlm1717 6y ago
Economics is being thrown out in favor of Modern Monetary Theory. Psychology has a reproducibility crisis. Epidemiology... isn't doing so well. And I have no problem throwing out sociology and anthropology.
- scarmig 6y ago
  All of these wildly miss the point. > Economics is being thrown out in favor of Modern Monetary Theory. MMT is many things, but it's most definitely not a rejection of the idea that human behavior can be modeled.
- dwaltrip 6y ago
  Don’t throw out the baby with the bathwater.

mabbo 6y ago

Nate Silver and 538 gave Trump a 28% chance of winning on Election night in 2016. Nobody else gave him even close to that kind of probability. Sam Wang, considered another very good polling expert said he would eat a bug if Trump won, giving him less than 1% of a chance (and he DID[0]). And here is the New Atlantic saying not to trust Nate Silver? Give me a break. Statistical models give probabilities, not certainties. [0]https://www.cnn.com/2016/11/12/politics/pollster-eats-bug-after-donald-trump-win/index.html https://www.cnn.com/2016/11/12/politics/pollster-eats-bug-af...

bassman9000 6y ago
https://projects.fivethirtyeight.com/2020-election-forecast/?cid=rrpromo https://projects.fivethirtyeight.com/2020-election-forecast/... 89% chance to Biden. We'll see in a few days.

bleah1000 6y ago

The article is way too long, and doesn't really talk about Nate Silver much. However, one interesting item is the idea that there is no way to verify these predictions. Because they are forecasting one time events, there is no way to validate if their models are in the least bit accurate. If you have a X% chance of event happening and it doesn't, were you wrong? No because it wasn't 100%, and even if you are right, it's not terribly meaningful. It would be interesting to see if a site like five thirty-eight modeled all elections results, and then see how accurate they were across all of their predictions. Maybe you would have to look at all predictions with > 70% chance of happening and see if they got 70% of those correct. However, in the end those predictions seem like they could be done by throwing darts at a dart board and we wouldn't really know.

scarmig 6y ago
It's not as dire as you suggest. To simplify a bit, you can take the set of ~75% predictions created by a model (which can be an actual model, or it can be just the meta model of whatever model Nate Silver wants us to trust right now), and then see what percentage of them were right. If that comes to 75%, then the model is well-calibrated, and it's a solid enough basis to assign a 75% probability to future 75% predictions, which is often as good as you can get when trying to make decisions and plans.
borramakot 6y ago
Boy, have I got the incredibly specific article for that exact question: https://fivethirtyeight.com/features/when-we-say-70-percent-it-really-means-70-percent/ https://fivethirtyeight.com/features/when-we-say-70-percent-...
- borramakot 6y ago
  I don't think this article provides strong evidence that they are well calibrated on the presidential election specifically (sample size N=3), or that they are correctly accounting for rare black swan events, but it does seem to imply that the criticisms about "538 claims victory no matter what because they always have non-zero probabilities" are oversimplified.