6 ms·

Nate Silver is a guy who got lucky one year, and managed to convince everyone he's a genius.

saghm 4y ago
Not making a judgment about his methods or not, but he was pretty accurate more than once; in 2008 he predicted 49/50 states correctly in the US presidential election, and in 2012 he predicted all 50 correctly. You might be right that he isn't as smart as people think, but "got lucky one year" is an exaggeration.
- keithwinstein 4y ago
  I'm a big fan of Nate Silver, but I think this idea that he "predicted N/50 states correctly" in 2008 and 2012 is mostly a myth. (1) The website doesn't "predict" a state when it reports that the probability of (e.g.) an Obama victory in Florida is 50.3%, which I believe is what 538 assessed in 2012. If you think that a state's probability distribution is 50/50 between Candidate X and Y, and the outcome is Candidate X, obviously you don't get any points for a good prediction. You didn't make a prediction. And if you say it's 50.3% probability, it's essentially the same situation. You didn't make (much of) a prediction. Which is okay! (2) In terms of Brier Scores or entropy scores, which compare the probability with the outcome and give more "points" the stronger the prediction was made (vs. what actually happened), FiveThirtyEight's predictions are generally just middle-of-the-pack compared with other websites and mechanisms of aggregating these polls. (See, e.g., https://web.math.princeton.edu/~sswang/Wang_Origins_of_Presidential_poll_aggregation_withfigures.pdf https://web.math.princeton.edu/~sswang/Wang_Origins_of_Presi...) There's no strong evidence that 538's predictions are much better or worse than other reasonable ways to weight and aggregate the recent polls that anybody else might come up with. (3) And... almost everything that tries to quantify the accuracy of the predictions is talking about the state of the 538 website ON THE MORNING OF THE ELECTION. But that's probably the least useful day to be looking at FiveThirtyEight. By then, you can just wait a few hours and get the actual results. :-) The predictions that are really valuable are the ones that are made some time in advance, e.g. one or two months. But (a) it's hard to directly measure the "accuracy" of a prediction like that, because... things change and we can't rerun the universe, and (b) it's not like the model is somehow time-invariant to be able to measure the accuracy indirectly by proxy; there are various "time until election" factors included in these model itself, the dynamics of the election itself probably have a "time until election" dependency, AND the website operators, including 538, are generally manually tweaking the model parameters or behavior in ways they don't always disclose as the election evolves. So basically... it's hard to say. I think 538 has done great work, and in general it's incredibly useful to have a consistent world model that you can rationally update to discuss the effect of recent news. The FiveThirtyEight political models are as good as any for that! But I don't think there's much strong evidence they're better.
  - saghm 4y ago
    That's fair! I probably should have realized that, given that I already knew that his 70-30 prediction of Clinton winning in 2016 didn't mean he "predicting wrong"; I'm guessing I was just less aware of how his models worked back in 2012 than I was by 2016, so I didn't really consider what it meant to "predict the states".
    - yowzadave 4y ago
      I think many people subconsciously conflate a "70% chance of winning" with a "70% of the vote" prediction, when the reality is much closer--a 70% chance of winning could very easily go the other way, given some systemic polling error and/or late-breaking news!
      - saghm 4y ago
        Yeah, it's definitely tricky to communicate. The chances of _either_ candidate getting 70% of the vote in any recent presidential election would be incredibly small; even 55% would be a huge anomaly. Most people wouldn't feel very enlightened looking at a probability distribution of potential vote percentages for each candidate on a CNN graphic though, especially when they'd basically just look like giant spikes around 48-52%.
  - dragonwriter 4y ago
    > But it’s hard to directly measure the “accuracy” of a prediction like that, because… things change and we can’t rerun the universe Its hard to measure the accuracy of a prediction like that, but with a big collection of predictions (and since every cycle Silver predicts a lot of individual elections – not just each of the state elections for presidential electors, including both the statewide electors and the by-district electors where those exist, but also house, senate, gubernatorial, and some other races – each election cycle, this exists) you can bucket them in similar probability buckets and say, for instance, and see how often things Silver predicted in each probability bucket came true vs. the probability range of the bucket, and get a good idea of the accuracy of the predictions.
test098 4y ago
> Silver was named one of The World's 100 Most Influential People by Time in 2009 after an election forecasting system he developed successfully predicted the outcomes in forty-nine of the fifty states in the 2008 U.S. presidential election.[2] His subsequent election forecasting systems accurately predicted the outcome of the 2012 and 2020 presidential elections with a high degree of accuracy, and - at the time - gave Presidential candidate Donald Trump the highest chance of success the day of the 2016 presidential election of reputable polls at the time. - https://en.wikipedia.org/wiki/Nate_Silver https://en.wikipedia.org/wiki/Nate_Silver
- dragonwriter 4y ago
  > gave Presidential candidate Donald Trump the highest chance of success the day of the 2016 presidential election of reputable polls at the time This makes a fundamental error of confusing “polls” (which is not a category Silver’s work falls into) with statistical predictions based on aggregates of polls (which is what it is.)
  - itsdrewmiller 4y ago
    I think it's poorly worded - there were a bunch of other aggregators at the time who said things like Clinton was 99% likely to win. IIRC 538 assumed that polling bias could be correlated between states while most analyses were assuming that each poll was completely independent, but that's probably an oversimplification given the vagaries of memory.
    - dllthomas 4y ago
      Your recollection matches mine, fwiw.
foxandmouse 4y ago
At FiveThirtyEight, their goal is to deliver high-quality quantitative analysis by utilizing record-level data. It's ironic that the type of comment you made is typically found on sports news websites that lacks the rigor and depth of a quality analysis.
dragonwriter 4y ago
> Nate Silver is a guy who got lucky one year, and managed to convince everyone he’s a genius. Nate Silver is a guy who is very good at applying statistical modelling to a number of domains (including election analysis, where he basically revolutionized the field with everyone else doing poll-aggregate-based-prediction – which no one was previously – following in his footsteps, but mostly doing a worse job), but who morphed into another generic center-right Democratic political talking head. Whether that’s a structural demand of Disney that will go away now that he is leaving Disney or whether its what he feels his groove is now…will be interesting to see.
- mardifoufs 4y ago
  Genuine question, since I dont really follow him outside of elections, but in what way is he center right? I'm not american and I'm an immigrant/poc so I'm not especially biaised to the right but he always came off to me as pretty centrist and "liberal". When did that change? (The 2020 election was the last time I really heard about him, and the most controversial takes he had was some very mild reservations on lockdowns and school closures... at a time when the entire world reopened schools except for the USA, which is why it didn't register like a hot/right wing take at all to me.)
  - deleted 4y ago
    [deleted]
cyanydeez 4y ago
He didn't get lucky. He found a trend and published it widely. He changed the very thing he was studying. Because politics was already a statistical game, his original models failed to reproduce. He clearly published a single result.
greesil 4y ago
His approach is pretty sound, and straightforward if you listen to his podcast. Proper treatment of fat tails via student t distributions and correlated variables with good (subjective) judgement on polling quality. I find that most people with negative views of Nate's approach are innumerate with regards to statistics. They don't criticize the model because they can't be bothered to understand it, or even basic probability, and certainly don't rank his performance vs other modelers which wouldn't require understanding the model.
- petulla 4y ago
  No: https://statmodeling.stat.columbia.edu/2020/10/24/reverse-engineering-the-problematic-tail-behavior-of-the-fivethirtyeight-presidential-election-forecast/ https://statmodeling.stat.columbia.edu/2020/10/24/reverse-en...
  - greesil 4y ago
    What are the consequences of this, in terms of prediction accuracy?
  - greesil 4y ago
    so basically what TFA says is if you build a new model on Nate's model outputs and use that to compute conditional probabilites of various improbable scenarios you see weird shit. State level polling data according to Nate isn't all that great (read: high uncertainty, moves around when a new poll comes out) unless it's a swing state. But hey when you go looking into noise you are going to see.... weird shit. Did you Google any other good blog posts?
trc001 3y ago
I’ve been saying some version of this to people for years. But in his defense, I think he’s been more the victim of the secular decline in polling quality than anything else.

Not making a judgment about his methods or not, but he was pretty accurate more than once; in 2008 he predicted 49/50 states correctly in the US presidential election, and in 2012 he predicted all 50 correctly. You might be right that he isn't as smart as people think, but "got lucky one year" is an exaggeration.

keithwinstein 4y ago
I'm a big fan of Nate Silver, but I think this idea that he "predicted N/50 states correctly" in 2008 and 2012 is mostly a myth. (1) The website doesn't "predict" a state when it reports that the probability of (e.g.) an Obama victory in Florida is 50.3%, which I believe is what 538 assessed in 2012. If you think that a state's probability distribution is 50/50 between Candidate X and Y, and the outcome is Candidate X, obviously you don't get any points for a good prediction. You didn't make a prediction. And if you say it's 50.3% probability, it's essentially the same situation. You didn't make (much of) a prediction. Which is okay! (2) In terms of Brier Scores or entropy scores, which compare the probability with the outcome and give more "points" the stronger the prediction was made (vs. what actually happened), FiveThirtyEight's predictions are generally just middle-of-the-pack compared with other websites and mechanisms of aggregating these polls. (See, e.g., https://web.math.princeton.edu/~sswang/Wang_Origins_of_Presidential_poll_aggregation_withfigures.pdf https://web.math.princeton.edu/~sswang/Wang_Origins_of_Presi...) There's no strong evidence that 538's predictions are much better or worse than other reasonable ways to weight and aggregate the recent polls that anybody else might come up with. (3) And... almost everything that tries to quantify the accuracy of the predictions is talking about the state of the 538 website ON THE MORNING OF THE ELECTION. But that's probably the least useful day to be looking at FiveThirtyEight. By then, you can just wait a few hours and get the actual results. :-) The predictions that are really valuable are the ones that are made some time in advance, e.g. one or two months. But (a) it's hard to directly measure the "accuracy" of a prediction like that, because... things change and we can't rerun the universe, and (b) it's not like the model is somehow time-invariant to be able to measure the accuracy indirectly by proxy; there are various "time until election" factors included in these model itself, the dynamics of the election itself probably have a "time until election" dependency, AND the website operators, including 538, are generally manually tweaking the model parameters or behavior in ways they don't always disclose as the election evolves. So basically... it's hard to say. I think 538 has done great work, and in general it's incredibly useful to have a consistent world model that you can rationally update to discuss the effect of recent news. The FiveThirtyEight political models are as good as any for that! But I don't think there's much strong evidence they're better.
- saghm 4y ago
  That's fair! I probably should have realized that, given that I already knew that his 70-30 prediction of Clinton winning in 2016 didn't mean he "predicting wrong"; I'm guessing I was just less aware of how his models worked back in 2012 than I was by 2016, so I didn't really consider what it meant to "predict the states".
  - yowzadave 4y ago
    I think many people subconsciously conflate a "70% chance of winning" with a "70% of the vote" prediction, when the reality is much closer--a 70% chance of winning could very easily go the other way, given some systemic polling error and/or late-breaking news!
    - saghm 4y ago
      Yeah, it's definitely tricky to communicate. The chances of _either_ candidate getting 70% of the vote in any recent presidential election would be incredibly small; even 55% would be a huge anomaly. Most people wouldn't feel very enlightened looking at a probability distribution of potential vote percentages for each candidate on a CNN graphic though, especially when they'd basically just look like giant spikes around 48-52%.
- dragonwriter 4y ago
  > But it’s hard to directly measure the “accuracy” of a prediction like that, because… things change and we can’t rerun the universe Its hard to measure the accuracy of a prediction like that, but with a big collection of predictions (and since every cycle Silver predicts a lot of individual elections – not just each of the state elections for presidential electors, including both the statewide electors and the by-district electors where those exist, but also house, senate, gubernatorial, and some other races – each election cycle, this exists) you can bucket them in similar probability buckets and say, for instance, and see how often things Silver predicted in each probability bucket came true vs. the probability range of the bucket, and get a good idea of the accuracy of the predictions.

test098 4y ago

> Silver was named one of The World's 100 Most Influential People by Time in 2009 after an election forecasting system he developed successfully predicted the outcomes in forty-nine of the fifty states in the 2008 U.S. presidential election.[2] His subsequent election forecasting systems accurately predicted the outcome of the 2012 and 2020 presidential elections with a high degree of accuracy, and - at the time - gave Presidential candidate Donald Trump the highest chance of success the day of the 2016 presidential election of reputable polls at the time. - https://en.wikipedia.org/wiki/Nate_Silver https://en.wikipedia.org/wiki/Nate_Silver

dragonwriter 4y ago
> gave Presidential candidate Donald Trump the highest chance of success the day of the 2016 presidential election of reputable polls at the time This makes a fundamental error of confusing “polls” (which is not a category Silver’s work falls into) with statistical predictions based on aggregates of polls (which is what it is.)
- itsdrewmiller 4y ago
  I think it's poorly worded - there were a bunch of other aggregators at the time who said things like Clinton was 99% likely to win. IIRC 538 assumed that polling bias could be correlated between states while most analyses were assuming that each poll was completely independent, but that's probably an oversimplification given the vagaries of memory.
  - dllthomas 4y ago
    Your recollection matches mine, fwiw.

foxandmouse 4y ago

At FiveThirtyEight, their goal is to deliver high-quality quantitative analysis by utilizing record-level data. It's ironic that the type of comment you made is typically found on sports news websites that lacks the rigor and depth of a quality analysis.

dragonwriter 4y ago

> Nate Silver is a guy who got lucky one year, and managed to convince everyone he’s a genius. Nate Silver is a guy who is very good at applying statistical modelling to a number of domains (including election analysis, where he basically revolutionized the field with everyone else doing poll-aggregate-based-prediction – which no one was previously – following in his footsteps, but mostly doing a worse job), but who morphed into another generic center-right Democratic political talking head. Whether that’s a structural demand of Disney that will go away now that he is leaving Disney or whether its what he feels his groove is now…will be interesting to see.

mardifoufs 4y ago
Genuine question, since I dont really follow him outside of elections, but in what way is he center right? I'm not american and I'm an immigrant/poc so I'm not especially biaised to the right but he always came off to me as pretty centrist and "liberal". When did that change? (The 2020 election was the last time I really heard about him, and the most controversial takes he had was some very mild reservations on lockdowns and school closures... at a time when the entire world reopened schools except for the USA, which is why it didn't register like a hot/right wing take at all to me.)
- deleted 4y ago
  [deleted]

cyanydeez 4y ago

He didn't get lucky. He found a trend and published it widely. He changed the very thing he was studying. Because politics was already a statistical game, his original models failed to reproduce. He clearly published a single result.

greesil 4y ago

His approach is pretty sound, and straightforward if you listen to his podcast. Proper treatment of fat tails via student t distributions and correlated variables with good (subjective) judgement on polling quality. I find that most people with negative views of Nate's approach are innumerate with regards to statistics. They don't criticize the model because they can't be bothered to understand it, or even basic probability, and certainly don't rank his performance vs other modelers which wouldn't require understanding the model.

petulla 4y ago
No: https://statmodeling.stat.columbia.edu/2020/10/24/reverse-engineering-the-problematic-tail-behavior-of-the-fivethirtyeight-presidential-election-forecast/ https://statmodeling.stat.columbia.edu/2020/10/24/reverse-en...
- greesil 4y ago
  What are the consequences of this, in terms of prediction accuracy?
- greesil 4y ago
  so basically what TFA says is if you build a new model on Nate's model outputs and use that to compute conditional probabilites of various improbable scenarios you see weird shit. State level polling data according to Nate isn't all that great (read: high uncertainty, moves around when a new poll comes out) unless it's a swing state. But hey when you go looking into noise you are going to see.... weird shit. Did you Google any other good blog posts?

trc001 3y ago

I’ve been saying some version of this to people for years. But in his defense, I think he’s been more the victim of the secular decline in polling quality than anything else.