7 ms·

How Good Are FiveThirtyEight Forecasts?

34 points by jacobedawson 7y ago

PaulHoule 7y ago
This 'calibration' is a missing link for machine learning. A prediction is much more valuable if you know how good of a prediction it is than if you don't.
- paulgb 7y ago
  I'd caution about putting too much stock in calibration, though. It's worth noting that you can achieve perfect calibration just by giving each of n candidates 1/n probability of winning.
  - robto 7y ago
    I don't think you can. If you give every candidate a 1/n probability in winning, but the actual true probability is less than 1% (I'm looking at you, Vermin Supreme[0]), then your calibration is going to be way off. Good calibration is having your 65% chance of victory be right 65% of the time. [0]https://en.wikipedia.org/wiki/Vermin_Supreme https://en.wikipedia.org/wiki/Vermin_Supreme
    - paulgb 7y ago
      Right, but say in 2016 you gave a 1/3 chance to Trump, 1/3 chance to Clinton, and 1/3 chance to Vermin Supreme. Exactly one of them will win, so your calibration for 1/3 will be perfect. If you do the same for elections with different numbers candidates, you can have good calibration overall without actually producing any information over the base rate.
      - robto 7y ago
        I disagree. Suppose you have an election with 3 candidates that runs every year. If you predict candidate A win at 33% confidence, but that candidate gets elected every year, your calibration will be way off. Calibration is about aggregating, so you can't really cheat it. Your 33% prediction has to be right 33% of the time - it's really hard to fake that. Maybe I'm misunderstanding your position, though.
        paulgb 7y ago
        That's right if you only predict the probability of candidate A, but if you are predicting 33% for all three candidates each year, your 33% prediction will consistently be right 1/3 of the time in aggregate.
empath75 7y ago
All of the people complaining about how Five Thirty Eight is a fraud because they "predicted" that Hillary Clinton would win should read this. They gave Trump a significant chance of winning, more than almost any other poll aggregator did. In fact, before the election, Nate made a post outlining all the reasons that Trump could win and gave him a 36% chance of doing so. They literally said that trump would win based on polls like that 36% of the time, so it shouldn't have been that shocking that he did. https://fivethirtyeight.com/features/election-update-dont-ignore-the-polls-clinton-leads-but-its-a-close-race/ https://fivethirtyeight.com/features/election-update-dont-ig... You might say that a forecast that's so ambivalent isn't useful, but if someone told you that there's a 36% chance of rain, you'd plan your beach trip accordingly. I don't see why predicting a 36% chance of the underdog winning the presidential race is any less informative.
- keymone 7y ago
  can downvoters elaborate on why they disagree?
  - ghaff 7y ago
    It's probably a confusing the map with the territory sort of thing. FiveThirtyEight predicted Clinton would probably win and didn't. Many people are (understandably) upset she didn't win and they're lashing out at FiveThirtyEight for having given them what turned out to be false hope.
    - pwinnski 7y ago
      As evidenced slightly in this thread already, my observation is that most of the critics actually wanted Trump to win and are gloating that he did.
    - erentz 7y ago
      It might also be something a bit different: There's a lot of people on the left who are angry with Clinton and the Democrat establishment for running a very technical/data-driven campaign that turned out to be very flawed and ran counter to common sense (even her husband was telling her to campaign in areas she was ignoring due to this "data"). The technically driven 538 sort of represents that attitude. Nate Silver has also started to stray into political punditry which is turning a lot of people off. People who liked him for his data are now having to hear where he stands on issues.
objektif 7y ago
They obviously got 16 very wrong. However, what is even more concerning is that Nate lately became a political hack, doing the ugly work for the democratic establishment. Just look at his twitter you can clearly see it. So I would take any "data science" he does with a grain of salt. Edit: An example of his tweets: "It's probably worth noting that while this group, Justice Democrats, calls Biden "out-of-touch" with the "center of energy" in the Democratic Party, only 26 of the 79 candidates it endorsed last year won their primaries, and only 7 of those went on to win the general election." Why didn’t he add the statistic that Biden already ran 2 times in the past and won 0 times.
- paulgb 7y ago
  I have seen his Twitter feed, do you have any specific examples? Obviously he's talking a lot about Democrats right now more than Republicans because Trump is the presumed Republican nominee, but covering the race isn't "doing ugly work for the Democratic establishment."
  - objektif 7y ago
    "It's probably worth noting that while this group, Justice Democrats, calls Biden "out-of-touch" with the "center of energy" in the Democratic Party, only 26 of the 79 candidates it endorsed last year won their primaries, and only 7 of those went on to win the general election." Please explain to me what is the point of such a tweet.
    - paulgb 7y ago
      It contextualizes a remark from a group that claims to speak for the party's electorate by providing hard numbers. That's what people who follow Nate expect from him on Twitter. Edit: to be fair, if memory serves JD mostly/entirely endorsed primary challenges rather than incumbents. The base rate for primary challengers is pretty low, so ~1/3 is a respectable win rate and I would concede that it's a little disingenuous not to mention that.
      - 7y ago
cljs-js-eval 7y ago
This gets to the core of Taleb's beef with FiveThirtyEight - what political forecasts are calibrated? The ones that were forecasted 6 months out, or the last forecast before the end of the election? They may only be calibrated when enough of the facts are in to make predicting easier. Still useful, but not calibrated in the way the general public might think.
- Symmetry 7y ago
  If you dive a bit more they look at each forcast at each time interval for which it is made. So they include the 12 month out one, the 11 month out one, etc when assessing calibration.
  - cljs-js-eval 7y ago
    Ah, that's very useful to know. I take my statement back then, this is better calibration than I thought.
joker3 7y ago
This is good in the sense that having poorly calibrated forecasts is bad, but it's not quite as strong as you might want it to be. Imagine that you're tasked with predicting the next state of a traffic light that just cycles through green, yellow and red over and over again. If at every step you forecast that there's a 33% chance of each color, you have a very well-calibrated forecaster, but it's clear that you can do better. FiveThirtyEight probably actually is doing better, but I suspect that they're not choosing to highlight those stronger metrics because they're complicated to explain.
- jsnell 7y ago
  But isn't that exactly what the last third of the article discusses?
petraeus 7y ago
Can anyone explain how 538 got the last election so wrong? Its quite obvious in hindsight their model is flawed, but Im not understanding how is it flawed.
- KeenFox 7y ago
  Did they? As far as I can recall 538 has always been clear that they deal in probabilities, not binary predictions. They gave him a small chance of winning for a reason, and it turned out chaos favored one of the low odds scenarios.
- fourmajor 7y ago
  If you're referring to either the midterms or the 2016 presidential election, they didn't get either particularly wrong.

34 points by jacobedawson 7y ago

This 'calibration' is a missing link for machine learning. A prediction is much more valuable if you know how good of a prediction it is than if you don't.

paulgb 7y ago
I'd caution about putting too much stock in calibration, though. It's worth noting that you can achieve perfect calibration just by giving each of n candidates 1/n probability of winning.
- robto 7y ago
  I don't think you can. If you give every candidate a 1/n probability in winning, but the actual true probability is less than 1% (I'm looking at you, Vermin Supreme[0]), then your calibration is going to be way off. Good calibration is having your 65% chance of victory be right 65% of the time. [0]https://en.wikipedia.org/wiki/Vermin_Supreme https://en.wikipedia.org/wiki/Vermin_Supreme
  - paulgb 7y ago
    Right, but say in 2016 you gave a 1/3 chance to Trump, 1/3 chance to Clinton, and 1/3 chance to Vermin Supreme. Exactly one of them will win, so your calibration for 1/3 will be perfect. If you do the same for elections with different numbers candidates, you can have good calibration overall without actually producing any information over the base rate.
    - robto 7y ago
      I disagree. Suppose you have an election with 3 candidates that runs every year. If you predict candidate A win at 33% confidence, but that candidate gets elected every year, your calibration will be way off. Calibration is about aggregating, so you can't really cheat it. Your 33% prediction has to be right 33% of the time - it's really hard to fake that. Maybe I'm misunderstanding your position, though.
      - paulgb 7y ago
        That's right if you only predict the probability of candidate A, but if you are predicting 33% for all three candidates each year, your 33% prediction will consistently be right 1/3 of the time in aggregate.

empath75 7y ago

All of the people complaining about how Five Thirty Eight is a fraud because they "predicted" that Hillary Clinton would win should read this. They gave Trump a significant chance of winning, more than almost any other poll aggregator did. In fact, before the election, Nate made a post outlining all the reasons that Trump could win and gave him a 36% chance of doing so. They literally said that trump would win based on polls like that 36% of the time, so it shouldn't have been that shocking that he did. https://fivethirtyeight.com/features/election-update-dont-ignore-the-polls-clinton-leads-but-its-a-close-race/ https://fivethirtyeight.com/features/election-update-dont-ig... You might say that a forecast that's so ambivalent isn't useful, but if someone told you that there's a 36% chance of rain, you'd plan your beach trip accordingly. I don't see why predicting a 36% chance of the underdog winning the presidential race is any less informative.

keymone 7y ago
can downvoters elaborate on why they disagree?
- ghaff 7y ago
  It's probably a confusing the map with the territory sort of thing. FiveThirtyEight predicted Clinton would probably win and didn't. Many people are (understandably) upset she didn't win and they're lashing out at FiveThirtyEight for having given them what turned out to be false hope.
  - pwinnski 7y ago
    As evidenced slightly in this thread already, my observation is that most of the critics actually wanted Trump to win and are gloating that he did.
  - erentz 7y ago
    It might also be something a bit different: There's a lot of people on the left who are angry with Clinton and the Democrat establishment for running a very technical/data-driven campaign that turned out to be very flawed and ran counter to common sense (even her husband was telling her to campaign in areas she was ignoring due to this "data"). The technically driven 538 sort of represents that attitude. Nate Silver has also started to stray into political punditry which is turning a lot of people off. People who liked him for his data are now having to hear where he stands on issues.

objektif 7y ago

They obviously got 16 very wrong. However, what is even more concerning is that Nate lately became a political hack, doing the ugly work for the democratic establishment. Just look at his twitter you can clearly see it. So I would take any "data science" he does with a grain of salt. Edit: An example of his tweets: "It's probably worth noting that while this group, Justice Democrats, calls Biden "out-of-touch" with the "center of energy" in the Democratic Party, only 26 of the 79 candidates it endorsed last year won their primaries, and only 7 of those went on to win the general election." Why didn’t he add the statistic that Biden already ran 2 times in the past and won 0 times.

paulgb 7y ago
I have seen his Twitter feed, do you have any specific examples? Obviously he's talking a lot about Democrats right now more than Republicans because Trump is the presumed Republican nominee, but covering the race isn't "doing ugly work for the Democratic establishment."
- objektif 7y ago
  "It's probably worth noting that while this group, Justice Democrats, calls Biden "out-of-touch" with the "center of energy" in the Democratic Party, only 26 of the 79 candidates it endorsed last year won their primaries, and only 7 of those went on to win the general election." Please explain to me what is the point of such a tweet.
  - paulgb 7y ago
    It contextualizes a remark from a group that claims to speak for the party's electorate by providing hard numbers. That's what people who follow Nate expect from him on Twitter. Edit: to be fair, if memory serves JD mostly/entirely endorsed primary challenges rather than incumbents. The base rate for primary challengers is pretty low, so ~1/3 is a respectable win rate and I would concede that it's a little disingenuous not to mention that.
    - 7y ago

cljs-js-eval 7y ago

This gets to the core of Taleb's beef with FiveThirtyEight - what political forecasts are calibrated? The ones that were forecasted 6 months out, or the last forecast before the end of the election? They may only be calibrated when enough of the facts are in to make predicting easier. Still useful, but not calibrated in the way the general public might think.

Symmetry 7y ago
If you dive a bit more they look at each forcast at each time interval for which it is made. So they include the 12 month out one, the 11 month out one, etc when assessing calibration.
- cljs-js-eval 7y ago
  Ah, that's very useful to know. I take my statement back then, this is better calibration than I thought.

joker3 7y ago

This is good in the sense that having poorly calibrated forecasts is bad, but it's not quite as strong as you might want it to be. Imagine that you're tasked with predicting the next state of a traffic light that just cycles through green, yellow and red over and over again. If at every step you forecast that there's a 33% chance of each color, you have a very well-calibrated forecaster, but it's clear that you can do better. FiveThirtyEight probably actually is doing better, but I suspect that they're not choosing to highlight those stronger metrics because they're complicated to explain.

jsnell 7y ago
But isn't that exactly what the last third of the article discusses?

petraeus 7y ago

Can anyone explain how 538 got the last election so wrong? Its quite obvious in hindsight their model is flawed, but Im not understanding how is it flawed.

KeenFox 7y ago
Did they? As far as I can recall 538 has always been clear that they deal in probabilities, not binary predictions. They gave him a small chance of winning for a reason, and it turned out chaos favored one of the low odds scenarios.
fourmajor 7y ago
If you're referring to either the midterms or the 2016 presidential election, they didn't get either particularly wrong.