4 ms·

Do you have an example of a time where Nate Silver has shown a misunderstanding of probability? Everything I've read by him has seemed informed.

baddox 8y ago
The best criticism I can come up with is that 538 probably doesn't do a sufficient job of explaining to its audience that their published probabilities are predictions for what would happen if the event occurred today. This seems to be one of Taleb's (and others') criticisms of their predictions: that they don't consider the probability of major unexpected events occurring between now and the event (like an athlete getting injured in the future, or the FBI announcing an investigation of a political candidate). But as I understand it, 538 is explicitly not attempting to consider those things, and is simply publishing probabilities of outcomes if the event were to happen now.
- ska 8y ago
  I think overall 538 did a pretty good job of communicating a difficult subject to a lay audience, but agree this point could perhaps be communicated better. I don't know of a good way to try and capture these sort of events meaningfully, do you?
- eridius 8y ago
  That seems like a pretty big difference to gloss over. Why are they publishing probabilities for if the event were to happen today? The event isn't happening today!
  - pohl 8y ago
    I know, right? Every time my meteorologist gives me a probability of rain on a day that it doesn't rain I ask the same question.
    - steve19 8y ago
      Your personal meteorologist presumably predicts rain on days in the future, when it could happen, not days in the past. Nate publishes predictions for events that will never occur pr(x wins | election today) when there is no election. It would be like me placing bets on a sports game that is not scheduled to be played.
    - dralley 8y ago
      Meteorologists don't try to predict the weather 2 months from now.
      - pohl 8y ago
        Nate hasn't been trying to predict elections that far out, either, to be fair. He has always said some variant of "if the election were held today" when discussing the model.
  - baddox 8y ago
    They don't exactly gloss over it. They have tons of articles explaining it, but of course the average reader is just going to see the probabilities and run with that, and 538 does have some responsibility to actively explain things upfront in their UI, which I don't think they do a good job at. > Why are they publishing probabilities for if the event were to happen today? The event isn't happening today! My guess is that it's extremely out of scope (and probably ultimately infeasible) to try to incorporate these things into their models. Sure, there's a non-zero chance of a candidate being killed by a meteorite, but I think it's fair to ignore that and, if it happens, admit that you didn't even attempt to include the probability of that into your model.
    - dragonwriter 8y ago
      > the average reader is just going to see the probabilities and run with that The putative “average reader” who does so will see the default view, which is a forecast, not the nowcast, which you must actively shift to in order to see. > and 538 does have some responsibility to actively explain things upfront in their UI, Like, on the view selector that you have to use to choose the nowcast instead of the default forecast, where the nowcast is described as “Now-cast / Who would win the election if it were held today”. I think they have that covered.
  - coliveira 8y ago
    Because this is what every election poll does. Nobody asks people if they would vote in such person a year from now. They always ask what is your voting preference today. If Nate Silver did something different, then he would be completely wrong.
    - eridius 8y ago
      If Nate Silver took the predictions for "if the event were to happen today" and pretended it was a prediction for the future event, he'd be wrong, but that's how most people interpret his predictions anyway so that makes him effectively wrong anyway. Exit polls are of course asking for voting preference today, because no person in their right mind would ever answer differently for "who would you vote for today?" versus "who will you vote for on November 4?" so it doesn't matter. The problem is, we don't care about what would happen if the election were to be run today. We care about what will happen when the election actually occurs. What we want is a prediction that takes the polls into account but also factors in all the other information we have that could be used to try and predict the future election (e.g. trying to identify whether changes in a candidates polling numbers are likely to be temporary or reflect a permanent shift, identifying historical trends that can be used to predict future behavior, etc). I realize that predicting the future is much harder than predicting the hypothetical of what would happen today, but predicting the future is the whole point of forecasting. There is value in being able to tease out the "accurate" voter sentiments today from the various polls and accounting for uncertainty and error, but that value is primarily in how we can then turn around and use that information to aid in future forecasting, and how candidates can use that data to gauge the effects of what they're doing. But without an attempt at forecasting the future, this is largely useless to the general population.
      - coliveira 8y ago
        > that's how most people interpret his predictions anyway so that makes him effectively wrong anyway. The only thing these people need is to learn what he's really doing, and then he will be immediately correct again! To me this seems like the simple and obvious solution to this debate.
  - thousandautumns 8y ago
    There are two types of people consuming these projections. I'll call them the sportsbook type and the newspaper reader. The sportsbook consumers want to know what the probabilities are for the actual event, when it occurs (election day). The newspaper consumers want to know what is going on currently and the country's political temperature. 538's forecasts, especially far from election day, are for the newspaper consumer. They provide insight into current political dynamics, and a quantitative look at how people view policy, the economy, where current fundraising is at, etc. in a single concise number that can serve as the basis for further discussion and analysis. They however aren't useful for the sportsbook consumer, because they don't account for the practically innumerable "unknown unknowns" that will occur between now and election day (scandals, terrorist attacks, fluctuations in the economy, flubs on the campaign trail, external activist efforts gaining traction). If they did account for all that uncertainty, then the predictions consistently be near 50-50 for the majority of the race, which not only is still uninteresting for the sportsbook types, but also means you no longer are providing value to the newspaper types. Its important to point out that, over time, the predictions become applicable to both readers. As election day nears, the number of "unknown unknowns" decreases, and a prediction for "what would happen if an election were held today" and "what will happen on election day" converges. So up until the few months and weeks leading up to election day, the predictions should be taken more as commentary, and then perceived as serious indications of what will occur closer to election day.
- whatisthiseven 8y ago
  Nate himself has explicitly addressed that complaint as a huge mistake that Nassim continues to make and refuses to admit: https://twitter.com/NateSilver538/status/1114160511208259584 https://twitter.com/NateSilver538/status/1114160511208259584 "p.s. You continue to cite a paper in which you critique our now-cast (a deliberately volatile hypothetical of what would happen in an election held today) and mistake it for our forecast but because you're such a stubborn idiot you won't admit to this very basic mistake."
- dragonwriter 8y ago
  > The best criticism I can come up with is that 538 probably doesn't do a sufficient job of explaining to its audience that their published probabilities are predictions for what would happen if the event occurred today I don't think that's true. While 538 does such predictions (the “nowcast”), they also predict actual results (the “polls-plus forecast” and the “polls-only forecast”), the default view is a forecast (I think the polls-plus usually, but I vaguely recall a chance to polls-only as the default during the 2016 cycle with concerns that there were reasons to suspect that some of the other signals on the polls-plus model had become less useful.) The “nowcast” is not the the default/headline number, so it would be very odd for a someone to mistakenly fixate on it thinking it was the forward-looking prediction.
  - thousandautumns 8y ago
    This is not correct, or at least it isn't any longer. The current predictions by 538 are the "lite" (polls only), the "classic" (polls + other data driven indicators) and the "deluxe" (polls + other data driven indicators + expert opinion). All of these are predictions based on "what would be the outcome of the election be were it to be held today". They just incorporate different data and complexity in their models. Previously, the reported predictions were the "nowcast", "polls-only", and "polls-plus". But despite their names, I believe those too, whatever 538 may have described them as, were "predictions for outcomes as if the event occurred today". In fact, were they not the case, then Taleb's argument would actually have merit, despite his obtuse attempts to explain it. Essentially, his argument is that, if you are predicting a particular outcome to an event (say, Clinton winning the election) a year from now with a certainty of 90%, and 3 months from now, your prediction is 75%, and 3 months after that is 85%, and then right before the election, is 70%, is evidence of a flawed model (keep in mind, I'm making these numbers up). If your prediction is 90% a year out, it should fluctuate much as time goes on, because 90% is very strong certainty. One may counter that as time goes on, public opinion may change, there are changes in news coverage, and external events like the economy or scandal can make a big impact, and all of that is absolutely true. But the point is that, if in light of new information coming forward, your model predictions repeatedly exhibit volatility, than your model failed to adequately account for the uncertainty as a result of future events, and you were never actually 90% sure in the first place. By necessity, a model predicting a distant event in which lots of unknown factors can cause it to fluctuate before it occurs will either be fairly stable as time goes on (meaning you indeed were correct that there was a high chance of a Clinton win), or should give much more hedged estimates of a win (meaning your probabilities a year out would be much closer to 50-50). Taleb's point is that 538's models are too volatile to actually be accurate when they say there is an X% chance someone wins an election a year, or 6 months from now. However, his argument does not apply in the case where the prediction is that there is an X% chance someone wins an election, were the election held today. Instead he seems to dismiss that that type of prediction is worthless, which is simply wrong and foolish in my opinion. There is lots to be learned about the current political climate from those predictions. So its entirely possible both parties are partially wrong and right. Taleb is correct that 538's models are too volatile to be accurate predictions far out from the election, but it assumes that the model is a prediction of what will happen on election day, rather than what would happen today. He also is throwing out the baby with the bathwater by insisting that that is the only kind of political prediction that has any value. On the other hand, 538 has not been very clear on if their forecasts are predictions of what happens on election day or what happens were election day today. If its the former, then their models have issues and Taleb has at least some validity to his argument. If its the latter, then Taleb's argument is no longer applicable.
- pdubbs90 8y ago
  > their published probabilities are predictions for what would happen if the event occurred today. This is manifestly untrue. For the 2016 election 538 had a "Now-cast" that they very clearly marked as "Who would win the election if it were held today" (see here: https://projects.fivethirtyeight.com/2016-election-forecast/ https://projects.fivethirtyeight.com/2016-election-forecast/) and their other models incorporated the fact that there's time left before the election. Due to people not finding the now-cast valuable they abandoned it for 2018. Nate Silver definitely incorporates uncertainty in the forecast (and talks about it frequently), but it's not modeled as a random walk. Rather they use historical data to look at what polling now has to say about an election however far away the election is. Taleb's criticism that they don't consider a black swan event might be valid, but Silver isn't just predicting the election now.
- gweinberg 8y ago
  But that's just nonsense. If something happened yesterday that has people all hot and bothered today but which nobody will give a shit about anymore by the time the next election rolls around, Silver should say "nobody will give a shit about this anymore by the time next election rolls around." Giving a numerical value of how it would effect the election if it were to happen tomorrow is gibberish on the level of "if my grandmother had wheels she'd be a wagon".
  - perl4ever 8y ago
    If you think someone is offering a choice of a hamburger or a shit sandwich, you can just choose the hamburger and stop complaining about the stuff on the menu you don't like.
- noelsusman 8y ago
  That's just not true. In 2016 they had several versions of the model. One was called the NowCast that very explicitly modeled the probability of winning the election today. They had a separate model that gave the probability of winning on election day, which included whatever their estimate was of uncertainty moving forward (including the probability of game-changing events, which is pretty low).
  - baddox 8y ago
    Fair enough. If I misunderstand their predictions then they definitely need to do a better job explaining them. :)

baddox 8y ago

The best criticism I can come up with is that 538 probably doesn't do a sufficient job of explaining to its audience that their published probabilities are predictions for what would happen if the event occurred today. This seems to be one of Taleb's (and others') criticisms of their predictions: that they don't consider the probability of major unexpected events occurring between now and the event (like an athlete getting injured in the future, or the FBI announcing an investigation of a political candidate). But as I understand it, 538 is explicitly not attempting to consider those things, and is simply publishing probabilities of outcomes if the event were to happen now.

ska 8y ago
I think overall 538 did a pretty good job of communicating a difficult subject to a lay audience, but agree this point could perhaps be communicated better. I don't know of a good way to try and capture these sort of events meaningfully, do you?
eridius 8y ago
That seems like a pretty big difference to gloss over. Why are they publishing probabilities for if the event were to happen today? The event isn't happening today!
- pohl 8y ago
  I know, right? Every time my meteorologist gives me a probability of rain on a day that it doesn't rain I ask the same question.
  - steve19 8y ago
    Your personal meteorologist presumably predicts rain on days in the future, when it could happen, not days in the past. Nate publishes predictions for events that will never occur pr(x wins | election today) when there is no election. It would be like me placing bets on a sports game that is not scheduled to be played.
  - dralley 8y ago
    Meteorologists don't try to predict the weather 2 months from now.
    - pohl 8y ago
      Nate hasn't been trying to predict elections that far out, either, to be fair. He has always said some variant of "if the election were held today" when discussing the model.
- baddox 8y ago
  They don't exactly gloss over it. They have tons of articles explaining it, but of course the average reader is just going to see the probabilities and run with that, and 538 does have some responsibility to actively explain things upfront in their UI, which I don't think they do a good job at. > Why are they publishing probabilities for if the event were to happen today? The event isn't happening today! My guess is that it's extremely out of scope (and probably ultimately infeasible) to try to incorporate these things into their models. Sure, there's a non-zero chance of a candidate being killed by a meteorite, but I think it's fair to ignore that and, if it happens, admit that you didn't even attempt to include the probability of that into your model.
  - dragonwriter 8y ago
    > the average reader is just going to see the probabilities and run with that The putative “average reader” who does so will see the default view, which is a forecast, not the nowcast, which you must actively shift to in order to see. > and 538 does have some responsibility to actively explain things upfront in their UI, Like, on the view selector that you have to use to choose the nowcast instead of the default forecast, where the nowcast is described as “Now-cast / Who would win the election if it were held today”. I think they have that covered.
- coliveira 8y ago
  Because this is what every election poll does. Nobody asks people if they would vote in such person a year from now. They always ask what is your voting preference today. If Nate Silver did something different, then he would be completely wrong.
  - eridius 8y ago
    If Nate Silver took the predictions for "if the event were to happen today" and pretended it was a prediction for the future event, he'd be wrong, but that's how most people interpret his predictions anyway so that makes him effectively wrong anyway. Exit polls are of course asking for voting preference today, because no person in their right mind would ever answer differently for "who would you vote for today?" versus "who will you vote for on November 4?" so it doesn't matter. The problem is, we don't care about what would happen if the election were to be run today. We care about what will happen when the election actually occurs. What we want is a prediction that takes the polls into account but also factors in all the other information we have that could be used to try and predict the future election (e.g. trying to identify whether changes in a candidates polling numbers are likely to be temporary or reflect a permanent shift, identifying historical trends that can be used to predict future behavior, etc). I realize that predicting the future is much harder than predicting the hypothetical of what would happen today, but predicting the future is the whole point of forecasting. There is value in being able to tease out the "accurate" voter sentiments today from the various polls and accounting for uncertainty and error, but that value is primarily in how we can then turn around and use that information to aid in future forecasting, and how candidates can use that data to gauge the effects of what they're doing. But without an attempt at forecasting the future, this is largely useless to the general population.
    - coliveira 8y ago
      > that's how most people interpret his predictions anyway so that makes him effectively wrong anyway. The only thing these people need is to learn what he's really doing, and then he will be immediately correct again! To me this seems like the simple and obvious solution to this debate.
- thousandautumns 8y ago
  There are two types of people consuming these projections. I'll call them the sportsbook type and the newspaper reader. The sportsbook consumers want to know what the probabilities are for the actual event, when it occurs (election day). The newspaper consumers want to know what is going on currently and the country's political temperature. 538's forecasts, especially far from election day, are for the newspaper consumer. They provide insight into current political dynamics, and a quantitative look at how people view policy, the economy, where current fundraising is at, etc. in a single concise number that can serve as the basis for further discussion and analysis. They however aren't useful for the sportsbook consumer, because they don't account for the practically innumerable "unknown unknowns" that will occur between now and election day (scandals, terrorist attacks, fluctuations in the economy, flubs on the campaign trail, external activist efforts gaining traction). If they did account for all that uncertainty, then the predictions consistently be near 50-50 for the majority of the race, which not only is still uninteresting for the sportsbook types, but also means you no longer are providing value to the newspaper types. Its important to point out that, over time, the predictions become applicable to both readers. As election day nears, the number of "unknown unknowns" decreases, and a prediction for "what would happen if an election were held today" and "what will happen on election day" converges. So up until the few months and weeks leading up to election day, the predictions should be taken more as commentary, and then perceived as serious indications of what will occur closer to election day.
whatisthiseven 8y ago
Nate himself has explicitly addressed that complaint as a huge mistake that Nassim continues to make and refuses to admit: https://twitter.com/NateSilver538/status/1114160511208259584 https://twitter.com/NateSilver538/status/1114160511208259584 "p.s. You continue to cite a paper in which you critique our now-cast (a deliberately volatile hypothetical of what would happen in an election held today) and mistake it for our forecast but because you're such a stubborn idiot you won't admit to this very basic mistake."
dragonwriter 8y ago
> The best criticism I can come up with is that 538 probably doesn't do a sufficient job of explaining to its audience that their published probabilities are predictions for what would happen if the event occurred today I don't think that's true. While 538 does such predictions (the “nowcast”), they also predict actual results (the “polls-plus forecast” and the “polls-only forecast”), the default view is a forecast (I think the polls-plus usually, but I vaguely recall a chance to polls-only as the default during the 2016 cycle with concerns that there were reasons to suspect that some of the other signals on the polls-plus model had become less useful.) The “nowcast” is not the the default/headline number, so it would be very odd for a someone to mistakenly fixate on it thinking it was the forward-looking prediction.
- thousandautumns 8y ago
  This is not correct, or at least it isn't any longer. The current predictions by 538 are the "lite" (polls only), the "classic" (polls + other data driven indicators) and the "deluxe" (polls + other data driven indicators + expert opinion). All of these are predictions based on "what would be the outcome of the election be were it to be held today". They just incorporate different data and complexity in their models. Previously, the reported predictions were the "nowcast", "polls-only", and "polls-plus". But despite their names, I believe those too, whatever 538 may have described them as, were "predictions for outcomes as if the event occurred today". In fact, were they not the case, then Taleb's argument would actually have merit, despite his obtuse attempts to explain it. Essentially, his argument is that, if you are predicting a particular outcome to an event (say, Clinton winning the election) a year from now with a certainty of 90%, and 3 months from now, your prediction is 75%, and 3 months after that is 85%, and then right before the election, is 70%, is evidence of a flawed model (keep in mind, I'm making these numbers up). If your prediction is 90% a year out, it should fluctuate much as time goes on, because 90% is very strong certainty. One may counter that as time goes on, public opinion may change, there are changes in news coverage, and external events like the economy or scandal can make a big impact, and all of that is absolutely true. But the point is that, if in light of new information coming forward, your model predictions repeatedly exhibit volatility, than your model failed to adequately account for the uncertainty as a result of future events, and you were never actually 90% sure in the first place. By necessity, a model predicting a distant event in which lots of unknown factors can cause it to fluctuate before it occurs will either be fairly stable as time goes on (meaning you indeed were correct that there was a high chance of a Clinton win), or should give much more hedged estimates of a win (meaning your probabilities a year out would be much closer to 50-50). Taleb's point is that 538's models are too volatile to actually be accurate when they say there is an X% chance someone wins an election a year, or 6 months from now. However, his argument does not apply in the case where the prediction is that there is an X% chance someone wins an election, were the election held today. Instead he seems to dismiss that that type of prediction is worthless, which is simply wrong and foolish in my opinion. There is lots to be learned about the current political climate from those predictions. So its entirely possible both parties are partially wrong and right. Taleb is correct that 538's models are too volatile to be accurate predictions far out from the election, but it assumes that the model is a prediction of what will happen on election day, rather than what would happen today. He also is throwing out the baby with the bathwater by insisting that that is the only kind of political prediction that has any value. On the other hand, 538 has not been very clear on if their forecasts are predictions of what happens on election day or what happens were election day today. If its the former, then their models have issues and Taleb has at least some validity to his argument. If its the latter, then Taleb's argument is no longer applicable.
pdubbs90 8y ago
> their published probabilities are predictions for what would happen if the event occurred today. This is manifestly untrue. For the 2016 election 538 had a "Now-cast" that they very clearly marked as "Who would win the election if it were held today" (see here: https://projects.fivethirtyeight.com/2016-election-forecast/ https://projects.fivethirtyeight.com/2016-election-forecast/) and their other models incorporated the fact that there's time left before the election. Due to people not finding the now-cast valuable they abandoned it for 2018. Nate Silver definitely incorporates uncertainty in the forecast (and talks about it frequently), but it's not modeled as a random walk. Rather they use historical data to look at what polling now has to say about an election however far away the election is. Taleb's criticism that they don't consider a black swan event might be valid, but Silver isn't just predicting the election now.
gweinberg 8y ago
But that's just nonsense. If something happened yesterday that has people all hot and bothered today but which nobody will give a shit about anymore by the time the next election rolls around, Silver should say "nobody will give a shit about this anymore by the time next election rolls around." Giving a numerical value of how it would effect the election if it were to happen tomorrow is gibberish on the level of "if my grandmother had wheels she'd be a wagon".
- perl4ever 8y ago
  If you think someone is offering a choice of a hamburger or a shit sandwich, you can just choose the hamburger and stop complaining about the stuff on the menu you don't like.
noelsusman 8y ago
That's just not true. In 2016 they had several versions of the model. One was called the NowCast that very explicitly modeled the probability of winning the election today. They had a separate model that gave the probability of winning on election day, which included whatever their estimate was of uncertainty moving forward (including the probability of game-changing events, which is pretty low).
- baddox 8y ago
  Fair enough. If I misunderstand their predictions then they definitely need to do a better job explaining them. :)