8 ms·

This article is mischaracterizing Silver's model. The author writes: "if the election were to happen today, what is the probability of each candidate winning?"

This article is mischaracterizing Silver's model. The author writes: "if the election were to happen today, what is the probability of each candidate winning?" This is not at all what Silver's model does. Silver writes and makes clear that there is more variance in the model a few months out than a few days out. (In other words - Biden may be 90% now but four months ago if the polls were identical he would be at less than 90% since there would be more time for voter change and real world events.) I honestly don't even get what Taleb is truly arguing here. He is using a lot of statistical arguments but ultimately Silver is making a model to predict an outcome. It is like a weather forecast. I may say that four months from now the chance it rains in Los Angeles is 2%. However, the day before with more info I may be able to say it will rain 90% of the time. There isn't anything wrong with this, even though Taleb seems to suggest this wrong (from my read)

smallnamespace 6y ago
Taleb is saying that if your rain model four months out shows 2%, 8%, 3%, 9%, then it should have instead shown 5-6% that entire range instead. If your final prediction ends up at 90%, your path there should resemble a sort of random walk. Put another way, the 538 model doesn’t add enough variance when we’re still months out. This shouldn’t be surprising, as Silver himself emphasizes that their primary metric is ‘last minute’ prediction accuracy.
- pochamago 6y ago
  I think what many people miss, and what Silver isn't likely to admit, is that 538 is entertainment. If their prediction were 50% split up until the last month of the campaign it would be boring and nobody would check out his site. Higher variance is desirable to him, and he can justify the whole thing in his post mortems by only measuring last minute accuracy.
coliveira 6y ago
The problems with Silver's results is that it is from a "seat-of-the-pants" model. It is trying to do some form of data mining with the data, but there is little underlying foundations to guarantee that it is even close to reality. Regular polls are more transparent because they (at least the well done) will tell you the methodology and the statistical limitations of the research. You can't get that from what Silver is doing.
- edmundsauto 6y ago
  This is simply untrue. The 538 models are calibrated against a lot of previous real-world outcomes.
repsilat 6y ago
Guessing at what Teleb would say: if it rained on 2% of days, and weather forecasts are only good a week out, you'd expect a reasonable model to have the probability sit around 2% until a week or so before the prediction date, then move up or down, maybe with some noise. If you saw the probability swinging from 20% down to 0.1% a month out you should be suspicious. A 90% chance in the election a long way out is suspicious if we think that the chance effects are large and roughly even-handed. Regardless of assumptions about the distribution of chance events though, multiple large swings are themselves suspicious because they each by definition imply the observation of highly improbable events.
- pwinnski 6y ago
  We've had Trump as President for four years, and it seems like most people have forgotten how incredibly unlikely that seemed in August or even October. Trump himself was laying the groundwork for launching a TV network until things changed quite dramatically at the end of October with Jim Comey and some other factors. Taleb seems to be arguing that 90% confidence was crazy given what eventually happened, with the benefit of hindsight, but that does seem to ignore just how crazy the idea of a Trump presidency seemed before it happened.
  - edmundsauto 6y ago
    In the scope of this thread - Silver v. Taleb - there's no need to mention the layperson's memory of their internal prediction. We know that 538 had a 30% chance of a Trump victory, which is the relevant piece of information - not our perception.
    - pwinnski 6y ago
      I disagree. Taleb's key assertion seems to be that Silver should never have given Clinton a 90% chance of winning with the level of uncertainty in that race. But there were times when Trump having only a 10% of winning seemed completely justified, and I think to say otherwise at this point seems like projecting backward with hindsight.
smeeth 6y ago
Minor addendum, Silver altered his model to become more certain as the election drew near after 2016. In effect, it changed from a “what if the election were today?” Model to a “what will the outcome of the election on date ___ be?” Model just for this cycle. I can’t say why for certain, but it definitely could be that Taleb’s 2018 criticisms played a part.
- kadoban 6y ago
  That's incorrect. At _least_ as far back as 2016 and I believe quite a bit farther back the model was not "what if the election were today?" There even used to be a separate side-model that was explicitly that, the "nowcast" I think it was called. I suspect that 538 _never_ covered an election solely or even primarily with a "what if the election were today?" model, but my memory isn't that reliable back that far.
  - smeeth 6y ago
    You are correct, thank you. I updated another comment of mine to reflect this. Comparing them now though, Silver definitely changed the 2020 model to increase variance as a function of time to the election, which was Nassim's complaint in the first place.
- dragonwriter 6y ago
  > Minor addendum, Silver altered his model to become more certain as the election drew near after 2016. In effect, it changed from a “what if the election were today?” Model to a “what will the outcome of the election on date ___ be?” Model just for this cycle. False. Silver's headline model has always been a “what will the outcome on election day” model. (Silver has a secondary “what of the election were held today model”, as well, which was published as an explicitly alternative model in previous cycles, with explicit notes about the difference in what it did from the main model, and is still run and occasionally referenced but not published.)
vincentmarle 6y ago
> I honestly don't even get what Taleb is truly arguing here. He is using a lot of statistical arguments but ultimately Silver is making a model to predict an outcome. It is like a weather forecast. "Making models to predict" is precisely the fluff that Taleb is against. There is no "if elections were held today", there are no do-overs, there is only one election. And you either win or lose (binary price [1]). So any prediction that doesn't incorporate the events between making the prediction until the election actually occurs, is worthless, or to be more precise: 50/50 (arbitrage free). The probability only goes up when you start counting the actual votes on Election Day. [1] https://arxiv.org/pdf/1703.06351.pdf https://arxiv.org/pdf/1703.06351.pdf
- kelnos 6y ago
  If it's 50/50 up until election day, then it's pointless to try to predict. And maybe that's Taleb's point. But is there value in a prediction that does try to predict using polling data and other information? If so, then regardless of whether or not 548 "does it right" by Taleb's definition, it's valuable, at least to some people.
Tomminn 6y ago
"if the election were to happen today, what is the probability of each candidate winning?" This was a correct characterization of Nate Silver's model at the time Nassim Taleb made his critique. Since then, Silver has changed what the probabilities in his prediction model mean. This change was (briefly) discussed previously on HN [1]. For the record, I think Taleb was an asshole, but an asshole with an entirely valid point in this debate. And I think Silver reluctantly and quietly took it on board, and then went on to use bolt-on fudge factors so that the time-dependent predictions of his model roughly look like the time-dependent predictions generated by the rigorously defined approach (option price theory) that Taleb was calling for. [1] https://news.ycombinator.com/item?id=24164814 https://news.ycombinator.com/item?id=24164814
- ashtonkem 6y ago
  This was a correct description of one of 538’s models, the nowcast. It was never the one they recommended, and they did not bring it back later because it tended to confuse people. They actually made it very clear on their podcasts that the nowcast was a mistake.
  - Tomminn 6y ago
    Ahh, that's even worse for Nate Silver then, if the "polls-plus" and the "polls only" predictions were not assuming the election happened today (which it seems like they weren't, even in 2016 [1]), then Taleb's criticism is all the more accurate. Whatever Nate Silver was doing to incorporate future uncertainty in 2016 was clearly inadequate for all the reasons that Taleb pointed out (you could reliably make money betting on Silver's prediction odds to tend toward their time-smoothed average, implying that their time-smoothed average was a better representation of the probability than the probability itself). [1] https://web.archive.org/web/20160930201714/http://fivethirtyeight.com/features/a-users-guide-to-fivethirtyeights-2016-general-election-forecast/ https://web.archive.org/web/20160930201714/http://fivethirty...
    - dragonwriter 6y ago
      Taleb's criticism is inaccurate in any case; it would be a valid criticism of the model if, in a large number of election cycles, the 2016 shape was typical. Because what you'd expect a forecast model to do typically is to get gradually more certain over time, so if it consistently behaved the way the 2016 forecast did, that would be a problem. But, you'd also expect, in a large number of cycles, to have some where you get patterns where you do start getting runa of information that contradict the impact of prior information, and where you do have the kind of reverses seen in the 2016 cycle. And, while we don't have enough Presidential cycles to make firm descriptions of what a “typical” cycle would look like with Silver's model, we do know that both 2012 and 2020 look a lot more like you'd expect a typical cycle in a forecast model to look.
asdfasgasdgasdg 6y ago
The point of premise 2 is that an 80% probability in late June implies a very high level of confidence. Confidence that doesn't seem intuitively reasonable. I would say that few rational individuals would have taken a bet with 4 pays 1 odds that Biden would win on July 1. It would not be a very safe bet, because of the likelihood of unknown random events. Therefore, what Nate's model was predicting in July probably wasn't the probability of Biden winning on November 3rd. That doesn't mean the model's not useful, but if it is trying to predict what you think it is, then it's overconfident. As a non-huge fan of Taleb, and as actually a big fan of Silver, I think Taleb is right here.
Tomminn 6y ago
Taleb's criticism was that the chance it rains four months from now shouldn't reliably vary about a mean (say from 30% to 70% to 30% to 70% etc) in the four months leading up to that day. Because if it does, I can make risk-free money by betting it won't rain on days the payout for not-rain is high, and betting it will rain on days the payout for rain is high. If a time-smoothed average of your predictions is a better prediction of the outcome than your current prediction, then your current prediction is wrong. The idea that someone can improve on your current prediction by averaging your last 10 predictions is a sign that your methodology for generating "current predictions" is deeply suboptimal.

smallnamespace 6y ago

Taleb is saying that if your rain model four months out shows 2%, 8%, 3%, 9%, then it should have instead shown 5-6% that entire range instead. If your final prediction ends up at 90%, your path there should resemble a sort of random walk. Put another way, the 538 model doesn’t add enough variance when we’re still months out. This shouldn’t be surprising, as Silver himself emphasizes that their primary metric is ‘last minute’ prediction accuracy.

pochamago 6y ago
I think what many people miss, and what Silver isn't likely to admit, is that 538 is entertainment. If their prediction were 50% split up until the last month of the campaign it would be boring and nobody would check out his site. Higher variance is desirable to him, and he can justify the whole thing in his post mortems by only measuring last minute accuracy.

coliveira 6y ago

The problems with Silver's results is that it is from a "seat-of-the-pants" model. It is trying to do some form of data mining with the data, but there is little underlying foundations to guarantee that it is even close to reality. Regular polls are more transparent because they (at least the well done) will tell you the methodology and the statistical limitations of the research. You can't get that from what Silver is doing.

edmundsauto 6y ago
This is simply untrue. The 538 models are calibrated against a lot of previous real-world outcomes.

repsilat 6y ago

Guessing at what Teleb would say: if it rained on 2% of days, and weather forecasts are only good a week out, you'd expect a reasonable model to have the probability sit around 2% until a week or so before the prediction date, then move up or down, maybe with some noise. If you saw the probability swinging from 20% down to 0.1% a month out you should be suspicious. A 90% chance in the election a long way out is suspicious if we think that the chance effects are large and roughly even-handed. Regardless of assumptions about the distribution of chance events though, multiple large swings are themselves suspicious because they each by definition imply the observation of highly improbable events.

pwinnski 6y ago
We've had Trump as President for four years, and it seems like most people have forgotten how incredibly unlikely that seemed in August or even October. Trump himself was laying the groundwork for launching a TV network until things changed quite dramatically at the end of October with Jim Comey and some other factors. Taleb seems to be arguing that 90% confidence was crazy given what eventually happened, with the benefit of hindsight, but that does seem to ignore just how crazy the idea of a Trump presidency seemed before it happened.
- edmundsauto 6y ago
  In the scope of this thread - Silver v. Taleb - there's no need to mention the layperson's memory of their internal prediction. We know that 538 had a 30% chance of a Trump victory, which is the relevant piece of information - not our perception.
  - pwinnski 6y ago
    I disagree. Taleb's key assertion seems to be that Silver should never have given Clinton a 90% chance of winning with the level of uncertainty in that race. But there were times when Trump having only a 10% of winning seemed completely justified, and I think to say otherwise at this point seems like projecting backward with hindsight.

smeeth 6y ago

Minor addendum, Silver altered his model to become more certain as the election drew near after 2016. In effect, it changed from a “what if the election were today?” Model to a “what will the outcome of the election on date ___ be?” Model just for this cycle. I can’t say why for certain, but it definitely could be that Taleb’s 2018 criticisms played a part.

kadoban 6y ago
That's incorrect. At _least_ as far back as 2016 and I believe quite a bit farther back the model was not "what if the election were today?" There even used to be a separate side-model that was explicitly that, the "nowcast" I think it was called. I suspect that 538 _never_ covered an election solely or even primarily with a "what if the election were today?" model, but my memory isn't that reliable back that far.
- smeeth 6y ago
  You are correct, thank you. I updated another comment of mine to reflect this. Comparing them now though, Silver definitely changed the 2020 model to increase variance as a function of time to the election, which was Nassim's complaint in the first place.
dragonwriter 6y ago
> Minor addendum, Silver altered his model to become more certain as the election drew near after 2016. In effect, it changed from a “what if the election were today?” Model to a “what will the outcome of the election on date ___ be?” Model just for this cycle. False. Silver's headline model has always been a “what will the outcome on election day” model. (Silver has a secondary “what of the election were held today model”, as well, which was published as an explicitly alternative model in previous cycles, with explicit notes about the difference in what it did from the main model, and is still run and occasionally referenced but not published.)

vincentmarle 6y ago

> I honestly don't even get what Taleb is truly arguing here. He is using a lot of statistical arguments but ultimately Silver is making a model to predict an outcome. It is like a weather forecast. "Making models to predict" is precisely the fluff that Taleb is against. There is no "if elections were held today", there are no do-overs, there is only one election. And you either win or lose (binary price [1]). So any prediction that doesn't incorporate the events between making the prediction until the election actually occurs, is worthless, or to be more precise: 50/50 (arbitrage free). The probability only goes up when you start counting the actual votes on Election Day. [1] https://arxiv.org/pdf/1703.06351.pdf https://arxiv.org/pdf/1703.06351.pdf

kelnos 6y ago
If it's 50/50 up until election day, then it's pointless to try to predict. And maybe that's Taleb's point. But is there value in a prediction that does try to predict using polling data and other information? If so, then regardless of whether or not 548 "does it right" by Taleb's definition, it's valuable, at least to some people.

Tomminn 6y ago

"if the election were to happen today, what is the probability of each candidate winning?" This was a correct characterization of Nate Silver's model at the time Nassim Taleb made his critique. Since then, Silver has changed what the probabilities in his prediction model mean. This change was (briefly) discussed previously on HN [1]. For the record, I think Taleb was an asshole, but an asshole with an entirely valid point in this debate. And I think Silver reluctantly and quietly took it on board, and then went on to use bolt-on fudge factors so that the time-dependent predictions of his model roughly look like the time-dependent predictions generated by the rigorously defined approach (option price theory) that Taleb was calling for. [1] https://news.ycombinator.com/item?id=24164814 https://news.ycombinator.com/item?id=24164814

ashtonkem 6y ago
This was a correct description of one of 538’s models, the nowcast. It was never the one they recommended, and they did not bring it back later because it tended to confuse people. They actually made it very clear on their podcasts that the nowcast was a mistake.
- Tomminn 6y ago
  Ahh, that's even worse for Nate Silver then, if the "polls-plus" and the "polls only" predictions were not assuming the election happened today (which it seems like they weren't, even in 2016 [1]), then Taleb's criticism is all the more accurate. Whatever Nate Silver was doing to incorporate future uncertainty in 2016 was clearly inadequate for all the reasons that Taleb pointed out (you could reliably make money betting on Silver's prediction odds to tend toward their time-smoothed average, implying that their time-smoothed average was a better representation of the probability than the probability itself). [1] https://web.archive.org/web/20160930201714/http://fivethirtyeight.com/features/a-users-guide-to-fivethirtyeights-2016-general-election-forecast/ https://web.archive.org/web/20160930201714/http://fivethirty...
  - dragonwriter 6y ago
    Taleb's criticism is inaccurate in any case; it would be a valid criticism of the model if, in a large number of election cycles, the 2016 shape was typical. Because what you'd expect a forecast model to do typically is to get gradually more certain over time, so if it consistently behaved the way the 2016 forecast did, that would be a problem. But, you'd also expect, in a large number of cycles, to have some where you get patterns where you do start getting runa of information that contradict the impact of prior information, and where you do have the kind of reverses seen in the 2016 cycle. And, while we don't have enough Presidential cycles to make firm descriptions of what a “typical” cycle would look like with Silver's model, we do know that both 2012 and 2020 look a lot more like you'd expect a typical cycle in a forecast model to look.

asdfasgasdgasdg 6y ago

The point of premise 2 is that an 80% probability in late June implies a very high level of confidence. Confidence that doesn't seem intuitively reasonable. I would say that few rational individuals would have taken a bet with 4 pays 1 odds that Biden would win on July 1. It would not be a very safe bet, because of the likelihood of unknown random events. Therefore, what Nate's model was predicting in July probably wasn't the probability of Biden winning on November 3rd. That doesn't mean the model's not useful, but if it is trying to predict what you think it is, then it's overconfident. As a non-huge fan of Taleb, and as actually a big fan of Silver, I think Taleb is right here.

Tomminn 6y ago

Taleb's criticism was that the chance it rains four months from now shouldn't reliably vary about a mean (say from 30% to 70% to 30% to 70% etc) in the four months leading up to that day. Because if it does, I can make risk-free money by betting it won't rain on days the payout for not-rain is high, and betting it will rain on days the payout for rain is high. If a time-smoothed average of your predictions is a better prediction of the outcome than your current prediction, then your current prediction is wrong. The idea that someone can improve on your current prediction by averaging your last 10 predictions is a sign that your methodology for generating "current predictions" is deeply suboptimal.