14 ms·

Nassim Taleb vs. Nate Silver: who is right about election forecasting?

93 points by probe 6y ago

handmodel 6y ago
This article is mischaracterizing Silver's model. The author writes: "if the election were to happen today, what is the probability of each candidate winning?" This is not at all what Silver's model does. Silver writes and makes clear that there is more variance in the model a few months out than a few days out. (In other words - Biden may be 90% now but four months ago if the polls were identical he would be at less than 90% since there would be more time for voter change and real world events.) I honestly don't even get what Taleb is truly arguing here. He is using a lot of statistical arguments but ultimately Silver is making a model to predict an outcome. It is like a weather forecast. I may say that four months from now the chance it rains in Los Angeles is 2%. However, the day before with more info I may be able to say it will rain 90% of the time. There isn't anything wrong with this, even though Taleb seems to suggest this wrong (from my read)
- smallnamespace 6y ago
  Taleb is saying that if your rain model four months out shows 2%, 8%, 3%, 9%, then it should have instead shown 5-6% that entire range instead. If your final prediction ends up at 90%, your path there should resemble a sort of random walk. Put another way, the 538 model doesn’t add enough variance when we’re still months out. This shouldn’t be surprising, as Silver himself emphasizes that their primary metric is ‘last minute’ prediction accuracy.
  - pochamago 6y ago
    I think what many people miss, and what Silver isn't likely to admit, is that 538 is entertainment. If their prediction were 50% split up until the last month of the campaign it would be boring and nobody would check out his site. Higher variance is desirable to him, and he can justify the whole thing in his post mortems by only measuring last minute accuracy.
- coliveira 6y ago
  The problems with Silver's results is that it is from a "seat-of-the-pants" model. It is trying to do some form of data mining with the data, but there is little underlying foundations to guarantee that it is even close to reality. Regular polls are more transparent because they (at least the well done) will tell you the methodology and the statistical limitations of the research. You can't get that from what Silver is doing.
  - edmundsauto 6y ago
    This is simply untrue. The 538 models are calibrated against a lot of previous real-world outcomes.
- repsilat 6y ago
  Guessing at what Teleb would say: if it rained on 2% of days, and weather forecasts are only good a week out, you'd expect a reasonable model to have the probability sit around 2% until a week or so before the prediction date, then move up or down, maybe with some noise. If you saw the probability swinging from 20% down to 0.1% a month out you should be suspicious. A 90% chance in the election a long way out is suspicious if we think that the chance effects are large and roughly even-handed. Regardless of assumptions about the distribution of chance events though, multiple large swings are themselves suspicious because they each by definition imply the observation of highly improbable events.
  - pwinnski 6y ago
    We've had Trump as President for four years, and it seems like most people have forgotten how incredibly unlikely that seemed in August or even October. Trump himself was laying the groundwork for launching a TV network until things changed quite dramatically at the end of October with Jim Comey and some other factors. Taleb seems to be arguing that 90% confidence was crazy given what eventually happened, with the benefit of hindsight, but that does seem to ignore just how crazy the idea of a Trump presidency seemed before it happened.
    - edmundsauto 6y ago
      In the scope of this thread - Silver v. Taleb - there's no need to mention the layperson's memory of their internal prediction. We know that 538 had a 30% chance of a Trump victory, which is the relevant piece of information - not our perception.
      - pwinnski 6y ago
        I disagree. Taleb's key assertion seems to be that Silver should never have given Clinton a 90% chance of winning with the level of uncertainty in that race. But there were times when Trump having only a 10% of winning seemed completely justified, and I think to say otherwise at this point seems like projecting backward with hindsight.
- smeeth 6y ago
  Minor addendum, Silver altered his model to become more certain as the election drew near after 2016. In effect, it changed from a “what if the election were today?” Model to a “what will the outcome of the election on date ___ be?” Model just for this cycle. I can’t say why for certain, but it definitely could be that Taleb’s 2018 criticisms played a part.
  - kadoban 6y ago
    That's incorrect. At _least_ as far back as 2016 and I believe quite a bit farther back the model was not "what if the election were today?" There even used to be a separate side-model that was explicitly that, the "nowcast" I think it was called. I suspect that 538 _never_ covered an election solely or even primarily with a "what if the election were today?" model, but my memory isn't that reliable back that far.
    - smeeth 6y ago
      You are correct, thank you. I updated another comment of mine to reflect this. Comparing them now though, Silver definitely changed the 2020 model to increase variance as a function of time to the election, which was Nassim's complaint in the first place.
  - dragonwriter 6y ago
    > Minor addendum, Silver altered his model to become more certain as the election drew near after 2016. In effect, it changed from a “what if the election were today?” Model to a “what will the outcome of the election on date ___ be?” Model just for this cycle. False. Silver's headline model has always been a “what will the outcome on election day” model. (Silver has a secondary “what of the election were held today model”, as well, which was published as an explicitly alternative model in previous cycles, with explicit notes about the difference in what it did from the main model, and is still run and occasionally referenced but not published.)
- vincentmarle 6y ago
  > I honestly don't even get what Taleb is truly arguing here. He is using a lot of statistical arguments but ultimately Silver is making a model to predict an outcome. It is like a weather forecast. "Making models to predict" is precisely the fluff that Taleb is against. There is no "if elections were held today", there are no do-overs, there is only one election. And you either win or lose (binary price [1]). So any prediction that doesn't incorporate the events between making the prediction until the election actually occurs, is worthless, or to be more precise: 50/50 (arbitrage free). The probability only goes up when you start counting the actual votes on Election Day. [1] https://arxiv.org/pdf/1703.06351.pdf https://arxiv.org/pdf/1703.06351.pdf
  - kelnos 6y ago
    If it's 50/50 up until election day, then it's pointless to try to predict. And maybe that's Taleb's point. But is there value in a prediction that does try to predict using polling data and other information? If so, then regardless of whether or not 548 "does it right" by Taleb's definition, it's valuable, at least to some people.
- Tomminn 6y ago
  "if the election were to happen today, what is the probability of each candidate winning?" This was a correct characterization of Nate Silver's model at the time Nassim Taleb made his critique. Since then, Silver has changed what the probabilities in his prediction model mean. This change was (briefly) discussed previously on HN [1]. For the record, I think Taleb was an asshole, but an asshole with an entirely valid point in this debate. And I think Silver reluctantly and quietly took it on board, and then went on to use bolt-on fudge factors so that the time-dependent predictions of his model roughly look like the time-dependent predictions generated by the rigorously defined approach (option price theory) that Taleb was calling for. [1] https://news.ycombinator.com/item?id=24164814 https://news.ycombinator.com/item?id=24164814
  - ashtonkem 6y ago
    This was a correct description of one of 538’s models, the nowcast. It was never the one they recommended, and they did not bring it back later because it tended to confuse people. They actually made it very clear on their podcasts that the nowcast was a mistake.
    - Tomminn 6y ago
      Ahh, that's even worse for Nate Silver then, if the "polls-plus" and the "polls only" predictions were not assuming the election happened today (which it seems like they weren't, even in 2016 [1]), then Taleb's criticism is all the more accurate. Whatever Nate Silver was doing to incorporate future uncertainty in 2016 was clearly inadequate for all the reasons that Taleb pointed out (you could reliably make money betting on Silver's prediction odds to tend toward their time-smoothed average, implying that their time-smoothed average was a better representation of the probability than the probability itself). [1] https://web.archive.org/web/20160930201714/http://fivethirtyeight.com/features/a-users-guide-to-fivethirtyeights-2016-general-election-forecast/ https://web.archive.org/web/20160930201714/http://fivethirty...
      - dragonwriter 6y ago
        Taleb's criticism is inaccurate in any case; it would be a valid criticism of the model if, in a large number of election cycles, the 2016 shape was typical. Because what you'd expect a forecast model to do typically is to get gradually more certain over time, so if it consistently behaved the way the 2016 forecast did, that would be a problem. But, you'd also expect, in a large number of cycles, to have some where you get patterns where you do start getting runa of information that contradict the impact of prior information, and where you do have the kind of reverses seen in the 2016 cycle. And, while we don't have enough Presidential cycles to make firm descriptions of what a “typical” cycle would look like with Silver's model, we do know that both 2012 and 2020 look a lot more like you'd expect a typical cycle in a forecast model to look.
- asdfasgasdgasdg 6y ago
  The point of premise 2 is that an 80% probability in late June implies a very high level of confidence. Confidence that doesn't seem intuitively reasonable. I would say that few rational individuals would have taken a bet with 4 pays 1 odds that Biden would win on July 1. It would not be a very safe bet, because of the likelihood of unknown random events. Therefore, what Nate's model was predicting in July probably wasn't the probability of Biden winning on November 3rd. That doesn't mean the model's not useful, but if it is trying to predict what you think it is, then it's overconfident. As a non-huge fan of Taleb, and as actually a big fan of Silver, I think Taleb is right here.
- Tomminn 6y ago
  Taleb's criticism was that the chance it rains four months from now shouldn't reliably vary about a mean (say from 30% to 70% to 30% to 70% etc) in the four months leading up to that day. Because if it does, I can make risk-free money by betting it won't rain on days the payout for not-rain is high, and betting it will rain on days the payout for rain is high. If a time-smoothed average of your predictions is a better prediction of the outcome than your current prediction, then your current prediction is wrong. The idea that someone can improve on your current prediction by averaging your last 10 predictions is a sign that your methodology for generating "current predictions" is deeply suboptimal.
deleted 6y ago
[deleted]
FandangoRanger 6y ago
It's impossible to believe any forecast, nobody fessed up after 2016 nor did they explain why their models are wrong or attempt to update them. You're less than 24 hours away from the permanent death of the political polling and forecasting industry in the USA. Nate's already coping really hard on Twitter.
- ponker 6y ago
  Well pre-2016 there was the assumption that the American electorate basically is capable of self-governing. This assumption has been invalidated so while the polling industry hasn’t gone away it’s now more like the sports forecasting industry, where people are speculating on an essentially theatrical performance, than something substantively driving good governance.
- jjeaff 6y ago
  That's not true, there have been countless articles about why polling went wrong last time. Also, polling wasn't that wrong. Due to sampling bias, several swing states were off by as much as 5-6 percentage points. But national polling bad Hillary winning by something like 4% in the popular vote. Instead, she won the popular vote by 3%.
  - JumpCrisscross 6y ago
    > several swing states were off by as much as 5-6 percentage points Most state polls have margins of error ranging from 3 to 6 points. A 3-point MoE state poll is a really expensive poll.
    - timmytokyo 6y ago
      A properly conducted poll with 1000 respondents has a margin of error of 3.2%. There have been quite a few of these in battleground states this cycle.
- gdulli 6y ago
  Believing that occurrence of an event with a forecasted 30% chance (or 2% chance) of occurring invalidates a model is so fundamentally a misunderstanding of the topic.
MR4D 6y ago
FTA, “ I believe Nate Silver is answering a subtly different question with his election forecasts. Each data point that Silver produces is answering the question: if the election were to happen today, what is the probability of each candidate winning?” If that’s true, then Nate Silver just does poll analysis, not predictions. Predictions should be on the day of the election, not the day you’re reading their blog post. In other words, Taleb is right.
- throwawaygh 6y ago
  It's not true.
  - MR4D 6y ago
    What’s not true?
    - cyphar 6y ago
      The statement you quoted from the article (which is basically Taleb's criticism of Nate Silver) is not correct, the FiveThirtyEight prediction models are explicitly not answering the question "if the election was held today, what would happen". There are models like that on FiveThirtyEight, but the main question being answered is who is likely to win the actual election on the day it is held.
      - ashtonkem 6y ago
        There were models like that, they didn’t bring them back this year.
        deleted 6y ago
        [deleted]
        dragonwriter 6y ago
        > There were models like that, they didn’t bring them back this year. They weren't the headline forecast, but, yes, the NowCast existed (and still does, in fact, though it's not visible, just like the polls-only forecast model isn't visible any more; Silver references them periodically.) But it's inaccurate to criticize the headline forecast based on the idea that it is trying to do what the NowCast does, and it is dishonest to criticize something other than the main forecast and say it is a criticism of the forecast, but not specify the alternative forecast you are actually criticizing.
throwawaygh 6y ago
> the election were to happen today, what is the probability of each candidate winning? He's explicitly not doing this. Here's how I think about it. Silver is answering the question: "how much would the polls (as an aggregate) need to differ from the final result in order for Candidate X to win/lose, conditioned on some reasonable priors?" Taleb is pointing out that the polls could be really wrong in all sorts of ways that are impossible to predict a priori. The whole argument is sort of pointless from an intellectual/academic perspective. It's a war of public personalities more than anything else. It's both the case that Silver designed a good piece of software that does what it's supposed to do and also the case that Taleb's skepticism is valid. But then, that sort of skepticism of statistical models is always valid, and yet we use these models to great effect in all sorts of settings.
- coliveira 6y ago
  Taleb made a career from stating the obvious and making it appear as a smart discovery. I've read some of his writings and have never found anything really new.
  - seibelj 6y ago
    Totally agree, he comes off as a hack to me. But he’s famous and I’m not, so shrug
    - bitdizzy 6y ago
      Taleb is seriously proficient at probability which can be judged from his technical publications.
  - sillysaurusx 6y ago
    The same is true of pg, but his discoveries seem obvious only in hindsight. (I’m objecting to “the discoveries are obvious” as a putdown. The simplest discoveries are some of the hardest. Though I doubt this matters for political pundit analysis.)
    - 6y ago
egonschiele 6y ago
Incredibly click-baity title and incendiary first paragraph were hard to get past.
- deleted 6y ago
  [deleted]
fortenforge 6y ago
> I believe Nate Silver is answering a subtly different question with his election forecasts. Each data point that Silver produces is answering the question: if the election were to happen today, what is the probability of each candidate winning? As others have pointed out, this is definitely not what Nate's model was doing. In fact, he did have a version of the model that did exactly this, the so called "Now-Cast" and as expected it was even more volatile than the real model: https://projects.fivethirtyeight.com/2016-election-forecast/#now https://projects.fivethirtyeight.com/2016-election-forecast/...
- dragonwriter 6y ago
  Right, the main model intends, in fact, to be predicting the probability of various outcomes when the election is actually held, that's why there is accounting for non-poll fundamentals (economy and other things) that are observed to influence poll changes over time that phase out as time runs out for them to move the polls.
travisgriggs 6y ago
So... Taleb is a like purist programmer (pick your favorite paradigm) and Silver is more like a pragmatist programmer that uses a "hybrid" language model? There's values for both types in programmerdom, is there not the same for both in the world of statisticians?
stopachka 6y ago
“if I tell you an event has a 0% chance of occurring, I cannot change my mind and tell you tomorrow it now has a 50% chance of occurring. Otherwise I shouldn’t have told you it has a 0% chance in the first place” I think this is the key to Taleb’s argument, and it’s pretty damning for Mr Siver. If there’s high uncertainty, you can’t say whether someone’s probability to win is low. The uncertainty widens the probability. For example, Say I told you there’s a 2% chance it will rain Monday next year in New York - This would _have_ to be a bogus prediction. So much could happen until then. I have _high_ uncertainty. There’s no way that uncertainty is accounted for with 2%
- ashtonkem 6y ago
  Yeah, except both of them are modeling how people will vote, and people change their minds. Saying that one should’ve held a constant chance of a candidate winning no matter what the polls do is frankly insane. This stance basically requires that you pretend that the outcome is determined and that voters opinions either don’t matter or don’t change. It’s utter nonsense, especially since we know that in 2016 undecided voters broke late for Trump. If your model doesn’t react to that, what are you even modeling?
  - stopachka 6y ago
    Taleb’s point is not that your prediction should not change. His point is that if it does change, and frequently at that, the purported 90% chance at t(n) is bogus. That prediction needed to account for the volatility
- cortesoft 6y ago
  The prediction should have more data to it about variance, but it isn't necessarily wrong. We have good data about rain, I am sure we could find out what percentage of the time it rains in New York on a given day during that time of the year. There might be high variance, but that doesn't mean it is wrong to say there is a 2% chance it rains on a particular Monday. This is how insurance works, how sports betting works, how option pricing works. Just because something has high variance doesn't mean we can't give probabilities that work out over a large number of samples.
ramraj07 6y ago
Your argument doesn't hold water at all. Today's blurb in the forecast page - "First, the forecast is now totally polls-based; that is, any advantage our forecast gave Trump for the economy or for being an incumbent is no longer factored in. Second, the uncertainty around how much the polls will change between today and Election Day is also no longer an issue." - that means Nate Silver is explicitly saying his forecast is always trying to predict what is going to happen on November 3. Heres an alternative hypothesis that involves no fancy math just anthropology - Taleb is right that the stats don't make sense, but Nate is also not an idiot - it's just in his financial best interests to make a website that can throw out confidence inducing pseudonumbers that will make people like us visit it every few hours for months at a time every two years. Perhaps he knows perfectly well the farce he's pulling (at least at a fundamental level though he might have convinced himself otherwise as academics always do) but just plays the game to make sure he's relevant. And thus, we are all the idiots.
rramach 6y ago
Nice summary of the Nassim-Nate debate. A formal analysis/discussion of election forecasting is in this paper: http://www.stat.columbia.edu/~gelman/research/published/jdm200907b.pdf http://www.stat.columbia.edu/~gelman/research/published/jdm2... Edward says that Nate's prediction should be interpreted as: “If nothing else changes between now and the election, Joe Biden has a 85% chance of winning.” (Silver’s argument) But the problem is this prediction is not testable as election only occurs once at the end. Thus, one should use Nate's early predictions as pure entertainment. Nate's final predictions show that they are well-calibrated across all of his political predictions but it is hard to estimate how accurate his model predictions are just for presidential elections given he has predicted only a few so far (unless one assumes all elections have similar uncertainty, which is clearly not true).
baron_harkonnen 6y ago
I've recently come to believe that Taleb really doesn't understand probability. I know this sounds like some insane flippant remark, but I saw a video where he talked about the election forecast as a martingale bounded between 0 and 1 and then made some claims about variance with these bounds to back up his claim against Silver. For his claim to make sense you would have to view probabilities as linear, which is not simply not how probabilities work. An event that has a 0.01 chance of happening is an order of magnitude more likely than one that has as 0.001 chance. However the different between and 0.5 and 0.501 is essentially negligible (hence probability is not linear because 0.001 does not mean the same everywhere). I've generally given Taleb a pass as a bit of a crank that doesn't care about the philosophical interpretation of probability, but after seeing him misunderstand probability in such a major way I've started writing him off entirely. edit: I should add I'm not particularly more convinced that Silver really understands all that much either, just that this is all two very loud people that don't know much. Neither should be taken seriously. This isn't Bradly Efron arguing with Andrew Gelman.
- inimino 6y ago
  Based on what I've read of Taleb, if you think he doesn't understand some patently obvious facts about probability, it's going to take more than saying that you watched a video and didn't think it made sense for me to give your claim any weight.
- amw-zero 6y ago
  There is a very clear flaw in your mathematical reasoning here, let's walk through it. Consider your second probability comparison: the difference between a probability of 0.500 and 0.501. You have this correct, the difference is 0.001. Now reconsider your first statement: that a probability of 0.010 is an order of magnitude more likely than a probability of 0.001. This is true, but is completely irrelevant because it is a completely different comparison. A more appropriate comparison would be to compare probabilities of 0.010 and 0.011. The difference of these is again 0.001. You compared proverbial apples to oranges, and you cannot trust the conclusions you make from invalid comparisons. A difference in probability of 0.001 does mean the same thing no matter what the starting probability is. Probability is "linear" in this way. Now that we know this to be true, I would encourage you to rethink your position on Nassim Taleb. He understands non-linearity quite well, in fact I would argue that his entire purpose on Earth (at least in his eyes) is to remind people about non-linearity as a concept. It comes across in all of his books and most of his public appearances. On the matter of probability, I trust him very deeply. Whether or not I trust him is also inconsequential, because he has serious mathematical literacy and you can evaluate his propositions for yourself.
smeeth 6y ago
Edit: as pointed out in the replies, I was wrong that the model was "explicitly" a now cast, and I would like future readers of my comment to know that without deleting things. I would still like to note that there was clearly a change in the 2020 model to increase variance as a function of time to the election when compared to the 2016 model. I think this is substantively the same. Some of the top comments are making the same mistake, probably because they aren't aware of this twitter beef history (and I envy them), so I'm putting this in a separate comment. These comments are all pointing out that Taleb's criticism is of something that Silver's models don't do: > the election were to happen today, what is the probability of each candidate winning? What people seem to be forgetting, is that Silver's 2016 model explicitly did this. Taleb made his criticisms starting in 2018 (even published an options pricing paper on the subject) then Silver changed his model for the 2020 cycle so that it no longer was an "if the election were today" model. I can't say for sure whether or not Silver changed his model because of Taleb, buuuuut he deserves at least a little credit here. Yes, I know Taleb is a colossal asshole people love to hate, but he's usually right about these sorts of things. Here is a pretty fair and balanced take on it for those who would like to read more: https://towardsdatascience.com/why-you-should-care-about-the-nate-silver-vs-nassim-taleb-twitter-war-a581dce1f5fc https://towardsdatascience.com/why-you-should-care-about-the...
- moultano 6y ago
  This isn't true. He had both a "nowcast" (if the election were held today) and a prediction of the final result. The only change in that regard between 2016 and now was to stop publishing the nowcast. Edit: Even after your edit I don't think the claim is true. The probabilities in this election didn't change much because the campaign season was comparatively uneventful and the polling didn't change much, not because of a fundamental change in the model. 2016 had tons of bombshell events which is what caused the chaos in the prediction model.
- ashtonkem 6y ago
  One of 538’s models did that in 2016, not all of them. They made it very clear that they did not recommend the “nowcast” because it tended to confuse people.
- 6y ago
person_of_color 6y ago
Why don’t we use blockchain for accurate, anonymous polling? Rewards in BTC?
deleted 6y ago
[deleted]
tmsh 6y ago
I think you can either: * make predictions with a range * or make predictions with a certainty coefficient (p value, etc.) Predicting a single value without either is not a truly quantified "prediction." It's weird now not getting that can lead to confusion. I agree with others in that those who don't communicate both values of information aren't being 100% clear.
shalmanese 6y ago
I don’t understand why there’s so much attempted theory when we can just directly compare Nate’s model against a market, in this case, the presidential prediction market. What you found for 2016 was that the market both had about the same volatility over time as Nate’s model AND was more wrong about the end result, thus showing that the variances in the model were likely driven by actual uncertainty, not spurious over confidence.
- aYsY4dDQ2NrcNzA 6y ago
  I don’t trust the prediction markets. https://freakonomics.com/2008/10/02/manipulation-in-political-prediction-markets/ https://freakonomics.com/2008/10/02/manipulation-in-politica... http://web.archive.org/web/20090130093117/http://www.fivethirtyeight.com/2008/09/intrade-betting-is-suspcious.html http://web.archive.org/web/20090130093117/http://www.fivethi...
  - ashtonkem 6y ago
    And nor should you. But one can compare 538’s model against models made by other statisticians.
  - shalmanese 6y ago
    But the question that was being asked was whether there was an arbitrage opportunity available which, regardless of whether the market is manipulated or not, has an empirical answer, not merely a theoretical one.
  - elie222 6y ago
    That's one case in 2008. If volume and stakes are low then sure, they're not too reliable. If you don't think prediction markets aren't great predictors you should make bets against that market. If you're a better predictor than the market you should make good money. In practice most people can't outguess the market. Some experts can and do make good money. The reason markets are a good predictor because as soon as an expert sees the market make a bad prediction, they'll make a bet against the market. This in turn will move the price towards the true price. Assuming there are enough experts out there with large enough pockets (which is true for popular bets) then the market will quickly reflect the experts' view of events which is by definition very hard to beat (you have to be more expert than the experts).
thaumaturgy 6y ago
The disagreement between Taleb and Silver wheels on technical minutiae in statistics, but the discussions around them by lay-people seem to be about something else altogether. The majority of 538's ad impressions come from people who start F5-mobbing the site during election time because they want to know before anyone else who's going to win. (A tiny few might be from people trying to figure out which races are worth donating towards.) So that puts 538 into this awkward position where their audience is demanding something that 538 is unable to provide, but if they get too in-your-face in saying, "look, that's not how this works", then they stand to lose a lot of ad revenue during peak season. That leads them to do very silly things, like bury this: > A 10 percent chance of winning, which is what our forecast gives Trump, is roughly the same as the odds that it’s raining in downtown Los Angeles. And it does rain there. ...in the same page as this: > We simulate the election 40,000 times to see who wins most often. And that's some bullshit. Silver and the rest of his staff over the last few months have spent a CVS receipt's worth of text on what went wrong with their predictions -- er, sorry, "models" -- in 2016, and why this year is different, and how uncertainty works, and why they're offering no guarantees really, "but please do keep coming back, and hey check out this cool new simulator thingy where you too can make guesses that are about as accurate as ours". I almost really wouldn't care, except for one kind of big problem with 538: It is a political observer-effect in action. Journos, hacks, and other newspeople trying to get one more page impression or soundbite before the end of the day keep referencing 538. When 538 says a candidate is doing well according to their models, it can and almost certainly is changing people's behavior. Likewise when 538 says a candidate is doing poorly. That's hugely problematic and, annoyingly, the more accurate 538 becomes, the more pronounced and dangerous this effect will be.
- ashtonkem 6y ago
  538 gave Trump one of the highest chances of winning from a non-partisan outfit; 1/3 if I recall correctly. Far better than, say, the NYT that gave him 1% odds. I’m not sure how you can look at that and declare them to be hacks.
- Traster 6y ago
  There's a small problem with your comment- 538 didn't predict the 2016 election wrong. They gave Trump a roughly 30% chance of winning. 30% happens. That's all. People act like 538 said "Clinton will win!" but that's absolutely not what they did.
dragonwriter 6y ago
> Taleb insists that Clinton never should’ve received a probability of winning of 90%. Even if polls were heavily in favor of Clinton at the time, he says Silver should’ve taken into account the uncertainty that polls would change over the next few months leading up to the election, or the possibility of major news breaking. Silver's model fairly explicitly does address both the likely direction and the uncertainty in future poll movements, based on historical evidence. The 2016 model results were more volatile than you'd like to see, but it's at least plausible that's because 2016 was an unusual cycle out in the statistical tail in behavior. The 2020 forecast has behaved in a more conventional manner with mild noisy ups and downs early on, and then a smooth progression of greater certainty, the 2012 projection was similar. While this obviously is way to few to generalize from (but better than try to assess from the behavior of 2016 alone), it's consistent with the idea that the model usually behaves in the way you'd expect a forecasting model to behave, and 2016 was just a collision of unlikely events.
ceilingcorner 6y ago
Silver has such an army of fanboys that it’s impossible to even remotely criticize 538 without getting downvoted into oblivion. It’s a weird sort of fanatical statistical religious belief. The fact of the matter is that election polling is not really accurate anymore. People are afraid to publicly admit their support of a certain candidate, even in a supposedly anonymous phone call. Considering that people have been doxxed online for this, I really don’t blame them for being skeptical and private. That’s not even mentioning the social aspects of the media. Even if Trump were ahead in all the polls, do you think Silver and other journalist-statisticians would report it as such? I doubt it. Pundits seems to consistently miss larger geopolitical trends, like Brexit, offshoring of jobs to east Asia, rise of right wing parties in Europe and India, and so on. These qualitative trends will end up being far more influential than a collection of polls.
- deleted 6y ago
  [deleted]
75dvtwin 6y ago
I actually think both are wrong (Taleb little bit less so) Because they are using empirical evidence (eg previous data) from 'experiments' that are effectively, unrelated to current situation. In my personal view, of course, Most previous elections in the last 30+ years, were between RINOs (Republican In Name Only) and Democrats. Those are pretty much two fractions of the same bribe-taking, accountability-avoiding global Cartel. This Election is between the representative of citizens of the Republic (Trump) and the Cartel's candidate. This really did not happen from the time Bushes-then-Clintons took over both parties and turned them into co-existing fractions of one party. - - - Another analogy: we do not use same methods and design constraints when building anti-lock breaks control software, as we do when we build a daily revenue reporting system. - - - Frame of reference is different basically. And we do not know the tensor(s) that help us to move between the coordinate systems. - - - So we are not going to be able to use even 2016 polls (because then folks did not realize how Corrupt the RINOs+Dems are ) - - - My prediction is that Trump wins, GOP takes House and retains Senate. This will also be the largest percent wise vote for a GOP president, by Latino and African American voters. - - - Will check back in 9 days or so, to see if I was right ! Cheers.
kodyo 6y ago
I lie to pollsters.
deleted 6y ago
[deleted]
jgalt212 6y ago
I think Don Rumsfeld could definitely help bridge the gap between Nassim and Nate. https://en.wikipedia.org/wiki/There_are_known_knowns https://en.wikipedia.org/wiki/There_are_known_knowns
Traster 6y ago
There is a lot of misunderstanding about now-casts and probability in this thread, but for me this discussion between two quite able mathematically minded poeple comes down to the exact same thing as what idiots argue with Nate Silver about. Nate Silver has a model that predicts the likely outcome of the election based on poll data, economic data, and a few others factors. Taleb saw Clinton's predicted victory peaked at 90% and says that's wrong - because the uncertainty is so high that you can't be 90% confident. Even if Clinton were ahead at that point in the election there was a greater than 10% chance of something changing before election day. But this is just the idiots critique - You said it was 90% would happen but the 10% happened! Well, yes, that happens 10% of the time, and if you're making lots of predictions it happens often. But let's examine the critique, was Clinton a 90% favourite to win? Well probably, I mean in order for Trump to win from that position not only did he need a decent polling error in his favour he also needed the almost unique situation of the FBI director announcing he was re-opening an investigation into Clinton. We can actually see from the polls the effect this had. That sounds like something, in combination with a polling miss, and in combination with a favourable distribution of Trump's vote by state combined to give Trump a win. It seems reasonable to me to fit that firmly under a 10% probability of happening. If that was in your list of likely scenarios at the point the model was at 90% then you are massively over-estimating how often shocks like that happen. The 90% criticism of 538 only works if you can provide a reasonable discussion of why you think that what happened between the 90% and the result had more than a 10% chance of happening. In fact I think 90% of Nassim's problem is that he says things like "When someone says are event and its opposite are extremely possible I infer either 1) 50/50 or 2) the predictor is shamelessly hedging, in other words, BS." He's totally wrong. Nate says "Extremely Possible" because when he says X is 75% people hear "X is 100%", and if Nassim had actually paid as much attention to 538 as he should before making that criticism, then he would know that. Every single election cycle 538 have struggled against the fact that people don't understand probabilities intuitively.

93 points by probe 6y ago

This article is mischaracterizing Silver's model. The author writes: "if the election were to happen today, what is the probability of each candidate winning?" This is not at all what Silver's model does. Silver writes and makes clear that there is more variance in the model a few months out than a few days out. (In other words - Biden may be 90% now but four months ago if the polls were identical he would be at less than 90% since there would be more time for voter change and real world events.) I honestly don't even get what Taleb is truly arguing here. He is using a lot of statistical arguments but ultimately Silver is making a model to predict an outcome. It is like a weather forecast. I may say that four months from now the chance it rains in Los Angeles is 2%. However, the day before with more info I may be able to say it will rain 90% of the time. There isn't anything wrong with this, even though Taleb seems to suggest this wrong (from my read)

smallnamespace 6y ago
Taleb is saying that if your rain model four months out shows 2%, 8%, 3%, 9%, then it should have instead shown 5-6% that entire range instead. If your final prediction ends up at 90%, your path there should resemble a sort of random walk. Put another way, the 538 model doesn’t add enough variance when we’re still months out. This shouldn’t be surprising, as Silver himself emphasizes that their primary metric is ‘last minute’ prediction accuracy.
- pochamago 6y ago
  I think what many people miss, and what Silver isn't likely to admit, is that 538 is entertainment. If their prediction were 50% split up until the last month of the campaign it would be boring and nobody would check out his site. Higher variance is desirable to him, and he can justify the whole thing in his post mortems by only measuring last minute accuracy.
coliveira 6y ago
The problems with Silver's results is that it is from a "seat-of-the-pants" model. It is trying to do some form of data mining with the data, but there is little underlying foundations to guarantee that it is even close to reality. Regular polls are more transparent because they (at least the well done) will tell you the methodology and the statistical limitations of the research. You can't get that from what Silver is doing.
- edmundsauto 6y ago
  This is simply untrue. The 538 models are calibrated against a lot of previous real-world outcomes.
repsilat 6y ago
Guessing at what Teleb would say: if it rained on 2% of days, and weather forecasts are only good a week out, you'd expect a reasonable model to have the probability sit around 2% until a week or so before the prediction date, then move up or down, maybe with some noise. If you saw the probability swinging from 20% down to 0.1% a month out you should be suspicious. A 90% chance in the election a long way out is suspicious if we think that the chance effects are large and roughly even-handed. Regardless of assumptions about the distribution of chance events though, multiple large swings are themselves suspicious because they each by definition imply the observation of highly improbable events.
- pwinnski 6y ago
  We've had Trump as President for four years, and it seems like most people have forgotten how incredibly unlikely that seemed in August or even October. Trump himself was laying the groundwork for launching a TV network until things changed quite dramatically at the end of October with Jim Comey and some other factors. Taleb seems to be arguing that 90% confidence was crazy given what eventually happened, with the benefit of hindsight, but that does seem to ignore just how crazy the idea of a Trump presidency seemed before it happened.
  - edmundsauto 6y ago
    In the scope of this thread - Silver v. Taleb - there's no need to mention the layperson's memory of their internal prediction. We know that 538 had a 30% chance of a Trump victory, which is the relevant piece of information - not our perception.
    - pwinnski 6y ago
      I disagree. Taleb's key assertion seems to be that Silver should never have given Clinton a 90% chance of winning with the level of uncertainty in that race. But there were times when Trump having only a 10% of winning seemed completely justified, and I think to say otherwise at this point seems like projecting backward with hindsight.
smeeth 6y ago
Minor addendum, Silver altered his model to become more certain as the election drew near after 2016. In effect, it changed from a “what if the election were today?” Model to a “what will the outcome of the election on date ___ be?” Model just for this cycle. I can’t say why for certain, but it definitely could be that Taleb’s 2018 criticisms played a part.
- kadoban 6y ago
  That's incorrect. At _least_ as far back as 2016 and I believe quite a bit farther back the model was not "what if the election were today?" There even used to be a separate side-model that was explicitly that, the "nowcast" I think it was called. I suspect that 538 _never_ covered an election solely or even primarily with a "what if the election were today?" model, but my memory isn't that reliable back that far.
  - smeeth 6y ago
    You are correct, thank you. I updated another comment of mine to reflect this. Comparing them now though, Silver definitely changed the 2020 model to increase variance as a function of time to the election, which was Nassim's complaint in the first place.
- dragonwriter 6y ago
  > Minor addendum, Silver altered his model to become more certain as the election drew near after 2016. In effect, it changed from a “what if the election were today?” Model to a “what will the outcome of the election on date ___ be?” Model just for this cycle. False. Silver's headline model has always been a “what will the outcome on election day” model. (Silver has a secondary “what of the election were held today model”, as well, which was published as an explicitly alternative model in previous cycles, with explicit notes about the difference in what it did from the main model, and is still run and occasionally referenced but not published.)
vincentmarle 6y ago
> I honestly don't even get what Taleb is truly arguing here. He is using a lot of statistical arguments but ultimately Silver is making a model to predict an outcome. It is like a weather forecast. "Making models to predict" is precisely the fluff that Taleb is against. There is no "if elections were held today", there are no do-overs, there is only one election. And you either win or lose (binary price [1]). So any prediction that doesn't incorporate the events between making the prediction until the election actually occurs, is worthless, or to be more precise: 50/50 (arbitrage free). The probability only goes up when you start counting the actual votes on Election Day. [1] https://arxiv.org/pdf/1703.06351.pdf https://arxiv.org/pdf/1703.06351.pdf
- kelnos 6y ago
  If it's 50/50 up until election day, then it's pointless to try to predict. And maybe that's Taleb's point. But is there value in a prediction that does try to predict using polling data and other information? If so, then regardless of whether or not 548 "does it right" by Taleb's definition, it's valuable, at least to some people.
Tomminn 6y ago
"if the election were to happen today, what is the probability of each candidate winning?" This was a correct characterization of Nate Silver's model at the time Nassim Taleb made his critique. Since then, Silver has changed what the probabilities in his prediction model mean. This change was (briefly) discussed previously on HN [1]. For the record, I think Taleb was an asshole, but an asshole with an entirely valid point in this debate. And I think Silver reluctantly and quietly took it on board, and then went on to use bolt-on fudge factors so that the time-dependent predictions of his model roughly look like the time-dependent predictions generated by the rigorously defined approach (option price theory) that Taleb was calling for. [1] https://news.ycombinator.com/item?id=24164814 https://news.ycombinator.com/item?id=24164814
- ashtonkem 6y ago
  This was a correct description of one of 538’s models, the nowcast. It was never the one they recommended, and they did not bring it back later because it tended to confuse people. They actually made it very clear on their podcasts that the nowcast was a mistake.
  - Tomminn 6y ago
    Ahh, that's even worse for Nate Silver then, if the "polls-plus" and the "polls only" predictions were not assuming the election happened today (which it seems like they weren't, even in 2016 [1]), then Taleb's criticism is all the more accurate. Whatever Nate Silver was doing to incorporate future uncertainty in 2016 was clearly inadequate for all the reasons that Taleb pointed out (you could reliably make money betting on Silver's prediction odds to tend toward their time-smoothed average, implying that their time-smoothed average was a better representation of the probability than the probability itself). [1] https://web.archive.org/web/20160930201714/http://fivethirtyeight.com/features/a-users-guide-to-fivethirtyeights-2016-general-election-forecast/ https://web.archive.org/web/20160930201714/http://fivethirty...
    - dragonwriter 6y ago
      Taleb's criticism is inaccurate in any case; it would be a valid criticism of the model if, in a large number of election cycles, the 2016 shape was typical. Because what you'd expect a forecast model to do typically is to get gradually more certain over time, so if it consistently behaved the way the 2016 forecast did, that would be a problem. But, you'd also expect, in a large number of cycles, to have some where you get patterns where you do start getting runa of information that contradict the impact of prior information, and where you do have the kind of reverses seen in the 2016 cycle. And, while we don't have enough Presidential cycles to make firm descriptions of what a “typical” cycle would look like with Silver's model, we do know that both 2012 and 2020 look a lot more like you'd expect a typical cycle in a forecast model to look.
asdfasgasdgasdg 6y ago
The point of premise 2 is that an 80% probability in late June implies a very high level of confidence. Confidence that doesn't seem intuitively reasonable. I would say that few rational individuals would have taken a bet with 4 pays 1 odds that Biden would win on July 1. It would not be a very safe bet, because of the likelihood of unknown random events. Therefore, what Nate's model was predicting in July probably wasn't the probability of Biden winning on November 3rd. That doesn't mean the model's not useful, but if it is trying to predict what you think it is, then it's overconfident. As a non-huge fan of Taleb, and as actually a big fan of Silver, I think Taleb is right here.
Tomminn 6y ago
Taleb's criticism was that the chance it rains four months from now shouldn't reliably vary about a mean (say from 30% to 70% to 30% to 70% etc) in the four months leading up to that day. Because if it does, I can make risk-free money by betting it won't rain on days the payout for not-rain is high, and betting it will rain on days the payout for rain is high. If a time-smoothed average of your predictions is a better prediction of the outcome than your current prediction, then your current prediction is wrong. The idea that someone can improve on your current prediction by averaging your last 10 predictions is a sign that your methodology for generating "current predictions" is deeply suboptimal.

deleted 6y ago

[deleted]

FandangoRanger 6y ago

It's impossible to believe any forecast, nobody fessed up after 2016 nor did they explain why their models are wrong or attempt to update them. You're less than 24 hours away from the permanent death of the political polling and forecasting industry in the USA. Nate's already coping really hard on Twitter.

ponker 6y ago
Well pre-2016 there was the assumption that the American electorate basically is capable of self-governing. This assumption has been invalidated so while the polling industry hasn’t gone away it’s now more like the sports forecasting industry, where people are speculating on an essentially theatrical performance, than something substantively driving good governance.
jjeaff 6y ago
That's not true, there have been countless articles about why polling went wrong last time. Also, polling wasn't that wrong. Due to sampling bias, several swing states were off by as much as 5-6 percentage points. But national polling bad Hillary winning by something like 4% in the popular vote. Instead, she won the popular vote by 3%.
- JumpCrisscross 6y ago
  > several swing states were off by as much as 5-6 percentage points Most state polls have margins of error ranging from 3 to 6 points. A 3-point MoE state poll is a really expensive poll.
  - timmytokyo 6y ago
    A properly conducted poll with 1000 respondents has a margin of error of 3.2%. There have been quite a few of these in battleground states this cycle.
gdulli 6y ago
Believing that occurrence of an event with a forecasted 30% chance (or 2% chance) of occurring invalidates a model is so fundamentally a misunderstanding of the topic.

MR4D 6y ago

FTA, “ I believe Nate Silver is answering a subtly different question with his election forecasts. Each data point that Silver produces is answering the question: if the election were to happen today, what is the probability of each candidate winning?” If that’s true, then Nate Silver just does poll analysis, not predictions. Predictions should be on the day of the election, not the day you’re reading their blog post. In other words, Taleb is right.

throwawaygh 6y ago
It's not true.
- MR4D 6y ago
  What’s not true?
  - cyphar 6y ago
    The statement you quoted from the article (which is basically Taleb's criticism of Nate Silver) is not correct, the FiveThirtyEight prediction models are explicitly not answering the question "if the election was held today, what would happen". There are models like that on FiveThirtyEight, but the main question being answered is who is likely to win the actual election on the day it is held.
    - ashtonkem 6y ago
      There were models like that, they didn’t bring them back this year.
      - deleted 6y ago
        [deleted]
      - dragonwriter 6y ago
        > There were models like that, they didn’t bring them back this year. They weren't the headline forecast, but, yes, the NowCast existed (and still does, in fact, though it's not visible, just like the polls-only forecast model isn't visible any more; Silver references them periodically.) But it's inaccurate to criticize the headline forecast based on the idea that it is trying to do what the NowCast does, and it is dishonest to criticize something other than the main forecast and say it is a criticism of the forecast, but not specify the alternative forecast you are actually criticizing.

throwawaygh 6y ago

> the election were to happen today, what is the probability of each candidate winning? He's explicitly not doing this. Here's how I think about it. Silver is answering the question: "how much would the polls (as an aggregate) need to differ from the final result in order for Candidate X to win/lose, conditioned on some reasonable priors?" Taleb is pointing out that the polls could be really wrong in all sorts of ways that are impossible to predict a priori. The whole argument is sort of pointless from an intellectual/academic perspective. It's a war of public personalities more than anything else. It's both the case that Silver designed a good piece of software that does what it's supposed to do and also the case that Taleb's skepticism is valid. But then, that sort of skepticism of statistical models is always valid, and yet we use these models to great effect in all sorts of settings.

coliveira 6y ago
Taleb made a career from stating the obvious and making it appear as a smart discovery. I've read some of his writings and have never found anything really new.
- seibelj 6y ago
  Totally agree, he comes off as a hack to me. But he’s famous and I’m not, so shrug
  - bitdizzy 6y ago
    Taleb is seriously proficient at probability which can be judged from his technical publications.
- sillysaurusx 6y ago
  The same is true of pg, but his discoveries seem obvious only in hindsight. (I’m objecting to “the discoveries are obvious” as a putdown. The simplest discoveries are some of the hardest. Though I doubt this matters for political pundit analysis.)
  - 6y ago

egonschiele 6y ago

Incredibly click-baity title and incendiary first paragraph were hard to get past.

deleted 6y ago
[deleted]

fortenforge 6y ago

> I believe Nate Silver is answering a subtly different question with his election forecasts. Each data point that Silver produces is answering the question: if the election were to happen today, what is the probability of each candidate winning? As others have pointed out, this is definitely not what Nate's model was doing. In fact, he did have a version of the model that did exactly this, the so called "Now-Cast" and as expected it was even more volatile than the real model: https://projects.fivethirtyeight.com/2016-election-forecast/#now https://projects.fivethirtyeight.com/2016-election-forecast/...

dragonwriter 6y ago
Right, the main model intends, in fact, to be predicting the probability of various outcomes when the election is actually held, that's why there is accounting for non-poll fundamentals (economy and other things) that are observed to influence poll changes over time that phase out as time runs out for them to move the polls.

travisgriggs 6y ago

So... Taleb is a like purist programmer (pick your favorite paradigm) and Silver is more like a pragmatist programmer that uses a "hybrid" language model? There's values for both types in programmerdom, is there not the same for both in the world of statisticians?

stopachka 6y ago

“if I tell you an event has a 0% chance of occurring, I cannot change my mind and tell you tomorrow it now has a 50% chance of occurring. Otherwise I shouldn’t have told you it has a 0% chance in the first place” I think this is the key to Taleb’s argument, and it’s pretty damning for Mr Siver. If there’s high uncertainty, you can’t say whether someone’s probability to win is low. The uncertainty widens the probability. For example, Say I told you there’s a 2% chance it will rain Monday next year in New York - This would _have_ to be a bogus prediction. So much could happen until then. I have _high_ uncertainty. There’s no way that uncertainty is accounted for with 2%

ashtonkem 6y ago
Yeah, except both of them are modeling how people will vote, and people change their minds. Saying that one should’ve held a constant chance of a candidate winning no matter what the polls do is frankly insane. This stance basically requires that you pretend that the outcome is determined and that voters opinions either don’t matter or don’t change. It’s utter nonsense, especially since we know that in 2016 undecided voters broke late for Trump. If your model doesn’t react to that, what are you even modeling?
- stopachka 6y ago
  Taleb’s point is not that your prediction should not change. His point is that if it does change, and frequently at that, the purported 90% chance at t(n) is bogus. That prediction needed to account for the volatility
cortesoft 6y ago
The prediction should have more data to it about variance, but it isn't necessarily wrong. We have good data about rain, I am sure we could find out what percentage of the time it rains in New York on a given day during that time of the year. There might be high variance, but that doesn't mean it is wrong to say there is a 2% chance it rains on a particular Monday. This is how insurance works, how sports betting works, how option pricing works. Just because something has high variance doesn't mean we can't give probabilities that work out over a large number of samples.

ramraj07 6y ago

Your argument doesn't hold water at all. Today's blurb in the forecast page - "First, the forecast is now totally polls-based; that is, any advantage our forecast gave Trump for the economy or for being an incumbent is no longer factored in. Second, the uncertainty around how much the polls will change between today and Election Day is also no longer an issue." - that means Nate Silver is explicitly saying his forecast is always trying to predict what is going to happen on November 3. Heres an alternative hypothesis that involves no fancy math just anthropology - Taleb is right that the stats don't make sense, but Nate is also not an idiot - it's just in his financial best interests to make a website that can throw out confidence inducing pseudonumbers that will make people like us visit it every few hours for months at a time every two years. Perhaps he knows perfectly well the farce he's pulling (at least at a fundamental level though he might have convinced himself otherwise as academics always do) but just plays the game to make sure he's relevant. And thus, we are all the idiots.

rramach 6y ago

Nice summary of the Nassim-Nate debate. A formal analysis/discussion of election forecasting is in this paper: http://www.stat.columbia.edu/~gelman/research/published/jdm200907b.pdf http://www.stat.columbia.edu/~gelman/research/published/jdm2... Edward says that Nate's prediction should be interpreted as: “If nothing else changes between now and the election, Joe Biden has a 85% chance of winning.” (Silver’s argument) But the problem is this prediction is not testable as election only occurs once at the end. Thus, one should use Nate's early predictions as pure entertainment. Nate's final predictions show that they are well-calibrated across all of his political predictions but it is hard to estimate how accurate his model predictions are just for presidential elections given he has predicted only a few so far (unless one assumes all elections have similar uncertainty, which is clearly not true).

baron_harkonnen 6y ago

I've recently come to believe that Taleb really doesn't understand probability. I know this sounds like some insane flippant remark, but I saw a video where he talked about the election forecast as a martingale bounded between 0 and 1 and then made some claims about variance with these bounds to back up his claim against Silver. For his claim to make sense you would have to view probabilities as linear, which is not simply not how probabilities work. An event that has a 0.01 chance of happening is an order of magnitude more likely than one that has as 0.001 chance. However the different between and 0.5 and 0.501 is essentially negligible (hence probability is not linear because 0.001 does not mean the same everywhere). I've generally given Taleb a pass as a bit of a crank that doesn't care about the philosophical interpretation of probability, but after seeing him misunderstand probability in such a major way I've started writing him off entirely. edit: I should add I'm not particularly more convinced that Silver really understands all that much either, just that this is all two very loud people that don't know much. Neither should be taken seriously. This isn't Bradly Efron arguing with Andrew Gelman.

inimino 6y ago
Based on what I've read of Taleb, if you think he doesn't understand some patently obvious facts about probability, it's going to take more than saying that you watched a video and didn't think it made sense for me to give your claim any weight.
amw-zero 6y ago
There is a very clear flaw in your mathematical reasoning here, let's walk through it. Consider your second probability comparison: the difference between a probability of 0.500 and 0.501. You have this correct, the difference is 0.001. Now reconsider your first statement: that a probability of 0.010 is an order of magnitude more likely than a probability of 0.001. This is true, but is completely irrelevant because it is a completely different comparison. A more appropriate comparison would be to compare probabilities of 0.010 and 0.011. The difference of these is again 0.001. You compared proverbial apples to oranges, and you cannot trust the conclusions you make from invalid comparisons. A difference in probability of 0.001 does mean the same thing no matter what the starting probability is. Probability is "linear" in this way. Now that we know this to be true, I would encourage you to rethink your position on Nassim Taleb. He understands non-linearity quite well, in fact I would argue that his entire purpose on Earth (at least in his eyes) is to remind people about non-linearity as a concept. It comes across in all of his books and most of his public appearances. On the matter of probability, I trust him very deeply. Whether or not I trust him is also inconsequential, because he has serious mathematical literacy and you can evaluate his propositions for yourself.

smeeth 6y ago

Edit: as pointed out in the replies, I was wrong that the model was "explicitly" a now cast, and I would like future readers of my comment to know that without deleting things. I would still like to note that there was clearly a change in the 2020 model to increase variance as a function of time to the election when compared to the 2016 model. I think this is substantively the same. Some of the top comments are making the same mistake, probably because they aren't aware of this twitter beef history (and I envy them), so I'm putting this in a separate comment. These comments are all pointing out that Taleb's criticism is of something that Silver's models don't do: > the election were to happen today, what is the probability of each candidate winning? What people seem to be forgetting, is that Silver's 2016 model explicitly did this. Taleb made his criticisms starting in 2018 (even published an options pricing paper on the subject) then Silver changed his model for the 2020 cycle so that it no longer was an "if the election were today" model. I can't say for sure whether or not Silver changed his model because of Taleb, buuuuut he deserves at least a little credit here. Yes, I know Taleb is a colossal asshole people love to hate, but he's usually right about these sorts of things. Here is a pretty fair and balanced take on it for those who would like to read more: https://towardsdatascience.com/why-you-should-care-about-the-nate-silver-vs-nassim-taleb-twitter-war-a581dce1f5fc https://towardsdatascience.com/why-you-should-care-about-the...

moultano 6y ago
This isn't true. He had both a "nowcast" (if the election were held today) and a prediction of the final result. The only change in that regard between 2016 and now was to stop publishing the nowcast. Edit: Even after your edit I don't think the claim is true. The probabilities in this election didn't change much because the campaign season was comparatively uneventful and the polling didn't change much, not because of a fundamental change in the model. 2016 had tons of bombshell events which is what caused the chaos in the prediction model.
ashtonkem 6y ago
One of 538’s models did that in 2016, not all of them. They made it very clear that they did not recommend the “nowcast” because it tended to confuse people.
6y ago

person_of_color 6y ago

Why don’t we use blockchain for accurate, anonymous polling? Rewards in BTC?

deleted 6y ago

[deleted]

tmsh 6y ago

I think you can either: * make predictions with a range * or make predictions with a certainty coefficient (p value, etc.) Predicting a single value without either is not a truly quantified "prediction." It's weird now not getting that can lead to confusion. I agree with others in that those who don't communicate both values of information aren't being 100% clear.

shalmanese 6y ago

I don’t understand why there’s so much attempted theory when we can just directly compare Nate’s model against a market, in this case, the presidential prediction market. What you found for 2016 was that the market both had about the same volatility over time as Nate’s model AND was more wrong about the end result, thus showing that the variances in the model were likely driven by actual uncertainty, not spurious over confidence.

aYsY4dDQ2NrcNzA 6y ago
I don’t trust the prediction markets. https://freakonomics.com/2008/10/02/manipulation-in-political-prediction-markets/ https://freakonomics.com/2008/10/02/manipulation-in-politica... http://web.archive.org/web/20090130093117/http://www.fivethirtyeight.com/2008/09/intrade-betting-is-suspcious.html http://web.archive.org/web/20090130093117/http://www.fivethi...
- ashtonkem 6y ago
  And nor should you. But one can compare 538’s model against models made by other statisticians.
- shalmanese 6y ago
  But the question that was being asked was whether there was an arbitrage opportunity available which, regardless of whether the market is manipulated or not, has an empirical answer, not merely a theoretical one.
- elie222 6y ago
  That's one case in 2008. If volume and stakes are low then sure, they're not too reliable. If you don't think prediction markets aren't great predictors you should make bets against that market. If you're a better predictor than the market you should make good money. In practice most people can't outguess the market. Some experts can and do make good money. The reason markets are a good predictor because as soon as an expert sees the market make a bad prediction, they'll make a bet against the market. This in turn will move the price towards the true price. Assuming there are enough experts out there with large enough pockets (which is true for popular bets) then the market will quickly reflect the experts' view of events which is by definition very hard to beat (you have to be more expert than the experts).

thaumaturgy 6y ago

The disagreement between Taleb and Silver wheels on technical minutiae in statistics, but the discussions around them by lay-people seem to be about something else altogether. The majority of 538's ad impressions come from people who start F5-mobbing the site during election time because they want to know before anyone else who's going to win. (A tiny few might be from people trying to figure out which races are worth donating towards.) So that puts 538 into this awkward position where their audience is demanding something that 538 is unable to provide, but if they get too in-your-face in saying, "look, that's not how this works", then they stand to lose a lot of ad revenue during peak season. That leads them to do very silly things, like bury this: > A 10 percent chance of winning, which is what our forecast gives Trump, is roughly the same as the odds that it’s raining in downtown Los Angeles. And it does rain there. ...in the same page as this: > We simulate the election 40,000 times to see who wins most often. And that's some bullshit. Silver and the rest of his staff over the last few months have spent a CVS receipt's worth of text on what went wrong with their predictions -- er, sorry, "models" -- in 2016, and why this year is different, and how uncertainty works, and why they're offering no guarantees really, "but please do keep coming back, and hey check out this cool new simulator thingy where you too can make guesses that are about as accurate as ours". I almost really wouldn't care, except for one kind of big problem with 538: It is a political observer-effect in action. Journos, hacks, and other newspeople trying to get one more page impression or soundbite before the end of the day keep referencing 538. When 538 says a candidate is doing well according to their models, it can and almost certainly is changing people's behavior. Likewise when 538 says a candidate is doing poorly. That's hugely problematic and, annoyingly, the more accurate 538 becomes, the more pronounced and dangerous this effect will be.

ashtonkem 6y ago
538 gave Trump one of the highest chances of winning from a non-partisan outfit; 1/3 if I recall correctly. Far better than, say, the NYT that gave him 1% odds. I’m not sure how you can look at that and declare them to be hacks.
Traster 6y ago
There's a small problem with your comment- 538 didn't predict the 2016 election wrong. They gave Trump a roughly 30% chance of winning. 30% happens. That's all. People act like 538 said "Clinton will win!" but that's absolutely not what they did.

dragonwriter 6y ago

> Taleb insists that Clinton never should’ve received a probability of winning of 90%. Even if polls were heavily in favor of Clinton at the time, he says Silver should’ve taken into account the uncertainty that polls would change over the next few months leading up to the election, or the possibility of major news breaking. Silver's model fairly explicitly does address both the likely direction and the uncertainty in future poll movements, based on historical evidence. The 2016 model results were more volatile than you'd like to see, but it's at least plausible that's because 2016 was an unusual cycle out in the statistical tail in behavior. The 2020 forecast has behaved in a more conventional manner with mild noisy ups and downs early on, and then a smooth progression of greater certainty, the 2012 projection was similar. While this obviously is way to few to generalize from (but better than try to assess from the behavior of 2016 alone), it's consistent with the idea that the model usually behaves in the way you'd expect a forecasting model to behave, and 2016 was just a collision of unlikely events.

ceilingcorner 6y ago

Silver has such an army of fanboys that it’s impossible to even remotely criticize 538 without getting downvoted into oblivion. It’s a weird sort of fanatical statistical religious belief. The fact of the matter is that election polling is not really accurate anymore. People are afraid to publicly admit their support of a certain candidate, even in a supposedly anonymous phone call. Considering that people have been doxxed online for this, I really don’t blame them for being skeptical and private. That’s not even mentioning the social aspects of the media. Even if Trump were ahead in all the polls, do you think Silver and other journalist-statisticians would report it as such? I doubt it. Pundits seems to consistently miss larger geopolitical trends, like Brexit, offshoring of jobs to east Asia, rise of right wing parties in Europe and India, and so on. These qualitative trends will end up being far more influential than a collection of polls.

deleted 6y ago
[deleted]

75dvtwin 6y ago

I actually think both are wrong (Taleb little bit less so) Because they are using empirical evidence (eg previous data) from 'experiments' that are effectively, unrelated to current situation. In my personal view, of course, Most previous elections in the last 30+ years, were between RINOs (Republican In Name Only) and Democrats. Those are pretty much two fractions of the same bribe-taking, accountability-avoiding global Cartel. This Election is between the representative of citizens of the Republic (Trump) and the Cartel's candidate. This really did not happen from the time Bushes-then-Clintons took over both parties and turned them into co-existing fractions of one party. - - - Another analogy: we do not use same methods and design constraints when building anti-lock breaks control software, as we do when we build a daily revenue reporting system. - - - Frame of reference is different basically. And we do not know the tensor(s) that help us to move between the coordinate systems. - - - So we are not going to be able to use even 2016 polls (because then folks did not realize how Corrupt the RINOs+Dems are ) - - - My prediction is that Trump wins, GOP takes House and retains Senate. This will also be the largest percent wise vote for a GOP president, by Latino and African American voters. - - - Will check back in 9 days or so, to see if I was right ! Cheers.

kodyo 6y ago

I lie to pollsters.

deleted 6y ago

[deleted]

jgalt212 6y ago

I think Don Rumsfeld could definitely help bridge the gap between Nassim and Nate. https://en.wikipedia.org/wiki/There_are_known_knowns https://en.wikipedia.org/wiki/There_are_known_knowns

Traster 6y ago

There is a lot of misunderstanding about now-casts and probability in this thread, but for me this discussion between two quite able mathematically minded poeple comes down to the exact same thing as what idiots argue with Nate Silver about. Nate Silver has a model that predicts the likely outcome of the election based on poll data, economic data, and a few others factors. Taleb saw Clinton's predicted victory peaked at 90% and says that's wrong - because the uncertainty is so high that you can't be 90% confident. Even if Clinton were ahead at that point in the election there was a greater than 10% chance of something changing before election day. But this is just the idiots critique - You said it was 90% would happen but the 10% happened! Well, yes, that happens 10% of the time, and if you're making lots of predictions it happens often. But let's examine the critique, was Clinton a 90% favourite to win? Well probably, I mean in order for Trump to win from that position not only did he need a decent polling error in his favour he also needed the almost unique situation of the FBI director announcing he was re-opening an investigation into Clinton. We can actually see from the polls the effect this had. That sounds like something, in combination with a polling miss, and in combination with a favourable distribution of Trump's vote by state combined to give Trump a win. It seems reasonable to me to fit that firmly under a 10% probability of happening. If that was in your list of likely scenarios at the point the model was at 90% then you are massively over-estimating how often shocks like that happen. The 90% criticism of 538 only works if you can provide a reasonable discussion of why you think that what happened between the 90% and the result had more than a 10% chance of happening. In fact I think 90% of Nassim's problem is that he says things like "When someone says are event and its opposite are extremely possible I infer either 1) 50/50 or 2) the predictor is shamelessly hedging, in other words, BS." He's totally wrong. Nate says "Extremely Possible" because when he says X is 75% people hear "X is 100%", and if Nassim had actually paid as much attention to 538 as he should before making that criticism, then he would know that. Every single election cycle 538 have struggled against the fact that people don't understand probabilities intuitively.