15 ms·

Why we should care about the Nate Silver vs. Nassim Taleb Twitter war

318 points by oska 8y ago

pizza 8y ago
If 538 is trying to do aleatory probabilities, then it is kinda like a broken clock - right sometimes, but as a tool for answers about future events of any complexity (i.e. the real world, where actions have consequences, and errors can be grave..), I'd avoid putting one's confidence in something that claims it has a simple/'linear' relationship to the real future. It is also a cop out to claim "everybody who thinks probabilities based on initial poll responses are not representative of eventual votes" must clearly not understand probability. Maybe I am straw-manning or just don't understand the actual disagreement. As a poll aggregator? I'm sure it's probably a fabulous tool for past, historical, observed data. The brand itself, probably even more valuable, as a symbol with some sort of social authority in certain circles, I guess. Anybody who reads my comment history will see I have obviously become a Taleb stan, but his approach seems right to me: "show me the money". Which in his case means mathematical papers of proofs primarily, actual money, second.
- wand3r 8y ago
  I mean obviously its hard to make predictions of complex things as the data from every angle is difficult to get and sometimes incorrect. I used to love Taleb but he is more of a broken cloc, many say if he stopped writing after black swan he would have gone down as great but some of his other points and politics have undermined him. I think Silver is actually pretty legit. He was over-hyped, then he was largely panned as a chrlatan but through and through he has been pretty consistent. The Obama elections he rose to prominance but he also did well in the most recent election. The absolute only news outlet saying that Trump had a statistical chance of winning. He hovered around 20-33% at peak while Huffington post told him he was an idiot. Many major outlets didn’t personally attack him obviously but had Trump in low single digits throughout the last weeks
  - rsj_hn 8y ago
    Taleb actually makes a great point here and published a pretty cool paper about it. The point is that if a probability of the event changes too much, you can arbitrage it. E.g. assume these are payoff odds and you can sell your position before the event. He then found some nice no-arbitrage conditions that "real" probabilities must meet and showed that Nate's predictive timeline allowed arbitrage. Unfortunately Nate never responded to this claim with anything other than "math is hard" and other nonsense. You can read the paper here: https://arxiv.org/pdf/1703.06351.pdf https://arxiv.org/pdf/1703.06351.pdf
    - lazyasciiart 8y ago
      But 538 was giving predictions for different events each day in the moving "now cast" so there was no arbitrage.
      - ericdykstra 8y ago
        The forecast still fluctuated way too much to be correct, as can be seen in Taleb's paper, which cites the "forecast" and not the "now cast."
    - nl 8y ago
      Is it a great point? He seems to miss the point of what 538 is doing. Their nowcasts are attempts to say what would happen if the election was today, which throws away the time-dependent uncertainty.
    - comex 8y ago
      I don’t have the background knowledge to properly understand the math in that paper. But I do know that you can’t simply set bounds a priori on how much a probability can change after additional information has been gathered; no matter how pathological the swings, you can design a system, and a series of observations of that system, that would make all the forecasts (i.e. conditional probabilities) correct. Thus the paper must be making additional assumptions. As far as I can tell, those include at least that the electoral process can be modeled as Brownian motion, which is a martingale, but is not the only type of martingale; in particular, I’d expect random motion to be a good model of typical polling drift, but a poor model of sudden polling swings caused by news events (which in reality are a large change caused by a single random event, not the sum of a series of small changes caused by independent events that just happen to mostly point in the same direction). I am not sure whether the assumptions also include an estimated value of `s` or volatility; the paper doesn’t seem to explain how it’s calculated, but maybe it can be derived from the raw polling results plus the assumption of Brownian motion? In any case, the paper says nothing explicitly about what data was used to produce the “rigorous updating” graph. I’d love if someone could explain this to me… Subjectively, it’s hard for me to believe that an unbiased forecast would truly be so utterly noncommittal until just before Election Day, or indeed that there’s enough data to answer that question, especially seemingly from just one election result. But my subjective impressions, of course, could be utterly wrong. I’m very curious whether or not this is the case.
- cm2012 8y ago
  Taleb is just mixing up the nowcast and the forecast, even after being corrected many times.
  - zimablue 8y ago
    I think he would say that the "nowcast" is so useless and deceptive to be malpractice, since everyone will interpret it as a forecast, and even if they didn't it's not in any way useful. Which I agree with (unsurprising since I've semi-invented it as an opinion for another person).
    - uncletaco 8y ago
      Silver has said before, at least in the podcast, that everyone at FiveThirtyEight hates the nowcast and its various incarnations but I think there's some pressure from the business side of things (for views, likes, shares, etc) to have something volatile for people to eat see day in and out.
    - thousandautumns 8y ago
      It's only useless and deceptive if you willfully do absolutely nothing to understand what it is, or how it differs from the other models 538 has, which apparently is the case for Taleb.
      - zimablue 8y ago
        Or in other words, you're 99% of people who will ever hear the number
        thousandautumns 8y ago
        It explains on the website what it is. And if you are going to rail against something, you should understand what you are railing against.
mehwoot 8y ago
He should not have accepted the honor if he didn’t call a winner in any of the states! And he didn't, as far as I recall. I remember him saying something to the effect of "that isn't that impressive, even a simple method looking at polls would have gotten nearly all the states correct". Don't confuse what other people focus on for something he is boasting about.
tareqak 8y ago
As a non-specialist, I like how the discussion boils down the picture at the end: https://cdn-images-1.medium.com/max/1600/1*D_tidaT-fHMY3DRLgjwekw.png https://cdn-images-1.medium.com/max/1600/1*D_tidaT-fHMY3DRLg... in that it is something I can understand.
- forkLding 8y ago
  Shows the brilliance of a true data scientist/statistician, condensing all the info in one graphic so that a layman can understand.
- cm2012 8y ago
  Nate Silver always calls the nowcast (blue line) for what it is - a forecast for if the election was held that day. The forecast, made a few days before election day, is the official forecast and the one that beats most other estimates.
  - MAXPOOL 8y ago
    I'm just eyeballing it, but it seems like linear trend line of the nowcast (blue line), would be very close to the official forecast (red dot).
  - OscarCunningham 8y ago
    That blue line is a record of Silver's forecast, not the nowcast. You can compare them here https://projects.fivethirtyeight.com/2016-election-forecast/ https://projects.fivethirtyeight.com/2016-election-forecast/
- chimeracoder 8y ago
  That's a really misleading graph, for a couple of reasons: 1. It conflates the forecast with the now-cast. The former is a prediction of what will happen on Election Day (whether that day is six months away or one day away). The latter is a prediction of what would happen if the election were held that day. In theory, those two would converge to the same value on the day of the election, though in practice, that wouldn't actually happen due to computational differences. 2. Silver never says that he "should be judged" by [just] the final result. In fact, he's gone out of his way to say that. The problem is, that's the only point on the graph where we can compare a model (predicted value) to the actual (observed value). There isn't an election on any of the other days, so even if both agreed to look at the now-cast and use that as grounds to evaluate the model, we still would only have one datapoint which is nonzero in both dimensions. In other words, we have a blue line, yes, but we only have a red dot. Taleb wants to extrapolate the red dot into a horizontal blue line, and then use that to judge Silver's model, which is ridiculous.
backpropaganda 8y ago
The difference between epistemic and aleatory uncertainty is not well understood even in the machine learning world. Most applications of machine learning to domains where uncertainty needs to be quantified use aleatory uncertainty, when epistemic uncertainty is the right one. I wish we had better ways to explain these concepts, and perhaps better terms too. Alternatives like intrinsic/extrinsic or irreducible/reducible don't really work either, because they make it seem like it's a dichotomy.
- pizza 8y ago
  I haven't read any of his works but I have a suspicion that Judea Pearl writes about causality in this niche with that inferential difference in mind.
tracer4201 8y ago
It's not clear to me what the author is getting at. 538 is good at aggregating polls and showing some probability. The author complains that's not enough because a non zero probability of the less probable outcome gives 538 too much cover when the less probable outcome occurs. Isn't that a shared characteristic of any other model dealing with probabilities? Is the author trying to say Nate Silver gets too much spotlight considering he doesn't guarantee outcomes? I didn't sell in 2008 and kept buying. I have the same strategy today. Not sure if that also makes me a guru if that's the bar (as the author applies it to Taleb). Disclaimer: not a mathematician
baddox 8y ago
I appreciate the stuff about two types of certainties, but where are they actually arguing anything remotely relevant like that? As far as I can tell, based on what I see bubble up to my twitter feed and what tweets are displayed in this post, the disagreement is based entirely on Talib’s claim that Silver switched his prediction from 80% to 50%. This really has nothing to do with actually understanding probability or prediction models. It’s just Talib sharing some other pundit’s deliberate misinterpretation of a single phrase deliberately taken out of context in an interview with Silver. I don’t think their disagreement has much to do with science or mathematics. I think the two guys have big online followings, each leaning heavily toward two different political ideologies, and the two leaders just have to try to beat each other up on a Twitter to decide which side is better.
mhb 8y ago
Shouldn't a guy like Taleb be able to propose a bet with Silver that demonstrates he (Taleb) is right?
- starchild_3001 8y ago
  Best solution proposed here! Put your money where your mouth is Taleb and Silver! Btw, I read 3 of Taleb's books, 1 from Nate Silver. Both are pretty good. Taleb is the more original, but also more often-objectionable.
  - dagw 8y ago
    Best solution proposed here So what should the bet be? As far as I can tell they're not disagreeing on anything that can easily be reduced to a bet
    - lixtra 8y ago
      Nate calls the chances like 80:20 Clinton:Trump. Taleb takes the side: 100,000 USD that Trump wins. If Trump wins, Nate pays 500,000 to Taleb. If Clinton wins, than Taleb pays 100,000 USD to Nate. Do this for each election.
      - dagw 8y ago
        But that doesn't seem to be the nature of their disagreement. Taleb's main complaint seems to be about how the swings in the prediction probabilities far out from the event indicates a fundamental flaw in how the underlying model uses probability. That being said it would be a fun little exercise to see who much money would be made if they bet had $1 on every race 538 predicted in 2016 and 2018 using the model odds.
        mhb 8y ago
        It doesn't need to be a simple bet. It could incorporate the volatility which Taleb might believe shouldn't be present in an unflawed model.
deleted 8y ago
[deleted]
BurritoAlPastor 8y ago
I'm infuriated that the author would allege that Taleb has a better way to do this but just links to a highly technical paper instead of addressing it.
Cd00d 8y ago
I would love to see the Nate Silver point by point response to this post. I think it would be humorous, but also add some insights into his proprietary black box. I listen to the 538 podcast every episode, and I think I agree with cm2012 that this is a case of mixing up the "nowcast" with the actual forecast.
deleted 8y ago
[deleted]
comex 8y ago
The article seems like an oversimplification of the underlying dispute. - It’s not necessary or even useful to define an arbitrary “decision boundary” unless you actually have to make a decision. From a Bayesian perspective, a probability stands for itself: 100% means the event is certain to occur, 50% means you have no idea, and numbers in between convey varying levels of certainty. In reality, 538’s predictions are not true Bayesian probabilities because they don’t take epistemic uncertainty into account, but that’s a totally different issue. - “Wild fluctuations in a prediction” from new information are absolutely a “normal part of forecasting” - sometimes. If I’m planning to flip two coins, the probability of getting two heads is 25%; but once I flip the first coin, the probability changes to either 50% (if I get heads) or 0% (if I get tails). In the case of Comey reopening the investigation, even if the model had included a probability of that happening, it would have been low and thus wouldn’t affect the overall forecast much. But once that low probability became a certainty, you would expect a sudden swing. The real question is whether 538’s predictions are more swingy than they logically should be (particularly earlier on), but again, that’s a different issue.
- kingkawn 8y ago
  Seems like in addition to whether they're more swingy than they should be, are they being presented in a way that is interpreted as more certain or authoritative than they can possibly claim?
  - c3534l 8y ago
    538 provides pretty in-depth discussions of it forecasts and changes in it. Far more than you would expect - and I think because 538 is ultimately a statistics blog at heart, not a political blog.
  - azernik 8y ago
    That is actually an issue they've acknowledged and tried to address. People also tended to see percentages and think they reflected vote percents. To address this, for the 2018 election cycle 538 started displaying their odds as numerical ratios (5 in 9, for example) to try to reduce the aura of certainty.
lalaithion 8y ago
> Practically what this means is that they do not predict a winner or looser but instead report a likelihood Yes, this is good. It means that you can evaluate their accuracy to a greater degree than % of times correct. > Further complicating the issue, these predictions are reported as point estimates They've learned from this, and now show their distribution of results very clearly. However, there's nothing mathematically fraught about reporting a prediction as a point estimate. > The problem is that models are not perfect replicas of the real world and are, as a matter of fact, always wrong in some way. ...yes? And so what? This is a fully general counterargument to all of science. > Predictions have two types of uncertainty; aleatory and epistemic. Don't say things like this with 100% certainty when your source itself says "the validity of this categorization is open to debate", and also when your source is Wikipedia. > However, as you can see, there is still a noticable variation of 2–5% of actual proportion to predictions. This is a signal of un-addressed epistemic uncertainty. It also means you cannot take one of these forecast probabilities at face value. Models aren't perfect. Also, 538 is very careful to address their epistemic uncertainty! They also try very hard to not change their models significantly after they publish them, in order to avoid letting their personal biases tinker with the results. This post goes out of it's way a to avoid mentioning that there are pretty well established ways of measuring the accuracy of predictions, instead using things like "look, it's not quite a line" and "look, the line went up and down before the election". If you're interested in actually reading about this from a mathematical viewpoint, read Taleb's paper, which has some actually interesting thoughts on why 538's algorithms are too eager, or read Madeka's paper, which includes comparisons of 538 to other people trying to make predictions, and finds that actually, they were better than most other news sources. https://arxiv.org/pdf/1703.06351.pdf https://arxiv.org/pdf/1703.06351.pdf https://arxiv.org/pdf/1704.02664.pdf https://arxiv.org/pdf/1704.02664.pdf
skolos 8y ago
Author does informative critique of 538. I'm not sure at all that it has anything to do with Silver vs Taleb twitter war though. It's more like premise for expressing author's own thoughts. However I totally dig what 538 does and appreciate their analysis. I understand that there is a complex underlying model with many parameters taken subjectively. The results of Monte Carlo runs of this model are very interesting to me. The fact that they describe distribution of outcomes and not just a single number is important property of a Monte Carlo model and I would have an issue if that wouldn't provide that info. If I would like to analyze complex event that has lots of moving parts and uncertainties I would also build a model to see what kind of distribution I would get. The fact that somebody published results of their model is quite useful. The analogy would be European and American weather models - no one say that their results are exact and everybody understand that uncertainty in result comes from inherent uncertainty of initial state as well as shortcuts and approximations each of the model takes. No one says that the results of these models are useless because of that. And everybody finds it valuable if weather forecast predicts rain tomorrow with 40% chance. So I don't see how political prediction is only valuable (by the words of the author) if it comes without probabilities attached to it.
mturmon 8y ago
This is an intellectually confused post. It is so unclear and muddled that only a charitable reader would suppose the author understands the mathematical and modeling issues here. I’m surprised the author published it in this form - you don’t want to jump into a controversy like this with such an unclear discussion. As just one example, the whole digression on a “decision boundary” is conceptually mistaken. Once you report a posterior probability, it’s up to the user to establish a simple real number threshold for placing a bet (if we’re talking about means to put skin in the game). The posterior you just learned holds all information, your only action is to threshold it. That’s the Neyman-Pearson lemma. If you allow generalizations to interval-valued probability, which might be sensible under these conditions, the situation gets more complicated. But the writer of the post did not mention this. Another place this came out is his initial discussion of aleatory vs. epistemic uncertainty — this can be done clearly, but here we read: > Aleatory uncertainty is concerned with the fundamental system (probability of rolling a six on a standard die). Epistemic uncertainty is concerned with the uncertainty of the system (how many sides does a die have? And what is the probability of rolling a six?). This is just not helpful. He has said “probability of rolling a six” twice. Source: do UQ in day job.
- hn_throwaway_99 8y ago
  Agreed. The more I read the more I got embarrassed for the author, mainly because his points were so ridiculously flimsy. I thought this was one of the worst: > Because FiveThirtyEight only predicts probabilities, they do not ever take an absolute stand on an outcome: No ‘skin in the game’ as Taleb would say. This is not, however, something their readers follow suit on. In the public eye, they (FiveThirtyEight) are judged on how many events with forecasted probabilities above and below 50% happened or didn’t respectively (in a binary setting). Saying "this is not something their readers follow suit on" - only if you're a reader who doesn't understand probability. In fact, Silver is constantly trying to get across how likely < 50% probabilities are with his football analogies (e.g. "a team down by 3 at the half actually winning") to emphasize how likely something like "20%" actually is. If anything, I love Silver's reality-based approach exactly because he does acknowledge that all he can do is offer a best estimate based on available evidence, instead of saying things with false-certainty and then basking in self-proclaimed genius when luck happened to run in his favor.
potatofarmer45 8y ago
Nate Silver built an aleatory model whereby low probability high effect events (Comey reopening the investigation on eve of the election) are discounted because if you add enough of these potential disruptive events the model will be 50%-50% and that has no value to anybody. Who wants to read an article that said "election forecast 50-50" every day. If a 538 model says one outcome is 75% probable, it really should come with a giant caveat saying "at current trends assuming nothing out of the ordinary occurs". Taleb's beef is that if that is the case, the 75% does not mean in reality this result is 75% likely to happen. What is a really interesting takeaway is that if in an event where the outcome you desire based appears unlikely, your job is introduce as much previously undefined or discounted uncertainty. Ideally you engineer a black swan event or at least do what you can to make it happen. This would be fun to model from a game-theory approach instead.
- wtallis 8y ago
  > whereby low probability high effect events (Comey reopening the investigation on eve of the election) are discounted because if you add enough of these potential disruptive events the model will be 50%-50% and that has no value to anybody. I don't think you can say this. The challenge is that it's impossible to even enumerate all the possible surprises that could drastically swing an election, and incorporating some of them into a model would require pulling numbers out of your ass for how to weight those unpredictable possibilities, and even picking which potential surprises to factor in is a similarly arbitrary decision for which there is insufficient evidence to provide guidance. But none of that means that attempting to factor in such possibilities will drive your model's predictions toward 50%; your predictions could end up almost anywhere in the unit interval depending on the value of the priors you pulled out of your ass, and your final number ends up saying more about your biases than about the state of available predictive evidence.
- waterhouse 8y ago
  > What is a really interesting takeaway is that if in an event where the outcome you desire based appears unlikely, your job is introduce as much previously undefined or discounted uncertainty. Ideally you engineer a black swan event or at least do what you can to make it happen. This would be fun to model from a game-theory approach instead. In chess, if you are behind, John Nunn's two recommended strategies are "grim defence" (if your opponent's advantage is not so large as to make it easy for them to force a win) and "create confusion": create complicated tactical situations and hope your opponent makes a mistake. The farther behind you are, the more appealing "create confusion" gets by comparison.
empath75 8y ago
Five thirty eight is pretty consistent with saying their forecasts more than a few days out are unreliable. They’re fairly accurate with with their final forecasts, though.
akivabamberger 8y ago
Why does this not just link to the Twitter war?
cperciva 8y ago
The author doesn't seem to understand how the 538 model works. The 538 model does take epistemic uncertainty into account; it just doesn't call it that. The model calls it "likelihood of polls moving by X% between now and election day". In 2016 they offered a "now-cast" which didn't include that, and answer the question "based on where the polls are right now, how likely is any given outcome if the election were held today?" while the "classic" model answered the question "based on where the polls are right now and the fact that unanticipated things will happen between now and the election, how likely is any given outcome?". For 2018, they dropped the "now-cast", because people didn't understand what it was saying. True, the epistemic uncertainty is limited -- it's based on past elections, because hey, 538 works with actual data. One could argue that rather than saying "71.4% Clinton 28.6% Trump", a better model would have said "71.3% Clinton 28.4% Trump 0.2% Election is cancelled due to nuclear war / natural disaster / etc" but I don't think anyone sensible is interpreting the model as excluding such extreme outcomes.
pmart123 8y ago
I liked the first chapter of "Fooled by Randomness," but I stopped reading the book when each chapter regurgitated the same points, each more nihilistic than the last. Perhaps Taleb should concentrate on the accuracy of his own predictions: https://www.businessinsider.com/taleb-every-single-human-being-should-short-treasuries-2010-2 https://www.businessinsider.com/taleb-every-single-human-bei... >>> "every single human being" should bet U.S. Treasury bonds will decline. >>> Short the S&P vs Long Gold, in a 5 to 1 ratio. By gold Taleb means a basket of precious metals including gold. The second trade would have lost $125k for every $10k invested in gold and $50k invested in SPY. Did he miscalculate the risk of a black swan event...?
- _imba_ 8y ago
  He gets predictions more than most. You should go read that book. There are many chapters on the incorrect weight people add to the outcome of correct predictions (or incorrect predictions, if that is what suits their agenda).
  - pmart123 8y ago
    For me, it'a less about the accuracy of his predictions, and more about his intellectual smugness rubbing me the wrong way. He has no problem with the hypocrisy of attacking Nate Silver over some supposed weaknesses in his election prediction model, but does not seem to care about the inaccuracies of his own public predictions. Nate Silver took political polls and hunches, which were far less accurate, and came up with something significantly better. Even in 2016, Fivethirteight's model showed the odds of Trump winning were not that remote versus major media outlets "guessing" the race was all but over. Election prediction models will always be inherently unstable in my mind. It's pretty much impossible to predict the when, if, how, who of an Anthony Weiner type scandal.
- Animats 8y ago
  I'd be more impressed with Taleb if he published the results for his Emperica fund for years other than 2008. Taleb's strategy was to buy options way out of the money. In boring years, the fund lost money. It did really well in 2008. Not enough numbers have been published to determine if his strategy was a net win over a business cycle. The win seems to have coming from the fund's short lifetime including 2008.
deleted 8y ago
[deleted]
einarvollset 8y ago
https://www.theguardian.com/books/2018/feb/25/skin-in-the-game-by-nassim-nicholas-taleb-digested-read https://www.theguardian.com/books/2018/feb/25/skin-in-the-ga...
013a 8y ago
The whole 2016 situation feels like many people are borderlining on delusion, Nate included. The Comey thing certainly helped Trump, but even the day of the election 538 was still forecasting a massive win for Hilary. How can the probabilities be that wrong? Because the polls were wrong. Or, at least, incomplete. It doesn't matter how good your algorithms are if the data isn't there. And 2016 proved that 538, and really most media outlets, don't have a grip on America. In other words, the existence of a black swan event in that particular system didn't really matter. Silver is a glorified modern fortune teller, who seems to have deluded himself into thinking that simply having data means you can prescribe meaningful probabilities to the future. Assigning probabilities to a coin flip is easy. Figuring out the most likely winner of a game of baseball is a little harder; there's a whole movie about a guy who was basically doing that in 2002. Politics is a completely different field. You can't possibly assemble all of the data necessary to be remotely confident in the probability of outcomes. And let's not forget unpredictable black swan events. But the readers love it. And, especially during the elections, the media would bring him out like a golden boy computer whiz, because he was saying the same things they were, but he's really smart and has the data and magic algorithms to back him up. And he was right that one time 8 years ago. He's about as pointless as Sean Hannity, with the one exception that, at least recently, Hannity was actually more correct about the future than Nate. It doesn't matter if you are right or wrong, if the reasons are wrong.
- wslack 8y ago
  I don't think delusion is a fair take: FiveThirtyEight Prediction: 48.5% to 44.9% Real outcome: 48.2% to 46.1% I agree that the horserace coverage is silly (especially given how polling operations use sampling and such), but its better to look at aggregate trends in polling than to ridiculously overcover outlier polls. I also certainly don't think its fair to say "28.6% chance he wins" is a "massive win" prediction. Silver's take was regarded as indefensibly right-leaning and attacked.
- nsgoetz 8y ago
  538 had almost a 1 in 3 chance that Trump won in their final prediction before the election. https://projects.fivethirtyeight.com/2016-election-forecast/ https://projects.fivethirtyeight.com/2016-election-forecast/
c1ccccc1 8y ago
I'm confused by the existence of this article. Comparing the accuracy of two probabilistic predictors is pretty much a solved problem. (At least in situations where the two predictors have both made predictions about a fairly large number of events in the same set, S.) Just add up the log scores for both predictors and see which does better. [1] [1] https://en.wikipedia.org/wiki/Scoring_rule#Logarithmic_scoring_rule https://en.wikipedia.org/wiki/Scoring_rule#Logarithmic_scori... Another thought: Although the author didn't lie about its meaning, I thought the graph "Stated Probabilities Compared with Average Portions" was visually misleading. At a casual glance, it appeared that 538 was all over the place with its senate predictions. But if you look closer, you can see that many of the red data points are at 0 or 0.5 or 1. This suggests that these data points represent the outcome of just 1 or 2 elections, which isn't really enough to say that 538 is doing particularly badly.
IIAOPSW 8y ago
Why should we care about any twitter war? Twitter is stupid. If it wasn't for a certain #1 individual user, the company would possibly be in bankruptcy by now.
macawfish 8y ago
I'm still waiting for somebody to talk about how tightly coupled Nate Silver's forecasts are to the political processes he attempts to forecast. What if Nate Silver were to run around on the football field before each play whispering in the players' ears about the how likely they are to win, according to his latest forecast. Imagine that they believed in him, or at least felt some degree of superstition when he came around. I just think there's something innately flawed about public forecasting of an election. Mathematically and maybe even ethically. I'm still trying to figure out how to articulate why I feel that way. It could even be argued that polling itself could influence opinions by introducing bias at a critical moment when someone is being asked to consider who they are voting for. What if that's the moment when they make up their mind? P.S. https://www.frontiersin.org/articles/10.3389/fphy.2015.00077/full https://www.frontiersin.org/articles/10.3389/fphy.2015.00077...
- dagw 8y ago
  I just think there's something innately flawed about public forecasting of an election. It used to be the case in France that you weren't allowed to publish polls two weeks before an election for just this reason. They eventually scrapped this with the argument that the all 'elites' had access to internal and private polls anyway so all you where doing was denying the 'people' the same access to information that the 'elites' had. It could even be argued that polling itself could influence opinions by introducing bias at a critical moment when someone is being asked to consider who they are voting for. This is a pretty well known effect, to the point that many political strategists try to avoid publishing or drawing attention to polls showing their candidate too far ahead in case it makes voters complacent and decide to stay home. The ideal situation is to make people convinced the polls show the candidates tied 50-50 going in to the election and that 'your' vote could easily swing the whole election.
  - mijamo 8y ago
    This is slightly wrong. It was initially one week, not two. It was changed because it was unconstitutional, as it limited freedom of expression too much, not because it was useless or anything to do with elite vs people. And it still exists, limited to the day of the election and the day before. Source: https://www.conseil-constitutionnel.fr/nouveaux-cahiers-du-conseil-constitutionnel/la-reglementation-des-sondages-et-l-election-presidentielle https://www.conseil-constitutionnel.fr/nouveaux-cahiers-du-c...
gaius 8y ago
No one should care about anything that happens on Twitter. It’s that simple.
zimablue 8y ago
To try and explain Taleb's argument as simply as possible: If someone tells you today that an event is 100% certain, and tomorrow tells you that it is impossible, then you intuitively will not trust their forecasting. If it's certain, then it's certain not to be impossible tomorrow Now if someone tells you something is a 90% chance today, and a 10% chance tomorrow, again you don't trust their predictions. There is a probability associated with "changes in probability". The probability of going from 100 to 0 should intuitively be 0%. The probability of going from 90 to 10 intuitively must be something like 20% (it's like 80 of the 90s "didn't happen", with a probability of 20). A key point is that you can add this up every day, if something goes from 90->10->90 then it's even more unlikely than going 90->10. So if you extend that intuition, put some real maths behind it, you can tell whether something is "likely to be a real probability" by the rate at which it changes. And if you have lots and lots of repeated predictions, you can be confident that something isn't a good probability (every timeseries has like a <10% chance of doing exactly what it does, so their combined probability [of being good probabilities] is like 0). Now Nate Silver accepts this, but says that he's not actually putting a probability on the event in the future, but some non-existent "probability on that event in the future if the future was now". But that doesn't correspond to anything useful or measurable (it's untestable!!), and most people will assume it follows the normal meaning of probability, and it's honestly just silly.
- OscarCunningham 8y ago
  > Now Nate Silver accepts this, but says that he's not actually putting a probability on the event in the future, but some non-existent "probability on that event in the future if the future was now". Could you link to where Silver discusses this? I'm interested in seeing his description of exactly what his numbers mean.
  - zimablue 8y ago
    https://fivethirtyeight.com/features/a-users-guide-to-fivethirtyeights-2016-general-election-forecast/ https://fivethirtyeight.com/features/a-users-guide-to-fiveth... What I'm referring to is the "now-cast", but his other two definitions both seem to shy away from saying "this is flat-out the probability we think of the election". The point is, you can redefine or choose a definition of probability if you want, but if it's less useful than the normal definition (and confusing to people!) then people are free to criticize your work on that basis. And there's a very useful, testable, mathematical definition of probability that allows us to equally assess everyone's predicting ability, and Nate Silver is dodging it. If you're interested in this subject, there's a non-mathematical discussion somewhere in Tetlock's book Superforecasting which is interesting in general.
OscarCunningham 8y ago
I think the difference in opinion is caused by Taleb modelling the election as though all of the uncertainty is caused by the fact that people might change their minds, whereas Silver is also including the uncertainty from polling errors. Silver's predictions fluctuate a lot as time passes. Taleb is assuming that all of these fluctuations are caused by the electorate changing who they're going to vote for. Then he says "if you knew people's opinions were as volatile as this, you shouldn't have been so confident earlier!", or words to that effect. If lots of people will change their minds, then early polling won't be very informative about the final result. But the fluctuations in Silver's predictions are actually caused by his changing certainty about how accurate the polls are. Silver does take into account that people might change their minds, but this isn't the cause of most changes in his predictions. So people actually change their minds a lot less often than Taleb's model of Silver's model suggests. This is why Silver can make strong predictions early on.
lambdadmitry 8y ago
I'm profoundly confused by the deep respect Taleb commands in the eyes of many people here, apparently. The guy has a few good ideas that he repeats ad nauseam, but he punches way over his own intellectual weight too often. A few especially egregious examples are his deranged GMO hate "paper" which is profoundly ignorant of the involved biology [1], his support of Syrian government's slaughter of civilians [2][3], and his Putinversteher views ([4] reads as a parody for anyone knowing anything about Putin and modern Russia). Combined with his unlimited arrogance, I don't understand how people can respect him intellectually. [1]: http://www.fooledbyrandomness.com/pp2 http://www.fooledbyrandomness.com/pp2 [2]: https://medium.com/opacity/the-syrian-war-condensed-a-more-rigorous-way-to-look-at-the-conflict-f841404c3b1d https://medium.com/opacity/the-syrian-war-condensed-a-more-r... [3]: https://medium.com/@ameraidi/nassim-taleb-is-wrong-on-syria-e1faa523824c https://medium.com/@ameraidi/nassim-taleb-is-wrong-on-syria-... [4]: https://twitter.com/nntaleb/status/773323163509850112 https://twitter.com/nntaleb/status/773323163509850112
jimnotgym 8y ago
All of which increases the probability that I will ignore Taleb's tweets. I follow people who are smart enough to disagree in a civil manner, and smart enough to recognise that there may be things one doesn't know. The way he talked to renowned classicist (and feminist, which seemed to be a problem for Taleb) Dame Mary Beard in another spat was disgusting. Trying to impose statistical certainty on ancient history ended up making him look pretty dumb to everyone but him.
- JabavuAdams 8y ago
  The problem is that the smart arrogant jerk may have the right answer sometimes when it counts, while being a pain in the ass the rest of the time. Better just to stay open to all feeds. He does come off as an arrogant dick, wow.
  - alexandercrohde 8y ago
    I used to look at things this way, with a "just filter out the rudeness" mindset. However in practical terms I think listening to somebody's rudeness does have a really big cost (to estimate: how much would you need to be paid to make it worth your time to listen to rudeness for an hour?)
lavrov 8y ago
I thought that the post was fairly unclear, but for me, it seems like the main argument against 538 is that it makes unfalsifiable claims about the probability of individual elections - picking a winner is a falsifiable claim, but assigning a probability always allows Silver to claim something like “even events with a 10% probability occur frequently” even if his model assigns a high likelihood of victory to the loser of an election. My favorite Nate Bronze takedown remains Carl Diggler[1]. [1] https://www.washingtonpost.com/amphtml/posteverything/wp/2016/05/09/our-fictional-pundit-predicted-more-correct-primary-results-than-nate-silver-did/ https://www.washingtonpost.com/amphtml/posteverything/wp/201...
vertline3 8y ago
Taleb's biggest thing is exposing holes in predictive models. Silver's biggest thing is making and using predictive models. So it's only natural for these two to quarrel.
alexandercrohde 8y ago
So does Taleb propose a better betting strategy? I'd like to see him put his money where is mouth is, rather than just his reputation.
mschuster91 8y ago
Of course the US elections are not as accurately predictable as e.g. German ones, or sports events. Not simply due to different pollster sources and their data quality (which Nate Silver tries to account for). The biggest and most unpredictable factor in US (and British) elections is their winner-takes-all system, with the second layer of the Electoral College in the US which does not account for popular votes either.
thousandautumns 8y ago
I've been following this spat since it reignited several weeks ago, but also since Taleb first attacked Nate on Twitter back before the 2016 elections, and it has been the final nail in the coffin as to any supposed credibility Taleb commands. The quandary that FiveThirtyEight and Silver are in is that the general public is stupendously bad with probabilities. Prior to the the 2016 election, Silver was personally being attacked, in some cases by major media organizations like the Huffington Post, and accused of tipping the scales towards Trump (for some inexplicable reason), since their models were giving him approximately a 1 in 4 chance of winning, since everyone knew it was a certainty Clinton would win. Then, in the months following the election, the narrative somehow switched, and the fact that Trump won despite only being given a 1 in 4 chance by FiveThirtyEight meant that Silver was now a hack and the model was wrong. This same criticism has been leveled by many right-wing media personalities, who had been trotting out Silver's 2016 predictions leading up to the election, as well as by the New York Times, whose own models were giving Clinton nearly a 90% chance of victory. So now, fast forward to this year's election, Silver is doing his best to drill into people's minds that, yes, their model showed a Democratic win in the house as the most likely outcome, but that absolutely doesn't not mean their model says it is an absolute certainty, or that other outcomes would be unusual. It isn't covering his ass, its educating the public on how probabilistic statements work. This is something Taleb should know. But Taleb has proven himself to be a hack and an intellectual parasite. He hasn't actually produced any ideas worth discussing since "Fooled By Randomness" which was nearly 20 years ago. Basically all of his works since then have been rehashing the same idea, or, as with Skin in the Game, pseudo-intellectual ramblings containing little to no interesting ideas, and what ideas do exist have been rehashed a million times by others. Since then, he has taken to making ad-hominem attacks and nonsensical critiques on others in the public view, often on Twitter. I hate bringing up Trump, as it happens far too often in discussions these days, but some of his methods of "debate" are strikingly similar: strange monikers ("klueless Nate"), schizophrenic jumps between topics/arguments, often indecipherable language, and a ridiculous volume of output. All of this combines to mask the fact that most of the time, he isn't actually saying anything, or at least anything sensible, but by shouting loud enough, long enough, and purposefully making his point difficult to identify, it almost becomes impossible to argue with him, and a non-insignificant number of people will assume he is right. Interestingly, I've also found many of his supporters to be similar to some of Trump's or other figures such as Musk. There is a cult of personality built up there where they have established the person as a visionary first, and therefore anything and everything they do or say is correct. Many of them parrot back the same, shallow catchphrases of his, that rarely contain anything insightful but are general enough that they can be thrown out in any situation: critics or dissidents are "Intellectual yet Idiots" or don't have "Skin in the Game". Note that the arguments of these critics or dissidents are almost never addressed. Unfortunately, the spat between Taleb and Silver (which actually started back prior to the 2016 elections), is exhibit A in why Taleb is an intellectual charlatan, and shouldn't be given the time of day. Silver isn't some perfect person either, but in general he has been a reliable voice in the realms of political prediction, fairly forthcoming on shortfalls, and his most damning criticisms all seem to be built upon critical failures in the understanding of probability, or a willful misrepresentation of his results.
coryfklein 8y ago
Re: the desire to choose a 50% (or other) decision threshold. It doesn't have to be a binary choice. Knowing the distinction between 65% and 80% improves your betmaking capabilities by allowing you to hedge, in some cases guaranteeing a profit [1] regardless of the odds and how they change. [1] https://help.smarkets.com/hc/en-gb/articles/115001431011-How-to-calculate-a-hedge-bet https://help.smarkets.com/hc/en-gb/articles/115001431011-How...
bshanks 8y ago
You could use the Brier score to compare how good various forecasters are at predicting something

318 points by oska 8y ago

If 538 is trying to do aleatory probabilities, then it is kinda like a broken clock - right sometimes, but as a tool for answers about future events of any complexity (i.e. the real world, where actions have consequences, and errors can be grave..), I'd avoid putting one's confidence in something that claims it has a simple/'linear' relationship to the real future. It is also a cop out to claim "everybody who thinks probabilities based on initial poll responses are not representative of eventual votes" must clearly not understand probability. Maybe I am straw-manning or just don't understand the actual disagreement. As a poll aggregator? I'm sure it's probably a fabulous tool for past, historical, observed data. The brand itself, probably even more valuable, as a symbol with some sort of social authority in certain circles, I guess. Anybody who reads my comment history will see I have obviously become a Taleb stan, but his approach seems right to me: "show me the money". Which in his case means mathematical papers of proofs primarily, actual money, second.

wand3r 8y ago
I mean obviously its hard to make predictions of complex things as the data from every angle is difficult to get and sometimes incorrect. I used to love Taleb but he is more of a broken cloc, many say if he stopped writing after black swan he would have gone down as great but some of his other points and politics have undermined him. I think Silver is actually pretty legit. He was over-hyped, then he was largely panned as a chrlatan but through and through he has been pretty consistent. The Obama elections he rose to prominance but he also did well in the most recent election. The absolute only news outlet saying that Trump had a statistical chance of winning. He hovered around 20-33% at peak while Huffington post told him he was an idiot. Many major outlets didn’t personally attack him obviously but had Trump in low single digits throughout the last weeks
- rsj_hn 8y ago
  Taleb actually makes a great point here and published a pretty cool paper about it. The point is that if a probability of the event changes too much, you can arbitrage it. E.g. assume these are payoff odds and you can sell your position before the event. He then found some nice no-arbitrage conditions that "real" probabilities must meet and showed that Nate's predictive timeline allowed arbitrage. Unfortunately Nate never responded to this claim with anything other than "math is hard" and other nonsense. You can read the paper here: https://arxiv.org/pdf/1703.06351.pdf https://arxiv.org/pdf/1703.06351.pdf
  - lazyasciiart 8y ago
    But 538 was giving predictions for different events each day in the moving "now cast" so there was no arbitrage.
    - ericdykstra 8y ago
      The forecast still fluctuated way too much to be correct, as can be seen in Taleb's paper, which cites the "forecast" and not the "now cast."
  - nl 8y ago
    Is it a great point? He seems to miss the point of what 538 is doing. Their nowcasts are attempts to say what would happen if the election was today, which throws away the time-dependent uncertainty.
  - comex 8y ago
    I don’t have the background knowledge to properly understand the math in that paper. But I do know that you can’t simply set bounds a priori on how much a probability can change after additional information has been gathered; no matter how pathological the swings, you can design a system, and a series of observations of that system, that would make all the forecasts (i.e. conditional probabilities) correct. Thus the paper must be making additional assumptions. As far as I can tell, those include at least that the electoral process can be modeled as Brownian motion, which is a martingale, but is not the only type of martingale; in particular, I’d expect random motion to be a good model of typical polling drift, but a poor model of sudden polling swings caused by news events (which in reality are a large change caused by a single random event, not the sum of a series of small changes caused by independent events that just happen to mostly point in the same direction). I am not sure whether the assumptions also include an estimated value of `s` or volatility; the paper doesn’t seem to explain how it’s calculated, but maybe it can be derived from the raw polling results plus the assumption of Brownian motion? In any case, the paper says nothing explicitly about what data was used to produce the “rigorous updating” graph. I’d love if someone could explain this to me… Subjectively, it’s hard for me to believe that an unbiased forecast would truly be so utterly noncommittal until just before Election Day, or indeed that there’s enough data to answer that question, especially seemingly from just one election result. But my subjective impressions, of course, could be utterly wrong. I’m very curious whether or not this is the case.
cm2012 8y ago
Taleb is just mixing up the nowcast and the forecast, even after being corrected many times.
- zimablue 8y ago
  I think he would say that the "nowcast" is so useless and deceptive to be malpractice, since everyone will interpret it as a forecast, and even if they didn't it's not in any way useful. Which I agree with (unsurprising since I've semi-invented it as an opinion for another person).
  - uncletaco 8y ago
    Silver has said before, at least in the podcast, that everyone at FiveThirtyEight hates the nowcast and its various incarnations but I think there's some pressure from the business side of things (for views, likes, shares, etc) to have something volatile for people to eat see day in and out.
  - thousandautumns 8y ago
    It's only useless and deceptive if you willfully do absolutely nothing to understand what it is, or how it differs from the other models 538 has, which apparently is the case for Taleb.
    - zimablue 8y ago
      Or in other words, you're 99% of people who will ever hear the number
      - thousandautumns 8y ago
        It explains on the website what it is. And if you are going to rail against something, you should understand what you are railing against.

mehwoot 8y ago

He should not have accepted the honor if he didn’t call a winner in any of the states! And he didn't, as far as I recall. I remember him saying something to the effect of "that isn't that impressive, even a simple method looking at polls would have gotten nearly all the states correct". Don't confuse what other people focus on for something he is boasting about.

tareqak 8y ago

As a non-specialist, I like how the discussion boils down the picture at the end: https://cdn-images-1.medium.com/max/1600/1*D_tidaT-fHMY3DRLgjwekw.png https://cdn-images-1.medium.com/max/1600/1*D_tidaT-fHMY3DRLg... in that it is something I can understand.

forkLding 8y ago
Shows the brilliance of a true data scientist/statistician, condensing all the info in one graphic so that a layman can understand.
cm2012 8y ago
Nate Silver always calls the nowcast (blue line) for what it is - a forecast for if the election was held that day. The forecast, made a few days before election day, is the official forecast and the one that beats most other estimates.
- MAXPOOL 8y ago
  I'm just eyeballing it, but it seems like linear trend line of the nowcast (blue line), would be very close to the official forecast (red dot).
- OscarCunningham 8y ago
  That blue line is a record of Silver's forecast, not the nowcast. You can compare them here https://projects.fivethirtyeight.com/2016-election-forecast/ https://projects.fivethirtyeight.com/2016-election-forecast/
chimeracoder 8y ago
That's a really misleading graph, for a couple of reasons: 1. It conflates the forecast with the now-cast. The former is a prediction of what will happen on Election Day (whether that day is six months away or one day away). The latter is a prediction of what would happen if the election were held that day. In theory, those two would converge to the same value on the day of the election, though in practice, that wouldn't actually happen due to computational differences. 2. Silver never says that he "should be judged" by [just] the final result. In fact, he's gone out of his way to say that. The problem is, that's the only point on the graph where we can compare a model (predicted value) to the actual (observed value). There isn't an election on any of the other days, so even if both agreed to look at the now-cast and use that as grounds to evaluate the model, we still would only have one datapoint which is nonzero in both dimensions. In other words, we have a blue line, yes, but we only have a red dot. Taleb wants to extrapolate the red dot into a horizontal blue line, and then use that to judge Silver's model, which is ridiculous.

backpropaganda 8y ago

The difference between epistemic and aleatory uncertainty is not well understood even in the machine learning world. Most applications of machine learning to domains where uncertainty needs to be quantified use aleatory uncertainty, when epistemic uncertainty is the right one. I wish we had better ways to explain these concepts, and perhaps better terms too. Alternatives like intrinsic/extrinsic or irreducible/reducible don't really work either, because they make it seem like it's a dichotomy.

pizza 8y ago
I haven't read any of his works but I have a suspicion that Judea Pearl writes about causality in this niche with that inferential difference in mind.

tracer4201 8y ago

It's not clear to me what the author is getting at. 538 is good at aggregating polls and showing some probability. The author complains that's not enough because a non zero probability of the less probable outcome gives 538 too much cover when the less probable outcome occurs. Isn't that a shared characteristic of any other model dealing with probabilities? Is the author trying to say Nate Silver gets too much spotlight considering he doesn't guarantee outcomes? I didn't sell in 2008 and kept buying. I have the same strategy today. Not sure if that also makes me a guru if that's the bar (as the author applies it to Taleb). Disclaimer: not a mathematician

baddox 8y ago

I appreciate the stuff about two types of certainties, but where are they actually arguing anything remotely relevant like that? As far as I can tell, based on what I see bubble up to my twitter feed and what tweets are displayed in this post, the disagreement is based entirely on Talib’s claim that Silver switched his prediction from 80% to 50%. This really has nothing to do with actually understanding probability or prediction models. It’s just Talib sharing some other pundit’s deliberate misinterpretation of a single phrase deliberately taken out of context in an interview with Silver. I don’t think their disagreement has much to do with science or mathematics. I think the two guys have big online followings, each leaning heavily toward two different political ideologies, and the two leaders just have to try to beat each other up on a Twitter to decide which side is better.

mhb 8y ago

Shouldn't a guy like Taleb be able to propose a bet with Silver that demonstrates he (Taleb) is right?

starchild_3001 8y ago
Best solution proposed here! Put your money where your mouth is Taleb and Silver! Btw, I read 3 of Taleb's books, 1 from Nate Silver. Both are pretty good. Taleb is the more original, but also more often-objectionable.
- dagw 8y ago
  Best solution proposed here So what should the bet be? As far as I can tell they're not disagreeing on anything that can easily be reduced to a bet
  - lixtra 8y ago
    Nate calls the chances like 80:20 Clinton:Trump. Taleb takes the side: 100,000 USD that Trump wins. If Trump wins, Nate pays 500,000 to Taleb. If Clinton wins, than Taleb pays 100,000 USD to Nate. Do this for each election.
    - dagw 8y ago
      But that doesn't seem to be the nature of their disagreement. Taleb's main complaint seems to be about how the swings in the prediction probabilities far out from the event indicates a fundamental flaw in how the underlying model uses probability. That being said it would be a fun little exercise to see who much money would be made if they bet had $1 on every race 538 predicted in 2016 and 2018 using the model odds.
      - mhb 8y ago
        It doesn't need to be a simple bet. It could incorporate the volatility which Taleb might believe shouldn't be present in an unflawed model.

deleted 8y ago

[deleted]

BurritoAlPastor 8y ago

I'm infuriated that the author would allege that Taleb has a better way to do this but just links to a highly technical paper instead of addressing it.

Cd00d 8y ago

I would love to see the Nate Silver point by point response to this post. I think it would be humorous, but also add some insights into his proprietary black box. I listen to the 538 podcast every episode, and I think I agree with cm2012 that this is a case of mixing up the "nowcast" with the actual forecast.

deleted 8y ago

[deleted]

comex 8y ago

The article seems like an oversimplification of the underlying dispute. - It’s not necessary or even useful to define an arbitrary “decision boundary” unless you actually have to make a decision. From a Bayesian perspective, a probability stands for itself: 100% means the event is certain to occur, 50% means you have no idea, and numbers in between convey varying levels of certainty. In reality, 538’s predictions are not true Bayesian probabilities because they don’t take epistemic uncertainty into account, but that’s a totally different issue. - “Wild fluctuations in a prediction” from new information are absolutely a “normal part of forecasting” - sometimes. If I’m planning to flip two coins, the probability of getting two heads is 25%; but once I flip the first coin, the probability changes to either 50% (if I get heads) or 0% (if I get tails). In the case of Comey reopening the investigation, even if the model had included a probability of that happening, it would have been low and thus wouldn’t affect the overall forecast much. But once that low probability became a certainty, you would expect a sudden swing. The real question is whether 538’s predictions are more swingy than they logically should be (particularly earlier on), but again, that’s a different issue.

kingkawn 8y ago
Seems like in addition to whether they're more swingy than they should be, are they being presented in a way that is interpreted as more certain or authoritative than they can possibly claim?
- c3534l 8y ago
  538 provides pretty in-depth discussions of it forecasts and changes in it. Far more than you would expect - and I think because 538 is ultimately a statistics blog at heart, not a political blog.
- azernik 8y ago
  That is actually an issue they've acknowledged and tried to address. People also tended to see percentages and think they reflected vote percents. To address this, for the 2018 election cycle 538 started displaying their odds as numerical ratios (5 in 9, for example) to try to reduce the aura of certainty.

lalaithion 8y ago

> Practically what this means is that they do not predict a winner or looser but instead report a likelihood Yes, this is good. It means that you can evaluate their accuracy to a greater degree than % of times correct. > Further complicating the issue, these predictions are reported as point estimates They've learned from this, and now show their distribution of results very clearly. However, there's nothing mathematically fraught about reporting a prediction as a point estimate. > The problem is that models are not perfect replicas of the real world and are, as a matter of fact, always wrong in some way. ...yes? And so what? This is a fully general counterargument to all of science. > Predictions have two types of uncertainty; aleatory and epistemic. Don't say things like this with 100% certainty when your source itself says "the validity of this categorization is open to debate", and also when your source is Wikipedia. > However, as you can see, there is still a noticable variation of 2–5% of actual proportion to predictions. This is a signal of un-addressed epistemic uncertainty. It also means you cannot take one of these forecast probabilities at face value. Models aren't perfect. Also, 538 is very careful to address their epistemic uncertainty! They also try very hard to not change their models significantly after they publish them, in order to avoid letting their personal biases tinker with the results. This post goes out of it's way a to avoid mentioning that there are pretty well established ways of measuring the accuracy of predictions, instead using things like "look, it's not quite a line" and "look, the line went up and down before the election". If you're interested in actually reading about this from a mathematical viewpoint, read Taleb's paper, which has some actually interesting thoughts on why 538's algorithms are too eager, or read Madeka's paper, which includes comparisons of 538 to other people trying to make predictions, and finds that actually, they were better than most other news sources. https://arxiv.org/pdf/1703.06351.pdf https://arxiv.org/pdf/1703.06351.pdf https://arxiv.org/pdf/1704.02664.pdf https://arxiv.org/pdf/1704.02664.pdf

skolos 8y ago

Author does informative critique of 538. I'm not sure at all that it has anything to do with Silver vs Taleb twitter war though. It's more like premise for expressing author's own thoughts. However I totally dig what 538 does and appreciate their analysis. I understand that there is a complex underlying model with many parameters taken subjectively. The results of Monte Carlo runs of this model are very interesting to me. The fact that they describe distribution of outcomes and not just a single number is important property of a Monte Carlo model and I would have an issue if that wouldn't provide that info. If I would like to analyze complex event that has lots of moving parts and uncertainties I would also build a model to see what kind of distribution I would get. The fact that somebody published results of their model is quite useful. The analogy would be European and American weather models - no one say that their results are exact and everybody understand that uncertainty in result comes from inherent uncertainty of initial state as well as shortcuts and approximations each of the model takes. No one says that the results of these models are useless because of that. And everybody finds it valuable if weather forecast predicts rain tomorrow with 40% chance. So I don't see how political prediction is only valuable (by the words of the author) if it comes without probabilities attached to it.

mturmon 8y ago

This is an intellectually confused post. It is so unclear and muddled that only a charitable reader would suppose the author understands the mathematical and modeling issues here. I’m surprised the author published it in this form - you don’t want to jump into a controversy like this with such an unclear discussion. As just one example, the whole digression on a “decision boundary” is conceptually mistaken. Once you report a posterior probability, it’s up to the user to establish a simple real number threshold for placing a bet (if we’re talking about means to put skin in the game). The posterior you just learned holds all information, your only action is to threshold it. That’s the Neyman-Pearson lemma. If you allow generalizations to interval-valued probability, which might be sensible under these conditions, the situation gets more complicated. But the writer of the post did not mention this. Another place this came out is his initial discussion of aleatory vs. epistemic uncertainty — this can be done clearly, but here we read: > Aleatory uncertainty is concerned with the fundamental system (probability of rolling a six on a standard die). Epistemic uncertainty is concerned with the uncertainty of the system (how many sides does a die have? And what is the probability of rolling a six?). This is just not helpful. He has said “probability of rolling a six” twice. Source: do UQ in day job.

hn_throwaway_99 8y ago
Agreed. The more I read the more I got embarrassed for the author, mainly because his points were so ridiculously flimsy. I thought this was one of the worst: > Because FiveThirtyEight only predicts probabilities, they do not ever take an absolute stand on an outcome: No ‘skin in the game’ as Taleb would say. This is not, however, something their readers follow suit on. In the public eye, they (FiveThirtyEight) are judged on how many events with forecasted probabilities above and below 50% happened or didn’t respectively (in a binary setting). Saying "this is not something their readers follow suit on" - only if you're a reader who doesn't understand probability. In fact, Silver is constantly trying to get across how likely < 50% probabilities are with his football analogies (e.g. "a team down by 3 at the half actually winning") to emphasize how likely something like "20%" actually is. If anything, I love Silver's reality-based approach exactly because he does acknowledge that all he can do is offer a best estimate based on available evidence, instead of saying things with false-certainty and then basking in self-proclaimed genius when luck happened to run in his favor.

potatofarmer45 8y ago

Nate Silver built an aleatory model whereby low probability high effect events (Comey reopening the investigation on eve of the election) are discounted because if you add enough of these potential disruptive events the model will be 50%-50% and that has no value to anybody. Who wants to read an article that said "election forecast 50-50" every day. If a 538 model says one outcome is 75% probable, it really should come with a giant caveat saying "at current trends assuming nothing out of the ordinary occurs". Taleb's beef is that if that is the case, the 75% does not mean in reality this result is 75% likely to happen. What is a really interesting takeaway is that if in an event where the outcome you desire based appears unlikely, your job is introduce as much previously undefined or discounted uncertainty. Ideally you engineer a black swan event or at least do what you can to make it happen. This would be fun to model from a game-theory approach instead.

wtallis 8y ago
> whereby low probability high effect events (Comey reopening the investigation on eve of the election) are discounted because if you add enough of these potential disruptive events the model will be 50%-50% and that has no value to anybody. I don't think you can say this. The challenge is that it's impossible to even enumerate all the possible surprises that could drastically swing an election, and incorporating some of them into a model would require pulling numbers out of your ass for how to weight those unpredictable possibilities, and even picking which potential surprises to factor in is a similarly arbitrary decision for which there is insufficient evidence to provide guidance. But none of that means that attempting to factor in such possibilities will drive your model's predictions toward 50%; your predictions could end up almost anywhere in the unit interval depending on the value of the priors you pulled out of your ass, and your final number ends up saying more about your biases than about the state of available predictive evidence.
waterhouse 8y ago
> What is a really interesting takeaway is that if in an event where the outcome you desire based appears unlikely, your job is introduce as much previously undefined or discounted uncertainty. Ideally you engineer a black swan event or at least do what you can to make it happen. This would be fun to model from a game-theory approach instead. In chess, if you are behind, John Nunn's two recommended strategies are "grim defence" (if your opponent's advantage is not so large as to make it easy for them to force a win) and "create confusion": create complicated tactical situations and hope your opponent makes a mistake. The farther behind you are, the more appealing "create confusion" gets by comparison.

empath75 8y ago

Five thirty eight is pretty consistent with saying their forecasts more than a few days out are unreliable. They’re fairly accurate with with their final forecasts, though.

akivabamberger 8y ago

Why does this not just link to the Twitter war?

cperciva 8y ago

The author doesn't seem to understand how the 538 model works. The 538 model does take epistemic uncertainty into account; it just doesn't call it that. The model calls it "likelihood of polls moving by X% between now and election day". In 2016 they offered a "now-cast" which didn't include that, and answer the question "based on where the polls are right now, how likely is any given outcome if the election were held today?" while the "classic" model answered the question "based on where the polls are right now and the fact that unanticipated things will happen between now and the election, how likely is any given outcome?". For 2018, they dropped the "now-cast", because people didn't understand what it was saying. True, the epistemic uncertainty is limited -- it's based on past elections, because hey, 538 works with actual data. One could argue that rather than saying "71.4% Clinton 28.6% Trump", a better model would have said "71.3% Clinton 28.4% Trump 0.2% Election is cancelled due to nuclear war / natural disaster / etc" but I don't think anyone sensible is interpreting the model as excluding such extreme outcomes.

pmart123 8y ago

I liked the first chapter of "Fooled by Randomness," but I stopped reading the book when each chapter regurgitated the same points, each more nihilistic than the last. Perhaps Taleb should concentrate on the accuracy of his own predictions: https://www.businessinsider.com/taleb-every-single-human-being-should-short-treasuries-2010-2 https://www.businessinsider.com/taleb-every-single-human-bei... >>> "every single human being" should bet U.S. Treasury bonds will decline. >>> Short the S&P vs Long Gold, in a 5 to 1 ratio. By gold Taleb means a basket of precious metals including gold. The second trade would have lost $125k for every $10k invested in gold and $50k invested in SPY. Did he miscalculate the risk of a black swan event...?

_imba_ 8y ago
He gets predictions more than most. You should go read that book. There are many chapters on the incorrect weight people add to the outcome of correct predictions (or incorrect predictions, if that is what suits their agenda).
- pmart123 8y ago
  For me, it'a less about the accuracy of his predictions, and more about his intellectual smugness rubbing me the wrong way. He has no problem with the hypocrisy of attacking Nate Silver over some supposed weaknesses in his election prediction model, but does not seem to care about the inaccuracies of his own public predictions. Nate Silver took political polls and hunches, which were far less accurate, and came up with something significantly better. Even in 2016, Fivethirteight's model showed the odds of Trump winning were not that remote versus major media outlets "guessing" the race was all but over. Election prediction models will always be inherently unstable in my mind. It's pretty much impossible to predict the when, if, how, who of an Anthony Weiner type scandal.
Animats 8y ago
I'd be more impressed with Taleb if he published the results for his Emperica fund for years other than 2008. Taleb's strategy was to buy options way out of the money. In boring years, the fund lost money. It did really well in 2008. Not enough numbers have been published to determine if his strategy was a net win over a business cycle. The win seems to have coming from the fund's short lifetime including 2008.

deleted 8y ago

[deleted]

einarvollset 8y ago

https://www.theguardian.com/books/2018/feb/25/skin-in-the-game-by-nassim-nicholas-taleb-digested-read https://www.theguardian.com/books/2018/feb/25/skin-in-the-ga...

013a 8y ago

The whole 2016 situation feels like many people are borderlining on delusion, Nate included. The Comey thing certainly helped Trump, but even the day of the election 538 was still forecasting a massive win for Hilary. How can the probabilities be that wrong? Because the polls were wrong. Or, at least, incomplete. It doesn't matter how good your algorithms are if the data isn't there. And 2016 proved that 538, and really most media outlets, don't have a grip on America. In other words, the existence of a black swan event in that particular system didn't really matter. Silver is a glorified modern fortune teller, who seems to have deluded himself into thinking that simply having data means you can prescribe meaningful probabilities to the future. Assigning probabilities to a coin flip is easy. Figuring out the most likely winner of a game of baseball is a little harder; there's a whole movie about a guy who was basically doing that in 2002. Politics is a completely different field. You can't possibly assemble all of the data necessary to be remotely confident in the probability of outcomes. And let's not forget unpredictable black swan events. But the readers love it. And, especially during the elections, the media would bring him out like a golden boy computer whiz, because he was saying the same things they were, but he's really smart and has the data and magic algorithms to back him up. And he was right that one time 8 years ago. He's about as pointless as Sean Hannity, with the one exception that, at least recently, Hannity was actually more correct about the future than Nate. It doesn't matter if you are right or wrong, if the reasons are wrong.

wslack 8y ago
I don't think delusion is a fair take: FiveThirtyEight Prediction: 48.5% to 44.9% Real outcome: 48.2% to 46.1% I agree that the horserace coverage is silly (especially given how polling operations use sampling and such), but its better to look at aggregate trends in polling than to ridiculously overcover outlier polls. I also certainly don't think its fair to say "28.6% chance he wins" is a "massive win" prediction. Silver's take was regarded as indefensibly right-leaning and attacked.
nsgoetz 8y ago
538 had almost a 1 in 3 chance that Trump won in their final prediction before the election. https://projects.fivethirtyeight.com/2016-election-forecast/ https://projects.fivethirtyeight.com/2016-election-forecast/

c1ccccc1 8y ago

I'm confused by the existence of this article. Comparing the accuracy of two probabilistic predictors is pretty much a solved problem. (At least in situations where the two predictors have both made predictions about a fairly large number of events in the same set, S.) Just add up the log scores for both predictors and see which does better. [1] [1] https://en.wikipedia.org/wiki/Scoring_rule#Logarithmic_scoring_rule https://en.wikipedia.org/wiki/Scoring_rule#Logarithmic_scori... Another thought: Although the author didn't lie about its meaning, I thought the graph "Stated Probabilities Compared with Average Portions" was visually misleading. At a casual glance, it appeared that 538 was all over the place with its senate predictions. But if you look closer, you can see that many of the red data points are at 0 or 0.5 or 1. This suggests that these data points represent the outcome of just 1 or 2 elections, which isn't really enough to say that 538 is doing particularly badly.

IIAOPSW 8y ago

Why should we care about any twitter war? Twitter is stupid. If it wasn't for a certain #1 individual user, the company would possibly be in bankruptcy by now.

macawfish 8y ago

I'm still waiting for somebody to talk about how tightly coupled Nate Silver's forecasts are to the political processes he attempts to forecast. What if Nate Silver were to run around on the football field before each play whispering in the players' ears about the how likely they are to win, according to his latest forecast. Imagine that they believed in him, or at least felt some degree of superstition when he came around. I just think there's something innately flawed about public forecasting of an election. Mathematically and maybe even ethically. I'm still trying to figure out how to articulate why I feel that way. It could even be argued that polling itself could influence opinions by introducing bias at a critical moment when someone is being asked to consider who they are voting for. What if that's the moment when they make up their mind? P.S. https://www.frontiersin.org/articles/10.3389/fphy.2015.00077/full https://www.frontiersin.org/articles/10.3389/fphy.2015.00077...

dagw 8y ago
I just think there's something innately flawed about public forecasting of an election. It used to be the case in France that you weren't allowed to publish polls two weeks before an election for just this reason. They eventually scrapped this with the argument that the all 'elites' had access to internal and private polls anyway so all you where doing was denying the 'people' the same access to information that the 'elites' had. It could even be argued that polling itself could influence opinions by introducing bias at a critical moment when someone is being asked to consider who they are voting for. This is a pretty well known effect, to the point that many political strategists try to avoid publishing or drawing attention to polls showing their candidate too far ahead in case it makes voters complacent and decide to stay home. The ideal situation is to make people convinced the polls show the candidates tied 50-50 going in to the election and that 'your' vote could easily swing the whole election.
- mijamo 8y ago
  This is slightly wrong. It was initially one week, not two. It was changed because it was unconstitutional, as it limited freedom of expression too much, not because it was useless or anything to do with elite vs people. And it still exists, limited to the day of the election and the day before. Source: https://www.conseil-constitutionnel.fr/nouveaux-cahiers-du-conseil-constitutionnel/la-reglementation-des-sondages-et-l-election-presidentielle https://www.conseil-constitutionnel.fr/nouveaux-cahiers-du-c...

gaius 8y ago

No one should care about anything that happens on Twitter. It’s that simple.

zimablue 8y ago

To try and explain Taleb's argument as simply as possible: If someone tells you today that an event is 100% certain, and tomorrow tells you that it is impossible, then you intuitively will not trust their forecasting. If it's certain, then it's certain not to be impossible tomorrow Now if someone tells you something is a 90% chance today, and a 10% chance tomorrow, again you don't trust their predictions. There is a probability associated with "changes in probability". The probability of going from 100 to 0 should intuitively be 0%. The probability of going from 90 to 10 intuitively must be something like 20% (it's like 80 of the 90s "didn't happen", with a probability of 20). A key point is that you can add this up every day, if something goes from 90->10->90 then it's even more unlikely than going 90->10. So if you extend that intuition, put some real maths behind it, you can tell whether something is "likely to be a real probability" by the rate at which it changes. And if you have lots and lots of repeated predictions, you can be confident that something isn't a good probability (every timeseries has like a <10% chance of doing exactly what it does, so their combined probability [of being good probabilities] is like 0). Now Nate Silver accepts this, but says that he's not actually putting a probability on the event in the future, but some non-existent "probability on that event in the future if the future was now". But that doesn't correspond to anything useful or measurable (it's untestable!!), and most people will assume it follows the normal meaning of probability, and it's honestly just silly.

OscarCunningham 8y ago
> Now Nate Silver accepts this, but says that he's not actually putting a probability on the event in the future, but some non-existent "probability on that event in the future if the future was now". Could you link to where Silver discusses this? I'm interested in seeing his description of exactly what his numbers mean.
- zimablue 8y ago
  https://fivethirtyeight.com/features/a-users-guide-to-fivethirtyeights-2016-general-election-forecast/ https://fivethirtyeight.com/features/a-users-guide-to-fiveth... What I'm referring to is the "now-cast", but his other two definitions both seem to shy away from saying "this is flat-out the probability we think of the election". The point is, you can redefine or choose a definition of probability if you want, but if it's less useful than the normal definition (and confusing to people!) then people are free to criticize your work on that basis. And there's a very useful, testable, mathematical definition of probability that allows us to equally assess everyone's predicting ability, and Nate Silver is dodging it. If you're interested in this subject, there's a non-mathematical discussion somewhere in Tetlock's book Superforecasting which is interesting in general.

OscarCunningham 8y ago

I think the difference in opinion is caused by Taleb modelling the election as though all of the uncertainty is caused by the fact that people might change their minds, whereas Silver is also including the uncertainty from polling errors. Silver's predictions fluctuate a lot as time passes. Taleb is assuming that all of these fluctuations are caused by the electorate changing who they're going to vote for. Then he says "if you knew people's opinions were as volatile as this, you shouldn't have been so confident earlier!", or words to that effect. If lots of people will change their minds, then early polling won't be very informative about the final result. But the fluctuations in Silver's predictions are actually caused by his changing certainty about how accurate the polls are. Silver does take into account that people might change their minds, but this isn't the cause of most changes in his predictions. So people actually change their minds a lot less often than Taleb's model of Silver's model suggests. This is why Silver can make strong predictions early on.

lambdadmitry 8y ago

I'm profoundly confused by the deep respect Taleb commands in the eyes of many people here, apparently. The guy has a few good ideas that he repeats ad nauseam, but he punches way over his own intellectual weight too often. A few especially egregious examples are his deranged GMO hate "paper" which is profoundly ignorant of the involved biology [1], his support of Syrian government's slaughter of civilians [2][3], and his Putinversteher views ([4] reads as a parody for anyone knowing anything about Putin and modern Russia). Combined with his unlimited arrogance, I don't understand how people can respect him intellectually. [1]: http://www.fooledbyrandomness.com/pp2 http://www.fooledbyrandomness.com/pp2 [2]: https://medium.com/opacity/the-syrian-war-condensed-a-more-rigorous-way-to-look-at-the-conflict-f841404c3b1d https://medium.com/opacity/the-syrian-war-condensed-a-more-r... [3]: https://medium.com/@ameraidi/nassim-taleb-is-wrong-on-syria-e1faa523824c https://medium.com/@ameraidi/nassim-taleb-is-wrong-on-syria-... [4]: https://twitter.com/nntaleb/status/773323163509850112 https://twitter.com/nntaleb/status/773323163509850112

jimnotgym 8y ago

All of which increases the probability that I will ignore Taleb's tweets. I follow people who are smart enough to disagree in a civil manner, and smart enough to recognise that there may be things one doesn't know. The way he talked to renowned classicist (and feminist, which seemed to be a problem for Taleb) Dame Mary Beard in another spat was disgusting. Trying to impose statistical certainty on ancient history ended up making him look pretty dumb to everyone but him.

JabavuAdams 8y ago
The problem is that the smart arrogant jerk may have the right answer sometimes when it counts, while being a pain in the ass the rest of the time. Better just to stay open to all feeds. He does come off as an arrogant dick, wow.
- alexandercrohde 8y ago
  I used to look at things this way, with a "just filter out the rudeness" mindset. However in practical terms I think listening to somebody's rudeness does have a really big cost (to estimate: how much would you need to be paid to make it worth your time to listen to rudeness for an hour?)

lavrov 8y ago

I thought that the post was fairly unclear, but for me, it seems like the main argument against 538 is that it makes unfalsifiable claims about the probability of individual elections - picking a winner is a falsifiable claim, but assigning a probability always allows Silver to claim something like “even events with a 10% probability occur frequently” even if his model assigns a high likelihood of victory to the loser of an election. My favorite Nate Bronze takedown remains Carl Diggler[1]. [1] https://www.washingtonpost.com/amphtml/posteverything/wp/2016/05/09/our-fictional-pundit-predicted-more-correct-primary-results-than-nate-silver-did/ https://www.washingtonpost.com/amphtml/posteverything/wp/201...

vertline3 8y ago

Taleb's biggest thing is exposing holes in predictive models. Silver's biggest thing is making and using predictive models. So it's only natural for these two to quarrel.

alexandercrohde 8y ago

So does Taleb propose a better betting strategy? I'd like to see him put his money where is mouth is, rather than just his reputation.

mschuster91 8y ago

Of course the US elections are not as accurately predictable as e.g. German ones, or sports events. Not simply due to different pollster sources and their data quality (which Nate Silver tries to account for). The biggest and most unpredictable factor in US (and British) elections is their winner-takes-all system, with the second layer of the Electoral College in the US which does not account for popular votes either.

thousandautumns 8y ago

I've been following this spat since it reignited several weeks ago, but also since Taleb first attacked Nate on Twitter back before the 2016 elections, and it has been the final nail in the coffin as to any supposed credibility Taleb commands. The quandary that FiveThirtyEight and Silver are in is that the general public is stupendously bad with probabilities. Prior to the the 2016 election, Silver was personally being attacked, in some cases by major media organizations like the Huffington Post, and accused of tipping the scales towards Trump (for some inexplicable reason), since their models were giving him approximately a 1 in 4 chance of winning, since everyone knew it was a certainty Clinton would win. Then, in the months following the election, the narrative somehow switched, and the fact that Trump won despite only being given a 1 in 4 chance by FiveThirtyEight meant that Silver was now a hack and the model was wrong. This same criticism has been leveled by many right-wing media personalities, who had been trotting out Silver's 2016 predictions leading up to the election, as well as by the New York Times, whose own models were giving Clinton nearly a 90% chance of victory. So now, fast forward to this year's election, Silver is doing his best to drill into people's minds that, yes, their model showed a Democratic win in the house as the most likely outcome, but that absolutely doesn't not mean their model says it is an absolute certainty, or that other outcomes would be unusual. It isn't covering his ass, its educating the public on how probabilistic statements work. This is something Taleb should know. But Taleb has proven himself to be a hack and an intellectual parasite. He hasn't actually produced any ideas worth discussing since "Fooled By Randomness" which was nearly 20 years ago. Basically all of his works since then have been rehashing the same idea, or, as with Skin in the Game, pseudo-intellectual ramblings containing little to no interesting ideas, and what ideas do exist have been rehashed a million times by others. Since then, he has taken to making ad-hominem attacks and nonsensical critiques on others in the public view, often on Twitter. I hate bringing up Trump, as it happens far too often in discussions these days, but some of his methods of "debate" are strikingly similar: strange monikers ("klueless Nate"), schizophrenic jumps between topics/arguments, often indecipherable language, and a ridiculous volume of output. All of this combines to mask the fact that most of the time, he isn't actually saying anything, or at least anything sensible, but by shouting loud enough, long enough, and purposefully making his point difficult to identify, it almost becomes impossible to argue with him, and a non-insignificant number of people will assume he is right. Interestingly, I've also found many of his supporters to be similar to some of Trump's or other figures such as Musk. There is a cult of personality built up there where they have established the person as a visionary first, and therefore anything and everything they do or say is correct. Many of them parrot back the same, shallow catchphrases of his, that rarely contain anything insightful but are general enough that they can be thrown out in any situation: critics or dissidents are "Intellectual yet Idiots" or don't have "Skin in the Game". Note that the arguments of these critics or dissidents are almost never addressed. Unfortunately, the spat between Taleb and Silver (which actually started back prior to the 2016 elections), is exhibit A in why Taleb is an intellectual charlatan, and shouldn't be given the time of day. Silver isn't some perfect person either, but in general he has been a reliable voice in the realms of political prediction, fairly forthcoming on shortfalls, and his most damning criticisms all seem to be built upon critical failures in the understanding of probability, or a willful misrepresentation of his results.

coryfklein 8y ago

Re: the desire to choose a 50% (or other) decision threshold. It doesn't have to be a binary choice. Knowing the distinction between 65% and 80% improves your betmaking capabilities by allowing you to hedge, in some cases guaranteeing a profit [1] regardless of the odds and how they change. [1] https://help.smarkets.com/hc/en-gb/articles/115001431011-How-to-calculate-a-hedge-bet https://help.smarkets.com/hc/en-gb/articles/115001431011-How...

bshanks 8y ago

You could use the Brier score to compare how good various forecasters are at predicting something