10 ms·
Agreed. The more I read the more I got embarrassed for the author, mainly because his points were so ridiculously flimsy. I thought this was one of the worst:
by hn_throwaway_99 8y ago
Agreed. The more I read the more I got embarrassed for the author, mainly because his points were so ridiculously flimsy. I thought this was one of the worst:
> Because FiveThirtyEight only predicts probabilities, they do not ever take an absolute stand on an outcome: No ‘skin in the game’ as Taleb would say. This is not, however, something their readers follow suit on. In the public eye, they (FiveThirtyEight) are judged on how many events with forecasted probabilities above and below 50% happened or didn’t respectively (in a binary setting).
Saying "this is not something their readers follow suit on" - only if you're a reader who doesn't understand probability. In fact, Silver is constantly trying to get across how likely < 50% probabilities are with his football analogies (e.g. "a team down by 3 at the half actually winning") to emphasize how likely something like "20%" actually is.
If anything, I love Silver's reality-based approach exactly because he does acknowledge that all he can do is offer a best estimate based on available evidence, instead of saying things with false-certainty and then basking in self-proclaimed genius when luck happened to run in his favor.
- VintageCool 8y agoThis "Twitter fight" began with Nate Silver talking to the media, trying to convey both his prediction and its uncertainty: https://thehill.com/homenews/sunday-talk-shows/414759-nate-silver-says-dems-could-retake-house-or-not-both-extremely https://thehill.com/homenews/sunday-talk-shows/414759-nate-s... > "So in the House we have Democrats with about a 4 in 5 chance of winning," Silver told ABC's "This Week." > "But no one should be surprised if they only win 19 seats and no one should be surprised if they win 51 seats," Silver added. "Those are both extremely possible, based on how accurate polls are in the real world." Dinesh D'Souza jumps on him for hedging his bets: > And just like that, an 80 percent chance for Democratic takeover of the House goes to 50-50. Nassim Taleb also jumps on him in the same way: > @DineshDSouza is right: you don't change a forecast from 80% to 50% under uncertainty. Second time klueless Nate Silver makes a mistake. > When someone says are event and its opposite are extremely possible I infer either 1) 50/50 or 2) the predictor is shamelessly hedging, in other words, BS. > 4- Things are actually worse: that @NateSilver538 doesn't get that if BOTH X & Non-X are "extremely possible = Probability converging to 50-50 is EXACTLY my problem w/his misunderstanding of probability in forecasting, & point of my paper. > 5- FOR THE RECORD I am not translating @DineshDSouza's point that it was 50-50: I ALSO understood ~50-50 DIRECTLY from Silver's "extremely" in the linked "The Hill". The Twitter fight happened on Nov 4, if you want to go look. In this article, the author notes that 538 only presents its model to readers in terms of probabilities, but the public tends to round their predictions up and down. > This is not, however, something their readers follow suit on. In the public eye, they (FiveThirtyEight) are judged on how many events with forecasted probabilities above and below 50% happened or didn’t respectively (in a binary setting) This isn't the public's fault, says the author, it's 538's fault for not clearly presenting the uncertainty in the prediction. Additionally, the author writes, the 538 polling-based model swung too wildly in the 2016 election. In doing so, it failed to present a single prediction on which it could be judged. It also failed to acknowledge that unforeseen events could change the polls. Personally, I'd argue that the swinginess of the conveyed the uncertainty of the polls. I spent the last month of 2016 citing Nate Silver to argue on Facebook and Reddit that people were overestimating the certainty of the election forecasts. Fifteen percent of the electorate was up for grabs with just two weeks left in the election! (about 7.5% undecided; and 7.5% saying they'd vote 3rd party) > This is Taleb’s primary argument; FiveThirtyEight’s predictions do not behave like probabilities that incorporate all uncertainty and should not be passed off as them. Nate Silver is being blamed both for trying to convey to the public that his predictions involve uncertainty and for the public not perceiving the uncertainties in his predictions.
- scythe 8y ago>In this article, the author notes that 538 only presents its model to readers in terms of probabilities, but the public tends to round their predictions up and down. The only way you could get "50-50" from that is if you read "the Democrats get 51 seats" as "the Democrats win the House", because without that inference, he didn't say anything about the chance that they win the House. He compared a very lopsided win to a slight loss, and those were equally likely.
- JadeNB 8y ago> He compared a very lopsided win to a slight loss, and those were equally likely. I'm not arguing with you, since the one thing I know is that there's no surer way to making a mistake than jumping in confidently on probabilistic issues, especially issues of interpretation rather than of pure mathematics; but I think you must have misspoke, or I must have misunderstood. The quoted sentence says 'extremely probable' (an absolute condition), not 'equally likely' (a relative condition): > "But no one should be surprised if they only win 19 seats and no one should be surprised if they win 51 seats," Silver added. "Those are both extremely possible, based on how accurate polls are in the real world."
- scythe 8y agoNot "extremely probable" -- it says "extremely possible", which is a malapropism or a new phrase. It's easy to mistake because it has no clear meaning. The comparison, though, does not appear to be D's win vs D's lose, but D's win by more than X vs D's lose, and the latter clearly (imo) suggests D is favored to win.
- JadeNB 8y ago> Not "extremely probable" -- it says "extremely possible", which is a malapropism or a new phrase. It's easy to mistake because it has no clear meaning. Indeed, despite quoting it in my response, I made exactly that mistake. Thanks!
- 8y ago
- Kurtz79 8y ago"Saying "this is not something their readers follow suit on" - only if you're a reader who doesn't understand probability." Unfortunately that is the more common case, and I agree on the author on that that most people take a binary stance on polls. It should not be the case for regular readers of a website whose main topic is statistical analysis, although I imagine that just before a major election they would see a large number of non-regulars just to check the polls.
- hn_throwaway_99 8y agoFine, but Silver goes to extreme ends in my opinion to explain what a probability means. During the 2016 presidential election he was emphasizing "Hey, it's less likely than the alternative, but there is a real possibility Trump will win. There is a very plausible path for him to wrap up the electoral votes." And afterwards, I think the results showed how accurate Silver was - he gave a much higher probability to Trump winning than anyone else IIRC, and the outcome was pretty much that: An unlikely probability (many of the deciding states were very much within any possible margin of error) that actually happened.
- afthonos 8y agoEven more confidence-inspiring: before the election, they had discussed what a path to victory for Trump would look like, and their most likely path is what ended up happening: the midwest went for Trump, in a correlated manner, which is exactly what other forecasters had failed to account for properly. If your model can help you predict how it's going to fail, I'm fairly impressed.
- scott_s 8y agoI agree. I think Silver laid it out clearly in a piece that people called him a troll for writing: Election Update: Yes, Donald Trump Has A Path To Victory, https://fivethirtyeight.com/features/election-update-yes-donald-trump-has-a-path-to-victory/ https://fivethirtyeight.com/features/election-update-yes-don.... Another political writer at 538, Harry Enten, had a similar story a few days later, Trump Is Just A Normal Polling Error Behind Clinton, https://fivethirtyeight.com/features/trump-is-just-a-normal-polling-error-behind-clinton/ https://fivethirtyeight.com/features/trump-is-just-a-normal-... I also listen to their politics podcast, and they constantly talk about how to prevent people from rounding up an 80% chance to a 100% chance. One change they made this cycle was to try say 4 out-of 5 chance (which is mathematically the same) to convey both the lack of precision and to make it more intuitive. I think it helps. Personally, when I talked to people about election outcomes, I phrased it as "Based on what I know, I will be surprised-but-not-shocked if x happens, and shocked if y happens" to differentiate between, say, a 20% chance and a 1% chance.
- coldtea 8y ago>Saying "this is not something their readers follow suit on" - only if you're a reader who doesn't understand probability Isn't that the very readership they capitalize on?
- bryanrasmussen 8y agoAs said he tries to help his readership understand how probability works, but in the end one may still ethically do business in one's chosen field without having to solve the problems of understanding the limits of that field for those uneducated in it. on edit: grammar was atrocious, still bad now.
- ggambetta 8y agoAuthor also doesn't seem to understand what "skin in the game" means.
- killjoywashere 8y ago> basking in self-proclaimed genius when luck happened to run in his favor. I have read read Black Swan and I think Taleb's central thesis is there are valid epistimic gaps in every model, and that suggest there is at least one family of hedge strategies where the goal is to think as creatively as possible about what we don't know, but what could be, so as to reframe that lack of knowledge as a plan to go tromping around in a jungle looking for holes. But to do so requires wide-ranging knowledge across many domains, which is hard to develop.
- mturmon 8y agoThis is a smart observation. There has been some theoretical work on the intersection of game theory and uncertainty quantification [1]. One part of it is a mathematical take on what you described - you have a system which is not pricing risk correctly, due over-confidence in model predictions. If so, there should be money to be made in placing bets on that system. [1] https://arc.aiaa.org/doi/pdf/10.2514/6.2017-1092 https://arc.aiaa.org/doi/pdf/10.2514/6.2017-1092
- golergka 8y ago> Saying "this is not something their readers follow suit on" - only if you're a reader who doesn't understand probability. But overwhelming majority of people don't.