12 ms·

> the election were to happen today, what is the probability of each candidate winning? He's explicitly not doing this. Here's how I think about it. Silver is

> the election were to happen today, what is the probability of each candidate winning? He's explicitly not doing this. Here's how I think about it. Silver is answering the question: "how much would the polls (as an aggregate) need to differ from the final result in order for Candidate X to win/lose, conditioned on some reasonable priors?" Taleb is pointing out that the polls could be really wrong in all sorts of ways that are impossible to predict a priori. The whole argument is sort of pointless from an intellectual/academic perspective. It's a war of public personalities more than anything else. It's both the case that Silver designed a good piece of software that does what it's supposed to do and also the case that Taleb's skepticism is valid. But then, that sort of skepticism of statistical models is always valid, and yet we use these models to great effect in all sorts of settings.

coliveira 6y ago
Taleb made a career from stating the obvious and making it appear as a smart discovery. I've read some of his writings and have never found anything really new.
- seibelj 6y ago
  Totally agree, he comes off as a hack to me. But he’s famous and I’m not, so shrug
  - bitdizzy 6y ago
    Taleb is seriously proficient at probability which can be judged from his technical publications.
- sillysaurusx 6y ago
  The same is true of pg, but his discoveries seem obvious only in hindsight. (I’m objecting to “the discoveries are obvious” as a putdown. The simplest discoveries are some of the hardest. Though I doubt this matters for political pundit analysis.)
  - JumpCrisscross 6y ago
    > The same is true of pg Y Combinator is a success. I find PG’s writings fantastic, but sure, some of it is more good writing than concept building. Taleb has no Y Combinator. His hedge fund failed. He’s just a talking head. That removes an element of grounding from his words.
    - t-writescode 6y ago
      Does it? I have no horse in this race, but with the rarity of a unicorn, it could be what amounts to luck. Out of billion people flipping a coin 30 times, a few of them may guess correctly all 30 times; but the books they write on coin flip guessing amount to meaningless drivel. My point is that success or failure in such a complex, edge-case, luck-based world may have very little to do with knowledge. PG, if you’re reading this, of course I mean no disrespect and I know little of the topic and nothing of this person at hand except that he seems to have run a failed hedge fund.
- raverbashing 6y ago
  Frequentists miss the obvious stuff all the time, Taleb is not smarter, it's other people who are tied in dogmatic thinking
- rramach 6y ago
  My favorite story when I hear someone dismiss an argument saying 'that is obvious': Lazarsfeld was writing about “The American Soldier”, a recently published study of over 600,000 servicemen, conducted by the research branch of the war department during and immediately after the second world war. To make his point, Lazarsfeld listed six findings that he claimed were representative of the report. Take number two: '“Men from rural backgrounds were usually in better spirits during their Army life than soldiers from city backgrounds.” “Aha,” says Lazarsfeld’s imagined reader, “that makes perfect sense. Rural men in the 1940s were accustomed to harsher living standards and more physical labour than city men, so naturally they had an easier time adjusting. Why did we need such a vast and expensive study to tell me what I already knew?” Why indeed. But Lazarsfeld then reveals the truth: all six of the “findings” were in fact the exact opposite of what the study found. It was city men, not rural men, who were happier during their army life. Of course, had the reader been told the real answers in the first place, they could just as easily have reconciled them with other things they already thought they knew: “City men are more used to working in crowded conditions and in corporations, with chains of command, strict standards of clothing, etiquette, and so on. That’s obvious!” But this is exactly the point Lazarsfeld was making. When every answer and its opposite appears equally obvious then, as he put it, “something is wrong with the entire argument of ‘obviousness'” More here: https://www.newscientist.com/article/mg21128210-100-the-human-paradox-that-is-common-sense/ https://www.newscientist.com/article/mg21128210-100-the-huma...
  - amscanne 6y ago
    Oh yes, I hate this kind of thing so much. People have so many pseudoscientific explanations and theories to support whatever random products and practices they like. I call these “huh, that makes sense” explanations, since that’s often what people say after hearing them. You even used the magic three words above.
    - ashtonkem 6y ago
      The related phenomenon in biology is called the “just so” fallacy; e.g. “X species evolved Y just so that they could do Z”. It’s a comforting story, but it will often lead you astray.
partiallypro 6y ago
I don't know much about Taleb, but one problem I've always had with Silver is that he's very smug about this models but they often enough don't work out. Then when they don't work out, instead of reflecting on his models to improve them (at least publicly,) he just tells people they don't understand probability or his models. His sports models are -especially- bad; no better than a coin flip in some instances. I know there are a ton of Silver fanboys, but surely you can agree he has a pretty big ego, which always makes me take him with a grain of salt. When it comes to politics his former work with Obama's campaign and DailyKos also make me question if he can be truly unbiased in his predictions, but that's a separate issue.
- throwawaygh 6y ago
  >> The whole argument is sort of pointless from an intellectual/academic perspective. It's a war of public personalities more than anything else.* > ...he's very smug... Right, it's just dueling personalities.
- Robotbeat 6y ago
  Whose models are better? There have been years where Nate Silver's model has called every single one of the 50 states correct as far as which POTUS candidate they'd vote for. See: https://mashable.com/2012/11/07/nate-silver-wins/ https://mashable.com/2012/11/07/nate-silver-wins/ He's not perfect (for instance, I think calling the 2016 model too certain early on is valid and I think some other people have basically equivalently-good models, but are less well known), but if confidence is smugness, then he has a right to be.
  - thaumasiotes 6y ago
    > Whose models are better? There have been years where Nate Silver's model has called every single one of the 50 states correct as far as which POTUS candidate they'd vote for. Well, this certainly isn't how you'd go about evaluating model quality. Calling which candidate a state will vote for is not a good task; it's too discrete to be a good evaluator. For the same reason, no political model produces this output. You predict the vote share given to each candidate -- a nice continuous variable -- you don't predict who will win. Calling the state is a gimmick on top of that (since victory is a function of vote share, it's easy to degrade your model's output into victory predictions) which serves only to make the statistics worse.
deleted 6y ago
[deleted]
tempestn 6y ago
I came here to say exactly the same thing. Silver and 538 are explicit about not doing this. Their models do include the uncertainty of future events, and do adjust closer to 50/50 given the same polls the further from the election it is. That's why Biden's odds have been ticking upwards slightly over the past couple weeks despite polls remaining pretty stable. So in theory, the forecast should be exactly what it appears: the percentage of the time the candidate would be expected to win given the situation they are in at the time of the forecast. So if on Oct. 14 it gives Biden a 72% chance, it means that a candidate in Biden's position, with his polling numbers, and the economic conditions and everything else that is the case on Oct. 14, would go on to win from there 72% of the time. I agree that it appears the volatility of the 2016 forecast was larger than you would expect given a correct model behaving that way. I expect that uncertainty is better expressed in the 2020 model. However it's certainly conceivable for such a model to swing correctly. IE, there's nothing impossible about a candidate having a 90% chance of winning on X date, then having circumstances change such that they only have a 50% chance at Y date. (In fact, if you ignore the discontinuity on election day, at least 10% of the time their odds would have to pass through 50% at some point, as they would go on to lose. And of course there would also be scenarios when it would dip to that level or below and yet still win.)
stopachka 6y ago
I think Taleb’s point is a bit more damning — implying the software does not do what it says it would. If all sorts of things can happen, where you predictions change widely, Taleb’s argument is that you can’t use a number like 90%. You have to include the error, which ends up saying “50%” for both before the election. This doesn’t mean that all prediction suffers from this effect, and you have to be skeptical everywhere. I don’t think that’s a fair characterization. Taleb has won by mispriced options. His main statement is that the true probability of this particular event is much closer to 50%
- cyphar 6y ago
  That's what confidence intervals are for. You cannot condense both your expectation value and uncertainty in a single percentage (it is not correct to say that the probability of an outcome is 50% if 80% of likely outcomes result in one candidate winning). Otherwise you'll end up with every 5-week weather forecast saying there's a 50% chance of rain.
  - stopachka 6y ago
    Indeed. Taleb’s point if I understand it correctly is that the uncertainty in this so high that the number comes realistically much closer to 50%.
    - cyphar 6y ago
      We're not in agreement here -- the solution to uncertainty in statistics is to model the uncertainty and treat them as confidence intervals, not throw your hands up and give everything a coin-flip probability. The probability isn't "realistically much closer to 50%" -- that's not how uncertainties in statistics work. It is possible that (if the uncertainty in the model was smaller) that the "real" probability is 50% but there's no way of knowing if that is the case.
      - stopachka 6y ago
        We can try going deeper. I may be making a dumb mistake: when I got to 538 — the main number I see is 89%. I don’t see 89% +- 40. What is his confidence interval?
Tomminn 6y ago
If I am expected to make a better prediction of the election outcome by averaging your last 10 predictions than by taking your current prediction, then your current prediction is suboptimal. This is Taleb's point, and it is solid as a rock.
- moultano 6y ago
  But he provides no evidence that this is the case. The 2016 election was extremely volatile due to what actually happened during it, not the modelling. There were several bombshell press events all throughout the campaign that dramatically shifted polls.
  - ethagknight 6y ago
    That’s exactly Taleb’s point, both in this instance and in every book Taleb has written. Those bombshells should have been included in the model. Fat tail risks. If the modeling doesn’t include the potential for dramatic unknowns that can DO happen, then the model is no good. In the Clinton/Trump election, Trump is a showman and a name caller focused on ratings, Clinton a career politician with accusations of dirt, the model should expect “big things can still happen today, tomorrow, next week” and the absence if big events yesterday or last election holds little value relative to the probability of a big event happening in the next 24 hours. Edit- I’m getting downvoted im guessing because I said Clinton has lots of dirt and called Trump obsessive, I’ve revised the comment to be less politically accusative. I’m not concerned with the politics, just interested in the obsession with these “predictions”
    - mcphage 6y ago
      > If the modeling doesn’t include the potential for dramatic unknowns that can DO happen, then the model is no good. Why do you think it didn't? It took a major news event breaking at the exact right time (too early, and people would realize it was meaningless, too late, and it wouldn't have time to get out). And even with that, Trump barely eked out a win. That seems unlikely-but-not-impossible, which seems to match with 538's estimations.

coliveira 6y ago

Taleb made a career from stating the obvious and making it appear as a smart discovery. I've read some of his writings and have never found anything really new.

seibelj 6y ago
Totally agree, he comes off as a hack to me. But he’s famous and I’m not, so shrug
- bitdizzy 6y ago
  Taleb is seriously proficient at probability which can be judged from his technical publications.
sillysaurusx 6y ago
The same is true of pg, but his discoveries seem obvious only in hindsight. (I’m objecting to “the discoveries are obvious” as a putdown. The simplest discoveries are some of the hardest. Though I doubt this matters for political pundit analysis.)
- JumpCrisscross 6y ago
  > The same is true of pg Y Combinator is a success. I find PG’s writings fantastic, but sure, some of it is more good writing than concept building. Taleb has no Y Combinator. His hedge fund failed. He’s just a talking head. That removes an element of grounding from his words.
  - t-writescode 6y ago
    Does it? I have no horse in this race, but with the rarity of a unicorn, it could be what amounts to luck. Out of billion people flipping a coin 30 times, a few of them may guess correctly all 30 times; but the books they write on coin flip guessing amount to meaningless drivel. My point is that success or failure in such a complex, edge-case, luck-based world may have very little to do with knowledge. PG, if you’re reading this, of course I mean no disrespect and I know little of the topic and nothing of this person at hand except that he seems to have run a failed hedge fund.
raverbashing 6y ago
Frequentists miss the obvious stuff all the time, Taleb is not smarter, it's other people who are tied in dogmatic thinking
rramach 6y ago
My favorite story when I hear someone dismiss an argument saying 'that is obvious': Lazarsfeld was writing about “The American Soldier”, a recently published study of over 600,000 servicemen, conducted by the research branch of the war department during and immediately after the second world war. To make his point, Lazarsfeld listed six findings that he claimed were representative of the report. Take number two: '“Men from rural backgrounds were usually in better spirits during their Army life than soldiers from city backgrounds.” “Aha,” says Lazarsfeld’s imagined reader, “that makes perfect sense. Rural men in the 1940s were accustomed to harsher living standards and more physical labour than city men, so naturally they had an easier time adjusting. Why did we need such a vast and expensive study to tell me what I already knew?” Why indeed. But Lazarsfeld then reveals the truth: all six of the “findings” were in fact the exact opposite of what the study found. It was city men, not rural men, who were happier during their army life. Of course, had the reader been told the real answers in the first place, they could just as easily have reconciled them with other things they already thought they knew: “City men are more used to working in crowded conditions and in corporations, with chains of command, strict standards of clothing, etiquette, and so on. That’s obvious!” But this is exactly the point Lazarsfeld was making. When every answer and its opposite appears equally obvious then, as he put it, “something is wrong with the entire argument of ‘obviousness'” More here: https://www.newscientist.com/article/mg21128210-100-the-human-paradox-that-is-common-sense/ https://www.newscientist.com/article/mg21128210-100-the-huma...
- amscanne 6y ago
  Oh yes, I hate this kind of thing so much. People have so many pseudoscientific explanations and theories to support whatever random products and practices they like. I call these “huh, that makes sense” explanations, since that’s often what people say after hearing them. You even used the magic three words above.
  - ashtonkem 6y ago
    The related phenomenon in biology is called the “just so” fallacy; e.g. “X species evolved Y just so that they could do Z”. It’s a comforting story, but it will often lead you astray.

partiallypro 6y ago

I don't know much about Taleb, but one problem I've always had with Silver is that he's very smug about this models but they often enough don't work out. Then when they don't work out, instead of reflecting on his models to improve them (at least publicly,) he just tells people they don't understand probability or his models. His sports models are -especially- bad; no better than a coin flip in some instances. I know there are a ton of Silver fanboys, but surely you can agree he has a pretty big ego, which always makes me take him with a grain of salt. When it comes to politics his former work with Obama's campaign and DailyKos also make me question if he can be truly unbiased in his predictions, but that's a separate issue.

throwawaygh 6y ago
>> The whole argument is sort of pointless from an intellectual/academic perspective. It's a war of public personalities more than anything else.* > ...he's very smug... Right, it's just dueling personalities.
Robotbeat 6y ago
Whose models are better? There have been years where Nate Silver's model has called every single one of the 50 states correct as far as which POTUS candidate they'd vote for. See: https://mashable.com/2012/11/07/nate-silver-wins/ https://mashable.com/2012/11/07/nate-silver-wins/ He's not perfect (for instance, I think calling the 2016 model too certain early on is valid and I think some other people have basically equivalently-good models, but are less well known), but if confidence is smugness, then he has a right to be.
- thaumasiotes 6y ago
  > Whose models are better? There have been years where Nate Silver's model has called every single one of the 50 states correct as far as which POTUS candidate they'd vote for. Well, this certainly isn't how you'd go about evaluating model quality. Calling which candidate a state will vote for is not a good task; it's too discrete to be a good evaluator. For the same reason, no political model produces this output. You predict the vote share given to each candidate -- a nice continuous variable -- you don't predict who will win. Calling the state is a gimmick on top of that (since victory is a function of vote share, it's easy to degrade your model's output into victory predictions) which serves only to make the statistics worse.

deleted 6y ago

[deleted]

tempestn 6y ago

I came here to say exactly the same thing. Silver and 538 are explicit about not doing this. Their models do include the uncertainty of future events, and do adjust closer to 50/50 given the same polls the further from the election it is. That's why Biden's odds have been ticking upwards slightly over the past couple weeks despite polls remaining pretty stable. So in theory, the forecast should be exactly what it appears: the percentage of the time the candidate would be expected to win given the situation they are in at the time of the forecast. So if on Oct. 14 it gives Biden a 72% chance, it means that a candidate in Biden's position, with his polling numbers, and the economic conditions and everything else that is the case on Oct. 14, would go on to win from there 72% of the time. I agree that it appears the volatility of the 2016 forecast was larger than you would expect given a correct model behaving that way. I expect that uncertainty is better expressed in the 2020 model. However it's certainly conceivable for such a model to swing correctly. IE, there's nothing impossible about a candidate having a 90% chance of winning on X date, then having circumstances change such that they only have a 50% chance at Y date. (In fact, if you ignore the discontinuity on election day, at least 10% of the time their odds would have to pass through 50% at some point, as they would go on to lose. And of course there would also be scenarios when it would dip to that level or below and yet still win.)

stopachka 6y ago

I think Taleb’s point is a bit more damning — implying the software does not do what it says it would. If all sorts of things can happen, where you predictions change widely, Taleb’s argument is that you can’t use a number like 90%. You have to include the error, which ends up saying “50%” for both before the election. This doesn’t mean that all prediction suffers from this effect, and you have to be skeptical everywhere. I don’t think that’s a fair characterization. Taleb has won by mispriced options. His main statement is that the true probability of this particular event is much closer to 50%

cyphar 6y ago
That's what confidence intervals are for. You cannot condense both your expectation value and uncertainty in a single percentage (it is not correct to say that the probability of an outcome is 50% if 80% of likely outcomes result in one candidate winning). Otherwise you'll end up with every 5-week weather forecast saying there's a 50% chance of rain.
- stopachka 6y ago
  Indeed. Taleb’s point if I understand it correctly is that the uncertainty in this so high that the number comes realistically much closer to 50%.
  - cyphar 6y ago
    We're not in agreement here -- the solution to uncertainty in statistics is to model the uncertainty and treat them as confidence intervals, not throw your hands up and give everything a coin-flip probability. The probability isn't "realistically much closer to 50%" -- that's not how uncertainties in statistics work. It is possible that (if the uncertainty in the model was smaller) that the "real" probability is 50% but there's no way of knowing if that is the case.
    - stopachka 6y ago
      We can try going deeper. I may be making a dumb mistake: when I got to 538 — the main number I see is 89%. I don’t see 89% +- 40. What is his confidence interval?

Tomminn 6y ago

If I am expected to make a better prediction of the election outcome by averaging your last 10 predictions than by taking your current prediction, then your current prediction is suboptimal. This is Taleb's point, and it is solid as a rock.

moultano 6y ago
But he provides no evidence that this is the case. The 2016 election was extremely volatile due to what actually happened during it, not the modelling. There were several bombshell press events all throughout the campaign that dramatically shifted polls.
- ethagknight 6y ago
  That’s exactly Taleb’s point, both in this instance and in every book Taleb has written. Those bombshells should have been included in the model. Fat tail risks. If the modeling doesn’t include the potential for dramatic unknowns that can DO happen, then the model is no good. In the Clinton/Trump election, Trump is a showman and a name caller focused on ratings, Clinton a career politician with accusations of dirt, the model should expect “big things can still happen today, tomorrow, next week” and the absence if big events yesterday or last election holds little value relative to the probability of a big event happening in the next 24 hours. Edit- I’m getting downvoted im guessing because I said Clinton has lots of dirt and called Trump obsessive, I’ve revised the comment to be less politically accusative. I’m not concerned with the politics, just interested in the obsession with these “predictions”
  - mcphage 6y ago
    > If the modeling doesn’t include the potential for dramatic unknowns that can DO happen, then the model is no good. Why do you think it didn't? It took a major news event breaking at the exact right time (too early, and people would realize it was meaningless, too late, and it wouldn't have time to get out). And even with that, Trump barely eked out a win. That seems unlikely-but-not-impossible, which seems to match with 538's estimations.