7 ms·

I have zero respect for election forecasters because I think they are not any better than astrologers. Nate silver has used the word probability in a way that i

I have zero respect for election forecasters because I think they are not any better than astrologers. Nate silver has used the word probability in a way that is not consistent with mathematical definition of probability so often in this context. Taleb despite being brilliant mathematician is actually pretty mediocre when it comes to real world. A lot of his "discoveries" such as "skin in the game" are well known to economists in the form of various theories. I once exchanged few tweets with him about his criticism of Pinker's claim that we are probably living in one of the most peaceful times in human history. At no point Pinker makes a claim it was inevitable or it being the evidence that peace will continue to exist. Taleb wrote a length paper on the topic which I found to be totally incoherent. When I asked questions his retort was that I do not understand statistics.

brian_cloutier 8y ago
Do you have an example of a time where Nate Silver has shown a misunderstanding of probability? Everything I've read by him has seemed informed.
- baddox 8y ago
  The best criticism I can come up with is that 538 probably doesn't do a sufficient job of explaining to its audience that their published probabilities are predictions for what would happen if the event occurred today. This seems to be one of Taleb's (and others') criticisms of their predictions: that they don't consider the probability of major unexpected events occurring between now and the event (like an athlete getting injured in the future, or the FBI announcing an investigation of a political candidate). But as I understand it, 538 is explicitly not attempting to consider those things, and is simply publishing probabilities of outcomes if the event were to happen now.
  - ska 8y ago
    I think overall 538 did a pretty good job of communicating a difficult subject to a lay audience, but agree this point could perhaps be communicated better. I don't know of a good way to try and capture these sort of events meaningfully, do you?
  - eridius 8y ago
    That seems like a pretty big difference to gloss over. Why are they publishing probabilities for if the event were to happen today? The event isn't happening today!
    - pohl 8y ago
      I know, right? Every time my meteorologist gives me a probability of rain on a day that it doesn't rain I ask the same question.
      - steve19 8y ago
        Your personal meteorologist presumably predicts rain on days in the future, when it could happen, not days in the past. Nate publishes predictions for events that will never occur pr(x wins | election today) when there is no election. It would be like me placing bets on a sports game that is not scheduled to be played.
      - dralley 8y ago
        Meteorologists don't try to predict the weather 2 months from now.
        pohl 8y ago
        Nate hasn't been trying to predict elections that far out, either, to be fair. He has always said some variant of "if the election were held today" when discussing the model.
    - baddox 8y ago
      They don't exactly gloss over it. They have tons of articles explaining it, but of course the average reader is just going to see the probabilities and run with that, and 538 does have some responsibility to actively explain things upfront in their UI, which I don't think they do a good job at. > Why are they publishing probabilities for if the event were to happen today? The event isn't happening today! My guess is that it's extremely out of scope (and probably ultimately infeasible) to try to incorporate these things into their models. Sure, there's a non-zero chance of a candidate being killed by a meteorite, but I think it's fair to ignore that and, if it happens, admit that you didn't even attempt to include the probability of that into your model.
  - whatisthiseven 8y ago
    Nate himself has explicitly addressed that complaint as a huge mistake that Nassim continues to make and refuses to admit: https://twitter.com/NateSilver538/status/1114160511208259584 https://twitter.com/NateSilver538/status/1114160511208259584 "p.s. You continue to cite a paper in which you critique our now-cast (a deliberately volatile hypothetical of what would happen in an election held today) and mistake it for our forecast but because you're such a stubborn idiot you won't admit to this very basic mistake."
  - dragonwriter 8y ago
    > The best criticism I can come up with is that 538 probably doesn't do a sufficient job of explaining to its audience that their published probabilities are predictions for what would happen if the event occurred today I don't think that's true. While 538 does such predictions (the “nowcast”), they also predict actual results (the “polls-plus forecast” and the “polls-only forecast”), the default view is a forecast (I think the polls-plus usually, but I vaguely recall a chance to polls-only as the default during the 2016 cycle with concerns that there were reasons to suspect that some of the other signals on the polls-plus model had become less useful.) The “nowcast” is not the the default/headline number, so it would be very odd for a someone to mistakenly fixate on it thinking it was the forward-looking prediction.
    - thousandautumns 8y ago
      This is not correct, or at least it isn't any longer. The current predictions by 538 are the "lite" (polls only), the "classic" (polls + other data driven indicators) and the "deluxe" (polls + other data driven indicators + expert opinion). All of these are predictions based on "what would be the outcome of the election be were it to be held today". They just incorporate different data and complexity in their models. Previously, the reported predictions were the "nowcast", "polls-only", and "polls-plus". But despite their names, I believe those too, whatever 538 may have described them as, were "predictions for outcomes as if the event occurred today". In fact, were they not the case, then Taleb's argument would actually have merit, despite his obtuse attempts to explain it. Essentially, his argument is that, if you are predicting a particular outcome to an event (say, Clinton winning the election) a year from now with a certainty of 90%, and 3 months from now, your prediction is 75%, and 3 months after that is 85%, and then right before the election, is 70%, is evidence of a flawed model (keep in mind, I'm making these numbers up). If your prediction is 90% a year out, it should fluctuate much as time goes on, because 90% is very strong certainty. One may counter that as time goes on, public opinion may change, there are changes in news coverage, and external events like the economy or scandal can make a big impact, and all of that is absolutely true. But the point is that, if in light of new information coming forward, your model predictions repeatedly exhibit volatility, than your model failed to adequately account for the uncertainty as a result of future events, and you were never actually 90% sure in the first place. By necessity, a model predicting a distant event in which lots of unknown factors can cause it to fluctuate before it occurs will either be fairly stable as time goes on (meaning you indeed were correct that there was a high chance of a Clinton win), or should give much more hedged estimates of a win (meaning your probabilities a year out would be much closer to 50-50). Taleb's point is that 538's models are too volatile to actually be accurate when they say there is an X% chance someone wins an election a year, or 6 months from now. However, his argument does not apply in the case where the prediction is that there is an X% chance someone wins an election, were the election held today. Instead he seems to dismiss that that type of prediction is worthless, which is simply wrong and foolish in my opinion. There is lots to be learned about the current political climate from those predictions. So its entirely possible both parties are partially wrong and right. Taleb is correct that 538's models are too volatile to be accurate predictions far out from the election, but it assumes that the model is a prediction of what will happen on election day, rather than what would happen today. He also is throwing out the baby with the bathwater by insisting that that is the only kind of political prediction that has any value. On the other hand, 538 has not been very clear on if their forecasts are predictions of what happens on election day or what happens were election day today. If its the former, then their models have issues and Taleb has at least some validity to his argument. If its the latter, then Taleb's argument is no longer applicable.
  - pdubbs90 8y ago
    > their published probabilities are predictions for what would happen if the event occurred today. This is manifestly untrue. For the 2016 election 538 had a "Now-cast" that they very clearly marked as "Who would win the election if it were held today" (see here: https://projects.fivethirtyeight.com/2016-election-forecast/ https://projects.fivethirtyeight.com/2016-election-forecast/) and their other models incorporated the fact that there's time left before the election. Due to people not finding the now-cast valuable they abandoned it for 2018. Nate Silver definitely incorporates uncertainty in the forecast (and talks about it frequently), but it's not modeled as a random walk. Rather they use historical data to look at what polling now has to say about an election however far away the election is. Taleb's criticism that they don't consider a black swan event might be valid, but Silver isn't just predicting the election now.
  - gweinberg 8y ago
    But that's just nonsense. If something happened yesterday that has people all hot and bothered today but which nobody will give a shit about anymore by the time the next election rolls around, Silver should say "nobody will give a shit about this anymore by the time next election rolls around." Giving a numerical value of how it would effect the election if it were to happen tomorrow is gibberish on the level of "if my grandmother had wheels she'd be a wagon".
    - perl4ever 8y ago
      If you think someone is offering a choice of a hamburger or a shit sandwich, you can just choose the hamburger and stop complaining about the stuff on the menu you don't like.
  - noelsusman 8y ago
    That's just not true. In 2016 they had several versions of the model. One was called the NowCast that very explicitly modeled the probability of winning the election today. They had a separate model that gave the probability of winning on election day, which included whatever their estimate was of uncertainty moving forward (including the probability of game-changing events, which is pretty low).
    - baddox 8y ago
      Fair enough. If I misunderstand their predictions then they definitely need to do a better job explaining them. :)
vertis 8y ago
I've found twitter not to be a good forum for this kind of thing. Leaving the current pair alone, I've had a few times where more public figures can't determine easily that you're not one of the trolls, that and the fact that they almost certainly don't follow you and have an understanding of your context and goodwill. For the most part, I've stopped following anyone that doesn't follow me in return on twitter. Taking these conversations to other platforms where possible as well (e.g. Slack/Telegram groups).
- backpropaganda 8y ago
  Is there any reason to believe that some other medium won't have this exact problem? For instance, if you were to meet a public figure at a public event, do you think they will behave any differently? To me it seems like this is more a problem of public figures having a high number of people who want to talk to them or troll them, than of Twitter as a medium.
LeanderK 8y ago
can you detail your cristicism of Nate Silver? I mean he's not a researcher trying to predict the election and somehow has to explain everythong the layman. So some derivation from the mathmatical definitions of probability are expected, i think (the mathematical definitions in its full glory are suprisingly complicated). I always like FiveTheirghtyEight, mostly because the plots are so beautiful and it never struck me as blatantly wrong.
- steve19 8y ago
  In many industries if we redefined probability it would be considered malpractice or fraud. We expect our doctors not to tell us we have a good chance of surviving surgery when we only have a 20% chance of surviving. I would be fraud (I assume) for a lottery to tell someone buying a ticket that they have a good chance of winning millions (0000000.1% probability that you will win but a 75% probability that it will be won tonight by someone ).
  - LeanderK 8y ago
    i thought more of a technical argument, if this comment is based on stuff like "Event A and not A are both extremely likely". I also think that both of your examples are not really useful, they don't violate the laws of probability. What a good chance is depends on more things than just the raw probabilities, for example i could phrase a ticket with a good chance as E(reward - price) > 0 (so its independent of its actual probability). I don't consider "Event A and not A are both extremely likely" as P(A) != 0.5, communicating probabilities is not a mathematical problem, especially if words like "extremely" are used. Maybe P(A)=0.3 might also be extremely likely for your target audience. Using your argument...what's a good chance of dying? It depends on the interpretation of the layperson and needs carful wording. I remain faithful that his actual models are sound.
    - thanatropism 8y ago
      Actual probabilities are uncertain. If you think of the probability distribution on probabilities (the Bayesian prior), the claim that "one of the extreme probabilities is more likely than a halfway one" makes sense, eg. [0] It makes more sense philosophically to make claims about one's model than about the real world, anyway. [0] https://en.wikipedia.org/wiki/Arcsine_distribution https://en.wikipedia.org/wiki/Arcsine_distribution
tptacek 8y ago
Ok. What were your opinions of the actual article we're commenting on? It's written by a Berkeley Math PhD, so I think it's unlikely that it includes a lot of misunderstandings of basic mathematical terms.
throw2016 8y ago
Piketty's extensive data on global wages raises serious questions about Pinker's narrative. And if there is still doubt Pinker's data sources [1] and interpretation are questionable to say the least. Here is Pinker himself admitting his selective use of data [2] Watch his answer at 4.30. This is extremely poor use of stats, and makes him look a side character apologist in Hunger Games. And here is an even more substantial criticism of Pinker's approach by Leon Wieseltier in the newrepublic [3]. [1] https://www.jasonhickel.org/blog/2019/2/3/pinker-and-global-poverty https://www.jasonhickel.org/blog/2019/2/3/pinker-and-global-... [2] https://www.youtube.com/watch?v=xAQ0BfcftBA https://www.youtube.com/watch?v=xAQ0BfcftBA [3] https://newrepublic.com/article/114548/leon-wieseltier-responds-steven-pinkers-scientism https://newrepublic.com/article/114548/leon-wieseltier-respo...

brian_cloutier 8y ago

Do you have an example of a time where Nate Silver has shown a misunderstanding of probability? Everything I've read by him has seemed informed.

baddox 8y ago
The best criticism I can come up with is that 538 probably doesn't do a sufficient job of explaining to its audience that their published probabilities are predictions for what would happen if the event occurred today. This seems to be one of Taleb's (and others') criticisms of their predictions: that they don't consider the probability of major unexpected events occurring between now and the event (like an athlete getting injured in the future, or the FBI announcing an investigation of a political candidate). But as I understand it, 538 is explicitly not attempting to consider those things, and is simply publishing probabilities of outcomes if the event were to happen now.
- ska 8y ago
  I think overall 538 did a pretty good job of communicating a difficult subject to a lay audience, but agree this point could perhaps be communicated better. I don't know of a good way to try and capture these sort of events meaningfully, do you?
- eridius 8y ago
  That seems like a pretty big difference to gloss over. Why are they publishing probabilities for if the event were to happen today? The event isn't happening today!
  - pohl 8y ago
    I know, right? Every time my meteorologist gives me a probability of rain on a day that it doesn't rain I ask the same question.
    - steve19 8y ago
      Your personal meteorologist presumably predicts rain on days in the future, when it could happen, not days in the past. Nate publishes predictions for events that will never occur pr(x wins | election today) when there is no election. It would be like me placing bets on a sports game that is not scheduled to be played.
    - dralley 8y ago
      Meteorologists don't try to predict the weather 2 months from now.
      - pohl 8y ago
        Nate hasn't been trying to predict elections that far out, either, to be fair. He has always said some variant of "if the election were held today" when discussing the model.
  - baddox 8y ago
    They don't exactly gloss over it. They have tons of articles explaining it, but of course the average reader is just going to see the probabilities and run with that, and 538 does have some responsibility to actively explain things upfront in their UI, which I don't think they do a good job at. > Why are they publishing probabilities for if the event were to happen today? The event isn't happening today! My guess is that it's extremely out of scope (and probably ultimately infeasible) to try to incorporate these things into their models. Sure, there's a non-zero chance of a candidate being killed by a meteorite, but I think it's fair to ignore that and, if it happens, admit that you didn't even attempt to include the probability of that into your model.
- whatisthiseven 8y ago
  Nate himself has explicitly addressed that complaint as a huge mistake that Nassim continues to make and refuses to admit: https://twitter.com/NateSilver538/status/1114160511208259584 https://twitter.com/NateSilver538/status/1114160511208259584 "p.s. You continue to cite a paper in which you critique our now-cast (a deliberately volatile hypothetical of what would happen in an election held today) and mistake it for our forecast but because you're such a stubborn idiot you won't admit to this very basic mistake."
- dragonwriter 8y ago
  > The best criticism I can come up with is that 538 probably doesn't do a sufficient job of explaining to its audience that their published probabilities are predictions for what would happen if the event occurred today I don't think that's true. While 538 does such predictions (the “nowcast”), they also predict actual results (the “polls-plus forecast” and the “polls-only forecast”), the default view is a forecast (I think the polls-plus usually, but I vaguely recall a chance to polls-only as the default during the 2016 cycle with concerns that there were reasons to suspect that some of the other signals on the polls-plus model had become less useful.) The “nowcast” is not the the default/headline number, so it would be very odd for a someone to mistakenly fixate on it thinking it was the forward-looking prediction.
  - thousandautumns 8y ago
    This is not correct, or at least it isn't any longer. The current predictions by 538 are the "lite" (polls only), the "classic" (polls + other data driven indicators) and the "deluxe" (polls + other data driven indicators + expert opinion). All of these are predictions based on "what would be the outcome of the election be were it to be held today". They just incorporate different data and complexity in their models. Previously, the reported predictions were the "nowcast", "polls-only", and "polls-plus". But despite their names, I believe those too, whatever 538 may have described them as, were "predictions for outcomes as if the event occurred today". In fact, were they not the case, then Taleb's argument would actually have merit, despite his obtuse attempts to explain it. Essentially, his argument is that, if you are predicting a particular outcome to an event (say, Clinton winning the election) a year from now with a certainty of 90%, and 3 months from now, your prediction is 75%, and 3 months after that is 85%, and then right before the election, is 70%, is evidence of a flawed model (keep in mind, I'm making these numbers up). If your prediction is 90% a year out, it should fluctuate much as time goes on, because 90% is very strong certainty. One may counter that as time goes on, public opinion may change, there are changes in news coverage, and external events like the economy or scandal can make a big impact, and all of that is absolutely true. But the point is that, if in light of new information coming forward, your model predictions repeatedly exhibit volatility, than your model failed to adequately account for the uncertainty as a result of future events, and you were never actually 90% sure in the first place. By necessity, a model predicting a distant event in which lots of unknown factors can cause it to fluctuate before it occurs will either be fairly stable as time goes on (meaning you indeed were correct that there was a high chance of a Clinton win), or should give much more hedged estimates of a win (meaning your probabilities a year out would be much closer to 50-50). Taleb's point is that 538's models are too volatile to actually be accurate when they say there is an X% chance someone wins an election a year, or 6 months from now. However, his argument does not apply in the case where the prediction is that there is an X% chance someone wins an election, were the election held today. Instead he seems to dismiss that that type of prediction is worthless, which is simply wrong and foolish in my opinion. There is lots to be learned about the current political climate from those predictions. So its entirely possible both parties are partially wrong and right. Taleb is correct that 538's models are too volatile to be accurate predictions far out from the election, but it assumes that the model is a prediction of what will happen on election day, rather than what would happen today. He also is throwing out the baby with the bathwater by insisting that that is the only kind of political prediction that has any value. On the other hand, 538 has not been very clear on if their forecasts are predictions of what happens on election day or what happens were election day today. If its the former, then their models have issues and Taleb has at least some validity to his argument. If its the latter, then Taleb's argument is no longer applicable.
- pdubbs90 8y ago
  > their published probabilities are predictions for what would happen if the event occurred today. This is manifestly untrue. For the 2016 election 538 had a "Now-cast" that they very clearly marked as "Who would win the election if it were held today" (see here: https://projects.fivethirtyeight.com/2016-election-forecast/ https://projects.fivethirtyeight.com/2016-election-forecast/) and their other models incorporated the fact that there's time left before the election. Due to people not finding the now-cast valuable they abandoned it for 2018. Nate Silver definitely incorporates uncertainty in the forecast (and talks about it frequently), but it's not modeled as a random walk. Rather they use historical data to look at what polling now has to say about an election however far away the election is. Taleb's criticism that they don't consider a black swan event might be valid, but Silver isn't just predicting the election now.
- gweinberg 8y ago
  But that's just nonsense. If something happened yesterday that has people all hot and bothered today but which nobody will give a shit about anymore by the time the next election rolls around, Silver should say "nobody will give a shit about this anymore by the time next election rolls around." Giving a numerical value of how it would effect the election if it were to happen tomorrow is gibberish on the level of "if my grandmother had wheels she'd be a wagon".
  - perl4ever 8y ago
    If you think someone is offering a choice of a hamburger or a shit sandwich, you can just choose the hamburger and stop complaining about the stuff on the menu you don't like.
- noelsusman 8y ago
  That's just not true. In 2016 they had several versions of the model. One was called the NowCast that very explicitly modeled the probability of winning the election today. They had a separate model that gave the probability of winning on election day, which included whatever their estimate was of uncertainty moving forward (including the probability of game-changing events, which is pretty low).
  - baddox 8y ago
    Fair enough. If I misunderstand their predictions then they definitely need to do a better job explaining them. :)

vertis 8y ago

I've found twitter not to be a good forum for this kind of thing. Leaving the current pair alone, I've had a few times where more public figures can't determine easily that you're not one of the trolls, that and the fact that they almost certainly don't follow you and have an understanding of your context and goodwill. For the most part, I've stopped following anyone that doesn't follow me in return on twitter. Taking these conversations to other platforms where possible as well (e.g. Slack/Telegram groups).

backpropaganda 8y ago
Is there any reason to believe that some other medium won't have this exact problem? For instance, if you were to meet a public figure at a public event, do you think they will behave any differently? To me it seems like this is more a problem of public figures having a high number of people who want to talk to them or troll them, than of Twitter as a medium.

LeanderK 8y ago

can you detail your cristicism of Nate Silver? I mean he's not a researcher trying to predict the election and somehow has to explain everythong the layman. So some derivation from the mathmatical definitions of probability are expected, i think (the mathematical definitions in its full glory are suprisingly complicated). I always like FiveTheirghtyEight, mostly because the plots are so beautiful and it never struck me as blatantly wrong.

steve19 8y ago
In many industries if we redefined probability it would be considered malpractice or fraud. We expect our doctors not to tell us we have a good chance of surviving surgery when we only have a 20% chance of surviving. I would be fraud (I assume) for a lottery to tell someone buying a ticket that they have a good chance of winning millions (0000000.1% probability that you will win but a 75% probability that it will be won tonight by someone ).
- LeanderK 8y ago
  i thought more of a technical argument, if this comment is based on stuff like "Event A and not A are both extremely likely". I also think that both of your examples are not really useful, they don't violate the laws of probability. What a good chance is depends on more things than just the raw probabilities, for example i could phrase a ticket with a good chance as E(reward - price) > 0 (so its independent of its actual probability). I don't consider "Event A and not A are both extremely likely" as P(A) != 0.5, communicating probabilities is not a mathematical problem, especially if words like "extremely" are used. Maybe P(A)=0.3 might also be extremely likely for your target audience. Using your argument...what's a good chance of dying? It depends on the interpretation of the layperson and needs carful wording. I remain faithful that his actual models are sound.
  - thanatropism 8y ago
    Actual probabilities are uncertain. If you think of the probability distribution on probabilities (the Bayesian prior), the claim that "one of the extreme probabilities is more likely than a halfway one" makes sense, eg. [0] It makes more sense philosophically to make claims about one's model than about the real world, anyway. [0] https://en.wikipedia.org/wiki/Arcsine_distribution https://en.wikipedia.org/wiki/Arcsine_distribution

tptacek 8y ago

Ok. What were your opinions of the actual article we're commenting on? It's written by a Berkeley Math PhD, so I think it's unlikely that it includes a lot of misunderstandings of basic mathematical terms.

throw2016 8y ago

Piketty's extensive data on global wages raises serious questions about Pinker's narrative. And if there is still doubt Pinker's data sources [1] and interpretation are questionable to say the least. Here is Pinker himself admitting his selective use of data [2] Watch his answer at 4.30. This is extremely poor use of stats, and makes him look a side character apologist in Hunger Games. And here is an even more substantial criticism of Pinker's approach by Leon Wieseltier in the newrepublic [3]. [1] https://www.jasonhickel.org/blog/2019/2/3/pinker-and-global-poverty https://www.jasonhickel.org/blog/2019/2/3/pinker-and-global-... [2] https://www.youtube.com/watch?v=xAQ0BfcftBA https://www.youtube.com/watch?v=xAQ0BfcftBA [3] https://newrepublic.com/article/114548/leon-wieseltier-responds-steven-pinkers-scientism https://newrepublic.com/article/114548/leon-wieseltier-respo...