12 ms·

Indeed. Taleb’s point if I understand it correctly is that the uncertainty in this so high that the number comes realistically much closer to 50%.

cyphar 6y ago
We're not in agreement here -- the solution to uncertainty in statistics is to model the uncertainty and treat them as confidence intervals, not throw your hands up and give everything a coin-flip probability. The probability isn't "realistically much closer to 50%" -- that's not how uncertainties in statistics work. It is possible that (if the uncertainty in the model was smaller) that the "real" probability is 50% but there's no way of knowing if that is the case.
- stopachka 6y ago
  We can try going deeper. I may be making a dumb mistake: when I got to 538 — the main number I see is 89%. I don’t see 89% +- 40. What is his confidence interval?
  - cyphar 6y ago
    If you want a more data-heavy version go to the graphs, then click on the "electoral votes" tab. The shaded part is the 80% confidence interval of each prediction. This is the historical graph form of the "every outcome in our simulations" chart underneath the dot chart. It should be noted that the dot chart at the top of the page does kind of outline the confidence interval (giving a realistic example of what kinds of outcomes might happen) but is obviously geared towards lay-people who don't know what a confidence interval is. > We simulate the election 40,000 times to see who wins most often. The sample of 100 outcomes below gives you a good idea of the range of scenarios our model thinks is possible. I think it would be better if the confidence intervals were better signposted, but it's possible that only the electoral college win margins have solid confidence intervals (I'll admit I don't know enough statistics to know if you could trivially transform the confidence intervals of the electoral college votes to win percentage). The senate and house predictions explicitly show the confidence interval in their main graphs.
    - stopachka 6y ago
      I think if you see 90% with 80% confidence interval, you can intuit that he isn't counting uncertainty in the tails. Consider, Would you be comfortable making a bet with someone, on a 90/10 payoff? This, I think is Taleb's central point. Silver's algorithm is not counting the uncertainty correctly.
  - dragonwriter 6y ago
    The 89% probability of victory is the result of the actual predicted electoral vote total and the uncertainty in that prediction. Placing an additional uncertainty on the uncertainty is...very much not understanding uncertainty.
    - stopachka 6y ago
      In this case, Taleb's argument stands: 89% implies way too little uncertainty
- inimino 6y ago
  That is precisely how probabilities work. If you have a simulation that gives an 80% chance of one outcome, but you only have 75% confidence that the assumptions that the simulation is based on, your actual prediction should be closer to 50%. This is not "throwing your hands up", it is simply correctly assessing the possibility of model error. There is no such thing as giving a probability and also giving a confidence interval, for an event with a binary outcome. Everything that your prediction can say about the outcome can be said with a single number between 0 and 1. To give an answer like "80% plus or minus 15%" just means you haven't finished calculating.
  - awb 6y ago
    Right, but I think Silver is saying that averaging high quality polls produces a high level of certainty. The only reason Trump has a 10% chance and not a 0% chance is that there's a 10% chance the polls are off by far more than they were in 2016 because Biden has consistently polled ahead of Trump in states totalling 270+ EV, plus he's polling at historically high levels nationally for a challenger and facing an incumbent with historically low approval ratings. I think Taleb is saying "there's so much uncertainty it should be a 50/50 race" and Silver is saying "there's uncertainty, but it's equally plausible in either direction, so let's go off of polling data and historical examples which looks like a 90/10 race". In the end they both should just bet their own predictions and let the winner emerge over time.
    - nwienert 6y ago
      It’s impossible to be so certain of polls because certain demographics may change how forthcoming they are with pollsters over time. Systemic bias is totally possible and can affect just one side, in a huge variety of ways, changing over time. This is just one example of a fat tail event you can’t predict but which you can expect, and given Nate has only called a couple elections, there’s no way he can have 90% confidence a model of the polls will be accurate.
      - awb 6y ago
        Models are backtested against the historical record where hundreds or thousands of elections (federal, state and local) can be used as training data. For example, a weighted average of polls has Biden up ~5% in PA. In state-wide elections, I wouldn't be surprised if candidates with that sized lead in PA (and states with similar size and socioeconomic demographics) end up winning about 85% of the time, which is what the 538 model has Biden's chances at in PA. That 15% uncertainty accounts for the bias you describe. If polling was perfect and there was no bias, a ~5% polling lead (and exceeding the margin of error) would be a 100% guaranteed victory. It would be great to see Taleb's forecast of federal and state elections so he has "skin in the game" as he likes to say. Assume you get points based on your % confidence, so if you predict there's a 55% chance of a particular outcome and you're right you get 55pts, and if you're wrong you lose 55pts. If Taleb pegs every race at 50/50 or 55/45 because you "can't predict elections", I doubt his score will be higher than Silver's. You'll end up with silly predictions like 40% chance of Trump winning Washington DC or 30% chance of Biden winning Wyoming, when in reality each have less than 1% chance of doing so. When you make reasonable, weighted predictions based on state polling, you see that Biden has leads outside the margin of error in states totaling 270 EV. So, the 10% chance Silver gives Trump is saying, hey there's a chance that the polls could be extremely biased in all 7 swing states in Trump's favor. But, if Trump loses even 1 of those swing states, there are so many safe D electoral votes that Biden will win.
awb 6y ago
Well, he's right in the sense that there's close to a 50% chance of a swing in either direction (blunders, dirt, health issues, polling errors, etc). However, when one candidate is up 8+% (even months out) against an incumbent with 4 years of net negative approval ratings, to call that a toss up is like saying the frontrunner is substantially more likely to blunder than the underdog. As Silver points out however, a Trump win is just as likely as the largest Democratic landslide victory in modern history. The polls could move either way, but the only way they move enough for Trump to win is a larger than 2016 polling error in his favor in all 7 battleground states, which doesn't sound like a 50% chance at all.
- cameldrv 6y ago
  To say Biden is up by 8 points is missing a lot of microstructure. In every state Trump needs except PA, he’s within about 2 points. In PA, the critical state, he’s down by about 4.5 points, but Biden’s lead has been diminishing rapidly in the past two weeks. These polls also rely on a turnout model. There’s no way to calibrate these models for a pandemic election where some people may be more scared to vote than others. I give Biden the edge but it’s nowhere near 90/10.
  - rurban 6y ago
    I don't know where you got your numbers from. I got them from Tanenbaums site https://www.electoral-vote.com/evp2020/Pres/Graphs/all.html https://www.electoral-vote.com/evp2020/Pres/Graphs/all.html and this shows that Trump would win all swing states and thus the election, if there was not the surprising swing in Texas from Rep to Dem lately, which broke Trump. You cannot really trust Tanenbaums summary as heavy Dem fanboy, which was wrong last time and is wrong this time, but the raw polling numbers are showing a clear picture. Early votes missed the Hunter Biden scandal, and Texas was champion in early votes.

cyphar 6y ago

We're not in agreement here -- the solution to uncertainty in statistics is to model the uncertainty and treat them as confidence intervals, not throw your hands up and give everything a coin-flip probability. The probability isn't "realistically much closer to 50%" -- that's not how uncertainties in statistics work. It is possible that (if the uncertainty in the model was smaller) that the "real" probability is 50% but there's no way of knowing if that is the case.

stopachka 6y ago
We can try going deeper. I may be making a dumb mistake: when I got to 538 — the main number I see is 89%. I don’t see 89% +- 40. What is his confidence interval?
- cyphar 6y ago
  If you want a more data-heavy version go to the graphs, then click on the "electoral votes" tab. The shaded part is the 80% confidence interval of each prediction. This is the historical graph form of the "every outcome in our simulations" chart underneath the dot chart. It should be noted that the dot chart at the top of the page does kind of outline the confidence interval (giving a realistic example of what kinds of outcomes might happen) but is obviously geared towards lay-people who don't know what a confidence interval is. > We simulate the election 40,000 times to see who wins most often. The sample of 100 outcomes below gives you a good idea of the range of scenarios our model thinks is possible. I think it would be better if the confidence intervals were better signposted, but it's possible that only the electoral college win margins have solid confidence intervals (I'll admit I don't know enough statistics to know if you could trivially transform the confidence intervals of the electoral college votes to win percentage). The senate and house predictions explicitly show the confidence interval in their main graphs.
  - stopachka 6y ago
    I think if you see 90% with 80% confidence interval, you can intuit that he isn't counting uncertainty in the tails. Consider, Would you be comfortable making a bet with someone, on a 90/10 payoff? This, I think is Taleb's central point. Silver's algorithm is not counting the uncertainty correctly.
- dragonwriter 6y ago
  The 89% probability of victory is the result of the actual predicted electoral vote total and the uncertainty in that prediction. Placing an additional uncertainty on the uncertainty is...very much not understanding uncertainty.
  - stopachka 6y ago
    In this case, Taleb's argument stands: 89% implies way too little uncertainty
inimino 6y ago
That is precisely how probabilities work. If you have a simulation that gives an 80% chance of one outcome, but you only have 75% confidence that the assumptions that the simulation is based on, your actual prediction should be closer to 50%. This is not "throwing your hands up", it is simply correctly assessing the possibility of model error. There is no such thing as giving a probability and also giving a confidence interval, for an event with a binary outcome. Everything that your prediction can say about the outcome can be said with a single number between 0 and 1. To give an answer like "80% plus or minus 15%" just means you haven't finished calculating.
- awb 6y ago
  Right, but I think Silver is saying that averaging high quality polls produces a high level of certainty. The only reason Trump has a 10% chance and not a 0% chance is that there's a 10% chance the polls are off by far more than they were in 2016 because Biden has consistently polled ahead of Trump in states totalling 270+ EV, plus he's polling at historically high levels nationally for a challenger and facing an incumbent with historically low approval ratings. I think Taleb is saying "there's so much uncertainty it should be a 50/50 race" and Silver is saying "there's uncertainty, but it's equally plausible in either direction, so let's go off of polling data and historical examples which looks like a 90/10 race". In the end they both should just bet their own predictions and let the winner emerge over time.
  - nwienert 6y ago
    It’s impossible to be so certain of polls because certain demographics may change how forthcoming they are with pollsters over time. Systemic bias is totally possible and can affect just one side, in a huge variety of ways, changing over time. This is just one example of a fat tail event you can’t predict but which you can expect, and given Nate has only called a couple elections, there’s no way he can have 90% confidence a model of the polls will be accurate.
    - awb 6y ago
      Models are backtested against the historical record where hundreds or thousands of elections (federal, state and local) can be used as training data. For example, a weighted average of polls has Biden up ~5% in PA. In state-wide elections, I wouldn't be surprised if candidates with that sized lead in PA (and states with similar size and socioeconomic demographics) end up winning about 85% of the time, which is what the 538 model has Biden's chances at in PA. That 15% uncertainty accounts for the bias you describe. If polling was perfect and there was no bias, a ~5% polling lead (and exceeding the margin of error) would be a 100% guaranteed victory. It would be great to see Taleb's forecast of federal and state elections so he has "skin in the game" as he likes to say. Assume you get points based on your % confidence, so if you predict there's a 55% chance of a particular outcome and you're right you get 55pts, and if you're wrong you lose 55pts. If Taleb pegs every race at 50/50 or 55/45 because you "can't predict elections", I doubt his score will be higher than Silver's. You'll end up with silly predictions like 40% chance of Trump winning Washington DC or 30% chance of Biden winning Wyoming, when in reality each have less than 1% chance of doing so. When you make reasonable, weighted predictions based on state polling, you see that Biden has leads outside the margin of error in states totaling 270 EV. So, the 10% chance Silver gives Trump is saying, hey there's a chance that the polls could be extremely biased in all 7 swing states in Trump's favor. But, if Trump loses even 1 of those swing states, there are so many safe D electoral votes that Biden will win.

awb 6y ago

Well, he's right in the sense that there's close to a 50% chance of a swing in either direction (blunders, dirt, health issues, polling errors, etc). However, when one candidate is up 8+% (even months out) against an incumbent with 4 years of net negative approval ratings, to call that a toss up is like saying the frontrunner is substantially more likely to blunder than the underdog. As Silver points out however, a Trump win is just as likely as the largest Democratic landslide victory in modern history. The polls could move either way, but the only way they move enough for Trump to win is a larger than 2016 polling error in his favor in all 7 battleground states, which doesn't sound like a 50% chance at all.

cameldrv 6y ago
To say Biden is up by 8 points is missing a lot of microstructure. In every state Trump needs except PA, he’s within about 2 points. In PA, the critical state, he’s down by about 4.5 points, but Biden’s lead has been diminishing rapidly in the past two weeks. These polls also rely on a turnout model. There’s no way to calibrate these models for a pandemic election where some people may be more scared to vote than others. I give Biden the edge but it’s nowhere near 90/10.
- rurban 6y ago
  I don't know where you got your numbers from. I got them from Tanenbaums site https://www.electoral-vote.com/evp2020/Pres/Graphs/all.html https://www.electoral-vote.com/evp2020/Pres/Graphs/all.html and this shows that Trump would win all swing states and thus the election, if there was not the surprising swing in Texas from Rep to Dem lately, which broke Trump. You cannot really trust Tanenbaums summary as heavy Dem fanboy, which was wrong last time and is wrong this time, but the raw polling numbers are showing a clear picture. Early votes missed the Hunter Biden scandal, and Texas was champion in early votes.