6 ms·

That is precisely how probabilities work. If you have a simulation that gives an 80% chance of one outcome, but you only have 75% confidence that the assumption

That is precisely how probabilities work. If you have a simulation that gives an 80% chance of one outcome, but you only have 75% confidence that the assumptions that the simulation is based on, your actual prediction should be closer to 50%. This is not "throwing your hands up", it is simply correctly assessing the possibility of model error. There is no such thing as giving a probability and also giving a confidence interval, for an event with a binary outcome. Everything that your prediction can say about the outcome can be said with a single number between 0 and 1. To give an answer like "80% plus or minus 15%" just means you haven't finished calculating.

awb 6y ago
Right, but I think Silver is saying that averaging high quality polls produces a high level of certainty. The only reason Trump has a 10% chance and not a 0% chance is that there's a 10% chance the polls are off by far more than they were in 2016 because Biden has consistently polled ahead of Trump in states totalling 270+ EV, plus he's polling at historically high levels nationally for a challenger and facing an incumbent with historically low approval ratings. I think Taleb is saying "there's so much uncertainty it should be a 50/50 race" and Silver is saying "there's uncertainty, but it's equally plausible in either direction, so let's go off of polling data and historical examples which looks like a 90/10 race". In the end they both should just bet their own predictions and let the winner emerge over time.
- nwienert 6y ago
  It’s impossible to be so certain of polls because certain demographics may change how forthcoming they are with pollsters over time. Systemic bias is totally possible and can affect just one side, in a huge variety of ways, changing over time. This is just one example of a fat tail event you can’t predict but which you can expect, and given Nate has only called a couple elections, there’s no way he can have 90% confidence a model of the polls will be accurate.
  - awb 6y ago
    Models are backtested against the historical record where hundreds or thousands of elections (federal, state and local) can be used as training data. For example, a weighted average of polls has Biden up ~5% in PA. In state-wide elections, I wouldn't be surprised if candidates with that sized lead in PA (and states with similar size and socioeconomic demographics) end up winning about 85% of the time, which is what the 538 model has Biden's chances at in PA. That 15% uncertainty accounts for the bias you describe. If polling was perfect and there was no bias, a ~5% polling lead (and exceeding the margin of error) would be a 100% guaranteed victory. It would be great to see Taleb's forecast of federal and state elections so he has "skin in the game" as he likes to say. Assume you get points based on your % confidence, so if you predict there's a 55% chance of a particular outcome and you're right you get 55pts, and if you're wrong you lose 55pts. If Taleb pegs every race at 50/50 or 55/45 because you "can't predict elections", I doubt his score will be higher than Silver's. You'll end up with silly predictions like 40% chance of Trump winning Washington DC or 30% chance of Biden winning Wyoming, when in reality each have less than 1% chance of doing so. When you make reasonable, weighted predictions based on state polling, you see that Biden has leads outside the margin of error in states totaling 270 EV. So, the 10% chance Silver gives Trump is saying, hey there's a chance that the polls could be extremely biased in all 7 swing states in Trump's favor. But, if Trump loses even 1 of those swing states, there are so many safe D electoral votes that Biden will win.
    - nwienert 6y ago
      You totally ignored the point. Overfitting to previous results doesn’t account for the fact that the facts on the ground are changing. I mean this year alone we have a once in a century pandemic, can you fit that to any other election?
      - awb 6y ago
        That's why you test predictions. Silver says 10% is enough uncertainty, Taleb says it's not. From the article: "Premise 1: If you give a probability, you must be willing to wager on it". Unless they're willing to publish and make a friendly bet on their predictions, it's just bloviating. Every day/year/election is unique, but only to a certain magnitude. Some elections have had active wars, some health crises, some criminal scandals, some terrorist attacks, etc. In the end, the polls attempt to account for those uncertainties and historically that's been a much better predictor than a coin flip. For example, Washington DC has never voted Republican since it won electoral votes in 1962. Polls have Biden winning 80-90% of the vote there. Even Taleb would agree there's a near 100% chance of Biden winning there, pandemic or not. Silver's methodology is just extrapolating that out to each state. When it gets to some tipping point states it's closer to 60/40 Biden or 70/30. If Taleb wants to create his own forecast, it would be interesting to track it's performance over time. I have a hard time believing that model closer to a coin-flip is going to outperform weighted polling data and historical precedent over multiple predictions.

awb 6y ago

Right, but I think Silver is saying that averaging high quality polls produces a high level of certainty. The only reason Trump has a 10% chance and not a 0% chance is that there's a 10% chance the polls are off by far more than they were in 2016 because Biden has consistently polled ahead of Trump in states totalling 270+ EV, plus he's polling at historically high levels nationally for a challenger and facing an incumbent with historically low approval ratings. I think Taleb is saying "there's so much uncertainty it should be a 50/50 race" and Silver is saying "there's uncertainty, but it's equally plausible in either direction, so let's go off of polling data and historical examples which looks like a 90/10 race". In the end they both should just bet their own predictions and let the winner emerge over time.

nwienert 6y ago
It’s impossible to be so certain of polls because certain demographics may change how forthcoming they are with pollsters over time. Systemic bias is totally possible and can affect just one side, in a huge variety of ways, changing over time. This is just one example of a fat tail event you can’t predict but which you can expect, and given Nate has only called a couple elections, there’s no way he can have 90% confidence a model of the polls will be accurate.
- awb 6y ago
  Models are backtested against the historical record where hundreds or thousands of elections (federal, state and local) can be used as training data. For example, a weighted average of polls has Biden up ~5% in PA. In state-wide elections, I wouldn't be surprised if candidates with that sized lead in PA (and states with similar size and socioeconomic demographics) end up winning about 85% of the time, which is what the 538 model has Biden's chances at in PA. That 15% uncertainty accounts for the bias you describe. If polling was perfect and there was no bias, a ~5% polling lead (and exceeding the margin of error) would be a 100% guaranteed victory. It would be great to see Taleb's forecast of federal and state elections so he has "skin in the game" as he likes to say. Assume you get points based on your % confidence, so if you predict there's a 55% chance of a particular outcome and you're right you get 55pts, and if you're wrong you lose 55pts. If Taleb pegs every race at 50/50 or 55/45 because you "can't predict elections", I doubt his score will be higher than Silver's. You'll end up with silly predictions like 40% chance of Trump winning Washington DC or 30% chance of Biden winning Wyoming, when in reality each have less than 1% chance of doing so. When you make reasonable, weighted predictions based on state polling, you see that Biden has leads outside the margin of error in states totaling 270 EV. So, the 10% chance Silver gives Trump is saying, hey there's a chance that the polls could be extremely biased in all 7 swing states in Trump's favor. But, if Trump loses even 1 of those swing states, there are so many safe D electoral votes that Biden will win.
  - nwienert 6y ago
    You totally ignored the point. Overfitting to previous results doesn’t account for the fact that the facts on the ground are changing. I mean this year alone we have a once in a century pandemic, can you fit that to any other election?
    - awb 6y ago
      That's why you test predictions. Silver says 10% is enough uncertainty, Taleb says it's not. From the article: "Premise 1: If you give a probability, you must be willing to wager on it". Unless they're willing to publish and make a friendly bet on their predictions, it's just bloviating. Every day/year/election is unique, but only to a certain magnitude. Some elections have had active wars, some health crises, some criminal scandals, some terrorist attacks, etc. In the end, the polls attempt to account for those uncertainties and historically that's been a much better predictor than a coin flip. For example, Washington DC has never voted Republican since it won electoral votes in 1962. Polls have Biden winning 80-90% of the vote there. Even Taleb would agree there's a near 100% chance of Biden winning there, pandemic or not. Silver's methodology is just extrapolating that out to each state. When it gets to some tipping point states it's closer to 60/40 Biden or 70/30. If Taleb wants to create his own forecast, it would be interesting to track it's performance over time. I have a hard time believing that model closer to a coin-flip is going to outperform weighted polling data and historical precedent over multiple predictions.