4 ms·

Turns out when statistics say there is a 5% chance of something happening, there's about a 5% chance of it happening. Also, for 2020, it turns out the actual v

by titzer 4y ago

Turns out when statistics say there is a 5% chance of something happening, there's about a 5% chance of it happening. Also, for 2020, it turns out the actual vote split between the two US presidential candidates was pretty much exactly in line with polls.

AlbertCory 4y ago
Your two paragraphs have no relationship. How it could possibly "turn out" that there's a 5% chance of Trump winning? Was the election run 100 times? The accuracy of the polls is another matter entirely. You could estimate the chances that your polling was wrong, given that the selection of the sample is a random variable, people lie, and intentions change, but that's not at all what the general public thinks the number means.
- astrange 4y ago
  > Your two paragraphs have no relationship. How it could possibly "turn out" that there's a 5% chance of Trump winning? Was the election run 100 times? You can calibrate it by looking at their other election predictions and seeing how generally on target it is. https://projects.fivethirtyeight.com/checking-our-work/ https://projects.fivethirtyeight.com/checking-our-work/
  - AlbertCory 4y ago
    no, you can only "calibrate" his prediction abilities in general: "given 100 predictions by Nate Silver, how many will be correct?" that's what you just said. In other words, if we forgot what actually happened, picked one past prediction out of a hat, asked "so was he right?" and then checked the results, he would be correct 95% of the time. That's what you're saying.
    - dragonwriter 4y ago
      No, its saying... Well, here: https://projects.fivethirtyeight.com/checking-our-work/ https://projects.fivethirtyeight.com/checking-our-work/
      - AlbertCory 4y ago
        OK, I read that. It's telling that he uses MLB games, rather than Presidential elections, which are not covered at all in that article. Why? There are many thousands of MLB games. There are a few thousand US House elections. There are only a handful of Presidential elections. It is not possible to measure their accuracy on those with any statistical rigor.
        dragonwriter 4y ago
        > OK, I read that. Not very well. > It’s telling that he uses MLB games, rather than Presidential elections, which are not covered at all in that article. All the types of predictions 538 does are covered in the Brier skills chart on the overall summary; there’s detailed analysis covering each type of prediction (with further available breakdowns for subtypes available once you’ve gone into the type) available from the dropdown at the top of the page.
        AlbertCory 4y ago
        OK, I missed the dropdown at the top. There is not "detailed analysis" for Presidential elections, although I guess you mean that each state is a data point. All that graph shows is that their "probability" is roughly in the ballpark with huge errors in the middle of the range, which is probably the swing states we care about most. I'll suggest one concrete case where their probability of a candidate winning a 2-person race has some exact meaning: *You'll pick a number randomly from 1 to 100. If it's less than or equal to candidate A's "probability" of winning, then you bet $10,000 on A. Otherwise you bet $10,000 on B* (we need to put real skin in the game) Are you willing to abide by that rule? If not, you don't really believe the probability.
        dragonwriter 4y ago
        > There is not “detailed analysis” for Presidential elections There is, in fact, detailed analysis for Presidential elections. > although I guess you mean that each state is a data point. No, each individual prediction of each of the 56 (enumerated upthread) distinct elections for one or more Presidential electors is a data point. > Are you willing to abide by that rule? If not, you don’t really believe the probability. Well, no, not being willing to abide by that rule means either not being rich enough that winning and losing $10,000 are roughly symmetric in utility or not being an SBF-style nutball and demanding a non-zero risk premium. As it turns out, I am in both categories.
    - astrange 4y ago
      > no, you can only "calibrate" his prediction abilities in general: Yeah, but that's what we care about - the past has already happened so we only care about future events.
      - AlbertCory 4y ago
        OK, so maybe we're agreed: you can bet on his abilities in one election. Let's say he's been right 95% of the time (I don't know if that's true) and we believe that's likely to continue, knowing nothing except "this is an election and Silver's predicted the result." Then if he says "Hillary is likely to win" we can have 95% confidence he's right. If he says "Hillary has an 80% chance of winning" we ignore the 80, and just observe that it's more than 50.
        astrange 4y ago
        It's a bit more flexible than that, though - rather than ignore 80% vs 70% and make it all up or down, you can let him predict a few different events in a row, add up the errors, and see how off they are. Or if you do want to review the past, you can look at the error for a category of elections or that entire year rather than his whole prediction career.
        AlbertCory 4y ago
        > predict a few different events in a row, add up the errors, and see how off they are. I'm not seeing a formula there.
        dllthomas 4y ago
        How about here? https://en.wikipedia.org/wiki/Scoring_rule https://en.wikipedia.org/wiki/Scoring_rule
        AlbertCory 4y ago
        that's the definition of a rule. > One could note the number of times that a 25% probability was quoted, over a long period, and compare this with the actual proportion of times that rain fell. it still depends on many samples, or "over a long period" in your doc. You can't escape the fact that there are only one or two samples, no matter how much math you throw around.
    - edmundsauto 4y ago
      Correctness is actually a pretty bad evaluation because it reduces the outcome to a binary. There are better measures like a brier score that IIRC, they have pretty rigorously evaluated on their own predictions.
- hnfong 4y ago
  Thanks. It's reassuring to know that there still people who understand what probabilities actually mean, especially for one-off events.
beerpls 4y ago
Now that’s a funny one With a straight face you’re going to say that in 2016 you saw a bunch of polls coming out showing trump winning? Absolutely not. Nobody, and I mean nobody, gave him a snowflakes chance. Everyone including Silver mentioned “the greatest election landslide in history”, or some variant. If there were polls showing trump winning they were hidden from ans downplayed in the msm. And to say anything else is a very poor lie.
- titzer 4y ago
  There weren't polls showing Trump winning the popular vote. And he didn't. https://projects.fivethirtyeight.com/2016-election-forecast/national-polls/ https://projects.fivethirtyeight.com/2016-election-forecast/... The last poll there from Nov 8 showed Clinton winning the popular vote by...wait for it...3.9 percent. She won the popular vote by...2.9 percent. National polls weren't far off at all. The electoral college was decided by less than 50,000 votes overall. No one predicted the margins in those key states would be so small. It was a razor-thin win. Statistics always have a margin of error and give probabilities. Polls didn't have to predict Trump winning. Instead, they give a probability of winning. FiveThirtyEight predicted a 28% chance of winning[2]. That's based on Monte Carlo simulations (that's where they run, e.g. 100 random elections and count how many times each outcome occurred). It actually doesn't matter if that estimate was low; it was high enough (1 in 4 chance) that the outcome wasn't insane. Even three days after the election, they already had an analysis of what they went wrong with polling[1]. I mean, you could read that or continue to trot out tired polarized arguments. [1] https://fivethirtyeight.com/features/why-fivethirtyeight-gave-trump-a-better-chance-than-almost-anyone-else/ https://fivethirtyeight.com/features/why-fivethirtyeight-gav... [2] https://projects.fivethirtyeight.com/2016-election-forecast/ https://projects.fivethirtyeight.com/2016-election-forecast/
  - beerpls 4y ago
    I love how the popular vote is only ever mentioned when it’s to defend the idea that all these analyses “were actually right” You’re not getting it. The analysis, the “28%” - those were wring. those were poor analyses Yea they came back after and made up some bs about “oh we know what we did wrong now!” When I was in school all the people who were cheating their way through math did the same thing. They couldn’t solve the problem, and the second you gave them the answer they magically “got it” Fortune telling is a business of luck. Unless you can convince your followers that you’re right even when you’re clearly wrong

by titzer 4y ago

AlbertCory 4y ago

Your two paragraphs have no relationship. How it could possibly "turn out" that there's a 5% chance of Trump winning? Was the election run 100 times? The accuracy of the polls is another matter entirely. You could estimate the chances that your polling was wrong, given that the selection of the sample is a random variable, people lie, and intentions change, but that's not at all what the general public thinks the number means.

astrange 4y ago
> Your two paragraphs have no relationship. How it could possibly "turn out" that there's a 5% chance of Trump winning? Was the election run 100 times? You can calibrate it by looking at their other election predictions and seeing how generally on target it is. https://projects.fivethirtyeight.com/checking-our-work/ https://projects.fivethirtyeight.com/checking-our-work/
- AlbertCory 4y ago
  no, you can only "calibrate" his prediction abilities in general: "given 100 predictions by Nate Silver, how many will be correct?" that's what you just said. In other words, if we forgot what actually happened, picked one past prediction out of a hat, asked "so was he right?" and then checked the results, he would be correct 95% of the time. That's what you're saying.
  - dragonwriter 4y ago
    No, its saying... Well, here: https://projects.fivethirtyeight.com/checking-our-work/ https://projects.fivethirtyeight.com/checking-our-work/
    - AlbertCory 4y ago
      OK, I read that. It's telling that he uses MLB games, rather than Presidential elections, which are not covered at all in that article. Why? There are many thousands of MLB games. There are a few thousand US House elections. There are only a handful of Presidential elections. It is not possible to measure their accuracy on those with any statistical rigor.
      - dragonwriter 4y ago
        > OK, I read that. Not very well. > It’s telling that he uses MLB games, rather than Presidential elections, which are not covered at all in that article. All the types of predictions 538 does are covered in the Brier skills chart on the overall summary; there’s detailed analysis covering each type of prediction (with further available breakdowns for subtypes available once you’ve gone into the type) available from the dropdown at the top of the page.
        AlbertCory 4y ago
        OK, I missed the dropdown at the top. There is not "detailed analysis" for Presidential elections, although I guess you mean that each state is a data point. All that graph shows is that their "probability" is roughly in the ballpark with huge errors in the middle of the range, which is probably the swing states we care about most. I'll suggest one concrete case where their probability of a candidate winning a 2-person race has some exact meaning: *You'll pick a number randomly from 1 to 100. If it's less than or equal to candidate A's "probability" of winning, then you bet $10,000 on A. Otherwise you bet $10,000 on B* (we need to put real skin in the game) Are you willing to abide by that rule? If not, you don't really believe the probability.
        dragonwriter 4y ago
        > There is not “detailed analysis” for Presidential elections There is, in fact, detailed analysis for Presidential elections. > although I guess you mean that each state is a data point. No, each individual prediction of each of the 56 (enumerated upthread) distinct elections for one or more Presidential electors is a data point. > Are you willing to abide by that rule? If not, you don’t really believe the probability. Well, no, not being willing to abide by that rule means either not being rich enough that winning and losing $10,000 are roughly symmetric in utility or not being an SBF-style nutball and demanding a non-zero risk premium. As it turns out, I am in both categories.
  - astrange 4y ago
    > no, you can only "calibrate" his prediction abilities in general: Yeah, but that's what we care about - the past has already happened so we only care about future events.
    - AlbertCory 4y ago
      OK, so maybe we're agreed: you can bet on his abilities in one election. Let's say he's been right 95% of the time (I don't know if that's true) and we believe that's likely to continue, knowing nothing except "this is an election and Silver's predicted the result." Then if he says "Hillary is likely to win" we can have 95% confidence he's right. If he says "Hillary has an 80% chance of winning" we ignore the 80, and just observe that it's more than 50.
      - astrange 4y ago
        It's a bit more flexible than that, though - rather than ignore 80% vs 70% and make it all up or down, you can let him predict a few different events in a row, add up the errors, and see how off they are. Or if you do want to review the past, you can look at the error for a category of elections or that entire year rather than his whole prediction career.
        AlbertCory 4y ago
        > predict a few different events in a row, add up the errors, and see how off they are. I'm not seeing a formula there.
        dllthomas 4y ago
        How about here? https://en.wikipedia.org/wiki/Scoring_rule https://en.wikipedia.org/wiki/Scoring_rule
        AlbertCory 4y ago
        that's the definition of a rule. > One could note the number of times that a 25% probability was quoted, over a long period, and compare this with the actual proportion of times that rain fell. it still depends on many samples, or "over a long period" in your doc. You can't escape the fact that there are only one or two samples, no matter how much math you throw around.
  - edmundsauto 4y ago
    Correctness is actually a pretty bad evaluation because it reduces the outcome to a binary. There are better measures like a brier score that IIRC, they have pretty rigorously evaluated on their own predictions.
hnfong 4y ago
Thanks. It's reassuring to know that there still people who understand what probabilities actually mean, especially for one-off events.

beerpls 4y ago

Now that’s a funny one With a straight face you’re going to say that in 2016 you saw a bunch of polls coming out showing trump winning? Absolutely not. Nobody, and I mean nobody, gave him a snowflakes chance. Everyone including Silver mentioned “the greatest election landslide in history”, or some variant. If there were polls showing trump winning they were hidden from ans downplayed in the msm. And to say anything else is a very poor lie.

titzer 4y ago
There weren't polls showing Trump winning the popular vote. And he didn't. https://projects.fivethirtyeight.com/2016-election-forecast/national-polls/ https://projects.fivethirtyeight.com/2016-election-forecast/... The last poll there from Nov 8 showed Clinton winning the popular vote by...wait for it...3.9 percent. She won the popular vote by...2.9 percent. National polls weren't far off at all. The electoral college was decided by less than 50,000 votes overall. No one predicted the margins in those key states would be so small. It was a razor-thin win. Statistics always have a margin of error and give probabilities. Polls didn't have to predict Trump winning. Instead, they give a probability of winning. FiveThirtyEight predicted a 28% chance of winning[2]. That's based on Monte Carlo simulations (that's where they run, e.g. 100 random elections and count how many times each outcome occurred). It actually doesn't matter if that estimate was low; it was high enough (1 in 4 chance) that the outcome wasn't insane. Even three days after the election, they already had an analysis of what they went wrong with polling[1]. I mean, you could read that or continue to trot out tired polarized arguments. [1] https://fivethirtyeight.com/features/why-fivethirtyeight-gave-trump-a-better-chance-than-almost-anyone-else/ https://fivethirtyeight.com/features/why-fivethirtyeight-gav... [2] https://projects.fivethirtyeight.com/2016-election-forecast/ https://projects.fivethirtyeight.com/2016-election-forecast/
- beerpls 4y ago
  I love how the popular vote is only ever mentioned when it’s to defend the idea that all these analyses “were actually right” You’re not getting it. The analysis, the “28%” - those were wring. those were poor analyses Yea they came back after and made up some bs about “oh we know what we did wrong now!” When I was in school all the people who were cheating their way through math did the same thing. They couldn’t solve the problem, and the second you gave them the answer they magically “got it” Fortune telling is a business of luck. Unless you can convince your followers that you’re right even when you’re clearly wrong