42 ms·

Your two paragraphs have no relationship. How it could possibly "turn out" that there's a 5% chance of Trump winning? Was the election run 100 times? The accur

Your two paragraphs have no relationship. How it could possibly "turn out" that there's a 5% chance of Trump winning? Was the election run 100 times? The accuracy of the polls is another matter entirely. You could estimate the chances that your polling was wrong, given that the selection of the sample is a random variable, people lie, and intentions change, but that's not at all what the general public thinks the number means.

astrange 4y ago
> Your two paragraphs have no relationship. How it could possibly "turn out" that there's a 5% chance of Trump winning? Was the election run 100 times? You can calibrate it by looking at their other election predictions and seeing how generally on target it is. https://projects.fivethirtyeight.com/checking-our-work/ https://projects.fivethirtyeight.com/checking-our-work/
- AlbertCory 4y ago
  no, you can only "calibrate" his prediction abilities in general: "given 100 predictions by Nate Silver, how many will be correct?" that's what you just said. In other words, if we forgot what actually happened, picked one past prediction out of a hat, asked "so was he right?" and then checked the results, he would be correct 95% of the time. That's what you're saying.
  - dragonwriter 4y ago
    No, its saying... Well, here: https://projects.fivethirtyeight.com/checking-our-work/ https://projects.fivethirtyeight.com/checking-our-work/
    - AlbertCory 4y ago
      OK, I read that. It's telling that he uses MLB games, rather than Presidential elections, which are not covered at all in that article. Why? There are many thousands of MLB games. There are a few thousand US House elections. There are only a handful of Presidential elections. It is not possible to measure their accuracy on those with any statistical rigor.
      - dragonwriter 4y ago
        > OK, I read that. Not very well. > It’s telling that he uses MLB games, rather than Presidential elections, which are not covered at all in that article. All the types of predictions 538 does are covered in the Brier skills chart on the overall summary; there’s detailed analysis covering each type of prediction (with further available breakdowns for subtypes available once you’ve gone into the type) available from the dropdown at the top of the page.
        AlbertCory 4y ago
        OK, I missed the dropdown at the top. There is not "detailed analysis" for Presidential elections, although I guess you mean that each state is a data point. All that graph shows is that their "probability" is roughly in the ballpark with huge errors in the middle of the range, which is probably the swing states we care about most. I'll suggest one concrete case where their probability of a candidate winning a 2-person race has some exact meaning: *You'll pick a number randomly from 1 to 100. If it's less than or equal to candidate A's "probability" of winning, then you bet $10,000 on A. Otherwise you bet $10,000 on B* (we need to put real skin in the game) Are you willing to abide by that rule? If not, you don't really believe the probability.
        dragonwriter 4y ago
        > There is not “detailed analysis” for Presidential elections There is, in fact, detailed analysis for Presidential elections. > although I guess you mean that each state is a data point. No, each individual prediction of each of the 56 (enumerated upthread) distinct elections for one or more Presidential electors is a data point. > Are you willing to abide by that rule? If not, you don’t really believe the probability. Well, no, not being willing to abide by that rule means either not being rich enough that winning and losing $10,000 are roughly symmetric in utility or not being an SBF-style nutball and demanding a non-zero risk premium. As it turns out, I am in both categories.
        AlbertCory 4y ago
  - astrange 4y ago
    > no, you can only "calibrate" his prediction abilities in general: Yeah, but that's what we care about - the past has already happened so we only care about future events.
    - AlbertCory 4y ago
      OK, so maybe we're agreed: you can bet on his abilities in one election. Let's say he's been right 95% of the time (I don't know if that's true) and we believe that's likely to continue, knowing nothing except "this is an election and Silver's predicted the result." Then if he says "Hillary is likely to win" we can have 95% confidence he's right. If he says "Hillary has an 80% chance of winning" we ignore the 80, and just observe that it's more than 50.
      - astrange 4y ago
        It's a bit more flexible than that, though - rather than ignore 80% vs 70% and make it all up or down, you can let him predict a few different events in a row, add up the errors, and see how off they are. Or if you do want to review the past, you can look at the error for a category of elections or that entire year rather than his whole prediction career.
        AlbertCory 4y ago
        > predict a few different events in a row, add up the errors, and see how off they are. I'm not seeing a formula there.
        dllthomas 4y ago
        How about here? https://en.wikipedia.org/wiki/Scoring_rule https://en.wikipedia.org/wiki/Scoring_rule
        AlbertCory 4y ago
        that's the definition of a rule. > One could note the number of times that a 25% probability was quoted, over a long period, and compare this with the actual proportion of times that rain fell. it still depends on many samples, or "over a long period" in your doc. You can't escape the fact that there are only one or two samples, no matter how much math you throw around.
  - edmundsauto 4y ago
    Correctness is actually a pretty bad evaluation because it reduces the outcome to a binary. There are better measures like a brier score that IIRC, they have pretty rigorously evaluated on their own predictions.
hnfong 4y ago
Thanks. It's reassuring to know that there still people who understand what probabilities actually mean, especially for one-off events.

astrange 4y ago

> Your two paragraphs have no relationship. How it could possibly "turn out" that there's a 5% chance of Trump winning? Was the election run 100 times? You can calibrate it by looking at their other election predictions and seeing how generally on target it is. https://projects.fivethirtyeight.com/checking-our-work/ https://projects.fivethirtyeight.com/checking-our-work/

AlbertCory 4y ago
no, you can only "calibrate" his prediction abilities in general: "given 100 predictions by Nate Silver, how many will be correct?" that's what you just said. In other words, if we forgot what actually happened, picked one past prediction out of a hat, asked "so was he right?" and then checked the results, he would be correct 95% of the time. That's what you're saying.
- dragonwriter 4y ago
  No, its saying... Well, here: https://projects.fivethirtyeight.com/checking-our-work/ https://projects.fivethirtyeight.com/checking-our-work/
  - AlbertCory 4y ago
    OK, I read that. It's telling that he uses MLB games, rather than Presidential elections, which are not covered at all in that article. Why? There are many thousands of MLB games. There are a few thousand US House elections. There are only a handful of Presidential elections. It is not possible to measure their accuracy on those with any statistical rigor.
    - dragonwriter 4y ago
      > OK, I read that. Not very well. > It’s telling that he uses MLB games, rather than Presidential elections, which are not covered at all in that article. All the types of predictions 538 does are covered in the Brier skills chart on the overall summary; there’s detailed analysis covering each type of prediction (with further available breakdowns for subtypes available once you’ve gone into the type) available from the dropdown at the top of the page.
      - AlbertCory 4y ago
        OK, I missed the dropdown at the top. There is not "detailed analysis" for Presidential elections, although I guess you mean that each state is a data point. All that graph shows is that their "probability" is roughly in the ballpark with huge errors in the middle of the range, which is probably the swing states we care about most. I'll suggest one concrete case where their probability of a candidate winning a 2-person race has some exact meaning: *You'll pick a number randomly from 1 to 100. If it's less than or equal to candidate A's "probability" of winning, then you bet $10,000 on A. Otherwise you bet $10,000 on B* (we need to put real skin in the game) Are you willing to abide by that rule? If not, you don't really believe the probability.
        dragonwriter 4y ago
        > There is not “detailed analysis” for Presidential elections There is, in fact, detailed analysis for Presidential elections. > although I guess you mean that each state is a data point. No, each individual prediction of each of the 56 (enumerated upthread) distinct elections for one or more Presidential electors is a data point. > Are you willing to abide by that rule? If not, you don’t really believe the probability. Well, no, not being willing to abide by that rule means either not being rich enough that winning and losing $10,000 are roughly symmetric in utility or not being an SBF-style nutball and demanding a non-zero risk premium. As it turns out, I am in both categories.
        AlbertCory 4y ago
- astrange 4y ago
  > no, you can only "calibrate" his prediction abilities in general: Yeah, but that's what we care about - the past has already happened so we only care about future events.
  - AlbertCory 4y ago
    OK, so maybe we're agreed: you can bet on his abilities in one election. Let's say he's been right 95% of the time (I don't know if that's true) and we believe that's likely to continue, knowing nothing except "this is an election and Silver's predicted the result." Then if he says "Hillary is likely to win" we can have 95% confidence he's right. If he says "Hillary has an 80% chance of winning" we ignore the 80, and just observe that it's more than 50.
    - astrange 4y ago
      It's a bit more flexible than that, though - rather than ignore 80% vs 70% and make it all up or down, you can let him predict a few different events in a row, add up the errors, and see how off they are. Or if you do want to review the past, you can look at the error for a category of elections or that entire year rather than his whole prediction career.
      - AlbertCory 4y ago
        > predict a few different events in a row, add up the errors, and see how off they are. I'm not seeing a formula there.
        dllthomas 4y ago
        How about here? https://en.wikipedia.org/wiki/Scoring_rule https://en.wikipedia.org/wiki/Scoring_rule
        AlbertCory 4y ago
        that's the definition of a rule. > One could note the number of times that a 25% probability was quoted, over a long period, and compare this with the actual proportion of times that rain fell. it still depends on many samples, or "over a long period" in your doc. You can't escape the fact that there are only one or two samples, no matter how much math you throw around.
- edmundsauto 4y ago
  Correctness is actually a pretty bad evaluation because it reduces the outcome to a binary. There are better measures like a brier score that IIRC, they have pretty rigorously evaluated on their own predictions.

hnfong 4y ago

Thanks. It's reassuring to know that there still people who understand what probabilities actually mean, especially for one-off events.