3 ms·

If Silver said there was a 60% chance that each of 10 states would go to Obama, and they all went to Obama, by Silver's own standards he messed up. Only 6 of t

If Silver said there was a 60% chance that each of 10 states would go to Obama, and they all went to Obama, by Silver's own standards he messed up. Only 6 of them should have gone to Obama. Since he had so many that were much less than 99%, saying "he got them all right" means he didn't get it right by his own definition.

bryanlarsen 14y ago
Given 9 swing states, that gives 512 different possibilities. Nate ranked these possibilities. The actual result (assuming Florida ends up democratic) was his top pick, with a 20% probability. (Unfortunately, this site no longer appears to be available in its previous form). In 2008, he only got 49/50 of the states right. Yet pundits still call this a victory for Nate. To judge the accuracy of silver's prognostications, take the top 5 or so of his "most probable scenarios", which added to gether give a probability way above 50%. Did one of these scenarios manifest? Yes, his top scenario manifested. His second most probable scenario included a Republican Florida, which still might happen. So yes, Nate nailed it.
- pseut 14y ago
  Like I said, I'm happy for him to get credit for nailing it. But I'm pretty sure that conclusion's based on misunderstanding his forecasts. Unless I missed it somewhere, he's never said "VA is going to Obama, IA's going to Obama, NC's going to Romney, etc." If he has, please provide a link. For analogy, let's say I have a biased coin. I tell you that I'm pretty sure it comes up heads 75% of the time. Then we flip it and it comes up heads. Did I "nail it?" No. The coin coming up heads is stronger evidence in favor my "model" (P[heads] = 0.75) than if it had come up tails, but it's pretty far from conclusive. The logic doesn't change if I tell you that the probability of heads is 90%, 99.9%, or whatever. (but, if I say it's 99.9% and it comes up tails, I'll concede that the model's wrong. If the prediction is extreme enough, a single observation can invalidate the model). To extend the analogy, we only know if I have a good model for this biased coin if we flip it a bunch of times and roughly 3/4 of the outcomes are heads. If we can't do that, because the coin flip is a one-off event (stretching the analogy quite a bit, but whatever), we still could tell if I'm good at modeling coin flips: if you give me a bunch of different coins, I give you a probability of heads for each one, and then we see whether the proportion of heads after we flip each one is close to the average of the probabilities I gave (i.e., let B(i) be 1 if the ith coin is heads, 0 if it's tails. Then n^{-1/2} (B(1) - Prob[coin 1 is heads] + B(2) - Prob[coin 2 is heads] + ... + B(n) - Prob[coin n is heads]) obeys a CLT and -- if the probabilities are right -- it becomes normal with mean zero and known variance as n gets large). If I tell you that each probability is .75 or some other number greater than 0.5 but less than 1, and all the coins always come up heads than I'm not "nailing it." But, I want to emphasize, it's premature to say that that's what's going on with Silver's predictions. And from what I've read of his blog, he completely understands this and explains it well. edit: minor change for clarity in the first and third paragraphs.
  - ncallaway 14y ago
    I think there's a minor error in the analogy of "state results" to "coin flip". In the coin flip scenario each throw of the biased coins are independent. How one lands does not effect the other. The same is likely not true for how states end-up voting. This doesn't entirely invalidate your point (yes, if you "re-ran" the election repeatedly and he "nailed it" everytime we might conclude his probabilities were incorrect), but it does explain why if 10 "60% Obaba" states vote we might not necessarily expect 6 of them to come up for Obama...
    - pseut 14y ago
      > In the coin flip scenario each throw of the biased coins are independent. How one lands does not effect the other. The same is likely not true for how states end-up voting. For sure. If I have a point, it's that there's not enough information in any election to claim that he "nailed it." Since the outcome he said was most likely seems to have occurred, it definitely supports his model and his approach. But you'd need to see a longer track record than exists to be sure. (think of each coin flip as being a separate national election). Independence is a little bit of a red herring, though. There are LLNs and CLTs that allow for weak dependence between the observations, and any strong/systematic dependence between the states belongs in the model (and I think is in Silver's model, but I could be misremembering). And we would expect that 6 of 10 "60% Obama" states would come up for Obama. The interdependence is going to affect the variance but not the mean. So we should expect to see (say) 8 or more of 10 going to Obama happen more frequently than a naive binomial(.6, 10) distribution would predict.
  - bryanlarsen 14y ago
    No, he never said "VA is going to Obama, ...". He said "the probability of VA going to Obama and IA going to Obama and NC going to Romney ..." is X%. This used to be at the "Paths to Victory" on the NYT, but they've been collapsed now. We can say that somebody "nails" a prediction through both specificity and repetition. Nate had a modest amount of both. 50/50 and 49/50 are pretty specific. He did that well in 2008 and 2012, and similarly in 2010.
cynicalkane 14y ago
No, the uncertainty is correlated. Each individual state might have a 60% chance for Obama but all of them will tend to vote the same way.
josephlord 14y ago
You are assuming that they are all independent events. What if in the calculation to get to 60% the main risk factor was a late national swing rather than a local polling error?

bryanlarsen 14y ago

Given 9 swing states, that gives 512 different possibilities. Nate ranked these possibilities. The actual result (assuming Florida ends up democratic) was his top pick, with a 20% probability. (Unfortunately, this site no longer appears to be available in its previous form). In 2008, he only got 49/50 of the states right. Yet pundits still call this a victory for Nate. To judge the accuracy of silver's prognostications, take the top 5 or so of his "most probable scenarios", which added to gether give a probability way above 50%. Did one of these scenarios manifest? Yes, his top scenario manifested. His second most probable scenario included a Republican Florida, which still might happen. So yes, Nate nailed it.

pseut 14y ago
Like I said, I'm happy for him to get credit for nailing it. But I'm pretty sure that conclusion's based on misunderstanding his forecasts. Unless I missed it somewhere, he's never said "VA is going to Obama, IA's going to Obama, NC's going to Romney, etc." If he has, please provide a link. For analogy, let's say I have a biased coin. I tell you that I'm pretty sure it comes up heads 75% of the time. Then we flip it and it comes up heads. Did I "nail it?" No. The coin coming up heads is stronger evidence in favor my "model" (P[heads] = 0.75) than if it had come up tails, but it's pretty far from conclusive. The logic doesn't change if I tell you that the probability of heads is 90%, 99.9%, or whatever. (but, if I say it's 99.9% and it comes up tails, I'll concede that the model's wrong. If the prediction is extreme enough, a single observation can invalidate the model). To extend the analogy, we only know if I have a good model for this biased coin if we flip it a bunch of times and roughly 3/4 of the outcomes are heads. If we can't do that, because the coin flip is a one-off event (stretching the analogy quite a bit, but whatever), we still could tell if I'm good at modeling coin flips: if you give me a bunch of different coins, I give you a probability of heads for each one, and then we see whether the proportion of heads after we flip each one is close to the average of the probabilities I gave (i.e., let B(i) be 1 if the ith coin is heads, 0 if it's tails. Then n^{-1/2} (B(1) - Prob[coin 1 is heads] + B(2) - Prob[coin 2 is heads] + ... + B(n) - Prob[coin n is heads]) obeys a CLT and -- if the probabilities are right -- it becomes normal with mean zero and known variance as n gets large). If I tell you that each probability is .75 or some other number greater than 0.5 but less than 1, and all the coins always come up heads than I'm not "nailing it." But, I want to emphasize, it's premature to say that that's what's going on with Silver's predictions. And from what I've read of his blog, he completely understands this and explains it well. edit: minor change for clarity in the first and third paragraphs.
- ncallaway 14y ago
  I think there's a minor error in the analogy of "state results" to "coin flip". In the coin flip scenario each throw of the biased coins are independent. How one lands does not effect the other. The same is likely not true for how states end-up voting. This doesn't entirely invalidate your point (yes, if you "re-ran" the election repeatedly and he "nailed it" everytime we might conclude his probabilities were incorrect), but it does explain why if 10 "60% Obaba" states vote we might not necessarily expect 6 of them to come up for Obama...
  - pseut 14y ago
    > In the coin flip scenario each throw of the biased coins are independent. How one lands does not effect the other. The same is likely not true for how states end-up voting. For sure. If I have a point, it's that there's not enough information in any election to claim that he "nailed it." Since the outcome he said was most likely seems to have occurred, it definitely supports his model and his approach. But you'd need to see a longer track record than exists to be sure. (think of each coin flip as being a separate national election). Independence is a little bit of a red herring, though. There are LLNs and CLTs that allow for weak dependence between the observations, and any strong/systematic dependence between the states belongs in the model (and I think is in Silver's model, but I could be misremembering). And we would expect that 6 of 10 "60% Obama" states would come up for Obama. The interdependence is going to affect the variance but not the mean. So we should expect to see (say) 8 or more of 10 going to Obama happen more frequently than a naive binomial(.6, 10) distribution would predict.
- bryanlarsen 14y ago
  No, he never said "VA is going to Obama, ...". He said "the probability of VA going to Obama and IA going to Obama and NC going to Romney ..." is X%. This used to be at the "Paths to Victory" on the NYT, but they've been collapsed now. We can say that somebody "nails" a prediction through both specificity and repetition. Nate had a modest amount of both. 50/50 and 49/50 are pretty specific. He did that well in 2008 and 2012, and similarly in 2010.

cynicalkane 14y ago

No, the uncertainty is correlated. Each individual state might have a 60% chance for Obama but all of them will tend to vote the same way.

josephlord 14y ago

You are assuming that they are all independent events. What if in the calculation to get to 60% the main risk factor was a late national swing rather than a local polling error?