5 ms·

So, Nate Silver seems to deserve the even higher profile he'll have after this election, but his model is pretty explicitly not predicting the outcomes of any i

by pseut 14y ago

So, Nate Silver seems to deserve the even higher profile he'll have after this election, but his model is pretty explicitly not predicting the outcomes of any individual states. They're giving estimated probabilities of the various outcomes, and the (presumptive) outcome that happened was given a probability of ~20%, which was more likely than any other particular outcome, but a far cry from "calling the states." The big difference is that if this were to happen over and over again ("this" meaning that the most likely predicted outcome actually materializes), that would be (weak) evidence against his model -- if the model's right, he should "correctly call" the states only about 20% of the time. No more and no less. But good for him, because he's had a lot of unjustified criticism for the last month or so, so he's due some (slightly) unjustified praise.

trhtrsh 14y ago
Huh? http://fivethirtyeight.blogs.nytimes.com/ http://fivethirtyeight.blogs.nytimes.com/ "State-by-State Probabilities"
- pseut 14y ago
  Do any of them equal 1? Saying that >50% probability means that the model picks the state for Obama would be fine as a classification rule, but he's trying to estimate the probabilities, not forecast who will win. The ~20% comes from the electoral vote distribution, where you can back out the mix of states that gives 332 Electoral college votes.
- furyofantares 14y ago
  Take Florida, for example. He predicted a 50.3% chance of an Obama win. If Romney actually wins it, he's only a little bit wrong - counting him as 0/1 on Florida in this case wouldn't be accurate. If Obama wins it, he's only a little bit right - counting him as 1/1 on Florida here wouldn't be accurate. It's hard to quantify just how close he was. I give him props because his prediction was that Florida was very close and that turned out to be true no matter which way the state actually ends up going in the final result. Now take Virginia, which was one of the later states to be called. He had Obama at 79.4% chance to win. But it turned out to be reasonably close -- which makes me wonder if we really had enough data to justify such a large %. It's a highly polled state, so maybe we did and we just came somewhat close to hitting that 20% of the time where all the polling was inaccurate. Or maybe he had Obama's chances too high. Another way to judge a prediction model like this is to credit him with .794/1 for Virginia, rather that 1/1. For Florida he gets .503 if Obama wins, and .497 if Romney wins. That doesn't capture it perfectly, either, but at least it doesn't give him 0/1 in Florida if Romney wins even though he successfully predicted that the state was super close.
  - apl 14y ago
    http://en.wikipedia.org/wiki/Scoring_rule http://en.wikipedia.org/wiki/Scoring_rule
    - pseut 14y ago
      It's not clear whether you mean that link as a rejoinder or as a supporting argument for the parent. My take is that it supports the parent.
  - svdad 14y ago
    I think you're confusing probability of winning with winning margin. His 79.4% number is his estimate of the probability that Obama would win Virginia by any margin. I don't think it's correct to say he's "a little bit right" -- either his prediction is correct, or it isn't. The only way I know to gauge the accuracy of his actual probabilities would be to run the election multiple times -- but there might be a more statistically advanced technique I'm not aware of.
    - furyofantares 14y ago
      I'm not confusing the two - I'm saying the fact that Florida was so close indicates Nate may have been doing about as good as possible with the data available in Florida, while the fact that Virginia was so close might indicate that Nate might not have done as well as possible with the data there. Neither case is at all conclusive. It's possible the quality of the data was just different in the two states, due to random factors in sampling or random factors on election day. And you are right, we couldn't know without running it multiple times with everything the same except these random factors.
      - svdad 14y ago
        Whether the race was close has nothing to do with how likely one side was to win. If there are 100 voters and I know with absolute certainty that 51 of them are die-hard republicans and 49 of them are die-hard democrats, I'm probably justified in saying that the republican candidate has some large probability of winning, 90% or something depending on actuarial probability that some of the republicans die etc. But that's a 90% probability of winning by a 51/49 margin. Alternately if there are 10 republicans, 10 democrats, and 80 independents I may have no idea how the independents will vote so AFAIK it's equally likely that either candidate wins. The margin of victory has nothing to do with the probability of the outcome.
- danielweber 14y ago
  If Silver said there was a 60% chance that each of 10 states would go to Obama, and they all went to Obama, by Silver's own standards he messed up. Only 6 of them should have gone to Obama. Since he had so many that were much less than 99%, saying "he got them all right" means he didn't get it right by his own definition.
  - bryanlarsen 14y ago
    Given 9 swing states, that gives 512 different possibilities. Nate ranked these possibilities. The actual result (assuming Florida ends up democratic) was his top pick, with a 20% probability. (Unfortunately, this site no longer appears to be available in its previous form). In 2008, he only got 49/50 of the states right. Yet pundits still call this a victory for Nate. To judge the accuracy of silver's prognostications, take the top 5 or so of his "most probable scenarios", which added to gether give a probability way above 50%. Did one of these scenarios manifest? Yes, his top scenario manifested. His second most probable scenario included a Republican Florida, which still might happen. So yes, Nate nailed it.
    - pseut 14y ago
      Like I said, I'm happy for him to get credit for nailing it. But I'm pretty sure that conclusion's based on misunderstanding his forecasts. Unless I missed it somewhere, he's never said "VA is going to Obama, IA's going to Obama, NC's going to Romney, etc." If he has, please provide a link. For analogy, let's say I have a biased coin. I tell you that I'm pretty sure it comes up heads 75% of the time. Then we flip it and it comes up heads. Did I "nail it?" No. The coin coming up heads is stronger evidence in favor my "model" (P[heads] = 0.75) than if it had come up tails, but it's pretty far from conclusive. The logic doesn't change if I tell you that the probability of heads is 90%, 99.9%, or whatever. (but, if I say it's 99.9% and it comes up tails, I'll concede that the model's wrong. If the prediction is extreme enough, a single observation can invalidate the model). To extend the analogy, we only know if I have a good model for this biased coin if we flip it a bunch of times and roughly 3/4 of the outcomes are heads. If we can't do that, because the coin flip is a one-off event (stretching the analogy quite a bit, but whatever), we still could tell if I'm good at modeling coin flips: if you give me a bunch of different coins, I give you a probability of heads for each one, and then we see whether the proportion of heads after we flip each one is close to the average of the probabilities I gave (i.e., let B(i) be 1 if the ith coin is heads, 0 if it's tails. Then n^{-1/2} (B(1) - Prob[coin 1 is heads] + B(2) - Prob[coin 2 is heads] + ... + B(n) - Prob[coin n is heads]) obeys a CLT and -- if the probabilities are right -- it becomes normal with mean zero and known variance as n gets large). If I tell you that each probability is .75 or some other number greater than 0.5 but less than 1, and all the coins always come up heads than I'm not "nailing it." But, I want to emphasize, it's premature to say that that's what's going on with Silver's predictions. And from what I've read of his blog, he completely understands this and explains it well. edit: minor change for clarity in the first and third paragraphs.

by pseut 14y ago

trhtrsh 14y ago

Huh? http://fivethirtyeight.blogs.nytimes.com/ http://fivethirtyeight.blogs.nytimes.com/ "State-by-State Probabilities"

pseut 14y ago
Do any of them equal 1? Saying that >50% probability means that the model picks the state for Obama would be fine as a classification rule, but he's trying to estimate the probabilities, not forecast who will win. The ~20% comes from the electoral vote distribution, where you can back out the mix of states that gives 332 Electoral college votes.
furyofantares 14y ago
Take Florida, for example. He predicted a 50.3% chance of an Obama win. If Romney actually wins it, he's only a little bit wrong - counting him as 0/1 on Florida in this case wouldn't be accurate. If Obama wins it, he's only a little bit right - counting him as 1/1 on Florida here wouldn't be accurate. It's hard to quantify just how close he was. I give him props because his prediction was that Florida was very close and that turned out to be true no matter which way the state actually ends up going in the final result. Now take Virginia, which was one of the later states to be called. He had Obama at 79.4% chance to win. But it turned out to be reasonably close -- which makes me wonder if we really had enough data to justify such a large %. It's a highly polled state, so maybe we did and we just came somewhat close to hitting that 20% of the time where all the polling was inaccurate. Or maybe he had Obama's chances too high. Another way to judge a prediction model like this is to credit him with .794/1 for Virginia, rather that 1/1. For Florida he gets .503 if Obama wins, and .497 if Romney wins. That doesn't capture it perfectly, either, but at least it doesn't give him 0/1 in Florida if Romney wins even though he successfully predicted that the state was super close.
- apl 14y ago
  http://en.wikipedia.org/wiki/Scoring_rule http://en.wikipedia.org/wiki/Scoring_rule
  - pseut 14y ago
    It's not clear whether you mean that link as a rejoinder or as a supporting argument for the parent. My take is that it supports the parent.
- svdad 14y ago
  I think you're confusing probability of winning with winning margin. His 79.4% number is his estimate of the probability that Obama would win Virginia by any margin. I don't think it's correct to say he's "a little bit right" -- either his prediction is correct, or it isn't. The only way I know to gauge the accuracy of his actual probabilities would be to run the election multiple times -- but there might be a more statistically advanced technique I'm not aware of.
  - furyofantares 14y ago
    I'm not confusing the two - I'm saying the fact that Florida was so close indicates Nate may have been doing about as good as possible with the data available in Florida, while the fact that Virginia was so close might indicate that Nate might not have done as well as possible with the data there. Neither case is at all conclusive. It's possible the quality of the data was just different in the two states, due to random factors in sampling or random factors on election day. And you are right, we couldn't know without running it multiple times with everything the same except these random factors.
    - svdad 14y ago
      Whether the race was close has nothing to do with how likely one side was to win. If there are 100 voters and I know with absolute certainty that 51 of them are die-hard republicans and 49 of them are die-hard democrats, I'm probably justified in saying that the republican candidate has some large probability of winning, 90% or something depending on actuarial probability that some of the republicans die etc. But that's a 90% probability of winning by a 51/49 margin. Alternately if there are 10 republicans, 10 democrats, and 80 independents I may have no idea how the independents will vote so AFAIK it's equally likely that either candidate wins. The margin of victory has nothing to do with the probability of the outcome.
danielweber 14y ago
If Silver said there was a 60% chance that each of 10 states would go to Obama, and they all went to Obama, by Silver's own standards he messed up. Only 6 of them should have gone to Obama. Since he had so many that were much less than 99%, saying "he got them all right" means he didn't get it right by his own definition.
- bryanlarsen 14y ago
  Given 9 swing states, that gives 512 different possibilities. Nate ranked these possibilities. The actual result (assuming Florida ends up democratic) was his top pick, with a 20% probability. (Unfortunately, this site no longer appears to be available in its previous form). In 2008, he only got 49/50 of the states right. Yet pundits still call this a victory for Nate. To judge the accuracy of silver's prognostications, take the top 5 or so of his "most probable scenarios", which added to gether give a probability way above 50%. Did one of these scenarios manifest? Yes, his top scenario manifested. His second most probable scenario included a Republican Florida, which still might happen. So yes, Nate nailed it.
  - pseut 14y ago
    Like I said, I'm happy for him to get credit for nailing it. But I'm pretty sure that conclusion's based on misunderstanding his forecasts. Unless I missed it somewhere, he's never said "VA is going to Obama, IA's going to Obama, NC's going to Romney, etc." If he has, please provide a link. For analogy, let's say I have a biased coin. I tell you that I'm pretty sure it comes up heads 75% of the time. Then we flip it and it comes up heads. Did I "nail it?" No. The coin coming up heads is stronger evidence in favor my "model" (P[heads] = 0.75) than if it had come up tails, but it's pretty far from conclusive. The logic doesn't change if I tell you that the probability of heads is 90%, 99.9%, or whatever. (but, if I say it's 99.9% and it comes up tails, I'll concede that the model's wrong. If the prediction is extreme enough, a single observation can invalidate the model). To extend the analogy, we only know if I have a good model for this biased coin if we flip it a bunch of times and roughly 3/4 of the outcomes are heads. If we can't do that, because the coin flip is a one-off event (stretching the analogy quite a bit, but whatever), we still could tell if I'm good at modeling coin flips: if you give me a bunch of different coins, I give you a probability of heads for each one, and then we see whether the proportion of heads after we flip each one is close to the average of the probabilities I gave (i.e., let B(i) be 1 if the ith coin is heads, 0 if it's tails. Then n^{-1/2} (B(1) - Prob[coin 1 is heads] + B(2) - Prob[coin 2 is heads] + ... + B(n) - Prob[coin n is heads]) obeys a CLT and -- if the probabilities are right -- it becomes normal with mean zero and known variance as n gets large). If I tell you that each probability is .75 or some other number greater than 0.5 but less than 1, and all the coins always come up heads than I'm not "nailing it." But, I want to emphasize, it's premature to say that that's what's going on with Silver's predictions. And from what I've read of his blog, he completely understands this and explains it well. edit: minor change for clarity in the first and third paragraphs.