3 ms·

and turned out to be exactly correct in every meaningful way. Silver himself wouldn't call his predictions correct. He was not making thumbs-up v thumbs-down

and turned out to be exactly correct in every meaningful way. Silver himself wouldn't call his predictions correct. He was not making thumbs-up v thumbs-down predictions. When he said "60% chance of Obama winning this state," it means that 6 times out of 10 Obama would win and 4 times out of 10 Romney would win. If Obama won all 10 times, by his own account he would be wrong.

Wilduck 14y ago
> it means that 6 times out of 10 Obama would win and 4 times out of 10 Romney would win. If Obama won all 10 times, by his own account he would be wrong. That's a very frequentist perspective. I think the Bayesian interpretation is a little more sensible. Given a prior estimate, updated with the information we have, it is logical to assume that Obama has a better chance of winning. That is, Nate Silver's 60% doesn't mean that if the election in a given state were repeated, we'd see different results four times out of 10, but rather that his information in making the prediction was incomplete.
- danielweber 14y ago
  Nate Silver himself said (in a radio interview that I cannot locate despite much Googling :< ) that if he said something would happen 60% of the time and it was 10 out of 10 that he his figure was wrong and should have said 100%.
  - Wilduck 14y ago
    But you can't run the same election 10 times. Without hearing the interview, I'm going to guess he's talking about multiple states, all with 60% chance of winning, which puts us in a different place entirely. I guess I'm not disagreeing with the main point of your post. Just your use of the phrase "something would happen 60% of the time", as being oddly frequentist, when a Bayesian perspective is more appropriate here.
    - pseut 14y ago
      > But you can't run the same election 10 times. That makes assessment difficult, but is irrelevant for interpretation. If something happens 60% of the time, it happens with the same probability as drawing a red ball from an urn with 6 red balls and 4 green balls, whether it happens once or several times. Trying to determine the quality of a model using some observed data is inherently a frequentist exercise -- a dyed-in-the-wool Bayesian would take the election results, use them to update his or her posterior distributions, and carry on happily. (I know that no one would actually do this; no one who analyzes data is really a "pure" Bayesian or a "pure" frequentist).
jmilloy 14y ago
You're all over this thread with the same comment, and you're wrong. When an outcome has 60% likelihood of occurring, then it should occur 6 out of 10 times. But the election only happens once. The prediction isn't "wrong" when the outcome occurs 1 out of 1 time. In this case, you describe 10 separate events, each with 60% chance that a certain outcome occurs. So we have 10 separate outcomes that occurred 1 out of 1 times.
- danielweber 14y ago
  But the election only happens once. The prediction isn't "wrong" when the outcome occurs 1 out of 1 time. It's not a single event. Nate Silver made many many predictions about last night. And none[1] of them were binary up-down events. If he did make binary predictions, we could claim he got them "all right," but Silver was never doing that in the first place. Nor am I saying he should have. Try this: if Romney had won, would it mean that Nate Silver was wrong? Should we say, like Silver's critic Dylan Byers did, "it’s difficult to see how people can continue to put faith in the predictions of someone who has never given that candidate anything higher than a 41 percent chance"? No, we shouldn't. Because Silver didn't say "Obama would win." He said "Obama has a ~86% chance of winning." Silver gave a 14% chance of finding himself in a universe where Romney would win despite what his data was telling him. Silver made essentially no prediction about Florida -- and we shouldn't push him into making an up/down decision if he doesn't think he can. But everyone wants to force him into the binary prediction box. EDIT: took out snark [1] Okay, he was 100% on some states like New York, but so was everyone else, so those aren't interesting to talk about.
  - jmilloy 14y ago
    Ha! I've been thinking about this pretty hard and talking to a colleague at work, and changed my mind. You're right... he's not making predictions, he's making forecasts. And you can't say that a forecast was "right" or "wrong" based on which side of the forecast any one result landed. Assuming the n isn't just too small to say anything, the only two reasonable conclusions to choose between are a) in this set of events, the outcome in EVERY state happened to be from more likely side of the forecast, which together is unlikely, or b) Nate's forecasts are inaccurate, being skewed much more to the middle than they ought to be. Also, "all over this thread" was just meant to imply that I noticed the same comment from you more than once. Sorry for not being more specific.

Wilduck 14y ago

> it means that 6 times out of 10 Obama would win and 4 times out of 10 Romney would win. If Obama won all 10 times, by his own account he would be wrong. That's a very frequentist perspective. I think the Bayesian interpretation is a little more sensible. Given a prior estimate, updated with the information we have, it is logical to assume that Obama has a better chance of winning. That is, Nate Silver's 60% doesn't mean that if the election in a given state were repeated, we'd see different results four times out of 10, but rather that his information in making the prediction was incomplete.

danielweber 14y ago
Nate Silver himself said (in a radio interview that I cannot locate despite much Googling :< ) that if he said something would happen 60% of the time and it was 10 out of 10 that he his figure was wrong and should have said 100%.
- Wilduck 14y ago
  But you can't run the same election 10 times. Without hearing the interview, I'm going to guess he's talking about multiple states, all with 60% chance of winning, which puts us in a different place entirely. I guess I'm not disagreeing with the main point of your post. Just your use of the phrase "something would happen 60% of the time", as being oddly frequentist, when a Bayesian perspective is more appropriate here.
  - pseut 14y ago
    > But you can't run the same election 10 times. That makes assessment difficult, but is irrelevant for interpretation. If something happens 60% of the time, it happens with the same probability as drawing a red ball from an urn with 6 red balls and 4 green balls, whether it happens once or several times. Trying to determine the quality of a model using some observed data is inherently a frequentist exercise -- a dyed-in-the-wool Bayesian would take the election results, use them to update his or her posterior distributions, and carry on happily. (I know that no one would actually do this; no one who analyzes data is really a "pure" Bayesian or a "pure" frequentist).

jmilloy 14y ago

You're all over this thread with the same comment, and you're wrong. When an outcome has 60% likelihood of occurring, then it should occur 6 out of 10 times. But the election only happens once. The prediction isn't "wrong" when the outcome occurs 1 out of 1 time. In this case, you describe 10 separate events, each with 60% chance that a certain outcome occurs. So we have 10 separate outcomes that occurred 1 out of 1 times.

danielweber 14y ago
But the election only happens once. The prediction isn't "wrong" when the outcome occurs 1 out of 1 time. It's not a single event. Nate Silver made many many predictions about last night. And none[1] of them were binary up-down events. If he did make binary predictions, we could claim he got them "all right," but Silver was never doing that in the first place. Nor am I saying he should have. Try this: if Romney had won, would it mean that Nate Silver was wrong? Should we say, like Silver's critic Dylan Byers did, "it’s difficult to see how people can continue to put faith in the predictions of someone who has never given that candidate anything higher than a 41 percent chance"? No, we shouldn't. Because Silver didn't say "Obama would win." He said "Obama has a ~86% chance of winning." Silver gave a 14% chance of finding himself in a universe where Romney would win despite what his data was telling him. Silver made essentially no prediction about Florida -- and we shouldn't push him into making an up/down decision if he doesn't think he can. But everyone wants to force him into the binary prediction box. EDIT: took out snark [1] Okay, he was 100% on some states like New York, but so was everyone else, so those aren't interesting to talk about.
- jmilloy 14y ago
  Ha! I've been thinking about this pretty hard and talking to a colleague at work, and changed my mind. You're right... he's not making predictions, he's making forecasts. And you can't say that a forecast was "right" or "wrong" based on which side of the forecast any one result landed. Assuming the n isn't just too small to say anything, the only two reasonable conclusions to choose between are a) in this set of events, the outcome in EVERY state happened to be from more likely side of the forecast, which together is unlikely, or b) Nate's forecasts are inaccurate, being skewed much more to the middle than they ought to be. Also, "all over this thread" was just meant to imply that I noticed the same comment from you more than once. Sorry for not being more specific.