7 ms·

Nate Silver's calculations are certainly more sophisticated that a simple time-weighted average of legitimate polling numbers, but are they more accurate? His

Nate Silver's calculations are certainly more sophisticated that a simple time-weighted average of legitimate polling numbers, but are they more accurate? His baseball projection system, PECOTA, was extremely complex, but barely, barely outperformed a simple 3-year weighted average with an age component (Marcel), and some years was worse. Other projection systems that didn't take player comps into account were better overall than PECOTA (CHONE being the best, before the creator was hired away by a team).

3JPLW 14y ago
The difference? Sample size. In baseball, everyone has the same data. It's all the data available. There's no sampling going on; it's a projection from past performance. Which, in many ways, is way more difficult. His advantage in 538 is that he is using very sound statistical methods to combine multiple polls to increase the effective sample size and shrink confidence intervals. The complexities there are all related to increasing the individual pollster's historic performance and simulating the actual election.
- spicyj 14y ago
  > to combine multiple polls to increase the effective sample size and shrink confidence intervals How does this work? I was under the impression that to shrink the confidence interval by half you need 4x as many sample points, but there must be more sophisticated statistical methods at work here.
  - 3JPLW 14y ago
    His models have a memory (increase sample size by using a handful of day or week old data) and aggregate as many as 5 to 8 independent polls per state plus as many as a dozen national polls that factor into his simulations. That's easily four to sixteen times the sample size.
  - niggler 14y ago
    You can improve effective sample size by eliminating extraneous variables. One thing that Nate does is take advantage of implicit biases (for example, PPP polls lean slightly left)
    - waterlesscloud 14y ago
      PPP seems to have predicted 332 EV and correctly picked the winner in every state they polled. They didn't poll some of the obvious states...
- _delirium 14y ago
  I agree with that, but that's part of why I'm not convinced of his overall model's necessity: I think he gets the vast majority of his predictive power simply by being a good poll aggregator, who lays a reasonable statistical model over the polls. His model has a bunch of other stuff too, though: estimates of how current jobs numbers effect the election, "state fundamentals" based on demographic variables fit to historical election data, etc. I'm not sure these things are adding much predictive power, especially given that they're generally fit to sparse, heterogeneous, and noisy data sources (e.g. the 11 Presidential elections 1968-2008). On the other hand, one difficulty in evaluating it is that a good portion of the work his model is doing is not aimed at giving the point estimate, but the distribution and probabilities, to give an accurate probabilistic picture of the current state of the election at any given point, integrating various available data and historical trends. The goal, which is sensible, is to take a more rigorous approach to numbers you hear thrown around like, "when an incumbent is up by x% y weeks before the election, they win z% of the time". But even now in retrospect we can't easily evaluate how good those distributions were: we can see how well his point estimate fared, but with only one instance of how the election turned out, can't evaluate his overall picture's accuracy.
  - waterlesscloud 14y ago
    I'm not sure his model adds much in itself, but... I think he adds tremendous value in his write-ups. He explains what he's doing in pretty good detail, and provides some insight into what he believes it means. If you're seriously into it, you'll find places you disagree with him (same as it works in any other field when people are into it), but even if you're just casually interested you'll come away with some additional understanding. Or at the very least some additional questions of your own. And he manages to write it in such a way that general audiences can pick up some of it, and come away with the impression that there's a lot more there. Not saying it's not a true impression, just that the fact that they get the impression at all is what matters.
    - jimbokun 14y ago
      I believe Nate Silver is, by far, the best popularizer of statistical thinking in the U.S.
Osmium 14y ago
I think I speak for everyone when I say thank god politics is not like baseball :) But seriously, congratulations to Nate Silver. I think this vindicates his method, at least for America. He was badly wrong with the last British election, whether because of bad polling data or an insufficient understanding of the system, so I was apprehensive going into this one. But it seems like, for America at least, he's cracked it. I think it's inevitable considering the absurd quantity of polling data available that someone would eventually.
- bgilroy26 14y ago
  It shows how important it is to have experience and know your domain when you build a model and how understanding your subject matter is more important than cleverness. Mr. Silver is one of the most clever people we have on the public stage, but that doesn't mean he has a silver bullet for everything. Rather, it enables him to work in a field that he knows very well and knock it out of the park.
- lifeisstillgood 14y ago
  >> I think I speak for everyone when I say thank god politics is not like baseball :) politics is not nearly as important. :-)
- viviantan 14y ago
  Revenge of the Nerd. Sweet, sweet revenge :)
- bitcartel 14y ago
  2010 British Election (Number of seats) Silver predicted: Conservative 312, Labour 204, Libdem 103 Actual result : Conservative 307, Labour 258, Libdem 57
  - tehwalrus 14y ago
    This may have been because a) Clegg's performance in the first debate led to a lot of polls where LD support was high, but ultimately this support melted away by the election and the history bias kept polls wrong until the last minute, or b) because there were lots of Polly Toynbees telling pollsters "I'm voting lib dem" to make sure they were in with a chance in the coverage, but actually intending to vote Labour (she actually advocated this tactic of lying to pollsters in the Guardian.) Either way, it shows that statistical averages of what people say they'll do isn't always a good indicator of what they'll actually do when push comes to shove. Disclaimer: as a (now rather embarrassed) Lib Dem, I'd quite like these predictions to be more accurate so I don't get my hopes up again.
cma 14y ago
If you take all the simple weighted average with an X component and then selectively choose X after the fact, you can beat a lot of things.
kinofcain 14y ago
And Netflix was willing to pay $1m to someone who could beat their own algorithm by 10%. I haven't looked into Nate Silver's numbers, and certainly there would still be a reason for supporting him even if Romney had won tonight, but I think what most people are missing is that it's relatively straightforward to get pretty close to accurate, but incredibly difficult if not impossible to be completely accurate. The predictions, as they get better, are going to asymptotically approach the actual results.
- Ecio78 14y ago
  And Netflix was willing to pay $1m to someone who could beat their own algorithm by 10% Yet at the end they decided not to implement it http://www.forbes.com/sites/ryanholiday/2012/04/16/what-the-failed-1m-netflix-prize-tells-us-about-business-advice/ http://www.forbes.com/sites/ryanholiday/2012/04/16/what-the-...
  - smoyer 14y ago
    They may not have implemented it, but they now know a) what's possible, b) one possible way to achieve the goal and c) how much effort is involved. I think the article misses the point as research often yields an answer that isn't a winning business proposition (at least at the moment). I'd love to know whether there were any actions taken based on the contest.
  - gadders 14y ago
    They didn't implement it in the end? Doofuses (or doofii).
  - DennisP 14y ago
    True, but other articles have said the main reason is that their business changed. The contest was based on DVD rentals, now they emphasize streaming, and that changes viewing patterns...people skip around, sample several movies before settling on one, watch their favorite parts, etc.
    - phillc 14y ago
thewarrior 14y ago
I've been reading Fooled by randomness , so i would ask , what proportion of his forecasting record can be attributed to just plain luck ?
waterlesscloud 14y ago
It's looking like the Real Clear Politics Poll average is going to be 49/50. It was 48/50 in 2008 and 49/50 in 2004.
lftl 14y ago
Wes Colley, who runs one of the college football BCS ranking models, also did an electoral model. His model is much simpler, essentially taking the median poll result in each state. http://www.colleyrankings.com/election2012/ http://www.colleyrankings.com/election2012/ Their predictions were basically the same as Sliver's, missing only on Florida (they manually broke a tie in the model on Virginia so count that however you'd like).
daniel-cussen 14y ago
Maybe he got better.
pbreit 14y ago
Are evaluating baseball players and poll results even similar? One has a binary result, the other a spectrum. One is physically and mentally extremely difficult, the other a simple vote.
foobarqux 14y ago
Silver's model does seem unjustifiably complicated. For a simple model that managed to predict the electoral college see http://election.princeton.edu http://election.princeton.edu

3JPLW 14y ago

The difference? Sample size. In baseball, everyone has the same data. It's all the data available. There's no sampling going on; it's a projection from past performance. Which, in many ways, is way more difficult. His advantage in 538 is that he is using very sound statistical methods to combine multiple polls to increase the effective sample size and shrink confidence intervals. The complexities there are all related to increasing the individual pollster's historic performance and simulating the actual election.

spicyj 14y ago
> to combine multiple polls to increase the effective sample size and shrink confidence intervals How does this work? I was under the impression that to shrink the confidence interval by half you need 4x as many sample points, but there must be more sophisticated statistical methods at work here.
- 3JPLW 14y ago
  His models have a memory (increase sample size by using a handful of day or week old data) and aggregate as many as 5 to 8 independent polls per state plus as many as a dozen national polls that factor into his simulations. That's easily four to sixteen times the sample size.
- niggler 14y ago
  You can improve effective sample size by eliminating extraneous variables. One thing that Nate does is take advantage of implicit biases (for example, PPP polls lean slightly left)
  - waterlesscloud 14y ago
    PPP seems to have predicted 332 EV and correctly picked the winner in every state they polled. They didn't poll some of the obvious states...
_delirium 14y ago
I agree with that, but that's part of why I'm not convinced of his overall model's necessity: I think he gets the vast majority of his predictive power simply by being a good poll aggregator, who lays a reasonable statistical model over the polls. His model has a bunch of other stuff too, though: estimates of how current jobs numbers effect the election, "state fundamentals" based on demographic variables fit to historical election data, etc. I'm not sure these things are adding much predictive power, especially given that they're generally fit to sparse, heterogeneous, and noisy data sources (e.g. the 11 Presidential elections 1968-2008). On the other hand, one difficulty in evaluating it is that a good portion of the work his model is doing is not aimed at giving the point estimate, but the distribution and probabilities, to give an accurate probabilistic picture of the current state of the election at any given point, integrating various available data and historical trends. The goal, which is sensible, is to take a more rigorous approach to numbers you hear thrown around like, "when an incumbent is up by x% y weeks before the election, they win z% of the time". But even now in retrospect we can't easily evaluate how good those distributions were: we can see how well his point estimate fared, but with only one instance of how the election turned out, can't evaluate his overall picture's accuracy.
- waterlesscloud 14y ago
  I'm not sure his model adds much in itself, but... I think he adds tremendous value in his write-ups. He explains what he's doing in pretty good detail, and provides some insight into what he believes it means. If you're seriously into it, you'll find places you disagree with him (same as it works in any other field when people are into it), but even if you're just casually interested you'll come away with some additional understanding. Or at the very least some additional questions of your own. And he manages to write it in such a way that general audiences can pick up some of it, and come away with the impression that there's a lot more there. Not saying it's not a true impression, just that the fact that they get the impression at all is what matters.
  - jimbokun 14y ago
    I believe Nate Silver is, by far, the best popularizer of statistical thinking in the U.S.

Osmium 14y ago

I think I speak for everyone when I say thank god politics is not like baseball :) But seriously, congratulations to Nate Silver. I think this vindicates his method, at least for America. He was badly wrong with the last British election, whether because of bad polling data or an insufficient understanding of the system, so I was apprehensive going into this one. But it seems like, for America at least, he's cracked it. I think it's inevitable considering the absurd quantity of polling data available that someone would eventually.

bgilroy26 14y ago
It shows how important it is to have experience and know your domain when you build a model and how understanding your subject matter is more important than cleverness. Mr. Silver is one of the most clever people we have on the public stage, but that doesn't mean he has a silver bullet for everything. Rather, it enables him to work in a field that he knows very well and knock it out of the park.
lifeisstillgood 14y ago
>> I think I speak for everyone when I say thank god politics is not like baseball :) politics is not nearly as important. :-)
viviantan 14y ago
Revenge of the Nerd. Sweet, sweet revenge :)
bitcartel 14y ago
2010 British Election (Number of seats) Silver predicted: Conservative 312, Labour 204, Libdem 103 Actual result : Conservative 307, Labour 258, Libdem 57
- tehwalrus 14y ago
  This may have been because a) Clegg's performance in the first debate led to a lot of polls where LD support was high, but ultimately this support melted away by the election and the history bias kept polls wrong until the last minute, or b) because there were lots of Polly Toynbees telling pollsters "I'm voting lib dem" to make sure they were in with a chance in the coverage, but actually intending to vote Labour (she actually advocated this tactic of lying to pollsters in the Guardian.) Either way, it shows that statistical averages of what people say they'll do isn't always a good indicator of what they'll actually do when push comes to shove. Disclaimer: as a (now rather embarrassed) Lib Dem, I'd quite like these predictions to be more accurate so I don't get my hopes up again.

cma 14y ago

If you take all the simple weighted average with an X component and then selectively choose X after the fact, you can beat a lot of things.

kinofcain 14y ago

And Netflix was willing to pay $1m to someone who could beat their own algorithm by 10%. I haven't looked into Nate Silver's numbers, and certainly there would still be a reason for supporting him even if Romney had won tonight, but I think what most people are missing is that it's relatively straightforward to get pretty close to accurate, but incredibly difficult if not impossible to be completely accurate. The predictions, as they get better, are going to asymptotically approach the actual results.

Ecio78 14y ago
And Netflix was willing to pay $1m to someone who could beat their own algorithm by 10% Yet at the end they decided not to implement it http://www.forbes.com/sites/ryanholiday/2012/04/16/what-the-failed-1m-netflix-prize-tells-us-about-business-advice/ http://www.forbes.com/sites/ryanholiday/2012/04/16/what-the-...
- smoyer 14y ago
  They may not have implemented it, but they now know a) what's possible, b) one possible way to achieve the goal and c) how much effort is involved. I think the article misses the point as research often yields an answer that isn't a winning business proposition (at least at the moment). I'd love to know whether there were any actions taken based on the contest.
- gadders 14y ago
  They didn't implement it in the end? Doofuses (or doofii).
- DennisP 14y ago
  True, but other articles have said the main reason is that their business changed. The contest was based on DVD rentals, now they emphasize streaming, and that changes viewing patterns...people skip around, sample several movies before settling on one, watch their favorite parts, etc.
  - phillc 14y ago

thewarrior 14y ago

I've been reading Fooled by randomness , so i would ask , what proportion of his forecasting record can be attributed to just plain luck ?

waterlesscloud 14y ago

It's looking like the Real Clear Politics Poll average is going to be 49/50. It was 48/50 in 2008 and 49/50 in 2004.

lftl 14y ago

Wes Colley, who runs one of the college football BCS ranking models, also did an electoral model. His model is much simpler, essentially taking the median poll result in each state. http://www.colleyrankings.com/election2012/ http://www.colleyrankings.com/election2012/ Their predictions were basically the same as Sliver's, missing only on Florida (they manually broke a tie in the model on Virginia so count that however you'd like).

daniel-cussen 14y ago

Maybe he got better.

pbreit 14y ago

Are evaluating baseball players and poll results even similar? One has a binary result, the other a spectrum. One is physically and mentally extremely difficult, the other a simple vote.

foobarqux 14y ago

Silver's model does seem unjustifiably complicated. For a simple model that managed to predict the electoral college see http://election.princeton.edu http://election.princeton.edu