15 ms·
Was Nate Silver the Most Accurate 2012 Election Pundit?
- waterlesscloud 14y agoIsn't YouGov the XBox pollster? They're in the top group.
- keypusher 14y agoAlso interesting to their their margin of error is significantly lower than any of the other polling organizations. From what I gather on Wikipedia, they are internet-centric although I'm not sure if that includes XBox.
- backprojection 14y agoNote how NPR is one of the most right-biased in this result. It's pretty evident from years of listening that the NPR staff generally are progressives, and would left. So I think this result exemplifies how genuinely 'fair and balanced' NPR really is.
- FaceKicker 14y agoThroughout all the criticism from pundits of Nate Silver et al. as a liberal/commie/whatever for predicting that Obama would win, I couldn't help but wonder if there was even any merit to the idea that an Obama supporter would want to say that Obama is going to win the election. I would think predicting that Romney has a 10% chance of winning (making it tough but a distinct possibility he would win) as Nate Silver did would lead to some of the BEST possible voter turnout for Romney (making it the worst possible prediction strategically, if you want Obama elected). On the other hand, a prediction that Romney will win in a landslide would make many Romney supporters stay home. Is there actually evidence that higher poll numbers in favor of X lead to higher voter turnout from supporters of X? It seems like everyone takes that for granted but I've never seen any evidence that it's true.
- khuey 14y agoWell Romney supporters were certainly predicting a Romney landslide, so it's certainly plausible that Obama supporters would predict an Obama landslide. That's independent of whether or not that's a good idea in the game theory sense, of course.
- FaceKicker 14y agoYou're definitely right, and it really didn't make sense to me that they were doing that. I guess it's probably just a human nature thing more than anything else. I think my point still stands though: this doesn't showcase NPR's neutrality - perhaps they are in fact biased toward Obama and just lie more strategically than most Republican pundits. (Certainly not accusing them of that, only saying that I don't think we can glean much from the fact that they gave Romney higher numbers than he deserves.)
- showerst 14y agoIt's worth adding that Nate is on the record as saying he'd have voted for either Romney or Gary Johnson, if he would have voted. http://www.mentalfloss.com/blogs/archives/150042 http://www.mentalfloss.com/blogs/archives/150042
- pronoiac 14y agoThat's weird - I thought I read that he was personally pulling for Obama somewhere on 538. I'll go look for that.
- pronoiac 14y agoOkay, I found the reference someone cited as "Nate Silver openly rooting for Obama". It's from March 2008, before he joined the NYTimes: http://www.fivethirtyeight.com/2008_02_24_archive.html http://www.fivethirtyeight.com/2008_02_24_archive.html
- rms 14y agoI'm skeptical that that statement should be taken at face value. Nate Silver isn't the smoothest in in-person interviews.
- gamble 14y agoIt seems like most of the polling organizations that did the worst during this election did poorly because they were way off-base with their likely voter model. No one in the press seemed to notice the scale of the turn-out machine Obama constructed since 2010. Many organizations (including, apparently, Romney's internal pollsters) assumed a turnout somewhere between '08 and '10 levels, when the Democrats managed to match or beat 2008 turnout in critical areas. I'm not sure it's really a question of ideological bias so much as filtering the raw data through a backward-looking model.
- brown9-2 14y agoI think it is a huge mistake to equate "polls that tend to be biased towards Candidate A" with "this institution favors Candidate A's policies". The former can occur simply because of the assumptions that your model makes. Bias in "statistical bias" does not mean the same thing as "political bias".
- bo1024 14y agoI would like to see them add in David Rothschild at Yahoo[1], who's an expert in scoring rules and prediction markets and whose February (!) predictions were almost exactly on the money. [1] http://news.yahoo.com/blogs/signal/ http://news.yahoo.com/blogs/signal/
- PaulMest 14y agoHe works at Microsoft[1]. I have been following his predictions on PredictWise.com for the last four months. [1] http://www.linkedin.com/pub/david-rothschild/12/651/681 http://www.linkedin.com/pub/david-rothschild/12/651/681
- dreeves 14y agoJust to clarify, he recently moved from Yahoo Research to Microsoft Research when the former imploded. Same with David Pennock and others. The Yahoo folks in New York mostly switched to MSR (founding MSR-NYC) and the folks in California mostly switched to Google. (I used to be part of that research group but I left at the end of 2010 to start Beeminder.)
- bwood 14y agoIs it possible to be more accurate than nailing all fifty states?
- shardling 14y agoYOU DIDN'T READ THE ARTICLE DID YOU
- jfoutz 14y agoWell, maybe yes. One strategy for predicting the outcomes would be to paint a quarter red on one side and blue on the other, and assigning the outcome of that state to the outcome of the coin flip. That probably wouldn't be a very effective strategy, but it's possible to still nail all 50 states (actually, 51 in this case). Silver's method is much much more accurate than the coin toss method. Silver takes all the polls, even the crappy ones, and includes them in his calculation. If you're a bayesian you'd find this comforting because all of the evidence is included in the belief. There might be some handwringing about how important each poll really is. If you're not a bayesian, then you have some other weird strategy that might or might not work. So really, the predictions are nice, but what we're after is a system that produces good predictions. It's not clear that Mr. Silver's is the best. Perhaps there is some horrible flaw an evil agent could exploit that just didn't get tickled this election. It's tough to say.
- diminoten 14y ago> If you're a bayesian you'd find this comforting because all of the evidence is included in the belief. Off-topic, but this is my beef with the "Bayesians" - it's not all of the evidence; it's completely absurd to believe that all evidence can ever be accounted for, when considering anything.
- gwern 14y ago"More of the evidence" seems like pretty much the same thing...
- SeanLuke 14y agoSlate Magazine found two other pundits which were as accurate as Silver. http://www.slate.com/articles/news_and_politics/politics/2012/11/pundit_scorecard_checking_pundits_predictions_against_the_actual_results.html http://www.slate.com/articles/news_and_politics/politics/201...
- Sniffnoy 14y agoJosh Putnam is considered in the article, which looks at a finer level of detail than just correctly predicting the states.
- interconnector 14y agoActually the most accurate 2012 election pundit was Drew Linzer (http://votamatic.org/ http://votamatic.org/). Provided Florida goes Obama's way, he correctly predicted the electoral college - Obama 332, Romney 206.
- thedufer 14y agoDidn't Nate Silver say that was the most likely split? There was a post about it on HN for like the past 2 days. Edit: The confusion is probably that Silver was reporting the average electoral split as his prediction, when the mode is more important in what you're talking about. His average was almost never a number that was actually possible, since he was quoting them to the nearest 1/10th of a vote, so its kind of unfair to punish him for not getting it exactly right.
- interconnector 14y agoYou're right, I didn't realize Nate Silver was using an average. Also after reading the article, it looks like not just predictions for the 2012 presidential electoral vote, but other votes such as the Senate races are also being compared.
- waterlesscloud 14y agoOn his Electoral Vote histogram, he'd had 332 as a big spike for weeks. The day before the election it was over 20% for that one outcome.
- peteretep 14y agoDid you read the article? Drew Linzer is one of the specific people it rates Nate Silver against.
- nhebb 14y agoThis compares top lines. I think a comparison of turnout model accuracy would be more informative. Most of the models that erred predicted that the 2012 turn out would lean less Democratic than the 2008 turn out model, based on the 2010 mid-term turnouts and a (mis)perceived dampening of enthusiasm among Democrats and increased enthusiasm among Republicans. Based on exit polling, there was a drop off of 7 million white voters, and I don't think anyone who predicted that.
- waterlesscloud 14y agoStill a bit early to tell how many people voted in total. Probably need to wait a week or two for the final result. Actually, that makes this whole exercise premature, the rankings may change once the final data is in.
- ams6110 14y agoThe reasons for the low turnout will be interesting to hear. Nationally, nearly 3 million fewer votes for Romney in '12 than McCain in '08, and around 10 million fewer votes for Obama. Polls showed the election to be fairly close, so that would not tend to explain either "giving up" or "overconfidence" as a reason person didn't vote.
- swang 14y agoArticle doesn't mention Sam Wang. His confidence level for Obama winning was at 99% http://election.princeton.edu/ http://election.princeton.edu/
- peteretep 14y agoOooh, I wonder what "he" might choose for a username on a social media site!
- jrockway 14y agoA reply below says that the username is a coincidence. I'm inclined to believe it. Why use your real name to troll HN over a pretty obscure topic that's not even related to the site's userbase?
- swang 14y agoI can guarantee, with a near 100% confidence rating, that I am not Sam Wang. Merely coincidental that we have the same last name and first initial. I mean there has to be at least 100 million people in the world with the surname Wang so the odds that we are the same person are pretty slim!
- crayola 14y ago"I mean there has to be at least 100 million people in the world with the surname Wang so the odds that we are the same person are pretty slim!" Except this seems to ignore that P(comment about Sam Wang | I am Sam Wang) is much higher than P(comment about Sam Wang | I am not Sam Wang) :-)
- gamble 14y agoHe mentions in the comments that Wang's website doesn't appear to give the raw numbers he needs to analyze its accuracy.
- gwern 14y agoRight. Wang seems to now have released raw numbers but did so after I finished my numbers and gave OP the go-ahead. I plan to add Wang and some other stuff today.
- esurc 14y agoThanks. I've been trying to find a good way to understand the accuracy of Silver's predictions. What's a fair benchmark? This article offers up a "coin flip" for each state, computing that such a coin flip would have a Brier score of 0.25. (The Brier score is a mean-squared error between outcome (1 or 0) and the percent certainty of the prediction in that outcome. If a coin flip is the model, each state's result of 1 or 0 would be in error by 0.5. The mean squared error would be 1/51 * 0.25 * 51 = 0.25.) But... that seems like too generous a benchmark. Take the simple model: "assume 100% likelihood that state X will vote for the same party as it did in 2008." That guarantees that deeply red or blue states will vote the same way, so it takes the non-battlegrounds out of the equation. With this model, there would only have been 2/51 errors. This simple lazy model achieves a Brier score of 0.039, beating Intrade and the poll average computed in this article quite badly. After working through this, I'm still impressed by Silver and the other quant predictions. But I'm more concerned about media that rely too much on reporting a single polls result as "news" rather than as part of a larger tapestry. Then again, it's the maligned media polls that are the raw input to Silver and the other models. Unless the media keeps funding the polls, the quality of these more advanced models will suffer.
- gwern 14y agoThanks for the suggestion. I've added your benchmark suggestion (along with a bunch of other fixes and new data like a 2008 RMSE benchmark along the lines of your Brier suggestion) into the R document: https://docs.google.com/document/d/1Rnmx8UZAe25YdxkVQbIVwBI0M-e6VARrjb0KdgMEVhk/edit https://docs.google.com/document/d/1Rnmx8UZAe25YdxkVQbIVwBI0... (I don't know when the new numbers will go live on the blog; Luke handles that.)
- Osmium 14y agoGreat article, but I disagree with the colouring on the first graph: if reality was within the poll's margin of error, I don't think it should be coloured, because that implies a bias that (probably) isn't actually there.
- Osmium 14y agoDoes anybody know more about YouGov's methodology? On the face of it, I'm suspicious of their very low margin of error which seems substantially better than any other poll out there, but you can't deny that their polling was accurate. Another thing that looks odd on that graph: the given polling numbers from Washington Times/Politico/Monmouth/Newsmax/Gravis/Fox/CNN/ARG all look identical despite their differing margins of error (which suggests their source data is different). What's going on there?
- sethg 14y agoHere’s a good article from the YouGov site that describes their methodology: http://today.yougov.com/news/2012/10/23/obama-stays-ahead-just/ http://today.yougov.com/news/2012/10/23/obama-stays-ahead-ju...
- Osmium 14y agoThanks for the link. In short, it seems like only poll people they can confirm are actually registered to vote based on the fact that: "According to US census data, just 71% of eligible Americans are registered to vote. In 2008, almost 90% of those who were registered did vote. So in any poll, it is vital to know which respondents are on the register." But there're a few more interesting subtleties in there too, so it's worth a read.
- chmullig 14y agoYouGov does online polling, which allows them to get very large sample sizes for national polls. I believe they had on the order of 36,000 for their final poll. Compare that to others with just ~1000.
- gwern 14y ago> Another thing that looks odd on that graph: the given polling numbers from Washington Times/Politico/Monmouth/Newsmax/Gravis/Fox/CNN/ARG all look identical despite their differing margins of error (which suggests their source data is different). What's going on there? I dunno. IIRC, Drew Linzer of Votomatic even worried on his blog that the polling numbers were too close and that pollers might be fudging their numbers to be more similar to each other (which would lead to substantial overconfidence in estimates). Still, the final results seem pretty accurate, so...
- brown9-2 14y agoIt's worth noting that if you used the 2008 results as your 2012 prediction, you would have gotten 49/51 on this scale.
- gwern 14y agoYeah, but if you simply predicted 2008, you'd get a mediocre Brier score on your state victory predictions (because it would punish you for getting 49 compared to everyone who got 50 or 51), and the RMSE is even worse: the margins were different and the electoral vote & popular vote very different from 2008.