6 ms·
How Can We Tell Which Election Forecasts Are True?
- ISL 10y agoBy doing the experiment and letting the election happen. Experiment is the arbiter of truth.
- Scarblac 10y agoBut say, a forecast says candidates A, B and C have 40%, 35% and 25% chance of winning. Now the election happens, and one of them wins. Was the forecast true?
- danmaz74 10y agoHow many times can you repeat the election?? :D
- huherto 10y agoThat is what was thinking. But not really. The most difficult part is to assign probabilities to the individual state races. The final probability is done using a bootstrapping technique(a simulation). So you really have 50 predictions that you can evaluate after the election.
- Scarblac 10y agoBut it's further complicated by the fact that the elections aren't independent.
- basch 10y agothat is what confidence is for (https://en.wikipedia.org/wiki/Brier_score https://en.wikipedia.org/wiki/Brier_score) "The reason for this is that pundits (the better ones, anyway) don’t just predict election outcomes, they also tell you how confident they are in each of their predictions. For example, Silver gave Obama a 50.3% chance of winning in Florida. That’s pretty damn close to 50/50 or “even odds.” So if Romney had won Florida, Silver would have been wrong, but only a little wrong. In contrast, Silver’s forecast was 92% confident that Rick Berg would win a Senate seat in North Dakota, but Berg lost. For that prediction, Silver was a lot wrong. Still, predictions with 92% confidence should be wrong 8% of the time (otherwise, that 92% confidence is underconfident), and Silver made a lot of correct predictions. So how can we tell who did best? We need a method that accounts not just for the predicted outcomes, but also for the confidence of each prediction. There are many ways you can score predictions to reward accuracy and punish arrogance, but most of them are gameable, for example by overstating one’s true beliefs. The methods which aren’t cheatable — where you score best if you are honest — are all called “proper scoring rules.” One of the most common proper scoring rules is the Brier score. A Brier score is simply a number between 0 and 1, and as with golf, a lower score is better." http://rationality.org/2012/11/09/was-nate-silver-the-most-accurate-2012-election-pundit/ http://rationality.org/2012/11/09/was-nate-silver-the-most-a... "Also note that Wang & Ferguson got a better Brier score than Silver despite getting Florida (barely) wrong while Silver got Florida right. The Atlantic Wire gave Wang & Ferguson only a “Silver Star” for this reason, but our more detailed analysis shows that Wang & Ferguson probably should have gotten a “Gold Star.”"
- Scarblac 10y agoLet's say if some forecaster says a candidate has a 60% of winning in each state, and in fact he wins all of them. Does that means that the forecaster was 40% wrong?
- zeeboo 10y agoIn some sense, yes. A proper scoring rule would give that forecaster less points than one that forecasts a 90% chance of winning when they win all of them.
- hsitz 10y ago"So if Romney had won Florida, Silver would have been wrong, but only a little wrong." No, Silver would not have been wrong at all, or at least we'll never know. He doesn't predict a winner; he gives probabilities of victory. It would have been perfectly consistent with Obama having a 50.3% chance of winning (or 63.5% or 32.8% or 99.9% or whatever probability you want to assign) and then for Romney to have won. Even after the election was over, a Florida victory for Romney would not have meant that Silver's pre-election probability assignment of 50.3% to Obama was wrong. That's not how probabilities work. Think of sporting events. A huge underdog with only, say, a 2% chance of winning may beat the favorite. This does not mean that it had better than a 2% chance of winning before the event. Depending on the reason for the underdog's victory, it may not even mean that the underdog has better than a 2% chance of winning if they play again. One in a million events happen, and it doesn't mean that their chances of happening were greater than one in a million.
- basch 10y agoread the last quoted sentence
- hsitz 10y agoNot sure which sentence you're referring to. And I'm not saying there's no way to tell which probability assignments are better. I'm just saying that the mere fact that an event assigned a lower probability happens (e.g., if Romney would have beaten Obama in Florida) does not mean the assignments were wrong. This is consistent with Silver's own explanations of what he's doing, and (I'm pretty sure ) he is careful to say he doesn't make "predictions". (Actually, not sure, it may be merely that he often clarifies that a different result than predicted does not make the probability assignment that prompted the prediction "wrong". The prediction and the probability analysis it's based on are different things.) I'm not saying that the Brier Score can't differentiate between better and worse probability assignments, but I am saying it's confusing the issue if it calls probability assignments "right" or "wrong". They may be "better" or "worse" than each other, but not "right" or "wrong". This may be using language in more technical manner, but it seems to me (and to Silver) to be an important distinction.
- ronald_raygun 10y agoYou can assign a score based off of what they said, and how confident they were. Check out this article https://terrytao.wordpress.com/2016/06/01/how-to-assign-partial-credit-on-an-exam-of-true-false-questions/ https://terrytao.wordpress.com/2016/06/01/how-to-assign-part...
- chimeracoder 10y ago> Presidential election forecasts are historically successful and appear to be highly precise. Yet they’re often contradictory. Eh, not really. Primarily elections (at all levels) are hard to predict for a number of reasons, but the presidential general election is about as straightforward as it gets. Different forecasts vary with respect to the exact estimate and the degrees of confidence, but all the scientific, polls based approaches tend to predict the same range of outcomes. I wouldn't really call that "contradictory" for two models to give the same basic prediction but with slight differences in the errors and exact estimate.
- ctdonath 10y agoThere are a number of emails leaked from various election polling companies which indicate intent to spin the results and induce a particular outcome.
- mach5 10y agosource? id like to read this
- bendykstra 10y agoUnsurprisingly, those turned out to be fabrications. Regardless, these models aggregate dozens to hundreds of polling organizations. If Monmouth tried to 'create a narrative,' as described in the purported memo it would just end up looking out of step and would lose credibility. > The "leaked memo" is a small test of readers' credulity. Written in apocalyptic terms, with strange capitalization ("the Liberal base is demoralized — but will become more enthusiastic the more Hillary is seen as an iconoclast who will overturn the last foundations of Western Culture"), it was uploaded to Scribd by a pro-Trump website with the confidence-inspiring name of RealTrueNews. For a short while, it was credited to Nate Silver; his name was blacked out, rendering it slightly less odd that someone who did not work for Monmouth was credited on a Monmouth polling memo. https://www.washingtonpost.com/news/post-politics/wp/2016/09/21/fake-polling-memo-prompts-the-question-how-much-are-partisans-willing-to-believe/ https://www.washingtonpost.com/news/post-politics/wp/2016/09...
- 1337biz 10y agoThat would explain why campaign intern polling is always such a sacred ground.
- maxerickson 10y agoThere are simpler reasons. Being able to direct the polling is useful (spend resources on a given issue or not). Being able to spend more polling resources on close races. Having guaranteed access to the internals of the poll. Etc.
- basch 10y agoI find Silver's celebrity worship slightly irritating. >FiveThirtyEight and the PEC both predicted the outcome of the 2012 presidential race with spectacular accuracy. Markos Moulitsas and Drew Linzer (Daily Kos and Votamatic respectively) were both more accurate than Silver (https://www.dailykos.com/story/2012/11/14/1161465/-More-accurate-than-Nate-Silver https://www.dailykos.com/story/2012/11/14/1161465/-More-accu... http://rationality.org/2012/11/09/was-nate-silver-the-most-accurate-2012-election-pundit/ http://rationality.org/2012/11/09/was-nate-silver-the-most-a...) AND they teamed up this year (http://elections.dailykos.com/app/elections/2016/office/president http://elections.dailykos.com/app/elections/2016/office/pres...) AND The Upshot is showing their forecast as the most confident in its projections (http://www.nytimes.com/interactive/2016/upshot/presidential-polls-forecast.html#other-forecasts http://www.nytimes.com/interactive/2016/upshot/presidential-...) Why does Silver continue to be rewarded with praise of spectacular accuracy while Markos Moulitsas and Drew Linzer don't get mentioned? Hell, Silver and Wong are products OF Daily Kos. How can you write an entire article about forecasting accuracy and not even link to their site? At least The Upshot respects them enough to include them.
- scottlocklin 10y agoMoulitsas and Linzer don't use Silver's publicity firm. The models they all use are fairly questionable.
- basch 10y agoespn?
- wnevets 10y agocould it be that they were affiliated with the dailykos?
- matt4077 10y agoWell he was also quite good in 2008, and his writing in 2006 to 2012 was a must-read. Since then I've felt less tempted to refresh 538 every ten minutes, but that probably has less to do with a decline in his writing and more with the fact that I had already learned a lot from him and there wasn't much new. Apart from any measures of accuracy, he does deserve the recognition to have invented the genre. Back in '08 there were I believe two people doing it – him and Tanenbaum's http://www.electoral-vote.com http://www.electoral-vote.com
- cpr 10y agoRead to the end: http://www.zerohedge.com/news/2016-10-11/first-post-debate-poll-gives-hillary-significant-lead-and-familiar-problem-emerges http://www.zerohedge.com/news/2016-10-11/first-post-debate-p... Edit: Correction noted. Post debate NBC/WSJ poll was conducted by group specifically paid by the Clinton campaign. Not exactly what you'd want for a major poll.
- akhilcacharya 10y ago...ZeroHedge? Really?
- mikestew 10y agoYou would do better to just make your argument rather than burp up a random link and say, "Read it!"
- losteverything 10y agoA question I recall asking a Director was why were there so many PhDs in our marketing group. They were all statisticians he said. They were brought in to support decisions marketing made because as the Director said "people believe PhDs more than us. They make our decisions believable." Same for using JD Power. Totally result driven. They got a lot of money to produce a result. So to me anytime I see polls the only thing that matters is the money trail. As far as elections go, we can see the accuracy in a month.
- matt4077 10y agoI like how your argument flows from anti-intellectualism to conspiracies, only to end up at fatalism. Luckily, there are people motivated by such questions. And even it may not make a difference to me except for it's entertainment value here, the impact statistics have had on our quality of life is undeniable. There was so much invention in statistics and operations research in WW2 that you can make a credible case that it was the ally's decisive advantage. Since then, medical research comes to mind as field where it has been useful to have people who don't answer "dunno – why don't you wait a month and see" but "I think this may be possible."
- losteverything 10y agoSome things you can change the outcome by making decisions partially based on statistics. Like who to choose for a hip replacement. Do you go to a hospital that does only orthopedics or a general one? I recall the event as a young marketer / techie because I always thought I wanted a PhD. But when I saw how far they were away from business decisions being only support I knew a PhD at this company was not for me. There are people who earn a living with polls. This is a not-so-bad way to spend ones time. There are worse.
- 6stringmerc 10y agoUm, we can tell after the election is over? I mean, that's generally what a forecast is tested by - the outcome versus the prediction. Every scenario is different, so relying on one model is only good for one test I think. Not saying building models is useless, I just think the analytical part is flawed in general because it still relies on humans to respond and humans are...difficult... I know a guy who correctly predicted the Obama/Romney popular vote outcome within 1% of each's actual take IIRC. As of now, he's got Clinton at 47% and Trump 34%, based on nothing more than personal outlook, hunches, and mulling over various polling data (he had 47% Clinton to 41% Trump just before the Conventions). Point being it might be just as likely an irrational human can point out human behavior (en masse, e.g. voting) on par with a statistical model, given some particular circumstances.
- 1337biz 10y agoHey, don't tell the kids Santa Clause is not real. We need to hold up the believe that there is some magical wizard who is the only one knowing how to do polling the correct way.
- matt4077 10y agoThe problem with that it only tells us how good they were on election day. It'd be great to somehow evaluate their earlier predictions, which is probably only possible by looking at an aggregate of election. (or maybe, since there are predictions and results for every state and every senate race, it's actually possible to use those for an evaluation. Note that it's not as easy as scoring a "won by trump" in favor of a forecaster who was giving trump a 60% chance. You need a measure that expects your 60% predictions to be wrong 40% of the time, like the Brier score)
- lintiness 10y agoit's a binary event, so differences in degree are going to be difficult to measure (especially in outcome). poker has an interesting add: don't measure your play by outcome; get the process right and results will follow. if you focus on outcome, you're likely to believe you played poorly when in reality you were just unlucky (something unlikely happened).
- tomjen3 10y agoBased on past history, we can assume they have to give Trump a higher than average of the current polls chance to win.
- umanwizard 10y agoWhat do you mean?
- lern_too_spel 10y agoSilver consistently underestimated Trump's chances in the primaries. He has yet to explain what he did wrong then or how he has fixed it now. If you don't want a Trump presidency, Silver's current projections shouldn't be very comforting.
- umanwizard 10y agoRight, but a lot of other poll aggregators who don't have that baggage are also predicting a Clinton win, many with higher confidence than Nate Silver.
- lern_too_spel 10y agoDid their polling analysis correctly predict Trump's nomination?
- maxxxxx 10y agoWhy do people care so much about polling and forecasts? Why not spend that effort on looking at issues, voting for the right people and after the election see who won? This seems a total waste of time.
- wnevets 10y agoA lot of voters will split their votes to prevent one party from having too much control. For example if hillary is going to win in a landslide they'll vote for a republican congress. However aren't most things in life a total waste of time, like posting on HN complaining about total waste of time?
- basch 10y agothe same reason fantasy football can be more fun than watching the games? according to the forecasters the election is already over, who's model is best is the only interesting undecided component left. right now we have an election between a Blowhard Reality Star and a Crony Crook, where most of the population would prefer neither. so combining the realities of the choice already made (barring an insane scandal) with an election between a douche and turd sandwich, statistical models are about the only semi-honest thing we can turn to. do you really think this election is about "the issues" and not who is better at pandering? both of their campaigns are designed to tell people what they want to hear, and figure out the rest later.
- lintiness 10y agoyou made me happy. people need to turn this shit off. i keep seeing a commercial from somebody acutely interested in hillary getting elected: it pretends trump makes fun of kids with spinal cancer. that pretty much sums up the discourse right now, and it's not good for anybody or anything -- certainly not the direction of the nation.
- basch 10y agoit's all a distraction to give the illusion of offering participation. people can go home and feel "i contributed" having fulfilled their civic duty. everyone paying any amount of attention knows its a rigged game ten ways to sunday. democracy in general is a feedback loop between the press and politicians with the voting population as pawns/fodder. journalists and politicians have one shared concern: staying relevant and needed.
- skywhopper 10y agoI would ask instead: what does it even mean to be "right" about a statistical probability of a future event? What does 54% chance mean vs an 84% chance when you're talking about something that's six weeks away? If the event happens or doesn't, that fact in itself does not tell you whether 54 or 84 was the right number. Each group creating predictions builds a mathematical model that simplifies reality into a set of numbers that necessarily have large uncertainty and error built in. The models are, by design, not intended to replicate reality. They are intended to be a smaller, simplified version, retaining as many of the relevant properties as possible, but with no illusions (one hopes) that they actually represent reality. (If they did, you'd expect them to make exact predictions, not percentage chances.) Presumably these teams do their own math "right", so the only useful question is "which model is _closer_ to reality?" But there's no way to know that, except to test them an infinite number of times, and ooops can't do that either...
- grandalf 10y agoI'm very glad someone made this point.
- platz 10y agoIt is very simple. All one has to do is imagine the election being repeated an infinity number of times under a stationary-non-changing distribution.
- chimeracoder 10y ago> It is very simple. It's actually not simple at all! This has been one of the most contentious questions of statistics for over a century. > All one has to do is imagine the election being repeated an infinity number of times under a stationary-non-changing distribution. That's using a frequentist interpretation of probability, which is not the only interpretation. (Nor is it, in fact, the interpretation used by e.g. Nate Silver, who is one of the statisticians mentioned in the article). (The frequentist interpretation, incidentally, has been shown to be self-contradictory, for reasons that involve continuous mathematics that I won't get into. In other words, you can construct situations that have nonsensical interpretations under the frequentist approach. That doesn't mean frequentist statistics aren't useful, but - like any model - the approach has limitations.)
- ythl 10y agoYou can't. A forecast is just that - a prediction.