8 ms·
Nate Silver at 538 talks about the whole dissecting polls deal, or "unskewing" them. All polls must make methodological choices, and all of those choices have
by aetherson 10y ago
Nate Silver at 538 talks about the whole dissecting polls deal, or "unskewing" them. All polls must make methodological choices, and all of those choices have advantages and disadvantages. Spending a lot of time trying to dissect those choices and passing judgment on them is not as productive as:
1. Looking at aggregates of lots of polls.
2. Looking at the variance that a poll captures from one iteration of it to another.
Or at least, so he claims. Obviously, he runs a poll aggregator, using a model that heavily weights the trendline of individual polls, so he has a dog in this fight.
- erickhill 10y agoHis dog won pretty handily in the last election, though. It's not an exact science, but his methodology seemed to eliminate a fair amount of doubt from the equation.
- marricks 10y agoHave you seen the comparisons to Princeton's Election Conesertium which did something similar but without the "special sauce"? He did a little worse, which is a bit of evidence saying Sam's more straight numbers are better than Nate's judgment.
- SubiculumCode 10y agoor random error.
- marricks 10y agoDefinitely possible, that's why I said "a bit"
- throwaway729 10y agoI think it will be really interesting to see what happens in election forecasting if Trump wins this election -- doubly so if there's no major scandal for Clinton between now and election day.
- jacquesm 10y agoWhat happens with election forecasting will be the least of our collective problems.
- kahrkunne 10y ago>doubly so if there's no major scandal for Clinton between now and election day. Unlikely, considering there's one every other day
- Scarblac 10y agoRepeating the same thing every other day doesn't count.
- inimino 10y agoIf the polls stay as they are and the election goes the other way, it means that polling as practiced is somehow fundamentally broken, which would be very surprising. If there's a gradual shift in the polls, I'm not sure what that would mean. Which scenario did you have in mind?
- ddalex 10y agoBRexit will likely want to have a word with you. I have a feeling that polling is used now a political weapon not a measure of reality.
- vintermann 10y agoNate Silver arguably lost a good deal of "prediction calibration points" by placing a very low chance on Trump getting the nomination in the first place.
- 1812calif 10y agoNot at all. A low probabilty does not make the prediction invalid because of the result.
- alva 10y agoNate Tin did eventually publish a post [0] outlining that his way off predictions (sub 2% at one point) [1] of the R. primaries were terribly affected by his bias. That was honourable of him to admit. Presumably he is trying to eliminate that bias going forward for the main event. [0] http://fivethirtyeight.com/features/how-i-acted-like-a-pundit-and-screwed-up-on-donald-trump/ http://fivethirtyeight.com/features/how-i-acted-like-a-pundi... [1] https://i.sli.mg/hF2vOz.jpg https://i.sli.mg/hF2vOz.jpg
- hilop 10y agoHis name is Nate Silver. Don't be obnoxious. 538 didn't fail to read data properly re Trump's nomination. Trump was a new phenomenon with no history to build a model from. 538's mistake was guessing instead of admitting they had insufficient data -- which is bad behavior, but not because their preferred methodology is bad.
- cwyers 10y agoBull. Their polling based model performed okay (although, at least through Super Tuesday when I looked at it, his polls-plus model didn't outperform his polls-only model, and neither significantly outperformed the RCP weighted average of polls). What failed during the primaries was Nate's "Party Decides" punditry.
- deleted 10y ago[deleted]
- fulldecent 10y agoLTCM failed too because they did not consider confidence intervals on its models.
- gozur88 10y agoThat wasn't the feat people make it out to be. In US elections there are only a handful of states that could go either way.
- matt4077 10y agoIt adds up pretty quickly. For 2008 and 2012 combined, he only got one state wrong. Say there are 10 swing states close to 50/50 chances, and all others are fixed. The chance of getting them all right, twice, are 0.5^(2*10) = 0.000000953674316. (that's actually not the best method to evaluate him, because he provides estimates of chances and could be better evaluated with something like the https://en.wikipedia.org/wiki/Brier_score https://en.wikipedia.org/wiki/Brier_score that checks for calibration as well, but it's more intuitive).
- gozur88 10y agoThere aren't ten swing states. It's probably closer to half that. And they aren't 50/50 either. I got 49 right the last election just by looking at the public polls in the days leading up to the vote.
- conistonwater 10y ago> so he has a dog in this fight. I don't understand this criticism. Usually when people bring this sort of thing up, they have a specific criticism in mind (e.g., Exxon Mobil on global warming): they have identified an opinion that is sufficiently, demonstrably, wrong, and then they go looking for a reason how someone might get it wrong, so one conclusion might be is "having a dog in the fight". But the logical chain here goes one way: first you demonstrate that they are wrong, then you ask why. But what's wrong with someone working with polls in a professional sense, expressing an opinion on best practices in polling and statistics? After all, stuff that Silver writes about is quite consistent with stuff other people write about. It should, at least in principle, be possible to address the relative merits of polling methodologies objectively, and people like Silver, and Andrew Gelman too (see his blog, in particular the xbox poststratification paper and the more recent differential nonresponse paper discussions), seem to be trying to do just that. But anyone professionally studying polls is going to have some kind of a dog in the fight, in the end. It seems hard to conclude anything based only on that.
- imh 10y agoThe disclaimer is that they make money if people believe they are right, so may be inclined to cherrypick facts. So when we repeat Exxon Mobil's stance on global warming, we say that their stance might not represent all the facts. It's a disclaimer that this isn't my own opinion, and that I don't 100% trust it to be fair.
- notfromhere 10y agoNate Silver's reputation stakes on being accurate, so anything outside of a perfect result isn't great. This is different than Exxon Mobil's stance on global warming, where they make money when the truth about global warming was obfuscated.
- inimino 10y agoNate Silver is a recognized expert in an obviously highly complex area, so the grandparent comment is making an argument from authority. Given that, the disclaimer about that authority's possible conflict of interest is warranted. It's not a criticism but it is relevant in evaluating an authority's opinion.
- alexbecker 10y agoHe's written about this poll specifically: http://fivethirtyeight.com/features/election-update-leave-the-la-times-poll-alone/ http://fivethirtyeight.com/features/election-update-leave-th...
- ianai 10y agoYou just criticized somebody for specialising in what they're talking about. Next you're going to tell me Hawking's biased about physics theories because he knows them and that's somehow a bad thing.
- dom0 10y agoWell technically yeah. The old saying goes "an old theory isn't thrown over by convincing people, but because their proponents die of old age" for a reason.
- aetherson 10y agoNo, but if Hawking tells me that there are two theories, "a" and "b," that compete to explain a given phenomenon, and "a" is right and "b" is wrong, and "a" is a theory that Hawking himself came up with, then I'll note that. Even if, as in this case, I think that despite the bias, Silver is right.
- Lerc 10y agoI think this can lead to the Wikipedia fallacy. If you reduce the credibility of information when you can perceive a means whereby the information could be compromised, you end up placing undue weight upon non-transparent information sources. People don't trust Wikipedia because anyone could have written it yet place more trust in books that may just have a veneer of authenticity.
- bmelton 10y agoAnd the fix for either is peer review. Obviously, a forefront-expert in a given field may not have the luxury of peers, making peer review more difficult, but as sub-peers attempt to dissect the work, they learn, and the more that attempt to dissect, the more sub-peers are elevated to an extent that they can either debunk or give approval.
- Spivak 10y agoIsn't that just a perception problem? If a source isn't transparent about their methods and biases then the standard should be even higher for demonstrating truth because we have no choice but to assume the worst forms of bias.
- JumpCrisscross 10y agoThe Princeton Election Consortium's model [1] has a better track record than 538. This year, it has been much less volatile than Silver's. It only measures state polls, and does so in a very transparent way. [1] http://election.princeton.edu http://election.princeton.edu
- firebones 10y agoI like Nate Silver. He broke new ground. But I've grown less enamored this most recent cycle because his predictions seem to be a lot more of the Nostradamus type--pre-analyze outcomes on three possible resolutions, then get credit for being right. Most of his wins come because polls have a tendency to tighten up to the actual numbers the closer they are to the election, as any latent bias gets washed out. He gets graded on more accurate information, not when uncertainty is high. (This isn't inherently a problem, but it does expose why he 100% missed the Trump primary phenomenon...he's trying to address it with "fundamentals" and other alternate models, but this is just more hedging so that at least one story makes sense.) He's not deceptive about this--I will give him that. It's all in the open. But it seems to amplify a kind of narrative fallacy if these mistakes aren't revisited anew (rather than just saying "as predicted, since outcome C actually happened, it was due to voter bloc X doing it as we said." But I can't complain about a pundit class not giving actionable data--that's not their job. Silver's job is to add data to the discussion, and to stimulate the discussion. And I give him a "B" for that. That said, this cycle, I've rediscovered Sam Wang at http://election.princeton.edu/ http://election.princeton.edu/ and think he maybe gives a purer approach to analysis, one not so much driven by clicks. He doesn't seem to hedge as much as Silver.
- brandmeyer 10y agoNate Silver's writing quality this cycle has definitely been worse than last cycle. IMO, it has quite a bit to do with running his own website, instead of working as a blog under somebody else. There is much more pressure for fivethirtyeight.com to produce frequent updates than there was for fivethirtyeight.blogs.nytimes.com. I get that a major justification for branching out was to provide more sports coverage and some sparser statistical coverage of other topics. But there just isn't enough daily news on the presidential polls to justify the article publication rate he's running this cycle.
- tiatia 10y agoNate Silver has been criticized a lot by Mish (http://mishtalk.com/ http://mishtalk.com/) and Mish was more often correct with his predictions than Nate. (What was Nate's prediction of Trump becoming the candidate? "Trump's chance of becoming the GOP nominee at 2 percent.") Nate is strongly biased in favor of Hillary. That is fine but that is not scientific. Nassim Taleb said about Nate: "55% "probability" for Trump then 20% 10 d later, not realizing their "probability" is too stochastic to be probability." and "So @FiveThirtyEight is showing us a textbook case on how to be totally clueless about probability yet make a business in it." Proceed with caution.
- matt4077 10y ago538 didn't give any predictions on the primaries. Yes, they wrote articles saying that he had no chance, but those are separate from their statistical predictions. And, regarding the volatility: the probabilities reported by 538 are closely linked to betting/prediction markets. Anyone with a better model could parlay into money quite easily.
- tiatia 10y ago"538 didn't give any predictions on the primaries." They gave predictions for every state during the primaries. Also see: https://mishtalk.com/2016/05/19/nate-silvers-self-serving-confession-reflections-on-being-wrong/ https://mishtalk.com/2016/05/19/nate-silvers-self-serving-co...
- corin_ 10y agoSomething happening after being given a statistically low chance of happening doesn't automatically mean the statistical model was wrong. If we are standing at a roulette table and I tell you there is a 2.7% chance the ball will land on 4 and it lands on 4, it doesn't mean my low % was incorrect.
- Houshalter 10y agoActually Nate Silver did criticize polls like this on his podcast. At least the choice of weighting based on who they voted for in the past election. Which is known to be very biased. 538 does grade the quality of polls and excludes/weights polls that have lots of problems.