18 ms·
Reverse-engineering the problematic tail behavior of Fivethirtyeight forecast
- lunchladydoris 6y agoNot to nitpick, but it's Andrew Gelman. Murray Gell-Mann is the physicist.
- kevmo 6y agoHaha I rolled in here to see if it was his son.
- deleted 6y ago[deleted]
- asplake 6y agoAn interesting read and a great concluding paragraph
- KKKKkkkk1 6y agoMy biggest issue is when people say that it's a probabilistic model, and therefore it wasn't wrong in 2016 because 28% chance of winning is pretty high and you don't get probabilities. Well, guess what, this kind of model that provides a probabilistic estimate on a future event that cannot be repeated cannot be validated or falsified. It's basically junk science (if it has any aspirations of being scientific).
- apsec112 6y agoYou can check their forecasts on Senate and House races, where we have a lot more data points. https://projects.fivethirtyeight.com/checking-our-work/ https://projects.fivethirtyeight.com/checking-our-work/
- Marazan 6y agoIf I tell you an unweighted 6 sided die only has a 1 in 6 chance of coming up 6 and we roll it once and it comes up a 6 then that is not junk science.
- bena 6y agoYeah, but what if we could never roll that particular die again? I think that's what he's talking about.
- mac01021 6y agoThis is just Frequentism vs Bayesianism, right? https://en.wikipedia.org/wiki/Probability_interpretations https://en.wikipedia.org/wiki/Probability_interpretations
- NhanH 6y agoYou treat the forecaster as the dice, not a particular forecast from them. You can measure their forecasts with reality.
- bena 6y agoYes, I'm not defending the original guy, I'm just stating what I believe his argument to be. He's basically saying he doesn't believe the die is fair/unweighted. So stating the odds of a fair die is kind of immaterial to his point. We need to demonstrate to the guy that the die is fair. Someone else posted a link (I think) of 538 going over on how accurate they were. Whether their odds bore out. Here it is: https://projects.fivethirtyeight.com/checking-our-work/ https://projects.fivethirtyeight.com/checking-our-work/ Basically what they did was bucket every prediction by odds. If they predicted 70/30, it went in that bucket. And they're "right" at about the rate of their predictions. In other words, for every 70/30 prediction they made, the people/teams with 70% chance to win, won about 70% of the time. That shows that 538 in this case is pretty decent at calculating odds.
- bart_spoon 6y agoYes, which is why it’s foolish to talk about the outcome of that single race. You are right that there will never be another 2016 election between Clinton and Trump. However, that is only one of hundreds of forecasts made by 538 across multiple election cycles, so we can see how often their probabilistic outcomes align with actual outcomes. The fact that their track record is fairly good across all of these is evidence suggesting that Trump’s win may be more likely to be the realization of a less likely outcome predicted by their model, rather than a fundamental problem of their model itself.
- stjohnswarts 6y agoSo all statistics and probabilities are junk science because they can't predict the future 100% of the time? Surely you can't be asserting that...
- civilized 6y agoOr... if you think for a second about what probabilities are supposed to mean, there is an obvious way to check if a probabilistic forecaster is accurate https://projects.fivethirtyeight.com/checking-our-work/ https://projects.fivethirtyeight.com/checking-our-work/
- cwhiz 6y agoNo one is purporting election forecasting to be scientific.
- bigbubba 6y agoSome of your sibling comments seem to suggest otherwise.
- cwhiz 6y agoWell, HN commenters don’t speak for the people doing election forecasting. I think Silver would describe it as educated guesswork.
- bigbubba 6y agoI suppose you meant that election forecasters don't claim election forecasting is a science, not 'no one.' I don't mean to quibble, this meaning was not at all clear in your original comment, nor did I expect that meaning from the context of the comment you responded to.
- HeavenFox 6y agoThis is basically the frequentist vs bayesian debate. Looks like you are firmly in the former camp :)
- jsnell 6y agoSee also this Twitter thread from Nate Cohn (who has no dog in the fight): https://twitter.com/Nate_Cohn/status/1320042092694065153 https://twitter.com/Nate_Cohn/status/1320042092694065153
- happytoexplain 6y agoThat's a pretty bold claim to make about essentially anybody in regards to the US presidential election. Not that I don't believe his account is even-handed and valuable, just that I'm curious what makes you say he has no dog in the fight.
- noirbot 6y agoI think the third post in that thread really nails the underlying issue here. We, as humans, know that it would be silly to think that some configurations could happen. We know there's no reason to expect Biden to win Alabama outside of him winning nearly every state. A statistical model only has a vague idea of context/the real world. It looks at polls (and probably not really that many polls of Alabama or Mississippi or Alaska) and sees that, statistically, Biden should win 3% of the time or so. It doesn't have a specific world set of events in mind that would cause that, it just knows that that's how the numbers go, and thus may lead to weird circumstances in the grander results because it has to make the world match the numbers in these small corners.
- preommr 6y ago> It didn't take very long to do the analysis. But it did then take another hour or so to write it up. It's very interesting to see how long it takes people to do things. I am amazed that entire article took 1 hour to type up. I've spent entire afternoons trying to write shallower pieces of work.
- chrisfrantz 6y agoIt looks like it was written as a single stream of conscience. While I couldn’t write that article, if I hit a flow state and was interested in the topic, it seems possible.
- jacquesm 6y agoThe trick is to think before you write. The same goes for programming. If you already know what you are going to write or build then you can reach very high apparent productivity, the time spent on thinking about it isn't accounted for.
- deleted 6y ago[deleted]
- deleted 6y ago[deleted]
- i_love_limes 6y agoI think a lot of people (myself included when I've felt the pressure to) lowball how long things like this take because a) it makes me look smart and b) people could judge of they knew who much time I actually wasted on it.
- taeric 6y agoI think you read the parent in the opposite direction than intended. It was praise for getting this done so quickly, by my read. Granted, i could just be misreading this post. :)
- aakilfernandes 6y agoCould this just be a result of low sampling by the author? If you reduce a sample to only include some tiny edge case, the resulting data points are going to be weird in random ways.
- RivieraKid 6y agoNo, there's enough data to determine all between-state error correlations. Edit: Why the downvote? Each of the between-state correlations can be calculated from 40,000 datapoints.
- nullc 6y agoMeh. If you fit a model and don't explicitly constrain against "un-physical" results like negative correlations, you'll end up with them. Constraining against them won't improve your models fit (usually by definition), and it doesn't always improve robustness (at least for situations near average)-- because they're acting to debias the model in ways that you otherwise don't have enough degrees of freedom to address. A negative correlation here is also potentially historically supported, in the sense that sometimes DEM/GOP candidates are philosophically reversed in some way relevant to the state. As in, "The only way a GOP would get elected in X is if they had the DEM position on subject Y which would make them lose state Z, who cares as much about that subject as X but in the opposite direction." Now-- it doesn't seem likely case in this election (e.g. Trump is not (currently) a massively pro-choice republican), so it probably shouldn't apply here-- but it's isn't hard for me to imagine how a negative correlation might show up out of the historical data.
- RivieraKid 6y ago> If you fit a model and don't explicitly constrain against "un-physical" results like negative correlations, you'll end up with them. The Economist model does exactly that, and all of their correlations are positive. I recommend reading their methodology, they know what they're doing (I wouldn't say the same about 538). Andrew Gelman has developed some of the Bayesian methods and software that people like Nate Silver use, he's the main author of what's considered a reference book on Bayesian statistics.
- noirbot 6y agoI think the question is if it matters to the predictive accuracy of the model. Just because it puts out results you can't envision actually happening on the margins doesn't mean they can't happen, or that they can't be valuable in presenting a holistic result. It's clear that the models are tuned differently, but from Silver's replies in the PS's, it seems that he's ok with these artifacts being part of the model.
- RivieraKid 6y ago
- mathattack 6y agoThe first problem with the model is it grossly missed the last election. I still think it’s good reading.
- hn_throwaway_99 6y agoAre you referring to the 2016 election? If so, you are wrong. 538 gave Trump a higher chance of winning than pretty much every independent pollster.
- Ibethewalrus 6y agonope, you're wrong. 538 gave Hillary a 71.4% chance of winning https://projects.fivethirtyeight.com/2016-election-forecast/ https://projects.fivethirtyeight.com/2016-election-forecast/
- wtallis 6y agoThe assertion you're responding to was not about Trump's odds in an absolute sense, but relative to other pollsters and forecasters. So pointing out the precise value FiveThirtyEight assigned to that outcome doesn't refute anything until you compare it to somebody else's number.
- patrickthebold 6y agoThe GP is saying 538 did better compared to other news outlets. You need to also provide a source for another news outlet giving Hillary less than 71.4% if you want to show that the GP is wrong.
- Ibethewalrus 6y agothanks. gave a reread to the comment I replied to and makes sense
- deleted 6y ago[deleted]
- deleted 6y ago
- mrtnmcc 6y agoFiveThirtyEight has a page where you can choose winning states (condition on a certain outcome) and it will regenerate the prediction map, https://projects.fivethirtyeight.com/trump-biden-election-map/?cid=abcnews https://projects.fivethirtyeight.com/trump-biden-election-ma... This appears to be the what Andrew Gelman is also trying to do with their raw data. At the bottom of the 538 page it says, " If you choose enough unlikely outcomes, we’ll eventually wind up with so few simulations remaining that we can’t produce accurate results. When that happens, we go back to our full set of simulations and run a series of regressions to see how your scenario might look if it turned up more often." I interpret that as running a regression (linear?) and extrapolating it out to the tail where the conditioning is happening. This should eliminate the issue Andrew is seeing?
- paulgb 6y agoFrom the plots Andrew posted, it looks like the problem is not just sample size and that (some) individual state pairs have inverse correlations, e.g. https://statmodeling.stat.columbia.edu/wp-content/uploads/2020/10/Screen-Shot-2020-10-23-at-11.26.59-PM.png https://statmodeling.stat.columbia.edu/wp-content/uploads/20...
- mrtnmcc 6y agoI'd argue negative correlation on conditionals distributions can be reasonable here. In that particular WA-MS example, if Trump suddenly took more liberal positions and somehow won WA (e.g., announces he's pro abortion), he would in fact be more at risk of losing Mississippi. The idea that these two states are in play already is fringe and would require some major idealogical (or other third variable) shifts.
- amscanne 6y agoBut then the correlations with the other 48 states are broken. In that insane scenario, Mississippi now votes for Biden (because I guess he’s suddenly come out as pro-life as well), but Alaska still goes for Trump. The negative correlations don’t make sense. Maybe it’s a small problem and the model is solid overall, but... I don’t think you can justify that one effect.
- RivieraKid 6y agoI'm surprised that a lot of people almost worship the 538 model, when there is a better model with more competent people behind it. Economist is also more open about how their model works and it's open-source. After reading the methodology I was pretty impressed. Edit: Why am I being downvoted?
- abnry 6y agoWhile numerically literate, I don't understand the details of the 538 or economist models. What I do know is that 538's model has had a great track record. It gave Trump one of the highest chances of winning in 2016. It did very well in prior elections. And both models are essentially predicting the same results: ~10% chance of Trump winning.
- ceilingcorner 6y agoHow is being slightly less wrong than everyone else "having a great track record"? Serious question. Because I find it hard to take any of them seriously after the debacle that was 2016.
- abnry 6y agoThey said Trump had a 1 in 4 chance. That's very high. NYT had something like 1 in 20 chance for Trump.
- ceilingcorner 6y agoThat doesn't really answer my question. It only indicates that they were slightly less wrong than every other media source, not that they have a good model. If I had a laptop that only worked 1/4th of the time, rather than 1/20th of the time, would that make it a reliable laptop? I don't think so.
- delecti 6y agoIt's not "wrong" to predict a low chance for something that eventually happens. Unlikely events can happen.
- newfeatureok 6y agoThe thing about this election is that what if Trump himself is some sort of wildcard that can't really be properly forecasted in the polls? Why is there so much fascination with polls to begin with? I understand that there are betting markets, but it seems sort of silly. If you had a 100% accurate poll, for instance, then what would be the purpose of the actual election?
- ceilingcorner 6y agoThe soothsayers and fortune tellers never went out of business, they just adopted a new name.
- techsupporter 6y ago> Why is there so much fascination with polls to begin with? Polling is useful for candidates and gives them ideas on where to target outreach and spending. For the rest of us, it gives us something to watch. With Presidential campaigns running for almost two years in advance of the actual voting weeks, there’s a huge gap between when the thing starts and when we see results. This way, people have something to fill the time. Even now that voting has started, we are still another 10 days until the voting is done and likely another 7 after that until we have a sufficient count to know who has been elected. That’s a long time for a populace that’s worried, distracted, and interested, especially since so many of us live in states where—due to the mechanics of a broken election system—we can’t do much to influence the national outcome.
- unwoundmouse 6y agoNow that you bring that up, if we had a 100% accurate poll that would be really good for productivity wouldn't it? Perhaps it wouldn't give voters the same feeling of self-determination but it'd save a lot of resources in fundraising, going out to vote, counting votes
- bigbubba 6y agoThe sensation of self determination is the entire point of democracy though. We don't use democracy because we think masses of people are particularly wise; we use these systems because they feel more fair than the alternatives and that perception of fairness produces good results (peaceful power transitions.)
- cwhiz 6y agoNot sure if it’s wrong to put this here, but here is a link to their election forecast. https://projects.economist.com/us-2020-forecast/president https://projects.economist.com/us-2020-forecast/president You can compare this to the 538 model and see where these two teams and forecasts disagree.
- ta8645 6y agoCan someone help me understand what odds like this mean in the context of an election? The model says that Trump has a 1 in 10 chance of winning. With a fair 10-sided die it makes sense that you have a 1 in 10 chance of any given side rolling face up. But what is the die that is being rolled in these election statistics? What is the "chance" element that is being predicted?
- weerd 6y agoThe odds for a face on a d10 would be 1 in 9 (1:9). It's different than probability (1/10 = .1)
- aecay 6y agoIt's typical to use "to" rather than "in" when discussing odds. So the odds of getting a 1 on a 10-sided dice are 9 to 1 against (odds are also typically specified with the larger number first, because of the overlap between mathematical odds and betting-shop odds). And the probability of it happening is 1 in 10. (I suspect that counter-pedantry on these lines might be part of why your post is getting downvoted; I wasn't one of the downvoters fwiw.)
- weerd 6y agoTIL, thank you. Looking back, not a great post. Probably deserves the votes.
- aakilfernandes 6y agoYou're comparing two scenarios, one in which you know all the facts, and one in which you don't. In the dice toss scenario, we know everything relevant. In the election scenario, we don't. A model like this is attempting to say "these are the rules we think exist. Based on the rules, and assuming the data is off by some random distribution, here's what we think could happen". What different forecasters disagree about is what the rules are. For example, the relevance of certain demographic characteristics and the potential variance between polling (conducted prior to the election) and actual election results. There's a huge amount of assumptions, and forecasters disagree on those assumptions. We have very little historical data (polling is very recent) and even with complete historical data, future elections do not always conform to past elections.
- an_opabinia 6y agoAndrew Gelman designed the 538 model in 2007. Nate Silver authored an adjustment to polls used in that model. Polls have more impact if they are more representative of statewide turnout among demographic things he chose like “black” and “low income.” This is why his predictions were so accurate for Obama’s 2008 and 2012 elections, and likely why they were so inaccurate in 2016. Gelman’s own grad student is the only person to have academically published this approach, in a paper about polling Xbox Live users. These guys sort of make a thing that is the same in many more ways than it is different. Why not just share the code is the biggest question?
- umanwizard 6y agoNate Silver said Trump had a 1 in 3 chance, which basically means one shouldn’t be surprised no matter the result. I’m not sure where this “all the polls were so far off in 2016!!” narrative comes from, but it’s wrong.
- vanattab 6y agoThe narrative comes from the medias inaccurate and misleading coverage of the polls in 2016. Many news outlets all but declared Clinton president before the election.
- ordinaryperson 6y agoBut the media is not Nate Silver. He said Trump had about as much chance of winning as the Cubs had that year of wining the World Series, and obviously both happened. Silver did a nice writeup of the whole experience: https://fivethirtyeight.com/features/the-real-story-of-2016/ https://fivethirtyeight.com/features/the-real-story-of-2016/
- untog 6y agoNate was the outlier in that respect. But it’s true that the polls aren’t weren’t all that inaccurate in 2016: a bunch of important swing states were within the margin of error and Trump won some important states by very small margins. The mistake in 2016, IMO was a) the extrapolation that came from those polls and b) people paying way too much attention to national polls, which have very little connection to electoral outcomes, given the electoral college. Also perhaps c) the larger public not “getting” statistics in the way they’ve been presented. The NYT had, if I recall, Clinton at 90% chance of winning. That still means that in one of every ten flips of a coin is a Trump win. But people read “90% chance” as “definite win”. I don’t actually know what anyone should or could do about that.
- justinzollars 6y agoEvery single one of these models will break down this year. We are living in an unprecedented time. I can't understand how we can model how many people will vote, when we don't even know how many people have moved out of cities this year. Half of my friends have left San Francisco - if as many people left Philadelphia, The Twin cities, Milwaukee or Pittsburgh, then that really effects the outcome.
- ceilingcorner 6y agoI've made similar points on topics like this, and bar none, every single time, it is downvoted into oblivion. People seem to have a difficult time with forecasting data that goes against their preferred outcomes.
- justinzollars 6y agoI'm not trying to get points, or keep points. I'm just trying to point out that the models are not going to work this year and its crazy to assume they will.
- evgen 6y agoThe models will work just fine. We are talking about large numbers here, and your small sample of anecdotes does not mean anything. No, there is no mass exodus. No, it is not going to change the results. Yes, the current polls are more accurate than just about any in history and we have an abundance of high-quality state level polls to back up the predictions. If you want to know where the models are going to break down it is more likely that the state-level polling has over-corrected for the factors that caused them to miss the swing to Trump in 2016 and Biden's numbers are even better than what polls are saying (a prediction based on looking at polling at the congressional district level and then seeing how that differs from state-level polling -- the numbers are the district level are closer to national numbers than the lower statewide numbers for swing states.)
- justinzollars 6y ago
- olliej 6y agoIt seems like the behavior between WA and MS could just be statistics saying that WA and MS always[1] vote for the opposite candidate, rather than considering a massive sudden change in the direction that one of them votes in. E.g. it's not reflecting who they vote, just who they most vehemently disagree with. I'm not sure why that kind of interstate correlation should impact predictions? <incoherent rambling :D> IANAS but it feels like these correlations were added to compensate for the failure in 2016 to recognize that state A going one way implied that state B would also go that way. It "feels" like a more correct approach would be to compute some kind of error/weakness measure in a states polls by bringing in those of its geographical neighbors and incorporating the polling error of that entire block vs prior years. Or something. The intuition I'm having difficulty conveying is that actual voting correlation is based on neighboring states only because you've got bubbles of ideology that aren't strictly cut along state lines. If strength of opinion in a bubble is going one way, then you'll see that mostly in the state at the center of the bubble, but the bubble still spreads into neighboring states, and a "stronger" bubble could push it geographically further into those neighbouring states, and/or could increase the bias in areas inside the bubble. </rambling> [1] "Always" == most recent history
- RivieraKid 6y ago> I'm not sure why that kind of interstate correlation should impact predictions? 538 has low positive correlations between states on average, which actually has a big impact, it increases overall uncertainty (and therefore Trump's win probability). Why? If the states are not correlated, you usually end up with a few states going off the rails, like Trump winning Colorado without any nationwide swing.
- DavidSJ 6y agoOther way around: uncorrelated errors tend to cancel each other, correlated errors tend to reinforce each other.
- RivieraKid 6y agoWell Gelman claims the opposite.
- yk 6y agoSomewhat interesting, however the guy lost me more and more the longer he argues. So, the various anomalies in the dataset are somewhat interesting, but having weird outliers in the margins is an entirely expected effect. Just because there are not many datapoints. So when you filter for something marginal like Trump winning New Jersey, then the statistical error increases and therefore it is entirely unsurprising that something weird happens. Thankfully, these systems are designed to work with probabilities, and these outliers are weighted down. Additionally, getting worked up about a 3% chance of Biden winning Alabama. I mean, what does a 3% chance even mean for a one off event, compared to a 5% chance or a .3% chance? I know fully well, that it means I should bet $100 if I can get more than $3000 payout, but the trouble is that is only if we bet often enough. (Perhaps often enough on different things.) For a one off thing, the important part is, it is with a very high degree of certainty a loss of $100. So any claims that Bidens chances of winning are too high should be regarded with high suspicion. Also, I listened eralier to Nate Silver's model talk [0], where he discusses quite a few problems with low quality polls in some states. [0] https://fivethirtyeight.com/features/politics-podcast-national-and-district-level-polls-disagree-about-how-close-the-2020-election-is/ https://fivethirtyeight.com/features/politics-podcast-nation...
- RivieraKid 6y ago> Just because there are not many datapoints. There are more than enough data points to determine the between-state error correlations, many of which seem to be very off. > Additionally, getting worked up about a 3% chance The weird between-state correlations actually have a large effect, they increase state and nationwide uncertainty and as a result Trump has a higher chance of winning.
- noirbot 6y agoIt seems more likely that it's actually the other way. Nate Silver has specifically said he built the model to have relatively high uncertainty, especially with the volatility of this year, so this seems more like the outcome of intentional decisions to not let the model be overly confident.
- deleted 6y ago[deleted]
- klyrs 6y agoThe negative correlation between NJ and AK is curious. What I'd like to see here (and in all of these forecast sites) is some confidence analysis. If you pick Trump in NJ, looking at the plots, you've selected a tiny fraction of the data to examine. Who cares if the predicted value is wonky; show me the confidence interval! Treating this as a tea-leaf reading (that is, deliberately searching for meaning via free association, without investing it with a truth value) I'm reminded of the "own the libs" meme. I see folks on foxnews.com comments bragging about it; I see lefties complaining about it, but I suspect that it's overblown and not actually a driver behind people's decision-making. But that's what comes up for me when I see "NJ goes Trump" forcing "AK goes Biden". I'm amused by the resulting thought experiment... if dems started airing "socialists for Trump" campaigns in otherwise safe GOP states, would it move the needle there? Even sillier: if you aired those ads in NJ, would it move the needle in AK?
- runawaybottle 6y agoThis is the 2008 election result county map in NY: https://upload.wikimedia.org/wikipedia/commons/thumb/1/1f/New_york_presidential_results_2008.svg/440px-New_york_presidential_results_2008.svg.png https://upload.wikimedia.org/wikipedia/commons/thumb/1/1f/Ne... This is it in 2016: https://upload.wikimedia.org/wikipedia/commons/thumb/e/e6/New_York_Presidential_Election_Results_2016.svg/440px-New_York_Presidential_Election_Results_2016.svg.png https://upload.wikimedia.org/wikipedia/commons/thumb/e/e6/Ne... Long Island is really red there. It’s really hard to say how a democratic stronghold like NYC and something literally a 45 min train ride next to it could vote so differently. Long Islanders are not separate from NYCers, they commute to and work in the city. To your question, could experiments work in similar situations like this across the country for either side? I think so in the next 50 years as demographics shift (and I don’t think it’s as simple as urban liberals taking over, people do become more conservative as they get older). God knows the dynamic at work between NYC and Long Island in 2016, but it’s obvious things are in flux. I’ll make a bold prediction here. If Long Island is that red again, yeah, you better believe the typical rust belt states are staying red.
- sampo 6y agoFunny thing, can't submit this to r/politics, as they seem to have a tightly curated whitelist of allowed domains that must "Be notable, as defined by our domain notability guidelines. Notable domains will consist of news organizations, research organizations, political advocacy groups, governmental agencies / bodies, and political parties." And apparently columbia.edu does not fulfill those criteria.
- thetinguy 6y agoDid you try asking the moderators to add it to the whitelist?
- sampo 6y agoIn a way. They have a form for suggesting new domains to the whitelist: https://goo.gl/forms/lRQikA1rI0bVbKCl1 https://goo.gl/forms/lRQikA1rI0bVbKCl1
- secondcoming 6y agoDon't bother, /r/politics is one of the most censored places on the internet. It's best avoided. Despite its description don't expect any actual adult discussion of politics there.
- noirbot 6y agoTo me, this mostly tracks with what 538 has said on record about how their model works and the design philosophy behind parts of it. To me, what Nate means when he says "directionally the right approach in terms of our model's takeaways" is that these sorts of wild and unintuitive outcomes are part of the point of the way the model is constructed. Specifically, that when you get off into the weird situations like Trump winning Washington state, it's likely something incredibly weird has happened - something that likely has no historical precedent, so it may actually be a more sane thing to do to assume that now almost everything is backwards and Biden would win a bunch of states he shouldn't either. To me, this points to a general willingness in the 538 model to just go "who knows" and build in some room for insane things to happen on the fringes. The Friday podcast episode about the 538 model specifically mentions that they have large/fat tails on their distribution that make it nearly impossible for someone to get over 95% chances of winning on a national level, and these sorts of wild results seem like the outcome of that. If you bake in an assumption that there's always a 5% chance of something crazy happening, that chance has to come from something in the data somewhere that reflects the ability of that to happen numerically, and thus will have numerical outcomes that seem impossible.
- drblast 6y agoYeah I think for Trump to win washington state, he'd have to do something to appeal to voters there in a way that would likely cause his red state base to abandon him. The negative correlation makes sense when we think about how difficult it is for everyone in Washington to suddenly turn conservative and everyone in Mississippi to turn liberal. Much more likely is that the crazy thing is that the candidate or circumstances changed in some way. It makes more sense if we ask...if a candidate wins NJ what is the chance they also won AK?
- Invictus0 6y agoI think there's nothing Trump can do to win Washington; rather, more accurate to say there are many things Biden could do to lose Washington.
- Traster 6y ago> I’d think that if Trump were to win New Jersey or, even more so, California, that this would most likely happen only as part of a national landslide of the sort envisioned by Scott Adams or whatever. That's a valid intuition to have but you can also clearly make the argument that if Trump wins California you're in such a weird scenario that using the traditional wisdom about correlation is dangerous. The point that 538 have tried repeatedly to make is that firstly: if you're conservative in your level of confidence you'll give a higher likelihood to outliers, and secondly: It's not particularly useful to focus on whether X has a 3% or 4% chance. If Trump wins California, we aren't going to be talking about whether the chance was 3% or 0.3% we're going to be talking about that Nuclear explosion that wiped out 25million Californians. For the same logic the reason that Trump winning Alaska given winning New Jersey is lower than given losing New Jersy is because your sample size is rubbish. The chance of Trump winning Alaska given losing New Jersey is an accurate number, the number of Trump winning Alaska given winning New Jersey is like saying "How likely is it Trump wins Alaska given the UK gains US statehood" it's like.... well... if that happens then we're so far outside of what the model thinks can happen then you should be that we're just gonna say it's 50:50 - because who the hell knows. It's not like saying "Oh well if X swing state goes blue, Y will probably follow", the scenarios in this article are so bizarre that the model should rightly be very cautious and probably default to either refusing to give an answer or just default to 50:50 or the same probably ignoring that data. The implicit bias in this analysis seems to be that if NJ went Red that would be because Trump won by a big margin, but that's not a likely enough scenario to actually get numbers for, and is so unlikely that things like "The supreme court threw out all the ballots for inner city areas" start to become valid possibilities.
- noirbot 6y agoIt is worth noting that they explicitly aren't accounting for any election tampering/throwing out ballots in the model, so that final hypothetical isn't something they're factoring in. Your "an explosion kills everyone in LA" is closer to the sort of things that the model is "considering" in so much as a pile of statistical code has any understanding of what could happen in the real world to cause the outcome it's putting forward.
- evgen 6y ago
- altdatathrow 6y agopolls and votes don’t matter when one side controls faithless electors and the supreme court. none of these models are accounting for that black swan.
- phonebucket 6y agoThis is undoubtedly useful to know when playing around with different scenarios. However, for my use of 538, I’m perfectly happy to ignore such scenarios (such as Trump taking New Jersey). I can call the election in his favour by myself in these scenarios without needing the model.
- peteradio 6y agoThe problem is that 538 is correctly factoring the voting fuckery done at the state level which ruins the voting correlations. Gelman seems to be modelling fair elections - ha! Now the question, how did 538 come up with the correct model which takes into account vote manipulations at the state level? /s
- TTPrograms 6y agoFor a post on a blog presumably focused on statistics the author seems oddly concerned with a model that predicts odd events as being "possible". The difference between "possible" and "impossible" is \eps << 1 - there's no real distinction to be made there in practical statistical terms. Likewise asserting that California with a 3% chance of going Trump is absurd is an unreasonable degree of overconfidence. Assuming maximizing expected return, the author is implying that they would be willing to take a bet that Trump would lose California with odds >> 97::3, i.e. presumably they would take a bet where I bet $1 to every $99 they bet. To be critical of a model based on outcomes it predicts with tiny probability you need truly remarkably biased priors.
- artichokes 6y agoI don't know if I'm reading your comment wrong but I would happily take that bet- I think most people would. Want to make it? The election result of California isn't a matter of probability; it's an empirical matter that the number of people who will vote for Biden in California far exceeds the number that will vote for Trump.
- TTPrograms 6y agoTo you and the other comment - as much as I dislike Taleb's rhetoric, this is precisely the sort of bet he has made a lot of money on. People round rare event probability down to zero, and if you bet against them enough with sufficiently extreme odds you'll eventually (and in expectation) hit a home run. I would be more than happy to make this bet with anyone willing to take the other side - as in literally, find a modern middle-man system and I'm game. You may want to be aware that you have provided me a nontrivial arbitrage opportunity, as the odds on predictit are closer to 7:93 : https://www.predictit.org/markets/detail/6611 https://www.predictit.org/markets/detail/6611
- runawaybottle 6y agoBut would Taleb specifically make a one time bet on one particular Black swan? Isn’t the idea the same as venture capitalism, where this idea only works if you do it with all possible black swans? You’d have to be crazy to take this one specific bet, you can only realistically take all possible improbable bets.
- vitus 6y agoOne thing to note: 538 uses a t-distribution (and calls out on regularly in their podcast that this yields much heavier tails than a normal distribution). Even 40,000 samples is not enough to characterize the tails. So, it seems to me that the entire article is predicated on a faulty conjecture, namely that 538 uses a mixture of a normal distribution with an independent heavy-tailed one. (It's not explicitly stated what the author thinks the base model is, but I think "normal" is a reasonable guess.) I'd be interested in seeing a reverse-engineering analysis of 538's choice of distribution parameters, and extrapolation from there to see if these pathologies still arise with (much) larger samples. ... That said, ultimately, the choice of how fat to make the tails is a modeling decision, and how the models behave outside the regime of interest isn't as important as how they behave within the operating region. There are key ways we can evaluate goodness of fit once we have results (e.g. bias, MSE) which we can use to determine just how wrong the model was as a predictor, and chances are pretty good that we won't see, say, Trump winning NJ, so we won't actually be able to validate the tail correlation with the vote in PA. But we will be able to validate the correlation in margin between PA and NJ. Maybe 538's tails are too fat, and every prediction in the 80-95% range ends up going as predicted. Or maybe they're not fat enough, and some races in the 99% bucket end up going the opposite way. Point is, we won't know for sure which models were the best predictors until we can verify the predictions. (see: all models are wrong, etc. Newtonian mechanics work great as long as your objects are big and slow, for instance.)
- screye 6y agoI disagree with the author on the idea that tail is too fat for isolated anomalies. There are most certainly events that can happen, which may lead to a red California or a blue Alabama. Presidential assassination, war, video proof of something incredibly heinous (pedophilia?), etc. can absolutely lead to these outcomes. You don't even have to go that far back. Nixon and Reagan flipped states like no-one's business. I do however agree, that 538's state-state correlation model seems weak. California and Alabama would only flip during a wave, and that wave would consume any and all states. The fact that 538's model doesn't strongly show that pattern is a failing of it. But, it is not clear if a model that inaccurately models the unlikeliest of events (california flipping while Florida stays blue), does not necessarily mean that it is terrible predictor of it's primary target (Presidential likelihoods). As a data scientist, I can totally understand Nate's hesitation. Do you impose strong priors on the model to reflect strong domain intuition or do build a model that best characterizes the data it is based on. In the presence of infinite data, you should abandon all domain based priors. For single digit data points, priors are essential. For any number of data in between, it is anyone's best guess.
- yodon 6y agoI've always liked Enrico Fermi's attitude on this. When you're Enrico Fermi, you get to say things like "One data point gives you a curve. Two data points gives you the distribution about the curve."
- pierrefermat1 6y agoCurious is there a source for this? It is meant as satire?
- yodon 6y agoThe source is a long ago personal comment from my dad, who worked with and was personal friends with Fermi. As to whether it's sarcasm, I'd describe it more as an inside joke with just a touch of the self-aware intellectual arrogance Physicists are famous for (see Lord Rutherford's "All science is either Physics or stamp collecting"). Trying to explain a joke is always dangerous so hopefully what follows won't be a mistake, but here goes: The way I have always interpreted the first sentence is to hear Fermi saying that when you are doing an experiment, you should have a deep understanding of the family of curves or behaviors the system under test is expected to follow, including the full range of curves that would follow from interactions that are wildly different from what you might naively expect. If you really understand the Physics of the system (this is where it starts to blend the line from advice to a joke), the measurement of one single data point should be enough to tell you which of those possible curves describes the actual behavior of the system (the joke being both that it's funny to be that arrogant and that it's obvious to anyone you'd tell this joke to that mathematically you need at least two points to determine a line, so by saying one point gives you not just a line but a curve the speaker is purposely going over the top for fun). Moving into the second sentence ("two points gives you the distribution about the curve") takes the statement into full-on joke mode, with the comment shifting from an observation about knowing your Physics to an insider dig at the relationship between theoretical Physicists and experimental Physicists (Fermi being one of the greatest theoretical Physicists of all time). When he says that two points gives you a distribution about the curve, he's saying he as a theoretical Physicist understands the underlying Physics of the system better than the experimentalist understands the noise in their experimental hardware, or alternatively that the noise in the hardware is sufficiently uninteresting as to be irrelevant to him. The former view would simply be arrogance but leaving the second option open circles the joke back to include a bit of insider self deprecating humor in that he's purposely ignoring experimental error, a thing theoretical Physicists are famous for doing.
- lordnacho 6y agoI only briefly scanned this, but it sounds like there's some correlations that turn up when conditioning on specific scenarios? Doesn't that sound like Berkson's Paradox?
- fisherjeff 6y agoI’ve decided to start my own election forecasting site that only ever gives 50/50 odds. Then I’ll just have to wait for the next 2016-style underdog win and my inevitable victory lap in the press as The Guy Who Called the Election.
- sequoia 6y agoAs a US voter I am very frustrated With and tired of this obsession with election forecasting. Is it going to influence whether or not you go out and vote? If not, what is the point of it? What value does something like fivethirtyeight add to our democracy, if any? Is this motivation the same as that of diving deep into baseball stats or Star Wars starship engineering, just like “nerding out” for its own sake? Contrast the voter who never looks at any of these polls with one who keeps up with them daily. Is the latter voter better off in some way? Is this just about trying to read the tea leaves so you can strut and preen later about having been correct, should the dice roll be in your favor? My concern is that these things are distracting and may actually dissuade some people from voting because they think they “don’t have to.” Here’s an idea: everyone go vote for whoever you think the best candidate is regardless of what a stack of polls say. Someone set me straight here, what is the point of all this stuff.
- ncw96 6y ago> What value does something like fivethirtyeight add to our democracy, if any? One possible use of polls and election models is for helping people who want to donate to candidates determine which races are closest and where their money is most likely to have an impact. > Here’s an idea: everyone go vote for whoever you think the best candidate is regardless of what a stack of polls say. Because the US does not have ranked choice voting in most elections, polls are useful to determine which candidates are viable. If your preferred candidate is only polling at 5%, they are pretty unlikely to win, so you might want to vote instead for whichever of the leading candidates you find most agreeable.
- xnyan 6y ago>Here’s an idea: everyone go vote for whoever you think the best candidate is regardless of what a stack of polls say. On the one hand, I think I get your sentiment. On the other, I mean, we are all just solitary individuals floating through this life. Almost all of our important decisions are make at least in part (or more in some cases) dependent on the thoughts and actions of others. That's natural, right? You do otherwise in your life? If the above is true, it makes total sense why one out of 7 billion plus people would want to understand the choices of others before making theirs.