17 ms·

On the data nerd side I continue to be shocked at how people misinterpret the certainty of polls/forecasts. Forecasts give us probability distributions based on

On the data nerd side I continue to be shocked at how people misinterpret the certainty of polls/forecasts. Forecasts give us probability distributions based on historical polling error data. Not infallible predictions the expected value will 100% happen. It’s fairly revealing of society’s general innumeracy, just as it was 4 years ago when Trump won.

fullshark 6y ago
What % chance did the polls/forecast give the republicans gaining seats in the house (I actually don't know but from reading pundits it sounds like very little)? I think the interesting thing this cycle is betting markets were pretty much also at odds with pollsters/forecasts on Trump's chances, basically baking in a pro-trump (R?) polling error. Perhaps there really are issues with some of these forecasts / uncertainty estimates?
- softwaredoug 6y ago
  I think I only saw projections for Dems retaining control (which was high). I think GOP gaining seats was always in the cards. I think it’s interesting too how some of the qualitative predictions (like Sabatos crystal ball) were pretty close to the outcome in the Pres race.
  - evgen 6y ago
    In just about every model, but let's use 538 as the canonical example, there was a reasonable chance for the Dems to lose seats but a very, very low chance they would lose control. We are looking at a two or three standard deviation variance from expectations for loss of control but losing seats was well within the range of outcomes that should have surprised no one.
- jayd16 6y ago
  It wasn't that little at all. Eyeballing the 538 graph it was about a 38% chance of Republicans gaining in the house? Over a one in three chance. https://projects.fivethirtyeight.com/2020-election-forecast/house/ https://projects.fivethirtyeight.com/2020-election-forecast/...
  - fullshark 6y ago
    Hmm looks like 548 gave a 3% chance of R control and betting markets gave about a 15% chance on election night: https://www.predictit.org/markets/detail/4365/Which-party-will-control-the-House-after-2020-election https://www.predictit.org/markets/detail/4365/Which-party-wi... You can also try to work the probability distribution from this betting market and I'm sure they were more favorable to R than he was: https://www.predictit.org/markets/detail/6669/How-many-House-seats-will-Democrats-win-in-the-2020-election https://www.predictit.org/markets/detail/6669/How-many-House... That's a pretty big discrepancy regardless, did he ever give an explanation as to why he disagreed with betting markets so much? Seems like betting markets had a ~ one standard deviation pro Trump/R polling error baked into their pricing.
    - jayd16 6y ago
      >why he disagreed with betting markets Because betting markets are kind of dumb? Nate Silver has uttered those words a few times, anyway.
      - fullshark 6y ago
        They have a pretty long track record of being predictive.
        jayd16 6y ago
        Not in this case, clearly.
        fullshark 6y ago
        Actually i think the number of seats gained by republicans will be pretty close to the P50 of the betting market probability distribution. They seem to have done better than 538 at least there. It's impossible to verify the overall Biden/Trump victory probability mass function but you could look at state by state races and see who did better.
Fuzzwah 6y ago
As an outsider all I can think is that this is an issue with the US media. The easy story is talking about the latest poll. The more difficult, but actually useful story, is focusing on the policies being put forward by each side.
535188B17C93743 6y ago
I think the big issue was the "margin of error" in many of these cases. For example, when Quinnipiac did a poll a few days before the election [1], they said Biden was winning Ohio by 4 points with a +-2.5 point margin of error... which happened to be very far off. And this wasn't isolated-–if memory serves me, Wisconsin was even further off? [1] https://poll.qu.edu/florida/release-detail?ReleaseID=3683 https://poll.qu.edu/florida/release-detail?ReleaseID=3683
jonhohle 6y ago
But the polls then and now weren’t just bad, they were shockingly bad and not representative of any real population that matters in an election. I’d expect something like this from junior engineers looking at metrics for the first time, but these are supposedly the most respective organizations in their field that have failed spectacularly for the second presidential election in a row. What value do they possibly serve? Should Biden ultimately be declared the victor, I’m concerned that the public will never receive the postmortem on what went wrong and how it happened again that it deserves.
- evgen 6y ago
  I really have no idea where this idea that the polls failed comes from. There were only two bad calls this cycle and every other outcome was within the margin of error. It was pretty much the same case in 2016 where people who had no idea what they were talking about suddenly decided they were certain polling failed because they are unable to grasp the concept of margin of error and sample size. Polls can only guess about turnout and the try to work backwards from there. The turnout estimates were wrong but not shockingly so, and as a consequence a lot of polls ended up having the result be at the far end of their margin of error. Nothing went wrong. Polling is hard. Get over this idea that you can have some sort of certainty regarding an election until we actually hold the election.
  - js2 6y ago
    I know they were within the margin of error, but the polls were all systematically within the margin of error in the same direction. Did a single poll that was wrong predict at GOP win and end up a Democratic win?
    - srtjstjsj 6y ago
      Yes.
      - js2 6y ago
        I can’t find that via Google. Help me out here.
cratermoon 6y ago
Couldn't the problem be that polling is just numeromancy dressed up in fancy math?
- ironSkillet 6y ago
  The problem that has appeared again is that without a representative sample of information gathered, it doesn't matter how fancy your models are, you will not have accurate results. And quantifying how unrepresentative your sample is basically impossible to figure out in this situation.
  - cratermoon 6y ago
    > quantifying how unrepresentative your sample is basically impossible Yeah, so... numeromancy.
  - saberdancer 6y ago
    Maybe they should poll about what the person will vote, but rather what the persons opinions are in general and then estimate based on that who will they vote for. I am betting Facebook or Google would be able to better calculate the outcome than any polling firm. Bring in all the data they collect on each internet user, I am sure they can model the likelihood of voting and where the user is leaning.
will4274 6y ago
I'm on the nerd side too, but I was quite surprised. Sampling isn't the only source of error in election polling - people also change their minds. In this election, people decided who they were going to vote for historically early - the lowest portion of people in the history of exit polls indicated they decided in the week before the election. In theory, this effect should reduce the width of the probability distribution, and yet, the polls were off again, in the same direction as they are always off, by a little bit more than average. A little scrutiny here is worthwhile. In 2016, for example, the polls were off in part because pollsters weighted the black demographic in proportion to their voting behavior in 2008 and 2012 (when a black candidate was running, boosting turnout). The actual 2016 black turnout looked more similar to 2004 or 2000 - other elections where a black candidate wasn't running. Though I haven't dived into the 2020 polling data, it wouldn't surprise me if a similar effect were at play. It's also become noticeable harder to do polling. Once upon a time, everybody had a landline, so random dialing worked really well. Now, some people have more phones than others, and people are increasingly less willing to talk to pollsters, increasing the error rate. These are all important things to talk about - when you dismiss it as "oh yeah, polls are never perfect," you prematurely shut down the conversation. Don't let perfect be the enemy of the good. The polls could be better and it's important to talk about how.
- saalweachter 6y ago
  There is also "what polls directly measure" and "what we infer from them". Besides the usual adjustments for likely voter models, we also do things like infer how people will vote in House and Senate elections based on Presidential polling, because we do a lot more Presidential polling than down ballot, because money. So one early thing that seems to have happened this year (and again, we are still early in the process to be making final verdicts in "what happened" retrospectives) is that people split their ballot more than anticipated, which means that the inferences we were making about down ballot races based on Presidential performance were off. So not quite a polling error, but an example of uncertainty that may not have been correctly taken into account in forecasting.
- A4ET8a8uTh0 6y ago
  << These are all important things to talk about - when you dismiss it as "oh yeah, polls are never perfect," you prematurely shut down the conversation. Don't let perfect be the enemy of the good. The polls could be better and it's important to talk about how. I think the complaint is that here the polls were not just imperfect. They were at best, misleading raising questions over methodology ( lessons, apparently, not learned in 2016 ), wishful thinking, and their usefulness, or, at worst, attempts at influencing desired income. I can absolutely agree that polling is not an exact science, but now it is twice in a row that polling science has grossly miscalculated the mood of the nation.
js2 6y ago
I really want to know what went wrong with the polls (again). Shy Trump voter doesn’t explain how wrong the polls were about Graham or Collins, outside of even the margin of error. Accurate polls are important to running campaigns and to legislative positions. Selzer got her polls right in Iowa and she posited that she knew the local electorate better than other pollsters so she weighted correctly. Maybe Trafalgar Group has a better method or maybe it just weights GOP voters more heavily (we don’t know, it won’t disclose its methods). Or maybe Trump just breaks polling and the polls will be fine next time. Whatever the reason, I’d like to know, but I’m not sure we ever will.
- atoav 6y ago
  I grew up in a right wing province of Austria and saw that sort of wrong polling throughout my youth — which is why I wasn't surprised the slightest about things going as they did. It boils down to: a fraction of the electorate will vote for $bigotcandidate but would never ever admit it to anyone outside the voting booth, because they know it is wrong. They will tell people they will vote for $notbigotcandidate, but vote for him nonetheless. This fraction is not insignificant and you cannot easily represent it in polls without guessing the effect. In my home province it eas consistently around 10 to 15% over a decade.
  - js2 6y ago
    I don’t see how that explains the Collins/Gideon race, but I’m not a Mainer. Anyone here from Maine can chime in why a Mainer would lie about voting for Gideon?
    - gkop 6y ago
      Bigots turned out. And lean Collins.
    - fma 6y ago
      My same question. OK you can be claim to be a "good Republican" and vote Biden but (R) down ballot. Why were Senate and House races also wrong? Why did Lindsey Graham win by so much when the polls didn't give him that lead? In South Carolina, it's ok to say you'll vote for Graham.
dorkwood 6y ago
What you're saying is that any result with a non-zero chance of happening should be unsurprising to us, because it was still in the realm of possibility? If that's the case, what use are polls? What makes an election forecast based on polls more valuable than a coin flip?
- softwaredoug 6y ago
  Your coin flip analogy is spot on. But instead of a 1 in 2 probability, Trump had a 1 in 10 probability of winning. We knew this probability precisely because of polling data (and historical error). This wasn’t some rare weird outcome. This was probably a standard deviation or so closer to Trump in a forecast. Perfectly normal thing to happen.
  - ghayes 6y ago
    Sure, the model had value precisely because it said that even if Trump did better than the polls suggested, he would likely still lose due to the paths of victory Biden had. That said, the polls have shown, in many cases, poor predictive behavior of actual votes, and as such, we should not just brush this off by quoting the distribution of outcomes.
- jayd16 6y ago
  Results fall on a spectrum from surprising to unsurprising based on the modeled prediction and confidence.
- pessimizer 6y ago
  Polls are astrological race science, and the reason you'll never be able to convince anyone of this is adequately covered by the book When Prophecy Fails. I've come to think that their bias is only slightly ideological, but instead springs more from a self-serving impulse to mirror the current beliefs of the audience most likely to believe in polls i.e. people overly impressed by credentials. They enjoy the sheer volume of numbers, rather than their actual usefulness. Alternate phrasings of identical questions asked in polls creates differences in outcome vastly larger than the effects they claim to detect. Somehow Nate Silver managed to get rich by selling synthetic CDOs of arbitrary polls (to people with his identical political outlook.) The only real poll is to ask the same questions, in the same way, as the ballot sheet does. Ideally these should be asked as people are entering or leaving the voting booth, and people should be paid for their responses, in order not to bias yourself towards bored loudmouths with nothing to do. This would be good for accuracy, not for usefulness in anything but generating a suspicion of election fraud. Instead, they're trying to figure out what percentage of black people go to the polls in Missouri, and pretending that the self-selected black voter has more in common politically with the general black phone answerer than the Missouri voter. Relying on that arbitrary assumption is the entire basis of their field. edit: I haven't read https://site.pennpress.org/aha-2021/9780812250046/race-and-the-making-of-american-political-science/ https://site.pennpress.org/aha-2021/9780812250046/race-and-t... yet, but I heard an interview with the author around the time that it came out, and it's another hidden history of a field that has completely unearned legitimacy. "Political Science" might largely be considered an outgrowth of scientific racism.
phlakaton 6y ago
I think it's more fundamental than numeracy. If the polls suggest there is a 2/3 probability you are going to win an election, but you lose, does that mean the polls were wrong? What if the probability was 9/10? I think the problem is trying to make sense of outliers and uncertainty, and that, in an election with 150M+ variables, you will end up with a result that isn't quite what anyone predicted. In other words: it's not numeracy, it's epistemology.
- elpakal 6y ago
  Great comment, agreed. The wonderful books from Nassim Taleb ‘Black Swan’ & ‘Fooled by Randomness’ touch on this in a way that forever changed my thinking. ‘We just can’t predict’.
- srtjstjsj 6y ago
  Fivethirtyeight's polling averaging model underestimated Trump in all the swing states. Trump wine 2-3 states predicted for Biden, and 0 vice versa. States that Biden had a supposedly strong lead in turned out to be statistical ties in the official vote. The direction bias in the polls (about 20 of 25) is obvious, it's not symmetric random error.
  - phlakaton 6y ago
    To which you then have to ask: 1) were the discrepancies within the margin of error? and 2) to what extent are you justified in critiquing the polls based on a sample size of one? I understand that the voting percentage we saw nationally was within the margin of error that 538 predicted, and that the specific configuration of electoral votes we saw was also one of the outcomes considered by the models.
  - TheCoelacanth 6y ago
    Modeling correlated polling errors is basically what 538's models are all about. They model how far off the polls could end up being from the actual result based on past elections. The polls are never 100% accurate. The pollsters have to estimate how many of each type of voter will actually show up to vote and what the response rate for each type of voter was. It is not an exact science.
meroes 6y ago
You could also fit the same data with "people polled thought they could vote until their vote was preferentially suppressed". So the polls will never match outcomes. And that it happened in the same direction two cycles in a row leads credence. (I do not know why the forecasts are wrong, but I don't see why Parent is at all justified in his statement).
listenallyall 6y ago
if you're a data nerd, how can you NOT be shocked by so many results falling far outside the confidence interval (often multiple sigmas), and all in the same direction?
- deleted 6y ago
  [deleted]
- atoav 6y ago
  Well this measures how many people are willing to admit they would vote for Trump which is not surprisingly a lower number than people who will vote for him in an unobserved secret vote. Even if the pollsters are the most neutral nonjudgmental people on earth there will still be a fraction of the electorate that will (irrationally) shy away from telling them the truth. This effect is certainly real.
  - listenallyall 6y ago
    Ummm, absolutely not. Pollsters universally stated they either had "adjusted their models" after being embarrassed in 2016, or that the "hidden Trump voter" was a myth (usually combined with "Trump fans won't shut up about how much they support him"). Poll results were repeatedly presented as forecasts of actual election results.
    - atoav 6y ago
      I grew up in a right wing Austrian province which over a decade consistently polled 10 to 15% wrong (in favour of the right wing candidate). I know those people in person. They are people who will tell even their spouse they vote the left wing candidate, but after the 6th beer they will tell you how they really want the strong guy in charge (note: Hitler is from Austria, voting the rightwing guy can still be a taboo for certain people). So these people do exist. If it is just them who make up the difference — no idea, but they exist.
      - listenallyall 6y ago
Vaslo 6y ago
Everytime I read a comment like this I think of that line from Dumb and Dumber: “What are the chances of a guy like you and a girl like me... ending up together?” “Not good” “Not good like one in a hundred?” “I'd say more like one in a million.” “So you're you’re saying there's a chance!” I know enough about probability distributions and modeling to know you can improve your models to lower that uncertainty. And sorry, you are the Lloyd Christmas in this one.
beervirus 6y ago
Statistics can be pretty unintuitive. On top of that, most people have never learned any statistics.
jayd16 6y ago
Whats even more frustrating is recognizing the population difference in day of voters and mail in votes, expecting a "blue shift" as votes are counted, and then watching pundits chime in on Tuesday about surprising results. As far as I can tell the most surprising result was Trump's popularity among Cuban Floridians.
- evgen 6y ago
  Once Biden's team let Trump paint him as socialist I was not surprised to see the Cubanos go Trump (or the Venezuelans for the same reason) but I will be interested to see some of the breakdowns of the Puerto Rican vote in FL and some of the Central American Latino vote in Texas; I am betting there is going to be a reasonably large gender difference in some of those groups.
- srtjstjsj 6y ago
  Cubans are anti-Castro Republican. The young generation of pollsters forgot that Cubans aren't just "generic Hispanic".
tsimionescu 6y ago
You're picking the easiest number to misread - the overall "chance of being elected", which doesn't really mean anything. Much more interesting are the direct results predictions, especially per state where they matter most. Those are polls that give a particular result with a +- confidence interval. If the polls were to any extent competently done, you expect the result to be well within that margin of error. The reality though has been that the results have been well outside the margin of error, sometimes multiple sigmas from the actual result - that is simply unacceptable for a poll. The only conclusion should be that polling data can be completely ignored, as it doesn't seem to have any clear relationship with the final results.
- evgen 6y ago
  Individual polls were sometimes outside the margin, but collectively the polls were not far off the mark and collectively they were within the margin of error. This is why sites like 538 were able to model the various outcomes with some degree of certainty and ONCE AGAIN manage to get results within the range of expected probabilities. You really seem to have a hard time understanding both polling and probability and in particular the relationship between sample size and error. I suggest a bit more research.
  - disgruntledphd2 6y ago
    Dude, I really don't get why you're being so gung ho on this. The errors on the individual polls were way out of line with the actual results. When we regularise these results like 538 or the Economist did, we get slightly better results, but they were still not particularly accurate. This is a real problem, and not one that we should be ignoring. I gave the polling companies a pass after 2016, but that this happened again is extremely concerning. > This is why sites like 538 were able to model the various outcomes with some degree of certainty and ONCE AGAIN manage to get results within the range of expected probabilities. But they were super close to the edges of the intervals. Like the most expected number of electoral votes for Biden was 380-400, which is not what happened. > You really seem to have a hard time understanding both polling and probability and in particular the relationship between sample size and error. I suggest a bit more research. Please don't be dismissive of other people. Many people (including people who do stuff like this every day) were very surprised by the results. We definitely shouldn't tar and feather the pollsters, but I think it would be great if we could get an understanding of why the polls appear to have been systematically biased.
wesnerm2 6y ago
To be fair, we still don't know the loss of votes through mail-in voting. Mail-in voting is more convenient which would likely increase participation, but missing signatures, naked ballots, late or undelivered mail, etc can prevent those ballots from being counted. When there is a major difference in voting patterns between the two political parties, then polls may not actually be inaccurate. In this case, Democrats chose mail-in voting and Republicans, by and large, chose in-person after being directed by the president. Also, some percentage of mail-in ballots may not have reached their destination because of possible election interference through reduced post office efficiency--in particular, fewer mail sorting machines, lower hours, deprioritized mails, etc. In short, the polls don't take into account election interference in the post office or unsuccessful delivery. If most people from both parties were voting in-person, maybe we might have had more parity between the polls and election results.
einpoklum 6y ago
> Forecasts give us probability distributions based on historical polling error data. ... and a lot of assumptions about the supposed underlying probability spaces.
ehejsbbejsk 6y ago
I am pretty certain Facebook has the data to crunch very accurate predictions in elections large and small.
- fullshark 6y ago
  Based on what? I think part of the problem with our politics these days is confusing social media engagement / metrics (specifically twitter) with national sentiment. Seems like there's a large mass of unobserved "silent" Americans out there.
deleted 6y ago
[deleted]
tibiapejagala 6y ago
I don't get why people care so much about predictions vs actual results in case of presidential elections in the US. The rules look like they were designed for maximum entertainment and sudden changes. The fact that you can win with less votes. When the other person already has 250 electors, but then on the last second you flip Florida or Pennsylvania with a few thousand votes and bam! you are the president! That's why people love watching your elections so much. Or maybe also because of the fallout in their countries.
deleted 6y ago
[deleted]
bendmorris 6y ago
>Forecasts give us probability distributions based on historical polling error data. They would give you a probability distribution if they were truly random samples. But in reality, response rates are very low, and pollsters make up for this using demographic weighting and modeling - so the results don't actually represent any real population, but rather are extrapolated from people who respond onto a hypothetical population that the pollster thinks is likely to vote. This is why they are spectacularly and systematically off, more and more over time. It's not just variance.

fullshark 6y ago

What % chance did the polls/forecast give the republicans gaining seats in the house (I actually don't know but from reading pundits it sounds like very little)? I think the interesting thing this cycle is betting markets were pretty much also at odds with pollsters/forecasts on Trump's chances, basically baking in a pro-trump (R?) polling error. Perhaps there really are issues with some of these forecasts / uncertainty estimates?

softwaredoug 6y ago
I think I only saw projections for Dems retaining control (which was high). I think GOP gaining seats was always in the cards. I think it’s interesting too how some of the qualitative predictions (like Sabatos crystal ball) were pretty close to the outcome in the Pres race.
- evgen 6y ago
  In just about every model, but let's use 538 as the canonical example, there was a reasonable chance for the Dems to lose seats but a very, very low chance they would lose control. We are looking at a two or three standard deviation variance from expectations for loss of control but losing seats was well within the range of outcomes that should have surprised no one.
jayd16 6y ago
It wasn't that little at all. Eyeballing the 538 graph it was about a 38% chance of Republicans gaining in the house? Over a one in three chance. https://projects.fivethirtyeight.com/2020-election-forecast/house/ https://projects.fivethirtyeight.com/2020-election-forecast/...
- fullshark 6y ago
  Hmm looks like 548 gave a 3% chance of R control and betting markets gave about a 15% chance on election night: https://www.predictit.org/markets/detail/4365/Which-party-will-control-the-House-after-2020-election https://www.predictit.org/markets/detail/4365/Which-party-wi... You can also try to work the probability distribution from this betting market and I'm sure they were more favorable to R than he was: https://www.predictit.org/markets/detail/6669/How-many-House-seats-will-Democrats-win-in-the-2020-election https://www.predictit.org/markets/detail/6669/How-many-House... That's a pretty big discrepancy regardless, did he ever give an explanation as to why he disagreed with betting markets so much? Seems like betting markets had a ~ one standard deviation pro Trump/R polling error baked into their pricing.
  - jayd16 6y ago
    >why he disagreed with betting markets Because betting markets are kind of dumb? Nate Silver has uttered those words a few times, anyway.
    - fullshark 6y ago
      They have a pretty long track record of being predictive.
      - jayd16 6y ago
        Not in this case, clearly.
        fullshark 6y ago
        Actually i think the number of seats gained by republicans will be pretty close to the P50 of the betting market probability distribution. They seem to have done better than 538 at least there. It's impossible to verify the overall Biden/Trump victory probability mass function but you could look at state by state races and see who did better.

Fuzzwah 6y ago

As an outsider all I can think is that this is an issue with the US media. The easy story is talking about the latest poll. The more difficult, but actually useful story, is focusing on the policies being put forward by each side.

535188B17C93743 6y ago

I think the big issue was the "margin of error" in many of these cases. For example, when Quinnipiac did a poll a few days before the election [1], they said Biden was winning Ohio by 4 points with a +-2.5 point margin of error... which happened to be very far off. And this wasn't isolated-–if memory serves me, Wisconsin was even further off? [1] https://poll.qu.edu/florida/release-detail?ReleaseID=3683 https://poll.qu.edu/florida/release-detail?ReleaseID=3683

jonhohle 6y ago

But the polls then and now weren’t just bad, they were shockingly bad and not representative of any real population that matters in an election. I’d expect something like this from junior engineers looking at metrics for the first time, but these are supposedly the most respective organizations in their field that have failed spectacularly for the second presidential election in a row. What value do they possibly serve? Should Biden ultimately be declared the victor, I’m concerned that the public will never receive the postmortem on what went wrong and how it happened again that it deserves.

evgen 6y ago
I really have no idea where this idea that the polls failed comes from. There were only two bad calls this cycle and every other outcome was within the margin of error. It was pretty much the same case in 2016 where people who had no idea what they were talking about suddenly decided they were certain polling failed because they are unable to grasp the concept of margin of error and sample size. Polls can only guess about turnout and the try to work backwards from there. The turnout estimates were wrong but not shockingly so, and as a consequence a lot of polls ended up having the result be at the far end of their margin of error. Nothing went wrong. Polling is hard. Get over this idea that you can have some sort of certainty regarding an election until we actually hold the election.
- js2 6y ago
  I know they were within the margin of error, but the polls were all systematically within the margin of error in the same direction. Did a single poll that was wrong predict at GOP win and end up a Democratic win?
  - srtjstjsj 6y ago
    Yes.
    - js2 6y ago
      I can’t find that via Google. Help me out here.

cratermoon 6y ago

Couldn't the problem be that polling is just numeromancy dressed up in fancy math?

ironSkillet 6y ago
The problem that has appeared again is that without a representative sample of information gathered, it doesn't matter how fancy your models are, you will not have accurate results. And quantifying how unrepresentative your sample is basically impossible to figure out in this situation.
- cratermoon 6y ago
  > quantifying how unrepresentative your sample is basically impossible Yeah, so... numeromancy.
- saberdancer 6y ago
  Maybe they should poll about what the person will vote, but rather what the persons opinions are in general and then estimate based on that who will they vote for. I am betting Facebook or Google would be able to better calculate the outcome than any polling firm. Bring in all the data they collect on each internet user, I am sure they can model the likelihood of voting and where the user is leaning.

will4274 6y ago

I'm on the nerd side too, but I was quite surprised. Sampling isn't the only source of error in election polling - people also change their minds. In this election, people decided who they were going to vote for historically early - the lowest portion of people in the history of exit polls indicated they decided in the week before the election. In theory, this effect should reduce the width of the probability distribution, and yet, the polls were off again, in the same direction as they are always off, by a little bit more than average. A little scrutiny here is worthwhile. In 2016, for example, the polls were off in part because pollsters weighted the black demographic in proportion to their voting behavior in 2008 and 2012 (when a black candidate was running, boosting turnout). The actual 2016 black turnout looked more similar to 2004 or 2000 - other elections where a black candidate wasn't running. Though I haven't dived into the 2020 polling data, it wouldn't surprise me if a similar effect were at play. It's also become noticeable harder to do polling. Once upon a time, everybody had a landline, so random dialing worked really well. Now, some people have more phones than others, and people are increasingly less willing to talk to pollsters, increasing the error rate. These are all important things to talk about - when you dismiss it as "oh yeah, polls are never perfect," you prematurely shut down the conversation. Don't let perfect be the enemy of the good. The polls could be better and it's important to talk about how.

saalweachter 6y ago
There is also "what polls directly measure" and "what we infer from them". Besides the usual adjustments for likely voter models, we also do things like infer how people will vote in House and Senate elections based on Presidential polling, because we do a lot more Presidential polling than down ballot, because money. So one early thing that seems to have happened this year (and again, we are still early in the process to be making final verdicts in "what happened" retrospectives) is that people split their ballot more than anticipated, which means that the inferences we were making about down ballot races based on Presidential performance were off. So not quite a polling error, but an example of uncertainty that may not have been correctly taken into account in forecasting.
A4ET8a8uTh0 6y ago
<< These are all important things to talk about - when you dismiss it as "oh yeah, polls are never perfect," you prematurely shut down the conversation. Don't let perfect be the enemy of the good. The polls could be better and it's important to talk about how. I think the complaint is that here the polls were not just imperfect. They were at best, misleading raising questions over methodology ( lessons, apparently, not learned in 2016 ), wishful thinking, and their usefulness, or, at worst, attempts at influencing desired income. I can absolutely agree that polling is not an exact science, but now it is twice in a row that polling science has grossly miscalculated the mood of the nation.

js2 6y ago

I really want to know what went wrong with the polls (again). Shy Trump voter doesn’t explain how wrong the polls were about Graham or Collins, outside of even the margin of error. Accurate polls are important to running campaigns and to legislative positions. Selzer got her polls right in Iowa and she posited that she knew the local electorate better than other pollsters so she weighted correctly. Maybe Trafalgar Group has a better method or maybe it just weights GOP voters more heavily (we don’t know, it won’t disclose its methods). Or maybe Trump just breaks polling and the polls will be fine next time. Whatever the reason, I’d like to know, but I’m not sure we ever will.

atoav 6y ago
I grew up in a right wing province of Austria and saw that sort of wrong polling throughout my youth — which is why I wasn't surprised the slightest about things going as they did. It boils down to: a fraction of the electorate will vote for $bigotcandidate but would never ever admit it to anyone outside the voting booth, because they know it is wrong. They will tell people they will vote for $notbigotcandidate, but vote for him nonetheless. This fraction is not insignificant and you cannot easily represent it in polls without guessing the effect. In my home province it eas consistently around 10 to 15% over a decade.
- js2 6y ago
  I don’t see how that explains the Collins/Gideon race, but I’m not a Mainer. Anyone here from Maine can chime in why a Mainer would lie about voting for Gideon?
  - gkop 6y ago
    Bigots turned out. And lean Collins.
  - fma 6y ago
    My same question. OK you can be claim to be a "good Republican" and vote Biden but (R) down ballot. Why were Senate and House races also wrong? Why did Lindsey Graham win by so much when the polls didn't give him that lead? In South Carolina, it's ok to say you'll vote for Graham.

dorkwood 6y ago

What you're saying is that any result with a non-zero chance of happening should be unsurprising to us, because it was still in the realm of possibility? If that's the case, what use are polls? What makes an election forecast based on polls more valuable than a coin flip?

softwaredoug 6y ago
Your coin flip analogy is spot on. But instead of a 1 in 2 probability, Trump had a 1 in 10 probability of winning. We knew this probability precisely because of polling data (and historical error). This wasn’t some rare weird outcome. This was probably a standard deviation or so closer to Trump in a forecast. Perfectly normal thing to happen.
- ghayes 6y ago
  Sure, the model had value precisely because it said that even if Trump did better than the polls suggested, he would likely still lose due to the paths of victory Biden had. That said, the polls have shown, in many cases, poor predictive behavior of actual votes, and as such, we should not just brush this off by quoting the distribution of outcomes.
jayd16 6y ago
Results fall on a spectrum from surprising to unsurprising based on the modeled prediction and confidence.
pessimizer 6y ago
Polls are astrological race science, and the reason you'll never be able to convince anyone of this is adequately covered by the book When Prophecy Fails. I've come to think that their bias is only slightly ideological, but instead springs more from a self-serving impulse to mirror the current beliefs of the audience most likely to believe in polls i.e. people overly impressed by credentials. They enjoy the sheer volume of numbers, rather than their actual usefulness. Alternate phrasings of identical questions asked in polls creates differences in outcome vastly larger than the effects they claim to detect. Somehow Nate Silver managed to get rich by selling synthetic CDOs of arbitrary polls (to people with his identical political outlook.) The only real poll is to ask the same questions, in the same way, as the ballot sheet does. Ideally these should be asked as people are entering or leaving the voting booth, and people should be paid for their responses, in order not to bias yourself towards bored loudmouths with nothing to do. This would be good for accuracy, not for usefulness in anything but generating a suspicion of election fraud. Instead, they're trying to figure out what percentage of black people go to the polls in Missouri, and pretending that the self-selected black voter has more in common politically with the general black phone answerer than the Missouri voter. Relying on that arbitrary assumption is the entire basis of their field. edit: I haven't read https://site.pennpress.org/aha-2021/9780812250046/race-and-the-making-of-american-political-science/ https://site.pennpress.org/aha-2021/9780812250046/race-and-t... yet, but I heard an interview with the author around the time that it came out, and it's another hidden history of a field that has completely unearned legitimacy. "Political Science" might largely be considered an outgrowth of scientific racism.

phlakaton 6y ago

I think it's more fundamental than numeracy. If the polls suggest there is a 2/3 probability you are going to win an election, but you lose, does that mean the polls were wrong? What if the probability was 9/10? I think the problem is trying to make sense of outliers and uncertainty, and that, in an election with 150M+ variables, you will end up with a result that isn't quite what anyone predicted. In other words: it's not numeracy, it's epistemology.

elpakal 6y ago
Great comment, agreed. The wonderful books from Nassim Taleb ‘Black Swan’ & ‘Fooled by Randomness’ touch on this in a way that forever changed my thinking. ‘We just can’t predict’.
srtjstjsj 6y ago
Fivethirtyeight's polling averaging model underestimated Trump in all the swing states. Trump wine 2-3 states predicted for Biden, and 0 vice versa. States that Biden had a supposedly strong lead in turned out to be statistical ties in the official vote. The direction bias in the polls (about 20 of 25) is obvious, it's not symmetric random error.
- phlakaton 6y ago
  To which you then have to ask: 1) were the discrepancies within the margin of error? and 2) to what extent are you justified in critiquing the polls based on a sample size of one? I understand that the voting percentage we saw nationally was within the margin of error that 538 predicted, and that the specific configuration of electoral votes we saw was also one of the outcomes considered by the models.
- TheCoelacanth 6y ago
  Modeling correlated polling errors is basically what 538's models are all about. They model how far off the polls could end up being from the actual result based on past elections. The polls are never 100% accurate. The pollsters have to estimate how many of each type of voter will actually show up to vote and what the response rate for each type of voter was. It is not an exact science.

meroes 6y ago

You could also fit the same data with "people polled thought they could vote until their vote was preferentially suppressed". So the polls will never match outcomes. And that it happened in the same direction two cycles in a row leads credence. (I do not know why the forecasts are wrong, but I don't see why Parent is at all justified in his statement).

listenallyall 6y ago

if you're a data nerd, how can you NOT be shocked by so many results falling far outside the confidence interval (often multiple sigmas), and all in the same direction?

deleted 6y ago
[deleted]
atoav 6y ago
Well this measures how many people are willing to admit they would vote for Trump which is not surprisingly a lower number than people who will vote for him in an unobserved secret vote. Even if the pollsters are the most neutral nonjudgmental people on earth there will still be a fraction of the electorate that will (irrationally) shy away from telling them the truth. This effect is certainly real.
- listenallyall 6y ago
  Ummm, absolutely not. Pollsters universally stated they either had "adjusted their models" after being embarrassed in 2016, or that the "hidden Trump voter" was a myth (usually combined with "Trump fans won't shut up about how much they support him"). Poll results were repeatedly presented as forecasts of actual election results.
  - atoav 6y ago
    I grew up in a right wing Austrian province which over a decade consistently polled 10 to 15% wrong (in favour of the right wing candidate). I know those people in person. They are people who will tell even their spouse they vote the left wing candidate, but after the 6th beer they will tell you how they really want the strong guy in charge (note: Hitler is from Austria, voting the rightwing guy can still be a taboo for certain people). So these people do exist. If it is just them who make up the difference — no idea, but they exist.
    - listenallyall 6y ago

Vaslo 6y ago

Everytime I read a comment like this I think of that line from Dumb and Dumber: “What are the chances of a guy like you and a girl like me... ending up together?” “Not good” “Not good like one in a hundred?” “I'd say more like one in a million.” “So you're you’re saying there's a chance!” I know enough about probability distributions and modeling to know you can improve your models to lower that uncertainty. And sorry, you are the Lloyd Christmas in this one.

beervirus 6y ago

Statistics can be pretty unintuitive. On top of that, most people have never learned any statistics.

jayd16 6y ago

Whats even more frustrating is recognizing the population difference in day of voters and mail in votes, expecting a "blue shift" as votes are counted, and then watching pundits chime in on Tuesday about surprising results. As far as I can tell the most surprising result was Trump's popularity among Cuban Floridians.

evgen 6y ago
Once Biden's team let Trump paint him as socialist I was not surprised to see the Cubanos go Trump (or the Venezuelans for the same reason) but I will be interested to see some of the breakdowns of the Puerto Rican vote in FL and some of the Central American Latino vote in Texas; I am betting there is going to be a reasonably large gender difference in some of those groups.
srtjstjsj 6y ago
Cubans are anti-Castro Republican. The young generation of pollsters forgot that Cubans aren't just "generic Hispanic".

tsimionescu 6y ago

You're picking the easiest number to misread - the overall "chance of being elected", which doesn't really mean anything. Much more interesting are the direct results predictions, especially per state where they matter most. Those are polls that give a particular result with a +- confidence interval. If the polls were to any extent competently done, you expect the result to be well within that margin of error. The reality though has been that the results have been well outside the margin of error, sometimes multiple sigmas from the actual result - that is simply unacceptable for a poll. The only conclusion should be that polling data can be completely ignored, as it doesn't seem to have any clear relationship with the final results.

evgen 6y ago
Individual polls were sometimes outside the margin, but collectively the polls were not far off the mark and collectively they were within the margin of error. This is why sites like 538 were able to model the various outcomes with some degree of certainty and ONCE AGAIN manage to get results within the range of expected probabilities. You really seem to have a hard time understanding both polling and probability and in particular the relationship between sample size and error. I suggest a bit more research.
- disgruntledphd2 6y ago
  Dude, I really don't get why you're being so gung ho on this. The errors on the individual polls were way out of line with the actual results. When we regularise these results like 538 or the Economist did, we get slightly better results, but they were still not particularly accurate. This is a real problem, and not one that we should be ignoring. I gave the polling companies a pass after 2016, but that this happened again is extremely concerning. > This is why sites like 538 were able to model the various outcomes with some degree of certainty and ONCE AGAIN manage to get results within the range of expected probabilities. But they were super close to the edges of the intervals. Like the most expected number of electoral votes for Biden was 380-400, which is not what happened. > You really seem to have a hard time understanding both polling and probability and in particular the relationship between sample size and error. I suggest a bit more research. Please don't be dismissive of other people. Many people (including people who do stuff like this every day) were very surprised by the results. We definitely shouldn't tar and feather the pollsters, but I think it would be great if we could get an understanding of why the polls appear to have been systematically biased.

wesnerm2 6y ago

To be fair, we still don't know the loss of votes through mail-in voting. Mail-in voting is more convenient which would likely increase participation, but missing signatures, naked ballots, late or undelivered mail, etc can prevent those ballots from being counted. When there is a major difference in voting patterns between the two political parties, then polls may not actually be inaccurate. In this case, Democrats chose mail-in voting and Republicans, by and large, chose in-person after being directed by the president. Also, some percentage of mail-in ballots may not have reached their destination because of possible election interference through reduced post office efficiency--in particular, fewer mail sorting machines, lower hours, deprioritized mails, etc. In short, the polls don't take into account election interference in the post office or unsuccessful delivery. If most people from both parties were voting in-person, maybe we might have had more parity between the polls and election results.

einpoklum 6y ago

> Forecasts give us probability distributions based on historical polling error data. ... and a lot of assumptions about the supposed underlying probability spaces.

ehejsbbejsk 6y ago

I am pretty certain Facebook has the data to crunch very accurate predictions in elections large and small.

fullshark 6y ago
Based on what? I think part of the problem with our politics these days is confusing social media engagement / metrics (specifically twitter) with national sentiment. Seems like there's a large mass of unobserved "silent" Americans out there.

deleted 6y ago

[deleted]

tibiapejagala 6y ago

I don't get why people care so much about predictions vs actual results in case of presidential elections in the US. The rules look like they were designed for maximum entertainment and sudden changes. The fact that you can win with less votes. When the other person already has 250 electors, but then on the last second you flip Florida or Pennsylvania with a few thousand votes and bam! you are the president! That's why people love watching your elections so much. Or maybe also because of the fallout in their countries.

deleted 6y ago

[deleted]

bendmorris 6y ago

>Forecasts give us probability distributions based on historical polling error data. They would give you a probability distribution if they were truly random samples. But in reality, response rates are very low, and pollsters make up for this using demographic weighting and modeling - so the results don't actually represent any real population, but rather are extrapolated from people who respond onto a hypothetical population that the pollster thinks is likely to vote. This is why they are spectacularly and systematically off, more and more over time. It's not just variance.