13 ms·

I really have no idea where this idea that the polls failed comes from. There were only two bad calls this cycle and every other outcome was within the margin

by evgen 6y ago

I really have no idea where this idea that the polls failed comes from. There were only two bad calls this cycle and every other outcome was within the margin of error. It was pretty much the same case in 2016 where people who had no idea what they were talking about suddenly decided they were certain polling failed because they are unable to grasp the concept of margin of error and sample size. Polls can only guess about turnout and the try to work backwards from there. The turnout estimates were wrong but not shockingly so, and as a consequence a lot of polls ended up having the result be at the far end of their margin of error. Nothing went wrong. Polling is hard. Get over this idea that you can have some sort of certainty regarding an election until we actually hold the election.

js2 6y ago
I know they were within the margin of error, but the polls were all systematically within the margin of error in the same direction. Did a single poll that was wrong predict at GOP win and end up a Democratic win?
- srtjstjsj 6y ago
  Yes.
  - js2 6y ago
    I can’t find that via Google. Help me out here.
    - randallsquared 6y ago
      "poll predicted Trump win" brings back a number of results for me, including https://www.express.co.uk/news/world/1354506/us-election-2020-donald-trump-win-poll-democracy-institute-joe-biden-swing-states https://www.express.co.uk/news/world/1354506/us-election-202...
      - js2 6y ago
        Ah sorry, I was thinking of down ballot races and polls that would have had enough reputation to make it into the Economist or 538 polling averages. I'll do some more searching. The DI poll is basically a push poll with questions like "Who do you believe is telling the truth about alleged Biden family corruption?” https://democracyinstitute.org/poll-donald-trump-set-to-win-us-presidency-by-electoral-college-landslide/ https://democracyinstitute.org/poll-donald-trump-set-to-win-...
- TheCoelacanth 6y ago
  That has happened in basically every election. Errors in polls tend to be correlated across states. If they were independent, 538 would have been predicting a ~0% chance of Trump winning instead of ~10%.
mixologic 6y ago
If you look at RCP's polling average for Clinton in 2016, she outperformed that average for just about every race. But, what the pollsters got crazy wrong, was Trumps polling numbers, by wide margins, far outside the margin of error. I think they got it wrong this time too, and I think it all comes down to their methodology for actually getting a random sample of voters. Many polls are still married to live interviews and also to landline contact, and also tied to live interview polling - I think there is a partisan slant that they are not accounting for that includes a "propensity to answer a polling survey in the first place"
- function_seven 6y ago
  > I think there is a partisan slant that they are not accounting for that includes a "propensity to answer a polling survey in the first place" They're well aware of that slant, and try their damndest to correct for it. But apparently they just don't have a good model of how large or how volatile that propensity attribute is, and how it relates to political leanings. I was polled this year. One of the first questions I was asked is if I was answering the call on a landline or on a cell phone. They weight those differently. Also of course my basic demographic info (wage, race, marital status, education level, home ownership status, etc.) I think your whole comment is correct, but instead of not accounting for the propensity factor, they do... just badly :) I have no idea what the solution is to this. Maybe there isn't one, and instead we can get a silver-lining effect: If people trust the results of polls less, they're hopefully less likely to think the results are preordained. More voter turnout?
  - scythe 6y ago
    >I think your whole comment is correct, but instead of not accounting for the propensity factor, they do... just badly :) In particular, Trump was extremely critical of polls and mail-in voting, and we have pretty much confirmed that his statements caused his supporters to avoid mail-in voting, so it's not too much of a leap to suggest the same thing happened with polls. Notably, this is a different effect from 2016, where IIRC Trump appealed to "non-likely" voters whose responses were inappropriately discounted.
  - 6y ago
nefitty 6y ago
I think the main problem is that some forecasts gave an overwhelming amount of “hope” for certain races. This costs real money, as political operatives make decisions on where to spend money based on these forecasts. The closer a race is, the more likely more money will help shift the race.
- evgen 6y ago
  Unfortunately you cannot just wish away the cause-effect here. Did Democrats piss away a hundred million in South Carolina on a race Jamie Harrison was never going to win, or did Jamie Harrison get within striking distance because of that money and was the investment in party infrastructure and training of lower-level party coordinators going to pay off over the next decade? Texas did not turn blue, but the effort spent on Beto's race and now on MJ Hagar may build down-ticket strength and knowledgeable people who may end up turning the state in a few more cycles. When I grew up California and Virginia were red states, and I am sure people then also said money spent there was not going to move the needle and was a waste. When you have a billion+ to spend on races sometimes a little hope now pays off in a decade or two even if you lose this race.
  - srtjstjsj 6y ago
    California hasn't had 2 Republican Senators at the same time since 1971. California became Democrat when the Republican Party realigned to the Southern Strategy, not because of campaigning.
    - evgen 6y ago
      Prior to 1992 the only time California had two Democratic senators was post-Watergate and then prior to that single term for Tunney you have to go all the way back to 1863 to get two Dem senators. California had frequently elected Republican governors (including one who went on to become president), had house delegations that were majority Republican for stretches of time, and except for Johnson's win in 1964 CA was a solid batch of electors for the Republican presidential candidate up until Clinton. CA was a Republican state from the late 19th century up until the 1990s.
qppo 6y ago
One could argue the strategic mistakes of the Clinton campaign (ignoring the rust belt) and near fatal mistakes of the Biden campaign (which looks to be on issues swinging voters in battleground states) were driven by bad polling. When I hear the conversation about "the polls were wrong" it's never been "the predictions were outside the error margin" so much as "voter sentiment was not accurately measured in regions where voters were most likely to flip and why they would do so."
KZeillmann 6y ago
My issue with polling is that many races were well beyond the margin of error. The Senate polling in particular was bad this year. Sara Gideon was favored to win the Maine race in the polling, because there hadn't been a single poll showing Collins in the lead since July. She lost her race by 9 points. It's also strange to see the region makes a difference in the poll error. The polls in Minnesota were basically spot-on, but in Wisconsin (demographically very similar), the polling average was Biden +8, with one ABC news poll showing him +17, the kind of outlier result you'd expect with a +8 average. He's gonna win there by ~1 percentage point. There's something wrong with how a lot of these pollsters determine samples, or how they judge someone's likeliness to vote.
- bra-ket 6y ago
  it's not just polls that are wrong but likely this premature projection as well, tens of thousands of provisioned and military ballots are still not counted. Recounts are still to happen in most swing states. Lawsuits are pending. I wouldn't be surprised Trump actually wins.
  - evgen 6y ago
    Everyone would be surprised if Trump wins, because at this point it is so mathematically improbable that even gun-shy decision desks at major news networks are able to make the call without fearing they will look like idiots. Recounts have never moved a state election more than a thousand votes that I am aware of and Trump is behind by tens of thousands in the states that matter. Lawsuits won't do anything, because no on has standing to prevent someone else from legitimately voting according to the rules in place at the time of election, nor of preventing someone else from counting those votes as legally mandated. Trump lost, get over it.
    - paulddraper 6y ago
      > Lawsuits won't do anything Lawsuits could definitely do something. If there is fraud or misconduct, that can matter. Hell, the US spent two years investigating some Facebook ads and hackers. But right now there's no clear indication that will in fact happen.
scrooched_moose 6y ago
If you dig into the the margins they aren't looking very good, even if they got a fair number of eventual winners correct. Predicted vs Actual (FiveThirtyEight's averages vs NYTimes' current tally; Biden's margin is positive) PA: +4.7% vs +.5% (-4.2%) FL: +2.5 vs -3.4 (-5.9) TX: -1.5 vs -5.9 (-4.4) OH: -0.6 vs -8.1 (-7.5) PA: +4.7 vs +0.5 (-4.2) IA: -1.5 vs -8.2 (-6.7) NC: +1.7 vs -1.4 (-3.1) WI: +8.3 vs +0.6 (-7.7) GA: +0.9 vs +0.1 (-.8) MI: +8.0 vs +2.6 (-5.4) AZ: +2.6 vs +0.6 (-2.0) NV: +6.2 vs +2.0 (-4.2) They pretty consistently overpredicted Biden's margin by about 4-7% in almost all of the swing states. Even if that's within, or close to, the margin of error; there's a systemic issue if it's happening in almost every state. Edit: Added AZ and NV; and the difference
- paulddraper 6y ago
  ^ Exactly. The polls were at least as wrong in 2016, even if that error was less significant this time (because the polled margin started out bigger). Pollsters and poll-watchers downplaying this is not a good sign for future improvement. EDIT: Overall though, I'm not sure that's a loss. People pay way more attention to polls than they should anyway.
- OskarS 6y ago
  This is such a damning list. At some point, reporting on polls when they have this bad of a track-record serves no journalistic purpose and just confuses the public. Like, the discussions people were having in the days before the election about Biden's strength barely resembles reality. I respect FiveThirtyEight and their work, and I do think they're intellectually honest and generally speaking good at their jobs. But they can only be as good as their sources, and when the sources are this terrible, they shouldn't be reported on this, and not given the statistical and scientific sheen of authenticity 538 gives them. Like, that site is starting to have a net negative influence on the world, and when that is the case with a journalistic institution, what are you even doing?
  - paulddraper 6y ago
    > I respect FiveThirtyEight and their work, and I do think they're intellectually honest and generally speaking good at their jobs. But 538 chooses how to weight the polls. For example, they only gave Rasmussen (which was far closer on these) a C rating, preferring less accurate polls. FWIW, their articles clearly lean left [1], IDK if that affects their analysis/forecasts. [1] https://www.adfontesmedia.com/interactive-media-bias-chart-2/ https://www.adfontesmedia.com/interactive-media-bias-chart-2... similar to WaPo
tsimionescu 6y ago
The polls showed that Biden would win Florida by 2.2%, and in fact he lost by 3.3%. Is 5% within the normal error margin? There is no reason that polls would be significantly different from the final vote results in most circumstances, if they are competently run. https://projects.fivethirtyeight.com/polls/florida/ https://projects.fivethirtyeight.com/polls/florida/
- srtjstjsj 6y ago
  What does "competent" mean if none is competent? Are engineers incompetent because they can't put a person on Mars?
  - tsimionescu 6y ago
    Sure, if some engineers told you "we are putting this person on Mars with a 99.9 probability", and the rocket in fact reaches Venus, the engineers are incompetent. Also, polling in most European countries doesn't go this far wrong, on almost all polls. So it's not like "accurate polling is impossible", it's just the polls in the US that seem to have some problem (though there have been some spectacular failures in other countries as well, which itself may lead to some thoughts on the value of polls in general).
    - evgen 6y ago
      > polling in most European countries doesn't go this far wrong, on almost all polls. Brexit and UK elections would like to have a word with you... Polling everywhere sucks. Pick a country and chances are it either has no polling to speak of or else it has the same problems.
      - disgruntledphd2 6y ago
        YouGov are pretty good, mostly because they grew up in the Internet Age. But I agree with your general point.
- gpm 6y ago
ktta 6y ago
If the error skews in one specific direction for all the polls, then I don't think you can just excuse the polls by saying they're in the margin of error. This is some kind of non-trivial process problem.
sampo 6y ago
> I really have no idea where this idea that the polls failed comes from. For one, Andrew Gelman (professor of statistics and political science at Columbia University). Who knows more about the topic than anyone here in this HN discussion. https://statmodeling.stat.columbia.edu/2020/11/04/dont-kid-yourself-the-polls-messed-up-and-that-would-be-the-case-even-wed-forecasted-biden-losing-florida-and-only-barely-winning-the-electoral-college/ https://statmodeling.stat.columbia.edu/2020/11/04/dont-kid-y...
- evgen 6y ago
  He may know a lot about the topic, but has little experience in actually modeling elections. He also has skin in the game and an incentive to make sure everyone knows that his miss was completely not his fault and was in fact due to failures by someone else, someone over there who you don't know, yes, she lives in Canada...
  - throwawaylolx 6y ago
    So what statistical mistakes did this lead to in his analysis?
matt_kantor 6y ago
People in this thread might be interested in this podcast episode: https://www.iheart.com/podcast/105-stuff-you-should-know-26940277/episode/how-election-polling-works-and-doesnt-71597551/ https://www.iheart.com/podcast/105-stuff-you-should-know-269... They discuss a lot of the things brought up here and go into details about the actual mechanics of calling people, etc. It's an easy listen.
dillondoyle 6y ago
It was very very bad up and down the ballot. No shit one of our clients their poll was - i'm rounding here because this is my real name - +15%. They barely won <1%. I wish pollsters would move towards very very large digital samples. You can't get a 15 minute survey but head-to-head you can get a huge sample for not a lot of money. And don't try to weight it to what you THINK the turnout will be. Break out likely voter or not. Do not add any inference or opinion or 'math'
- evgen 6y ago
  All digital surveys skew heavily to a younger demographic and heavily male. You have basically captured the political opinions of the fraction of the potential electorate least likely to vote. How exactly is that worthwhile?
  - dillondoyle 6y ago
    I don't buy that, and even if I did the flip would be worse if you look phone contact rates. I've seen some good very large digital surveys. We do some - geared towards measuring ad recall/lift - and I find them valuable, especially a giant sample on only two recall + head to head questions You can always ask for gender/age and weight it but that's part of what I see as the problem. so many 'traditional' surveys i see make fairly large adjustments, lots of 'looking back' at past turnout and over fitting based on personal bias. Especially when weighting from very small sub-sample xtab e.g. hispanics. im not a pollster and that's just my still fairly-insider / polling adjacent insight Another problem with digital surveys is most of the firms do opt-in panels, like having people register to take surveys and get paid on mechanical turk. and then they weight from there. I think this is a problem. The surveys we do run inside of mobile ads, similar to Google / FB Brand Lift surveys they go after those who saw the ads or a truly random sample, not a small biased group of survey panel
titzer 6y ago
Slapping +/- 5% on top of a prediction that is systematically off by ~5% isn't "in the margin of error", it's just simply off. For one prediction, two predictions, no problem. They fall somewhere in this range with probability distribution (usually a Bell curve around the mean). But for hundreds, if not thousands, of polls to be so systematically off, it is not "within the margin of error" or a statistical fluke. If it was, you would expect a normal distribution around the mean of the polls. The mean was completely off. Put simply, it was sampling bias. Pollsters screwed up. They either sampled the wrong voters, the wrong areas, mispredicted who would turn out, or voters did not accurately report who they planned to vote for. Garbage in, garbage out. You can try to correct for sampling bias, but if you are blind to it, as your comment pushes more people to be, then you will fail.
- ehsankia 6y ago
  Right, but the job of 538 is exactly to deal with that, and they do take into account the fact that errors are correlated, so if one poll is off, chances are they all are. That's why their predictions are fairly conservative. They gave Florida a 1/3 chance for Trump and 1/10 for the presidency. Both those numbers make sense. Now you can argue that there's little utility if the error bars are so big, and you may have a point. The bigger issue here is that modern elections have been extremely close, and unfortunately we cannot get polling accurate enough.

by evgen 6y ago

js2 6y ago

I know they were within the margin of error, but the polls were all systematically within the margin of error in the same direction. Did a single poll that was wrong predict at GOP win and end up a Democratic win?

srtjstjsj 6y ago
Yes.
- js2 6y ago
  I can’t find that via Google. Help me out here.
  - randallsquared 6y ago
    "poll predicted Trump win" brings back a number of results for me, including https://www.express.co.uk/news/world/1354506/us-election-2020-donald-trump-win-poll-democracy-institute-joe-biden-swing-states https://www.express.co.uk/news/world/1354506/us-election-202...
    - js2 6y ago
      Ah sorry, I was thinking of down ballot races and polls that would have had enough reputation to make it into the Economist or 538 polling averages. I'll do some more searching. The DI poll is basically a push poll with questions like "Who do you believe is telling the truth about alleged Biden family corruption?” https://democracyinstitute.org/poll-donald-trump-set-to-win-us-presidency-by-electoral-college-landslide/ https://democracyinstitute.org/poll-donald-trump-set-to-win-...
TheCoelacanth 6y ago
That has happened in basically every election. Errors in polls tend to be correlated across states. If they were independent, 538 would have been predicting a ~0% chance of Trump winning instead of ~10%.

mixologic 6y ago

If you look at RCP's polling average for Clinton in 2016, she outperformed that average for just about every race. But, what the pollsters got crazy wrong, was Trumps polling numbers, by wide margins, far outside the margin of error. I think they got it wrong this time too, and I think it all comes down to their methodology for actually getting a random sample of voters. Many polls are still married to live interviews and also to landline contact, and also tied to live interview polling - I think there is a partisan slant that they are not accounting for that includes a "propensity to answer a polling survey in the first place"

function_seven 6y ago
> I think there is a partisan slant that they are not accounting for that includes a "propensity to answer a polling survey in the first place" They're well aware of that slant, and try their damndest to correct for it. But apparently they just don't have a good model of how large or how volatile that propensity attribute is, and how it relates to political leanings. I was polled this year. One of the first questions I was asked is if I was answering the call on a landline or on a cell phone. They weight those differently. Also of course my basic demographic info (wage, race, marital status, education level, home ownership status, etc.) I think your whole comment is correct, but instead of not accounting for the propensity factor, they do... just badly :) I have no idea what the solution is to this. Maybe there isn't one, and instead we can get a silver-lining effect: If people trust the results of polls less, they're hopefully less likely to think the results are preordained. More voter turnout?
- scythe 6y ago
  >I think your whole comment is correct, but instead of not accounting for the propensity factor, they do... just badly :) In particular, Trump was extremely critical of polls and mail-in voting, and we have pretty much confirmed that his statements caused his supporters to avoid mail-in voting, so it's not too much of a leap to suggest the same thing happened with polls. Notably, this is a different effect from 2016, where IIRC Trump appealed to "non-likely" voters whose responses were inappropriately discounted.
- 6y ago

nefitty 6y ago

I think the main problem is that some forecasts gave an overwhelming amount of “hope” for certain races. This costs real money, as political operatives make decisions on where to spend money based on these forecasts. The closer a race is, the more likely more money will help shift the race.

evgen 6y ago
Unfortunately you cannot just wish away the cause-effect here. Did Democrats piss away a hundred million in South Carolina on a race Jamie Harrison was never going to win, or did Jamie Harrison get within striking distance because of that money and was the investment in party infrastructure and training of lower-level party coordinators going to pay off over the next decade? Texas did not turn blue, but the effort spent on Beto's race and now on MJ Hagar may build down-ticket strength and knowledgeable people who may end up turning the state in a few more cycles. When I grew up California and Virginia were red states, and I am sure people then also said money spent there was not going to move the needle and was a waste. When you have a billion+ to spend on races sometimes a little hope now pays off in a decade or two even if you lose this race.
- srtjstjsj 6y ago
  California hasn't had 2 Republican Senators at the same time since 1971. California became Democrat when the Republican Party realigned to the Southern Strategy, not because of campaigning.
  - evgen 6y ago
    Prior to 1992 the only time California had two Democratic senators was post-Watergate and then prior to that single term for Tunney you have to go all the way back to 1863 to get two Dem senators. California had frequently elected Republican governors (including one who went on to become president), had house delegations that were majority Republican for stretches of time, and except for Johnson's win in 1964 CA was a solid batch of electors for the Republican presidential candidate up until Clinton. CA was a Republican state from the late 19th century up until the 1990s.

qppo 6y ago

One could argue the strategic mistakes of the Clinton campaign (ignoring the rust belt) and near fatal mistakes of the Biden campaign (which looks to be on issues swinging voters in battleground states) were driven by bad polling. When I hear the conversation about "the polls were wrong" it's never been "the predictions were outside the error margin" so much as "voter sentiment was not accurately measured in regions where voters were most likely to flip and why they would do so."

KZeillmann 6y ago

My issue with polling is that many races were well beyond the margin of error. The Senate polling in particular was bad this year. Sara Gideon was favored to win the Maine race in the polling, because there hadn't been a single poll showing Collins in the lead since July. She lost her race by 9 points. It's also strange to see the region makes a difference in the poll error. The polls in Minnesota were basically spot-on, but in Wisconsin (demographically very similar), the polling average was Biden +8, with one ABC news poll showing him +17, the kind of outlier result you'd expect with a +8 average. He's gonna win there by ~1 percentage point. There's something wrong with how a lot of these pollsters determine samples, or how they judge someone's likeliness to vote.

bra-ket 6y ago
it's not just polls that are wrong but likely this premature projection as well, tens of thousands of provisioned and military ballots are still not counted. Recounts are still to happen in most swing states. Lawsuits are pending. I wouldn't be surprised Trump actually wins.
- evgen 6y ago
  Everyone would be surprised if Trump wins, because at this point it is so mathematically improbable that even gun-shy decision desks at major news networks are able to make the call without fearing they will look like idiots. Recounts have never moved a state election more than a thousand votes that I am aware of and Trump is behind by tens of thousands in the states that matter. Lawsuits won't do anything, because no on has standing to prevent someone else from legitimately voting according to the rules in place at the time of election, nor of preventing someone else from counting those votes as legally mandated. Trump lost, get over it.
  - paulddraper 6y ago
    > Lawsuits won't do anything Lawsuits could definitely do something. If there is fraud or misconduct, that can matter. Hell, the US spent two years investigating some Facebook ads and hackers. But right now there's no clear indication that will in fact happen.

scrooched_moose 6y ago

If you dig into the the margins they aren't looking very good, even if they got a fair number of eventual winners correct. Predicted vs Actual (FiveThirtyEight's averages vs NYTimes' current tally; Biden's margin is positive) PA: +4.7% vs +.5% (-4.2%) FL: +2.5 vs -3.4 (-5.9) TX: -1.5 vs -5.9 (-4.4) OH: -0.6 vs -8.1 (-7.5) PA: +4.7 vs +0.5 (-4.2) IA: -1.5 vs -8.2 (-6.7) NC: +1.7 vs -1.4 (-3.1) WI: +8.3 vs +0.6 (-7.7) GA: +0.9 vs +0.1 (-.8) MI: +8.0 vs +2.6 (-5.4) AZ: +2.6 vs +0.6 (-2.0) NV: +6.2 vs +2.0 (-4.2) They pretty consistently overpredicted Biden's margin by about 4-7% in almost all of the swing states. Even if that's within, or close to, the margin of error; there's a systemic issue if it's happening in almost every state. Edit: Added AZ and NV; and the difference

paulddraper 6y ago
^ Exactly. The polls were at least as wrong in 2016, even if that error was less significant this time (because the polled margin started out bigger). Pollsters and poll-watchers downplaying this is not a good sign for future improvement. EDIT: Overall though, I'm not sure that's a loss. People pay way more attention to polls than they should anyway.
OskarS 6y ago
This is such a damning list. At some point, reporting on polls when they have this bad of a track-record serves no journalistic purpose and just confuses the public. Like, the discussions people were having in the days before the election about Biden's strength barely resembles reality. I respect FiveThirtyEight and their work, and I do think they're intellectually honest and generally speaking good at their jobs. But they can only be as good as their sources, and when the sources are this terrible, they shouldn't be reported on this, and not given the statistical and scientific sheen of authenticity 538 gives them. Like, that site is starting to have a net negative influence on the world, and when that is the case with a journalistic institution, what are you even doing?
- paulddraper 6y ago
  > I respect FiveThirtyEight and their work, and I do think they're intellectually honest and generally speaking good at their jobs. But 538 chooses how to weight the polls. For example, they only gave Rasmussen (which was far closer on these) a C rating, preferring less accurate polls. FWIW, their articles clearly lean left [1], IDK if that affects their analysis/forecasts. [1] https://www.adfontesmedia.com/interactive-media-bias-chart-2/ https://www.adfontesmedia.com/interactive-media-bias-chart-2... similar to WaPo

tsimionescu 6y ago

The polls showed that Biden would win Florida by 2.2%, and in fact he lost by 3.3%. Is 5% within the normal error margin? There is no reason that polls would be significantly different from the final vote results in most circumstances, if they are competently run. https://projects.fivethirtyeight.com/polls/florida/ https://projects.fivethirtyeight.com/polls/florida/

srtjstjsj 6y ago
What does "competent" mean if none is competent? Are engineers incompetent because they can't put a person on Mars?
- tsimionescu 6y ago
  Sure, if some engineers told you "we are putting this person on Mars with a 99.9 probability", and the rocket in fact reaches Venus, the engineers are incompetent. Also, polling in most European countries doesn't go this far wrong, on almost all polls. So it's not like "accurate polling is impossible", it's just the polls in the US that seem to have some problem (though there have been some spectacular failures in other countries as well, which itself may lead to some thoughts on the value of polls in general).
  - evgen 6y ago
    > polling in most European countries doesn't go this far wrong, on almost all polls. Brexit and UK elections would like to have a word with you... Polling everywhere sucks. Pick a country and chances are it either has no polling to speak of or else it has the same problems.
    - disgruntledphd2 6y ago
      YouGov are pretty good, mostly because they grew up in the Internet Age. But I agree with your general point.
gpm 6y ago

ktta 6y ago

If the error skews in one specific direction for all the polls, then I don't think you can just excuse the polls by saying they're in the margin of error. This is some kind of non-trivial process problem.

sampo 6y ago

> I really have no idea where this idea that the polls failed comes from. For one, Andrew Gelman (professor of statistics and political science at Columbia University). Who knows more about the topic than anyone here in this HN discussion. https://statmodeling.stat.columbia.edu/2020/11/04/dont-kid-yourself-the-polls-messed-up-and-that-would-be-the-case-even-wed-forecasted-biden-losing-florida-and-only-barely-winning-the-electoral-college/ https://statmodeling.stat.columbia.edu/2020/11/04/dont-kid-y...

evgen 6y ago
He may know a lot about the topic, but has little experience in actually modeling elections. He also has skin in the game and an incentive to make sure everyone knows that his miss was completely not his fault and was in fact due to failures by someone else, someone over there who you don't know, yes, she lives in Canada...
- throwawaylolx 6y ago
  So what statistical mistakes did this lead to in his analysis?

matt_kantor 6y ago

People in this thread might be interested in this podcast episode: https://www.iheart.com/podcast/105-stuff-you-should-know-26940277/episode/how-election-polling-works-and-doesnt-71597551/ https://www.iheart.com/podcast/105-stuff-you-should-know-269... They discuss a lot of the things brought up here and go into details about the actual mechanics of calling people, etc. It's an easy listen.

dillondoyle 6y ago

It was very very bad up and down the ballot. No shit one of our clients their poll was - i'm rounding here because this is my real name - +15%. They barely won <1%. I wish pollsters would move towards very very large digital samples. You can't get a 15 minute survey but head-to-head you can get a huge sample for not a lot of money. And don't try to weight it to what you THINK the turnout will be. Break out likely voter or not. Do not add any inference or opinion or 'math'

evgen 6y ago
All digital surveys skew heavily to a younger demographic and heavily male. You have basically captured the political opinions of the fraction of the potential electorate least likely to vote. How exactly is that worthwhile?
- dillondoyle 6y ago
  I don't buy that, and even if I did the flip would be worse if you look phone contact rates. I've seen some good very large digital surveys. We do some - geared towards measuring ad recall/lift - and I find them valuable, especially a giant sample on only two recall + head to head questions You can always ask for gender/age and weight it but that's part of what I see as the problem. so many 'traditional' surveys i see make fairly large adjustments, lots of 'looking back' at past turnout and over fitting based on personal bias. Especially when weighting from very small sub-sample xtab e.g. hispanics. im not a pollster and that's just my still fairly-insider / polling adjacent insight Another problem with digital surveys is most of the firms do opt-in panels, like having people register to take surveys and get paid on mechanical turk. and then they weight from there. I think this is a problem. The surveys we do run inside of mobile ads, similar to Google / FB Brand Lift surveys they go after those who saw the ads or a truly random sample, not a small biased group of survey panel

titzer 6y ago

Slapping +/- 5% on top of a prediction that is systematically off by ~5% isn't "in the margin of error", it's just simply off. For one prediction, two predictions, no problem. They fall somewhere in this range with probability distribution (usually a Bell curve around the mean). But for hundreds, if not thousands, of polls to be so systematically off, it is not "within the margin of error" or a statistical fluke. If it was, you would expect a normal distribution around the mean of the polls. The mean was completely off. Put simply, it was sampling bias. Pollsters screwed up. They either sampled the wrong voters, the wrong areas, mispredicted who would turn out, or voters did not accurately report who they planned to vote for. Garbage in, garbage out. You can try to correct for sampling bias, but if you are blind to it, as your comment pushes more people to be, then you will fail.

ehsankia 6y ago
Right, but the job of 538 is exactly to deal with that, and they do take into account the fact that errors are correlated, so if one poll is off, chances are they all are. That's why their predictions are fairly conservative. They gave Florida a 1/3 chance for Trump and 1/10 for the presidency. Both those numbers make sense. Now you can argue that there's little utility if the error bars are so big, and you may have a point. The bigger issue here is that modern elections have been extremely close, and unfortunately we cannot get polling accurate enough.