4 ms·

Trafalgar almost certainly got more states wrong (counts aren't all finished, but we do know likely winners in all states). Other sites like 538 predicted all

Trafalgar almost certainly got more states wrong (counts aren't all finished, but we do know likely winners in all states). Other sites like 538 predicted all states correctly, except NC and FL (50.5%/48.8% and 50.9%/48.4% respectively, both predictions wrongly favoring Biden [1]). Meanwhile, Trafalgar's polls put Trump up in NV, GA, PA, AZ, and MI (from their website, all late Oct / early Nov polls [2]). Once counts are final, we can more accurately compare margins and errors more specifically, but Trafalgar does not have a good record for 2020. Comparing models to a single pollster isn't completely even, but it's important to note that the Trafalgar polls were mostly outliers, disagreeing with the consensus in many states [3]. [1]https://projects.fivethirtyeight.com/2020-election-forecast/ https://projects.fivethirtyeight.com/2020-election-forecast/ [2]https://www.thetrafalgargroup.org/ https://www.thetrafalgargroup.org/ [3]https://projects.fivethirtyeight.com/polls/president-general/national/ https://projects.fivethirtyeight.com/polls/president-general... https://www.realclearpolitics.com/epolls/latest_polls/ https://www.realclearpolitics.com/epolls/latest_polls/

zimablue 6y ago
Number of states that you get wrong isn’t a great measure though right? If I call if for trump at 51-49 and you call it for Biden at 75-25 and Biden wins by 51-49 I was more correct than you were, to count “states called” is the wrong way to evaluate analysis in the same way that fptp is unrepresentatove of the popular vote, ironically.
- Green_man 6y ago
  I agree with you, and that's why i made effort to differentiate between margins vs winning. In my opinion, 2020's polls were a lot worse than 2016's, but because of Biden's much bigger lead (in the polls and in key states), the polling errors did not cross the threshold to be "wrong" in the winner take all system. Finally, there is an argument that even a Biden 75-25 call gives the correct outcome because each state is a winner take all system (with the exception of NE and ME, iirc). From a stats standpoint, the delta in that example make it very inaccurate, but from a model/communication standpoint, getting each state "right" (in binary terms) is very valuable. Trafalgar had completely different results than the rest of the consensus, and once counts are more finalized in non swing states (places like CA and NY take forever, but we don't pay attention because they are always blue), we can accurately compare the polling and real results between different pollsters and polling methodologies. Some experts have cautioned against using exit polls for this purpose (what is usually done for a quick read of polling accuracy), because exit polls are only measuring election day in person votes, and thus trend really red this year. Trafalgar had unconventional methodologies like a "shy trump voter" bias where they arbitrarily shifted their numbers to the right, with weirdly consistent results of Trump +3 in a bunch of close/blue states. Perhaps there is more complicated justification of these numbers in the background, but I'm concerned that even if their deltas end up better than the "normal" pollsters, they are generating an inferior product with overbaked data.
  - rayiner 6y ago
    They didn't just add a "shy Trump voter" factor:https://www.nationalreview.com/2020/10/another-pollster-sees-a-trump-win/ https://www.nationalreview.com/2020/10/another-pollster-sees... They tried to come up with approaches to get around the very effect the article talks about: Trump voters don't answer survey questions. The main differences in methodology were making fewer assumptions about Republican turnout, larger sample sizes, and different survey techniques. Trafalgar clearly missed some things: traditionally republican collar counties breaking hard for Biden. (My county didn't vote for Obama either time, but voted for Biden by 12, after voting for the republican governor a couple of years ago by 38.) But 538 had some insane misses this year in critical states like Wisconsin (off by 7.7), Ohio (off by 7.3), Florida (off by 5.9), etc. Finalizing counts in NY and CA isn't going to change those numbers--and Trafalgar wasn't analyzing them anyway.
    - Green_man 6y ago
      Like I said in an earlier comment, comparisons between polls and a model aren't completely fair. Part of the model's calculation is that 8.8 points or whatever requires an enormous polling error to swing for trump. I do agree that other polls should get criticism/improve their methodologies to avoid the ~5 point margin in FL, or the 7.7 margin you listed. And, I even think that they may not get as much criticism as their errors warrant because many of these states were off by 6+ points but still went for Biden. Still, Trafalgar has a unique methodology that should be understood better before they are extolled as the "best pollster". And, Trafalgar is not immune to similar polling errors, just in the other direction, and with a wrong result. The delta may be more important from a statistical methodology standpoint, but these polls are measuring winner take all states, and "The Trafalgar Group’s Robert Cahaly is an outlier among pollsters in that he thinks President Trump will carry Michigan, Pennsylvania, or both, and hence be reelected with roughly 280 electoral votes" is a pretty poor prediction based on their data. Does this mean that their data is necessarily poor? No, but it isn't a good sign.
nxmnxm99 6y ago
You're not looking at the deltas. > but it's important to note that the Trafalgar polls were mostly outliers, disagreeing with the consensus in many states Well, yeah?
- Green_man 6y ago
  We can't measure the deltas with as much confidence, because counting is far from finalized in many states (especially non swing states). But if trafalgar is the only group putting out polls where trump is ahead in several states, and then those states are lost, isn't that meaningful? Trafalgar was putting a "shy trump voter" bias into their polls, basically shoving the polls arbitrarily right. Some of that may have helped with polling errors that hurt other pollsters, but its clear they went too far, especially with many midwest states that ended up going blue.
rayiner 6y ago
Trafalgar got the margin much better than 538 in key states. Your list excludes several that 538 thought would be close but weren't: Ohio and Texas, and which ended up being close but 538 thought would be a blowout: Wisconsin. Putting those states in, you get Trafalgar erring in favor of Trump by 1.76 points, while 538 erred in favor of Biden by 4.37 points: https://docs.google.com/spreadsheets/d/e/2PACX-1vSulsnjJ96c2tiW6Zk_Bt4Ji2t1mht5V1vjX0hORgJhWiPpBmTC_8H_zWY275OXkwJF42Z_wIfxflUe/pubhtml https://docs.google.com/spreadsheets/d/e/2PACX-1vSulsnjJ96c2... The final count will change these numbers a little bit, but I think Trafalgar will have gotten closer to the final result in these key states by 2.5 points on average.
- Green_man 6y ago
  I'll be interested to see the final count and compare on all states, but Trafalgar's issues are deeper than just deltas in my opinion. They had an arbitrary "shy trump voter" factor, which may be justifiable, but any "shy voter" effect is tautologically immeasurable (and thus just a guess on Trafalgar's part). They had weirdly consistent results where Trump was up by 3 points in lots of purple/blue states, which doesn't line up with the differences we saw in other polls and in the results (so far). I agree that the polling errors are significant in many of the mainstream models, even in the states that were called correctly. These deltas so far are worse than 2016, and should be looked into. But _if_ Trafalgar ends up with a lower delta over all 50 states when this is all finished, they still gave the wrong result in most of the key states that matter for the election. That's significant, because the polls are measuring 50 winner take all systems, and 538 told us 90% chance Biden win, while Trafalgar told us that Trump was favored in almost any state that was worth watching this year. That was wrong, and Trafalgar deserves criticism for it.
  - rayiner 6y ago
    They didn't have an "arbitrary shy Trump voter" factor. That's not how they did it. They observed that Trump voters were less likely to answer polls and tried to get at the data other ways. Also, who cares about 50-state results? One of the key points Trafalgar's founder has made is that polling is a state-by-state exercise. All pollsters use data to estimate e.g. turnout among different demographics. National pollsters use things like exit polls, but Trafalgar digs into state-level voter registration data.

zimablue 6y ago

Number of states that you get wrong isn’t a great measure though right? If I call if for trump at 51-49 and you call it for Biden at 75-25 and Biden wins by 51-49 I was more correct than you were, to count “states called” is the wrong way to evaluate analysis in the same way that fptp is unrepresentatove of the popular vote, ironically.

Green_man 6y ago
I agree with you, and that's why i made effort to differentiate between margins vs winning. In my opinion, 2020's polls were a lot worse than 2016's, but because of Biden's much bigger lead (in the polls and in key states), the polling errors did not cross the threshold to be "wrong" in the winner take all system. Finally, there is an argument that even a Biden 75-25 call gives the correct outcome because each state is a winner take all system (with the exception of NE and ME, iirc). From a stats standpoint, the delta in that example make it very inaccurate, but from a model/communication standpoint, getting each state "right" (in binary terms) is very valuable. Trafalgar had completely different results than the rest of the consensus, and once counts are more finalized in non swing states (places like CA and NY take forever, but we don't pay attention because they are always blue), we can accurately compare the polling and real results between different pollsters and polling methodologies. Some experts have cautioned against using exit polls for this purpose (what is usually done for a quick read of polling accuracy), because exit polls are only measuring election day in person votes, and thus trend really red this year. Trafalgar had unconventional methodologies like a "shy trump voter" bias where they arbitrarily shifted their numbers to the right, with weirdly consistent results of Trump +3 in a bunch of close/blue states. Perhaps there is more complicated justification of these numbers in the background, but I'm concerned that even if their deltas end up better than the "normal" pollsters, they are generating an inferior product with overbaked data.
- rayiner 6y ago
  They didn't just add a "shy Trump voter" factor:https://www.nationalreview.com/2020/10/another-pollster-sees-a-trump-win/ https://www.nationalreview.com/2020/10/another-pollster-sees... They tried to come up with approaches to get around the very effect the article talks about: Trump voters don't answer survey questions. The main differences in methodology were making fewer assumptions about Republican turnout, larger sample sizes, and different survey techniques. Trafalgar clearly missed some things: traditionally republican collar counties breaking hard for Biden. (My county didn't vote for Obama either time, but voted for Biden by 12, after voting for the republican governor a couple of years ago by 38.) But 538 had some insane misses this year in critical states like Wisconsin (off by 7.7), Ohio (off by 7.3), Florida (off by 5.9), etc. Finalizing counts in NY and CA isn't going to change those numbers--and Trafalgar wasn't analyzing them anyway.
  - Green_man 6y ago
    Like I said in an earlier comment, comparisons between polls and a model aren't completely fair. Part of the model's calculation is that 8.8 points or whatever requires an enormous polling error to swing for trump. I do agree that other polls should get criticism/improve their methodologies to avoid the ~5 point margin in FL, or the 7.7 margin you listed. And, I even think that they may not get as much criticism as their errors warrant because many of these states were off by 6+ points but still went for Biden. Still, Trafalgar has a unique methodology that should be understood better before they are extolled as the "best pollster". And, Trafalgar is not immune to similar polling errors, just in the other direction, and with a wrong result. The delta may be more important from a statistical methodology standpoint, but these polls are measuring winner take all states, and "The Trafalgar Group’s Robert Cahaly is an outlier among pollsters in that he thinks President Trump will carry Michigan, Pennsylvania, or both, and hence be reelected with roughly 280 electoral votes" is a pretty poor prediction based on their data. Does this mean that their data is necessarily poor? No, but it isn't a good sign.

nxmnxm99 6y ago

You're not looking at the deltas. > but it's important to note that the Trafalgar polls were mostly outliers, disagreeing with the consensus in many states Well, yeah?

Green_man 6y ago
We can't measure the deltas with as much confidence, because counting is far from finalized in many states (especially non swing states). But if trafalgar is the only group putting out polls where trump is ahead in several states, and then those states are lost, isn't that meaningful? Trafalgar was putting a "shy trump voter" bias into their polls, basically shoving the polls arbitrarily right. Some of that may have helped with polling errors that hurt other pollsters, but its clear they went too far, especially with many midwest states that ended up going blue.

rayiner 6y ago

Trafalgar got the margin much better than 538 in key states. Your list excludes several that 538 thought would be close but weren't: Ohio and Texas, and which ended up being close but 538 thought would be a blowout: Wisconsin. Putting those states in, you get Trafalgar erring in favor of Trump by 1.76 points, while 538 erred in favor of Biden by 4.37 points: https://docs.google.com/spreadsheets/d/e/2PACX-1vSulsnjJ96c2tiW6Zk_Bt4Ji2t1mht5V1vjX0hORgJhWiPpBmTC_8H_zWY275OXkwJF42Z_wIfxflUe/pubhtml https://docs.google.com/spreadsheets/d/e/2PACX-1vSulsnjJ96c2... The final count will change these numbers a little bit, but I think Trafalgar will have gotten closer to the final result in these key states by 2.5 points on average.

Green_man 6y ago
I'll be interested to see the final count and compare on all states, but Trafalgar's issues are deeper than just deltas in my opinion. They had an arbitrary "shy trump voter" factor, which may be justifiable, but any "shy voter" effect is tautologically immeasurable (and thus just a guess on Trafalgar's part). They had weirdly consistent results where Trump was up by 3 points in lots of purple/blue states, which doesn't line up with the differences we saw in other polls and in the results (so far). I agree that the polling errors are significant in many of the mainstream models, even in the states that were called correctly. These deltas so far are worse than 2016, and should be looked into. But _if_ Trafalgar ends up with a lower delta over all 50 states when this is all finished, they still gave the wrong result in most of the key states that matter for the election. That's significant, because the polls are measuring 50 winner take all systems, and 538 told us 90% chance Biden win, while Trafalgar told us that Trump was favored in almost any state that was worth watching this year. That was wrong, and Trafalgar deserves criticism for it.
- rayiner 6y ago
  They didn't have an "arbitrary shy Trump voter" factor. That's not how they did it. They observed that Trump voters were less likely to answer polls and tried to get at the data other ways. Also, who cares about 50-state results? One of the key points Trafalgar's founder has made is that polling is a state-by-state exercise. All pollsters use data to estimate e.g. turnout among different demographics. National pollsters use things like exit polls, but Trafalgar digs into state-level voter registration data.