3 ms·

Do you have a reason to doubt their methodology? Pollsters working for news organizations with an editorial point of view still produce accurate poll. In the US

Do you have a reason to doubt their methodology? Pollsters working for news organizations with an editorial point of view still produce accurate poll. In the US, Fox News, which I think can be fairly characterized as an arm of the Republican Party, produces high quality polls that are not reflective of their overall orientation.

mike_hearn 3y ago
Unfortunately yes, there are lots of reasons to doubt. Some UK polling firms routinely return results that are clearly impossible, are called out on it publicly, but don't investigate or fix anything. The root causes of their unreliability are known but not well understood outside the polling industry itself. A good example of that happening recently was with a poll commissioned by the BBC from Savanta via Kings College London. Savanta is a member of the British Polling Council, and the poll supposedly canvassed people about misinformation related topics. The results claimed several things so absurd and implausible that once people noticed, Kings College London attempted to backtrack [1]. The results were very obviously indefensible garbage. Amongst other things, if you assumed the poll was representative you'd be required to believe that: • 15% of British people get "a great deal" or "a fair amount" of their information from Truth Social. • A quarter say COVID was a hoax. • Around 3.7 million people in the UK had attended protests about COVID and central bank digital currencies and "15 minute cities". No protests of any such size have been seen anywhere, needless to say. • Around 7.4 million people had heard of an obscure hobbyist conspiracy newspaper called The Light, and half of all the people who had heard of it had helped distribute it. In reality The Light has a tiny number of subscribers. The poll itself is clearly misinformation. Ironically, it was commissioned for the BBC's "Marianna in Conspiracyland" which promoted the poll relentlessly; the BBC has never accepted that the poll's results are nonsensical and simply ignored the problems. The Guardian also jumped all over it [2] without subjecting the results to even basic reality checks. The actual meaning of these results is that Savanta's panel has become extremely unrepresentative and the company is simply ignoring this problem. This is a problem that has been affecting all panel-polling companies and it's been getting worse with time, albeit Savanta seems to be truly hosed to an extent that maybe YouGov isn't (yet). Panels are deeply unrepresentative, in particular with time they become deeply skewed towards women and people who volunteer for things (for obvious reasons). They attempt to weight the results to make them more realistic, but the models are calibrated only for elections and other moments when they can get ground truth. Ask non-election related questions and you will routinely get absurd results that don't reflect the actual views of the population. The polling industry is in deep trouble as panels degrade over time (the rewards for taking part are meaningless), but are currently able to ignore it because people generally accept their results as gospel. There's Advanced Math™ involved so it must be correct, right? Still, I think eventually people will get more savvy about this. I certainly wouldn't take any poll promoted by the Guardian at face value given their disinterest in checking the results. [1] https://www.kcl.ac.uk/news/conspiracy-theory-research https://www.kcl.ac.uk/news/conspiracy-theory-research [2] https://www.theguardian.com/society/2023/jun/13/quarter-in-uk-believe-covid-was-a-hoax-poll-on-conspiracy-theories-finds https://www.theguardian.com/society/2023/jun/13/quarter-in-u...

mike_hearn 3y ago

Unfortunately yes, there are lots of reasons to doubt. Some UK polling firms routinely return results that are clearly impossible, are called out on it publicly, but don't investigate or fix anything. The root causes of their unreliability are known but not well understood outside the polling industry itself. A good example of that happening recently was with a poll commissioned by the BBC from Savanta via Kings College London. Savanta is a member of the British Polling Council, and the poll supposedly canvassed people about misinformation related topics. The results claimed several things so absurd and implausible that once people noticed, Kings College London attempted to backtrack [1]. The results were very obviously indefensible garbage. Amongst other things, if you assumed the poll was representative you'd be required to believe that: • 15% of British people get "a great deal" or "a fair amount" of their information from Truth Social. • A quarter say COVID was a hoax. • Around 3.7 million people in the UK had attended protests about COVID and central bank digital currencies and "15 minute cities". No protests of any such size have been seen anywhere, needless to say. • Around 7.4 million people had heard of an obscure hobbyist conspiracy newspaper called The Light, and half of all the people who had heard of it had helped distribute it. In reality The Light has a tiny number of subscribers. The poll itself is clearly misinformation. Ironically, it was commissioned for the BBC's "Marianna in Conspiracyland" which promoted the poll relentlessly; the BBC has never accepted that the poll's results are nonsensical and simply ignored the problems. The Guardian also jumped all over it [2] without subjecting the results to even basic reality checks. The actual meaning of these results is that Savanta's panel has become extremely unrepresentative and the company is simply ignoring this problem. This is a problem that has been affecting all panel-polling companies and it's been getting worse with time, albeit Savanta seems to be truly hosed to an extent that maybe YouGov isn't (yet). Panels are deeply unrepresentative, in particular with time they become deeply skewed towards women and people who volunteer for things (for obvious reasons). They attempt to weight the results to make them more realistic, but the models are calibrated only for elections and other moments when they can get ground truth. Ask non-election related questions and you will routinely get absurd results that don't reflect the actual views of the population. The polling industry is in deep trouble as panels degrade over time (the rewards for taking part are meaningless), but are currently able to ignore it because people generally accept their results as gospel. There's Advanced Math™ involved so it must be correct, right? Still, I think eventually people will get more savvy about this. I certainly wouldn't take any poll promoted by the Guardian at face value given their disinterest in checking the results. [1] https://www.kcl.ac.uk/news/conspiracy-theory-research https://www.kcl.ac.uk/news/conspiracy-theory-research [2] https://www.theguardian.com/society/2023/jun/13/quarter-in-uk-believe-covid-was-a-hoax-poll-on-conspiracy-theories-finds https://www.theguardian.com/society/2023/jun/13/quarter-in-u...