5 ms·

The original report is here: http://www.dailykos.com/story/2010/6/29/880179/-Research-2000:-Problems-in-plain-sight http://www.dailykos.com/story/2010/6/29/8801

by samdk 16y ago

The original report is here: http://www.dailykos.com/story/2010/6/29/880179/-Research-2000:-Problems-in-plain-sight http://www.dailykos.com/story/2010/6/29/880179/-Research-200... What I find most interesting about this is that the statistics used here really aren't complicated at all: the missing zeros are a pretty classic case of the human vision of random not matching up with reality, and the other issues they found are also pretty easy to understand at a high level. They go (a bit) deeper to show exactly why the problems they found are problems, but it's quite easily apparent that something is wrong when you look at the raw data. Problems in plain sight indeed.

aaronbrethorst 16y ago
For more on this sort of thing, check out 538's incredibly comprehensive takedown of Strategic Vision last September: http://www.fivethirtyeight.com/2009/09/strategic-vision-polls-exhibit-unusual.html http://www.fivethirtyeight.com/2009/09/strategic-vision-poll... http://www.fivethirtyeight.com/2009/09/comparison-study-unusual-patterns-in.html http://www.fivethirtyeight.com/2009/09/comparison-study-unus...
- mason55 16y ago
  Yeah I'm not sure how TechDirt didn't mention Nate. He started the whole thing.
_delirium 16y ago
What actually bugs me somewhat more about this case is that it seems they were only caught (if indeed it was faked) because they faked it badly. But what if a pollster fakes their polls more intelligently, such that they look legit to at least most basic statistical tests? In well-polled areas, especially (e.g. presidential races), it seems like it would be difficult to distinguish between a real pollster, and a fake pollster who took a bunch of other polls and ran them through a "generate random-looking result roughly similar to these other polls" function. As long as your distribution looks right, and your absolute numbers are roughly in line with the pack, it seems like it'd be hard to prove that you didn't actually robocall anyone.
- joeyh 16y ago
  Also, my understanding is that polls on Kos included more data than is typically provided. Which I think helped this analysis. So a more traditional poll could be faked just as badly and not be susceptable to being caught.
deleted 16y ago
[deleted]
mturmon 16y ago
The insight was in knowing to look at differences, week(n) - week(n-1). That was not the raw data. They also noticed that the percentages of "men in favor of Obama" and "women in favor of Obama" did not tend to be odd/even or even/odd numbers. They were even/even or odd/odd. That's obvious only in hindsight.

by samdk 16y ago

aaronbrethorst 16y ago

For more on this sort of thing, check out 538's incredibly comprehensive takedown of Strategic Vision last September: http://www.fivethirtyeight.com/2009/09/strategic-vision-polls-exhibit-unusual.html http://www.fivethirtyeight.com/2009/09/strategic-vision-poll... http://www.fivethirtyeight.com/2009/09/comparison-study-unusual-patterns-in.html http://www.fivethirtyeight.com/2009/09/comparison-study-unus...

mason55 16y ago
Yeah I'm not sure how TechDirt didn't mention Nate. He started the whole thing.

_delirium 16y ago

What actually bugs me somewhat more about this case is that it seems they were only caught (if indeed it was faked) because they faked it badly. But what if a pollster fakes their polls more intelligently, such that they look legit to at least most basic statistical tests? In well-polled areas, especially (e.g. presidential races), it seems like it would be difficult to distinguish between a real pollster, and a fake pollster who took a bunch of other polls and ran them through a "generate random-looking result roughly similar to these other polls" function. As long as your distribution looks right, and your absolute numbers are roughly in line with the pack, it seems like it'd be hard to prove that you didn't actually robocall anyone.

joeyh 16y ago
Also, my understanding is that polls on Kos included more data than is typically provided. Which I think helped this analysis. So a more traditional poll could be faked just as badly and not be susceptable to being caught.

deleted 16y ago

[deleted]

mturmon 16y ago

The insight was in knowing to look at differences, week(n) - week(n-1). That was not the raw data. They also noticed that the percentages of "men in favor of Obama" and "women in favor of Obama" did not tend to be odd/even or even/odd numbers. They were even/even or odd/odd. That's obvious only in hindsight.