7 ms·

Nate Silver Finds Unusual Patterns in a Polling Firm's Data. Was it fabricated?

98 points by gluejar 17y ago

DarkShikari 17y ago
Particularly interesting is the further analysis done in the comments where similar statistical methods are used to analyse other polling agencies' results. Nearly every single one seems to line up to look almost exactly like the Quinnipiac results, further damning Strategic Vision. But what interests me more is the question of exactly why this distribution comes up; it's not Benford's Law, since probabilities aren't a power-law distribution, yet it seems consistent across multiple sources.
callmeed 17y ago
With newspapers suffering so much, it's refreshing to see a blogger taking the time to perform investigations like this. We need more of this.
- cwan 17y ago
  My sense has been newspapers have been suffering so much because they don't take the time to perform investigations like this. This is hardly the only example with the Acorn videos quite recently for instance. We do definitely need more of this.
  - callmeed 17y ago
    I was thinking along the lines of their decline in ad/classified revenue-which is used to fund their investigative journalism. Are you saying the quality of old media investigations has declined independently?
    - cwan 17y ago
      The WSJ runs an online column called "Best of the Web". Granted, it slants conservative (which again, I am not - but it does poke fun at various media outlets/stories) but have a look at his point on the lack of coverage by the NYT on the issues facing the current Obama Administration: http://online.wsj.com/article/SB10001424052970204488304574435083027872174.html http://online.wsj.com/article/SB1000142405297020448830457443... I don't know if investigative journalism has declined independently. What I do think however is that there is far more investigative journalism happening outside of traditional media outlets which has made traditional media outlets considerably less important. For this reason, and the pervasive view that the media is biased (as many journalists themselves even admit on the Pew poll), I think they've seen readerships decline - and rightfully so.
    - anamax 17y ago
      How about a third possibility - that there never really was much investigative journalism?
  - philwelch 17y ago
    The trouble with the Acorn videos is that they're a partisan attempt to discredit the opposition. While the videos themselves seem legit, I'd rather not leave exposes of political groups up to their political enemies. Political enemies only want to discredit their opponents--they're not interested in truth, and for every Acorn video there's a Swift Boat smear campaign with no basis in fact.
    - cwan 17y ago
      Not sure you've got your facts right on the swiftboating campaign: http://www.pajamasmedia.com/instapundit-archive/archives/017139.php http://www.pajamasmedia.com/instapundit-archive/archives/017... (Glenn Reynolds isn't a conservative - nor am I). That being said, I don't know that you're right about the nature of the Acorn videos (though I would agree that they were ultimately used by partisans) - but even if you are, you seem to assume that journalists don't have some type of agenda (Jayson Blair, Dan Rather or even as far back as Pulitzer prize winner Walter Duranty). On the other side of it, I'm not sure your concerns that false partisan attacks will win the day are justified - look at the "birthers" who have been thoroughly discredited and marginalized (ie the ones who claim Obama wasn't born on American soil). At least with respect to Acorn, it has had a slew of scandals plague them and yet various levels of the US government had continued to trust them with rather important tasks - e.g. help at the IRS, census, etc.. In my view these were far worse offenses than the 'entrapment' that brought them down. There is no reason that larger media outlets shouldn't have pursued this story themselves - after all - the cost even of putting out the Acorn story was less than five thousand dollars. Meanwhile, supposedly reputable outlets like the NYT had trouble covering the story until the Obama Administration and Congress began to react and cut funding and ties to Acorn. I'm all for disclosure. Let's be open to criticism/investigations from anyone and everyone and let them be judged on their merits.
      - earl 17y ago
        No, Glenn Reynolds is a glibertarian who always votes, supports, and blogs about conservative causes. Also, the Acorn worker in question reported the two actors in the video to police: http://www.foxnews.com/story/0,2933,553423,00.html http://www.foxnews.com/story/0,2933,553423,00.html
  - eli 17y ago
    Let's be fair; that's not really why papers are hurting financially. It was a sudden and drastic drop in advertisers, not people switching to blogs. Newspapers used to make lots of money on ads from national retailers, auto dealers, financial companies, and real estate. Those are four of the worst possible groups to have as your clients.
    - cwan 17y ago
      I think this goes back to whether newspapers are selling their content or the medium. From what I recall, the general rule of thumb (used to be?) that a newspaper gets a third of revenues from advertisers, a third from classifieds and a third from subscriptions. The much broader problem that affects all of the above has been that readerships have been in decline - in some cases rather steep decline. Further, from the supply side of the argument, the supply of media that's available to advertisers has broadened considerably because of the internet. But there's always been demand for valuable content - and that's what I would suggest has been a problem. That for quite some time that demand was captive - people didn't have alternatives. Now they do, and now they're looking elsewhere. Even when the businesses in the retail, auto, financial and real estate sectors come back, traditional media won't - it's been in decline far longer than this current/past recession. This I think has a great deal to do with the value (or lack thereof) that people see in traditional media versus the alternative sources they can find increasingly available because of the net.
      - eli 17y ago
        I mostly agree, but I think it's important to note that readership has been on a steady decline for decades, but the ad market crashed very suddenly and drastically in just the last few years. And you are correct that many of these advertisers won't come back.
- deleted 17y ago
  [deleted]
jonknee 17y ago
A true statistics hacker. I love reading his analysis--he was the go to source during the election for anything poll related.
isamuel 17y ago
This is pretty damning stuff. In a well-functioning media environment, it would be sufficient to seriously damage the credibility of the pollster involved (Strategic Vision), or at least shift the burden to them to explain their behavior. Instead, we already see clues about what will happen: the pollster is threatening a lawsuit, which will be a sure loser but nonetheless drain resources and time from its target (either Nate himself or fivethirtyeight). The polls from the offending pollster will, I'm completely certain, be reported with utter credulity in the 2010 election cycle and beyond. The fact that the books are cooked will not even be mentioned in media reports, and will ultimately matter not at all. This is because we simply do not have a well-functioning media environment.
- cwan 17y ago
  I think their lack of disclosure has already severely damaged them. Ultimately, in this case it would seem the damage will be pretty brutal unless Nate Silver was wrong. Either they don't sue and don't disclose and the result is no one trusts them enough to pay for their polling (after all, what's the point in paying for a poll if it's going to come under question afterwards?) or they'll sue while not releasing the support for their polls/data on their merits - and that also opens them up to discovery which could be even more damaging. The best case scenario at this point is if they can publicly prove Nate Silver is wrong. And based on their policies in the past of not releasing the additional supporting data, I suspect this won't happen.
- smhinsey 17y ago
  It does nothing to help their credibility that they are basically riding the coat tails of another firm, called Strategic Vision Inc. (the firm he is challenging is Strategic Vision, LLC), which is something Nate mentions in one of his posts on the matter. That right there ought to serve as a pretty big negative indicator of credibility, but it doesn't seem to matter much to the media, if they even do enough research to notice that they aren't the "real" Strategic Vision.
danger 17y ago
Here's a bit more on the subject: http://news.ycombinator.com/item?id=846514 http://news.ycombinator.com/item?id=846514
kevinpet 17y ago
It seems to me that if there were one data source I would expect to not be consistent with Benford's law it would be polling data. Benford's law is usually seen in numbers that follow some skewed distribution. Polling data groups around maybe 45% depending on the number undecided. I don't know what I would expect, but polling extremely close races, or polling close to the election, I would expect to see lots of high digits (6 7 8 9). I wouldn't expect many 41 or 31 (those questions are uninteresting) so the 1s come from mostly 51s. He may be right, but I'm going to need a little more convincing if someone wants to assume that polling data would exhibit Benford's law.

98 points by gluejar 17y ago

Particularly interesting is the further analysis done in the comments where similar statistical methods are used to analyse other polling agencies' results. Nearly every single one seems to line up to look almost exactly like the Quinnipiac results, further damning Strategic Vision. But what interests me more is the question of exactly why this distribution comes up; it's not Benford's Law, since probabilities aren't a power-law distribution, yet it seems consistent across multiple sources.

callmeed 17y ago

With newspapers suffering so much, it's refreshing to see a blogger taking the time to perform investigations like this. We need more of this.

cwan 17y ago
My sense has been newspapers have been suffering so much because they don't take the time to perform investigations like this. This is hardly the only example with the Acorn videos quite recently for instance. We do definitely need more of this.
- callmeed 17y ago
  I was thinking along the lines of their decline in ad/classified revenue-which is used to fund their investigative journalism. Are you saying the quality of old media investigations has declined independently?
  - cwan 17y ago
    The WSJ runs an online column called "Best of the Web". Granted, it slants conservative (which again, I am not - but it does poke fun at various media outlets/stories) but have a look at his point on the lack of coverage by the NYT on the issues facing the current Obama Administration: http://online.wsj.com/article/SB10001424052970204488304574435083027872174.html http://online.wsj.com/article/SB1000142405297020448830457443... I don't know if investigative journalism has declined independently. What I do think however is that there is far more investigative journalism happening outside of traditional media outlets which has made traditional media outlets considerably less important. For this reason, and the pervasive view that the media is biased (as many journalists themselves even admit on the Pew poll), I think they've seen readerships decline - and rightfully so.
  - anamax 17y ago
    How about a third possibility - that there never really was much investigative journalism?
- philwelch 17y ago
  The trouble with the Acorn videos is that they're a partisan attempt to discredit the opposition. While the videos themselves seem legit, I'd rather not leave exposes of political groups up to their political enemies. Political enemies only want to discredit their opponents--they're not interested in truth, and for every Acorn video there's a Swift Boat smear campaign with no basis in fact.
  - cwan 17y ago
    Not sure you've got your facts right on the swiftboating campaign: http://www.pajamasmedia.com/instapundit-archive/archives/017139.php http://www.pajamasmedia.com/instapundit-archive/archives/017... (Glenn Reynolds isn't a conservative - nor am I). That being said, I don't know that you're right about the nature of the Acorn videos (though I would agree that they were ultimately used by partisans) - but even if you are, you seem to assume that journalists don't have some type of agenda (Jayson Blair, Dan Rather or even as far back as Pulitzer prize winner Walter Duranty). On the other side of it, I'm not sure your concerns that false partisan attacks will win the day are justified - look at the "birthers" who have been thoroughly discredited and marginalized (ie the ones who claim Obama wasn't born on American soil). At least with respect to Acorn, it has had a slew of scandals plague them and yet various levels of the US government had continued to trust them with rather important tasks - e.g. help at the IRS, census, etc.. In my view these were far worse offenses than the 'entrapment' that brought them down. There is no reason that larger media outlets shouldn't have pursued this story themselves - after all - the cost even of putting out the Acorn story was less than five thousand dollars. Meanwhile, supposedly reputable outlets like the NYT had trouble covering the story until the Obama Administration and Congress began to react and cut funding and ties to Acorn. I'm all for disclosure. Let's be open to criticism/investigations from anyone and everyone and let them be judged on their merits.
    - earl 17y ago
      No, Glenn Reynolds is a glibertarian who always votes, supports, and blogs about conservative causes. Also, the Acorn worker in question reported the two actors in the video to police: http://www.foxnews.com/story/0,2933,553423,00.html http://www.foxnews.com/story/0,2933,553423,00.html
- eli 17y ago
  Let's be fair; that's not really why papers are hurting financially. It was a sudden and drastic drop in advertisers, not people switching to blogs. Newspapers used to make lots of money on ads from national retailers, auto dealers, financial companies, and real estate. Those are four of the worst possible groups to have as your clients.
  - cwan 17y ago
    I think this goes back to whether newspapers are selling their content or the medium. From what I recall, the general rule of thumb (used to be?) that a newspaper gets a third of revenues from advertisers, a third from classifieds and a third from subscriptions. The much broader problem that affects all of the above has been that readerships have been in decline - in some cases rather steep decline. Further, from the supply side of the argument, the supply of media that's available to advertisers has broadened considerably because of the internet. But there's always been demand for valuable content - and that's what I would suggest has been a problem. That for quite some time that demand was captive - people didn't have alternatives. Now they do, and now they're looking elsewhere. Even when the businesses in the retail, auto, financial and real estate sectors come back, traditional media won't - it's been in decline far longer than this current/past recession. This I think has a great deal to do with the value (or lack thereof) that people see in traditional media versus the alternative sources they can find increasingly available because of the net.
    - eli 17y ago
      I mostly agree, but I think it's important to note that readership has been on a steady decline for decades, but the ad market crashed very suddenly and drastically in just the last few years. And you are correct that many of these advertisers won't come back.
deleted 17y ago
[deleted]

jonknee 17y ago

A true statistics hacker. I love reading his analysis--he was the go to source during the election for anything poll related.

isamuel 17y ago

This is pretty damning stuff. In a well-functioning media environment, it would be sufficient to seriously damage the credibility of the pollster involved (Strategic Vision), or at least shift the burden to them to explain their behavior. Instead, we already see clues about what will happen: the pollster is threatening a lawsuit, which will be a sure loser but nonetheless drain resources and time from its target (either Nate himself or fivethirtyeight). The polls from the offending pollster will, I'm completely certain, be reported with utter credulity in the 2010 election cycle and beyond. The fact that the books are cooked will not even be mentioned in media reports, and will ultimately matter not at all. This is because we simply do not have a well-functioning media environment.

cwan 17y ago
I think their lack of disclosure has already severely damaged them. Ultimately, in this case it would seem the damage will be pretty brutal unless Nate Silver was wrong. Either they don't sue and don't disclose and the result is no one trusts them enough to pay for their polling (after all, what's the point in paying for a poll if it's going to come under question afterwards?) or they'll sue while not releasing the support for their polls/data on their merits - and that also opens them up to discovery which could be even more damaging. The best case scenario at this point is if they can publicly prove Nate Silver is wrong. And based on their policies in the past of not releasing the additional supporting data, I suspect this won't happen.
smhinsey 17y ago
It does nothing to help their credibility that they are basically riding the coat tails of another firm, called Strategic Vision Inc. (the firm he is challenging is Strategic Vision, LLC), which is something Nate mentions in one of his posts on the matter. That right there ought to serve as a pretty big negative indicator of credibility, but it doesn't seem to matter much to the media, if they even do enough research to notice that they aren't the "real" Strategic Vision.

danger 17y ago

Here's a bit more on the subject: http://news.ycombinator.com/item?id=846514 http://news.ycombinator.com/item?id=846514

kevinpet 17y ago

It seems to me that if there were one data source I would expect to not be consistent with Benford's law it would be polling data. Benford's law is usually seen in numbers that follow some skewed distribution. Polling data groups around maybe 45% depending on the number undecided. I don't know what I would expect, but polling extremely close races, or polling close to the election, I would expect to see lots of high digits (6 7 8 9). I wouldn't expect many 41 or 31 (those questions are uninteresting) so the 1s come from mostly 51s. He may be right, but I'm going to need a little more convincing if someone wants to assume that polling data would exhibit Benford's law.