7 ms·

Benford's Law tests on Wikileaks data

30 points by agconway 16y ago

Symmetry 16y ago
I might be missing something, but how is it that overall the number of leading 2s is greater than the number of leading 1s, but not in any of the constituent regions?
- ars 16y ago
  It might be a form of http://en.wikipedia.org/wiki/Simpsons_paradox http://en.wikipedia.org/wiki/Simpsons_paradox
  - hiroaki 16y ago
    I don't think Simpsons paradox plays any role here, if I understand this correctly. In the first graph, the leading digit counts are not the things being summed up across all the regions. The weekly report counts are summed up across all regions and then the leading digits are counted.
- riffer 16y ago
  Could it be that the mean number of reports in a week is in the 2.5-3.5x10^x range? Why would he look at the leading number instead of the trailing number? He really ought to provide the exact dataset that he's using to generate the graphs.
  - btilly 16y ago
    Benford's law is about the distribution of the leading digit, not the trailing ones.
    - riffer 16y ago
      Right, but was his objective to use Benford's law, or was it to test whether the data was tampered with?
      - btilly 16y ago
        His objective was to use Benford's law to test whether the data as tampered with. (This is a standard use of Benford's law.)
zitterbewegung 16y ago
I thought that wikileaks eliminates certain information that would cause harm to a party mentioned in the leak?
- biturd 16y ago
  That is correct, there is a subset of ~15,000 records that are being withheld, which will be released when they no longer have a probability of being harmful to those they mention. Source: http://boingboing.net/2010/07/26/wikileaks-qa-with-ja.html http://boingboing.net/2010/07/26/wikileaks-qa-with-ja.html Boing Boing: We're told that there are more documents from this archive yet to be released by Wikileaks, some 15,000 of them as reported. Some have speculated that these could relate to Iraq. Can you comment more? Jacob Appelbaum: The 15,000 documents are part of the set of Afghanistan documents. They are being redacted for harm-minimizing purposes as requested by our source, and will be made available as is applicable with respect to the relevant security concerns. I would like to know much more about the process by which Wikileaks determines what is or is not safe/relevant to be released. While I support their efforts, they are walking a thin line in becoming the very thing they are working against. I only hope that the decision was done by experienced committee, but even that raises questions as to how do you vet that committee for the ability to gauge what is and is not releasable. How does one determine those people are even of adequate credentials/authority to read the information ahead of time? If I were our the US govt., I certainly would be playing up that issue as much as possible. On the surface it would not be hard to push general sentiment in the direction of Wikileaks playing information Gods with the data they have.
jongraehl 16y ago
Incredibly weak evidence.
- long 16y ago
  It's not weak evidence. There's no evidence. The guy doesn't seem to know how significance testing works.
djmdjm 16y ago
They should expect a bias, since Wikileaks have stated that some information has been held back deliberately. From http://wikileaks.org/wiki/Afghan_War_Diary,_2004-2010 http://wikileaks.org/wiki/Afghan_War_Diary,_2004-2010 "We have delayed the release of some 15,000 reports from the total archive as part of a harm minimization process demanded by our source. After further review, these reports will be released, with occasional redactions, and eventually in full, as the security situation in Afghanistan permits."
countersignaler 16y ago
this is a statistical blog post written by someone who has no idea how to do statistics. and even when told how to do it properly by a commenter, fails to accept the simple and overwhelming conclusion. why is this on the front page?
- Confusion 16y ago
  Perhaps because the data is interesting and we can draw the contrary conclusion: that there is no reason to suppose the data has been tampered with or fabricated?
mbateman 16y ago
> Also, as suggested in the comments, I used a chi-square goodness-of-fit test to see if the deviation is statistically significant, but it was not; with a p-value of 0.2303. Meaning we would fail to reject the null hypothesis: the observed data were a good fit for a Benford process. That said, the p-value is not so large as to suggest total adherence. Welcome to stats 101. If you get p > 0.05, it means your test failed to tell you anything. Which means your test failed to tell you about the likelihood of tampering.
- whyenot 16y ago
  You are on the right track, but no, it is more subtle than that. The test could still be meaningful depending on what probability of false positives you are willing to accept. [1] http://en.wikipedia.org/wiki/P-value http://en.wikipedia.org/wiki/P-value [2] http://en.wikipedia.org/wiki/Type_I_and_type_II_errors http://en.wikipedia.org/wiki/Type_I_and_type_II_errors
corruption 16y ago
An aptly named blog for the analysis.

30 points by agconway 16y ago

I might be missing something, but how is it that overall the number of leading 2s is greater than the number of leading 1s, but not in any of the constituent regions?

ars 16y ago
It might be a form of http://en.wikipedia.org/wiki/Simpsons_paradox http://en.wikipedia.org/wiki/Simpsons_paradox
- hiroaki 16y ago
  I don't think Simpsons paradox plays any role here, if I understand this correctly. In the first graph, the leading digit counts are not the things being summed up across all the regions. The weekly report counts are summed up across all regions and then the leading digits are counted.
riffer 16y ago
Could it be that the mean number of reports in a week is in the 2.5-3.5x10^x range? Why would he look at the leading number instead of the trailing number? He really ought to provide the exact dataset that he's using to generate the graphs.
- btilly 16y ago
  Benford's law is about the distribution of the leading digit, not the trailing ones.
  - riffer 16y ago
    Right, but was his objective to use Benford's law, or was it to test whether the data was tampered with?
    - btilly 16y ago
      His objective was to use Benford's law to test whether the data as tampered with. (This is a standard use of Benford's law.)

zitterbewegung 16y ago

I thought that wikileaks eliminates certain information that would cause harm to a party mentioned in the leak?

biturd 16y ago
That is correct, there is a subset of ~15,000 records that are being withheld, which will be released when they no longer have a probability of being harmful to those they mention. Source: http://boingboing.net/2010/07/26/wikileaks-qa-with-ja.html http://boingboing.net/2010/07/26/wikileaks-qa-with-ja.html Boing Boing: We're told that there are more documents from this archive yet to be released by Wikileaks, some 15,000 of them as reported. Some have speculated that these could relate to Iraq. Can you comment more? Jacob Appelbaum: The 15,000 documents are part of the set of Afghanistan documents. They are being redacted for harm-minimizing purposes as requested by our source, and will be made available as is applicable with respect to the relevant security concerns. I would like to know much more about the process by which Wikileaks determines what is or is not safe/relevant to be released. While I support their efforts, they are walking a thin line in becoming the very thing they are working against. I only hope that the decision was done by experienced committee, but even that raises questions as to how do you vet that committee for the ability to gauge what is and is not releasable. How does one determine those people are even of adequate credentials/authority to read the information ahead of time? If I were our the US govt., I certainly would be playing up that issue as much as possible. On the surface it would not be hard to push general sentiment in the direction of Wikileaks playing information Gods with the data they have.

jongraehl 16y ago

Incredibly weak evidence.

long 16y ago
It's not weak evidence. There's no evidence. The guy doesn't seem to know how significance testing works.

djmdjm 16y ago

They should expect a bias, since Wikileaks have stated that some information has been held back deliberately. From http://wikileaks.org/wiki/Afghan_War_Diary,_2004-2010 http://wikileaks.org/wiki/Afghan_War_Diary,_2004-2010 "We have delayed the release of some 15,000 reports from the total archive as part of a harm minimization process demanded by our source. After further review, these reports will be released, with occasional redactions, and eventually in full, as the security situation in Afghanistan permits."

countersignaler 16y ago

this is a statistical blog post written by someone who has no idea how to do statistics. and even when told how to do it properly by a commenter, fails to accept the simple and overwhelming conclusion. why is this on the front page?

Confusion 16y ago
Perhaps because the data is interesting and we can draw the contrary conclusion: that there is no reason to suppose the data has been tampered with or fabricated?

mbateman 16y ago

> Also, as suggested in the comments, I used a chi-square goodness-of-fit test to see if the deviation is statistically significant, but it was not; with a p-value of 0.2303. Meaning we would fail to reject the null hypothesis: the observed data were a good fit for a Benford process. That said, the p-value is not so large as to suggest total adherence. Welcome to stats 101. If you get p > 0.05, it means your test failed to tell you anything. Which means your test failed to tell you about the likelihood of tampering.

whyenot 16y ago
You are on the right track, but no, it is more subtle than that. The test could still be meaningful depending on what probability of false positives you are willing to accept. [1] http://en.wikipedia.org/wiki/P-value http://en.wikipedia.org/wiki/P-value [2] http://en.wikipedia.org/wiki/Type_I_and_type_II_errors http://en.wikipedia.org/wiki/Type_I_and_type_II_errors

corruption 16y ago

An aptly named blog for the analysis.