5 ms·

Chi-square goodness-of-fit, Benford’s law, and the Iranian election

31 points by mariorz 17y ago

earl 17y ago
Wow -- incanter looks terrible. All that code to run a chi2 test and a plot?
- liebke 17y ago
  The plots and chi-square tests are a single function each, it's the data preparation that takes up so much space. Alas, most statistical analyses involve a lot of preparatory steps, which are rarely shown in the final write-up.
  - earl 17y ago
    Since I recently did something similar in R: d <- read.csv(file='~/stuff/earlh/Iran_2009.csv', header=T, sep=',') lastDigit <- function(v){ v - 10*floor(v/10) } digits <- lastDigit( c(d$Ahmadinejad, d$Karroubi, d$Mousavi, d$Rezaee)) hist(digits, breaks=10) #chi2 gof tab <- table(digits) n <- length(digits) model <- chisq.test(x=tab, p=rep(0.1, 10)) model # hand generated -- check our work above ts <- 0 for(i in 1:length(tab)){ ts <- ts + ( tab[[i]] - 0.1*n)^2 / (0.1*n) } qchisq(p=1-0.076, df=9) and to be more specific: (def regions (sel votes :cols "Region")) (def ahmadinejad-votes (sel votes :cols "Ahmadinejad")) (def mousavi-votes (sel votes :cols "Mousavi")) (def rezai-votes (sel votes :cols "Rezai")) (def karrubi-votes (sel votes :cols "Karrubi")) # -or- attach(d) (def ahmadinejad (map first-digit ahmadinejad-votes)) (def mousavi (map first-digit mousavi-votes)) (def rezai (map first-digit rezai-votes)) (def karrubi (map first-digit karrubi-votes)) # -or- digits <- lastDigit( c(d$Ahmadinejad, d$Karroubi, d$Mousavi, d$Rezaee)) # not even attached -- could drop d$ etc. While it's obviously a matter of taste, it looks horridly verbose.
    - Confusion 17y ago
      Don't mistake explicitness for verbosity. Soon we'll see someone posting a Perl oneliner to do the same, claiming R is also verbose. Conciseness is not a goal, clarity is. if you have to expand the operations in your mind, you may well spend more time thinking about your code then when it's written fully, perhaps factored into a properly named function.
    - pygy 17y ago
      It could be done more concisely by using data structures and the (map fn struct) function, but it was expanded for clarty.
- fuzzmeister 17y ago
  JMP can do such analysis in a more GUI-oriented manner if you prefer. Here's a 30-day trial: http://www.sas.com/apps/demosdownloads/jmptrial8_PROD__sysdep.jsp?packageID=000503&jmpflag=Y http://www.sas.com/apps/demosdownloads/jmptrial8_PROD__sysde...
bint 17y ago
Commentators are “explaining” the Iran elections based on their own illusions, delusions, emotions, and vested interests. Whether or not the poll results predicting Ahmadinejad’s win are sound, there is, so far, no evidence beyond surmise that the election was stolen. However, there are credible reports that the CIA has been working for two years to destabilize the Iranian government. On May 23, 2007, Brian Ross and Richard Esposito reported on ABC News: “The CIA has received secret presidential approval to mount a covert “black” operation to destabilize the Iranian government, current and former officials in the intelligence community tell ABC News.” On May 27, 2007, the London Telegraph independently reported: “Mr. Bush has signed an official document endorsing CIA plans for a propaganda and disinformation campaign intended to destabilize, and eventually topple, the theocratic rule of the mullahs.” A few days previously, the Telegraph reported on May 16, 2007, that Bush administration neocon warmonger John Bolton told the Telegraph that a US military attack on Iran would “be a ‘last option’ after economic sanctions and attempts to foment a popular revolution had failed.” On June 29, 2008, Seymour Hersh reported in the New Yorker: “Late last year, Congress agreed to a request from President Bush to fund a major escalation of covert operations against Iran, according to current and former military, intelligence, and congressional sources. These operations, for which the President sought up to four hundred million dollars, were described in a Presidential Finding signed by Bush, and are designed to destabilize the country’s religious leadership.” --> http://www.counterpunch.org/roberts06192009.html http://www.counterpunch.org/roberts06192009.html Yep, a full blown infowar is what we have here.
- Devilboy 17y ago
  That's what I don't understand: Weeks before the election several polls (including ones by Western newspapers) predicted a 2 to 1 win for Ahmadinejad. These predictions were all fairly uncontroversial as far as I could tell. Now suddenly everywhere I look there's stories about young photogenic Iranians getting shot at while protesting the election results. I keep hearing about claims that the election was rigged but so far I've not seen anything concrete. The only way I can make sense of this is by assuming infowar / agent provocateur / other covert tactics employed by the USA. Iran has been a thorn in the side of America's middle-eastern foreign interests for a long time, and poking holes in it's democratic process is an important step. I can't even visit Reddit at the moment, every second story is about Americans getting angry at Ahmadinejad.
magoghm 17y ago
Walter Mebane uses Benford's Law on the second digit as a more reliable system to detect electoral fraud: http://www.umich.edu/~wmebane/pm06.pdf http://www.umich.edu/~wmebane/pm06.pdf Furthermore, it is interesting to note that he pointed out in 2006 that the results of the Mexican presidential election seemed fraudulent and he recommended a recount. But at that time the mainstream media seemed quite happy with the official winner, so they didn't talk about it, even when over two million Mexicans protested in Mexico City over the election results. The mainstream media decided to ignore all the statistical evidence of fraud, and just label the other candidate a a "sore loser".

31 points by mariorz 17y ago

earl 17y ago

Wow -- incanter looks terrible. All that code to run a chi2 test and a plot?

liebke 17y ago
The plots and chi-square tests are a single function each, it's the data preparation that takes up so much space. Alas, most statistical analyses involve a lot of preparatory steps, which are rarely shown in the final write-up.
- earl 17y ago
  Since I recently did something similar in R: d <- read.csv(file='~/stuff/earlh/Iran_2009.csv', header=T, sep=',') lastDigit <- function(v){ v - 10*floor(v/10) } digits <- lastDigit( c(d$Ahmadinejad, d$Karroubi, d$Mousavi, d$Rezaee)) hist(digits, breaks=10) #chi2 gof tab <- table(digits) n <- length(digits) model <- chisq.test(x=tab, p=rep(0.1, 10)) model # hand generated -- check our work above ts <- 0 for(i in 1:length(tab)){ ts <- ts + ( tab[[i]] - 0.1*n)^2 / (0.1*n) } qchisq(p=1-0.076, df=9) and to be more specific: (def regions (sel votes :cols "Region")) (def ahmadinejad-votes (sel votes :cols "Ahmadinejad")) (def mousavi-votes (sel votes :cols "Mousavi")) (def rezai-votes (sel votes :cols "Rezai")) (def karrubi-votes (sel votes :cols "Karrubi")) # -or- attach(d) (def ahmadinejad (map first-digit ahmadinejad-votes)) (def mousavi (map first-digit mousavi-votes)) (def rezai (map first-digit rezai-votes)) (def karrubi (map first-digit karrubi-votes)) # -or- digits <- lastDigit( c(d$Ahmadinejad, d$Karroubi, d$Mousavi, d$Rezaee)) # not even attached -- could drop d$ etc. While it's obviously a matter of taste, it looks horridly verbose.
  - Confusion 17y ago
    Don't mistake explicitness for verbosity. Soon we'll see someone posting a Perl oneliner to do the same, claiming R is also verbose. Conciseness is not a goal, clarity is. if you have to expand the operations in your mind, you may well spend more time thinking about your code then when it's written fully, perhaps factored into a properly named function.
  - pygy 17y ago
    It could be done more concisely by using data structures and the (map fn struct) function, but it was expanded for clarty.
fuzzmeister 17y ago
JMP can do such analysis in a more GUI-oriented manner if you prefer. Here's a 30-day trial: http://www.sas.com/apps/demosdownloads/jmptrial8_PROD__sysdep.jsp?packageID=000503&jmpflag=Y http://www.sas.com/apps/demosdownloads/jmptrial8_PROD__sysde...

bint 17y ago

Commentators are “explaining” the Iran elections based on their own illusions, delusions, emotions, and vested interests. Whether or not the poll results predicting Ahmadinejad’s win are sound, there is, so far, no evidence beyond surmise that the election was stolen. However, there are credible reports that the CIA has been working for two years to destabilize the Iranian government. On May 23, 2007, Brian Ross and Richard Esposito reported on ABC News: “The CIA has received secret presidential approval to mount a covert “black” operation to destabilize the Iranian government, current and former officials in the intelligence community tell ABC News.” On May 27, 2007, the London Telegraph independently reported: “Mr. Bush has signed an official document endorsing CIA plans for a propaganda and disinformation campaign intended to destabilize, and eventually topple, the theocratic rule of the mullahs.” A few days previously, the Telegraph reported on May 16, 2007, that Bush administration neocon warmonger John Bolton told the Telegraph that a US military attack on Iran would “be a ‘last option’ after economic sanctions and attempts to foment a popular revolution had failed.” On June 29, 2008, Seymour Hersh reported in the New Yorker: “Late last year, Congress agreed to a request from President Bush to fund a major escalation of covert operations against Iran, according to current and former military, intelligence, and congressional sources. These operations, for which the President sought up to four hundred million dollars, were described in a Presidential Finding signed by Bush, and are designed to destabilize the country’s religious leadership.” --> http://www.counterpunch.org/roberts06192009.html http://www.counterpunch.org/roberts06192009.html Yep, a full blown infowar is what we have here.

Devilboy 17y ago
That's what I don't understand: Weeks before the election several polls (including ones by Western newspapers) predicted a 2 to 1 win for Ahmadinejad. These predictions were all fairly uncontroversial as far as I could tell. Now suddenly everywhere I look there's stories about young photogenic Iranians getting shot at while protesting the election results. I keep hearing about claims that the election was rigged but so far I've not seen anything concrete. The only way I can make sense of this is by assuming infowar / agent provocateur / other covert tactics employed by the USA. Iran has been a thorn in the side of America's middle-eastern foreign interests for a long time, and poking holes in it's democratic process is an important step. I can't even visit Reddit at the moment, every second story is about Americans getting angry at Ahmadinejad.

magoghm 17y ago

Walter Mebane uses Benford's Law on the second digit as a more reliable system to detect electoral fraud: http://www.umich.edu/~wmebane/pm06.pdf http://www.umich.edu/~wmebane/pm06.pdf Furthermore, it is interesting to note that he pointed out in 2006 that the results of the Mexican presidential election seemed fraudulent and he recommended a recount. But at that time the mainstream media seemed quite happy with the official winner, so they didn't talk about it, even when over two million Mexicans protested in Mexico City over the election results. The mainstream media decided to ignore all the statistical evidence of fraud, and just label the other candidate a a "sore loser".