34 ms·
The drama in trying to convert election PDFs to Spreadsheets
- SergeAx 4y agoPdf is a very unfortunate format. It is proprietary, it is paper-oriented, its almost single goal is to keep precise printing layout. But for the last 30 years world didn't come up with anything that could compete.
- segfaultbuserr 4y agoPDF isn't the actual problem in this particular case. The documents here are photographs taken at different camera angles, embedded in PDFs.
- jxramos 4y agoI was going to say, using alt drag to select vertical columns is usually how I extract useable tables out from pdfs with embedded tables.
- londons_explore 4y agoIsn't things like this the reason that the UN provide election observers? By spot checking just a random 100 votes are correctly tallied, you can be pretty sure the outcome of the election is legit in a > 10M voter country.
- Someone 4y ago> By spot checking just a random 100 votes are correctly tallied How do you do that? I think the only error you could detect is when the tally has fewer votes for a party than what’s in that sample. If so, a fraudster could report 100 votes for every party, and add the remaining to whatever party they want to win.
- londons_explore 4y agoYou have to design the election system with this in mind. One such design would be for every vote to have a unique id. When announcing the results, you also publish a list of which vote ids were tallied for which candidate. Then you have 100 random ids, and the checkers watch those votes all the way from the voter casting them to the final tally.
- orf 4y agoFantastic story! Did the results get used in a claim?
- kevviiinn 4y agoWow what a cliffhanger, it sounds like they have to deal with the courts now. I hope we get an update https://www.msn.com/en-us/news/world/opposition-files-petition-against-nigerian-election-result/ar-AA18Uc7z https://www.msn.com/en-us/news/world/opposition-files-petiti...
- davedx 4y agoIncredible story. Some more background: https://ng.usembassy.gov/nigerias-2023-elections/ https://ng.usembassy.gov/nigerias-2023-elections/
- davedx 4y agoMore background. OP is an impressive entrepreneur! Massive kudos. https://markessien.com/projects/hotels-ng/ https://markessien.com/projects/hotels-ng/
- mattlutze 4y agoThis was thrilling. Sometimes, one person's bug is another person's feature :)
- vincheezel 4y agoI hope for (but do not expect) a positive outcome
- blntechie 4y agoWhat was the final result numbers from the transcription?
- toyg 4y agoThey're probably going to be similar to the 14k sample he tweeted: a solidified Labour Party getting 50-55% of the votes, and the establishment candidates splitting the rest.
- MontagFTB 4y agoSo the bug where the first voting sheet shown to a user was from the same 10% of the photos turned out to be a feature, serving as a CAPTCHA of sorts to weed out the bad actors from the good. If memory serves, some CAPTCHA techniques include showing two numbers to transcribe, where one’s value is already known. If that number is transcribed incorrectly, then the other number’s result isn’t used, and the CAPTCHA fails. Perhaps a similar technique may have also helped here?
- malborodog 4y agoCan you explain that again differently? I didn’t understand that captcha point. It feels important though.
- wodenokoto 4y agoOriginal captcha was built around transcribing text that ocr tools failed at So I give you two words to transcribe to prove you are human. I know one of them and I want to know the other.
- czx4f4bd 4y agoI think they're referring to the old reCAPTCHA v1 approach. From https://en.wikipedia.org/wiki/ReCAPTCHA https://en.wikipedia.org/wiki/ReCAPTCHA: > The original iteration of the service was a mass collaboration platform designed for the digitization of books, particularly those that were too illegible to be scanned by computers. The verification prompts utilized pairs of words from scanned pages, with one known word used as a control for verification, and the second used to crowdsource the reading of an uncertain word.
- Spare_account 4y agoThis approach was part of their strategy: >Then we started showing some results we knew to the bots - if they entered wrong numbers, we would stop accepting the results.
- didgetmaster 4y ago
- favaq 4y ago[flagged]
- clipper_janosch 4y agoWhat an exceptional story. You are a legend.
- djoldman 4y agoChecking one at random: https://docs.google.com/spreadsheets/d/1HhV9iJxXTU9liAZPIDoMOZ93L48uZRnnT13G-lqI150/edit#gid=786971613 https://docs.google.com/spreadsheets/d/1HhV9iJxXTU9liAZPIDoM... ...shows 0s in the first row for all candidate parties. But the corresponding photo shows votes for all three: https://inec-cvr-cache.s3.eu-west-1.amazonaws.com/cached/results/2676/result_46011_1677492816_thumb.jpg https://inec-cvr-cache.s3.eu-west-1.amazonaws.com/cached/res... I hope it's not a mistake and that there's some arcane law/technicality to explain it. edit: another mistake on row 21, LP should get 25 but it was credited to NNPP: https://docs.inecelectionresults.net/elections_prod/1292/state/1/lga/3120/ward/17712/pu/2707/2707-1677468711.pdf https://docs.inecelectionresults.net/elections_prod/1292/sta...
- dan-robertson 4y agoYeah looks weird. When I scrolled to a random part, the numbers seemed to line up. They didn’t say things were entirely correct though. Perhaps the data quality is sufficient for a challenge. Odd that the first rows seem more wrong though.
- hoseja 4y agoSilly, you don't malcount the actual votes, you brainwash the population and pervert the process until they vote the way you want them to, like in the advanced first world democracies.
- avodonosov 4y agoThat's not the worst case, if wise elite brainwashes (manufactures consent of) the population. Worse is when the elite is not so wise (sometimes plainly crazy), or the elite loses control to crazy people, adversaries. Or self-induced mass hysteria of the population. The direct "democracy" that very soon will inevitably be enabled by technology, poses great dangers in the situation where masses are so easily manupulateable, and their collective intelligence seems not raising above individual level, but degrading below it for some reason. Violent chaos, lynch courts, etc.
- prhrb 4y agoWhat a scam by the ruling political party
- jgtrosh 4y agoThe context should be dated to 2020, not 2023 Edit: it was now corrected, no need to downvote Great story! Looking forward to some follow up
- public_defender 4y agoI don't understand. The article says the SARS protests started in 2020 and the election was in 2023. This seems correct.
- jgtrosh 4y agoYes, it was now corrected
- dec0dedab0de 4y agoThis would have been a good use for hn style shadow banning. Especially if they didn't publish the current tally, then the original easy to detect bots may have never realized you were on to them
- thread_id 4y agoFantastic story. What an excellent example of democratization from technology. And also a perfect example of how the blade cuts both ways. Digital warriors battling it out in real time and the stakes are enormous. Great respect for Mark and his ingenuity and adaptive responses!!!!
- mmmuhd 4y agoElupee 75, To be frank, you did a great job and i am proud of someone from my country pulling this off, but the bitter truth is President Elect Bola Ahmed Tinibu won this election. Peter Obi's youth support is predominantly in the south, and Christian majority parts of the country, he clearly lack support in the Muslim north, where I am from. I voted for Kwankwaso though.
- deleted 4y ago[deleted]
- bmsleight_ 4y agoCan you expand on " he clearly lack support". Bonus points for facts over opinions.
- mmmuhd 4y agoClearly means even his Vice Presidential Candidate could not win his own polling unit, polling unit, not ward, not Local Government, not State. https://punchng.com/nigeriaelections2023-datti-loses-polling-unit-to-pdp/ https://punchng.com/nigeriaelections2023-datti-loses-polling...
- vuln 4y ago[flagged]
- sd9 4y agoI was naturally skeptical of the punchng article, so I crosschecked it against OP's CSV. The votes in the article do agree with OP's CSV (although the number of accredited voters differs slightly). The crosschecked results are in KADUNA_crosschecked on line 3800. The image is here: https://inec-cvr-cache.s3.eu-west-1.amazonaws.com/cached/results/56603/result_118723_1677530826_thumb.jpg https://inec-cvr-cache.s3.eu-west-1.amazonaws.com/cached/res... Accredited voters: 276, Registered voters: 750, APC: 98, LP: 54, PDP: 102, NNPP: 11 All that said, I don't think that the results for 276 voters in one polling unit in one ward in one local government area in one state is clear evidence that Obi lacks support. If anything, the fact that OP's CSV matches a (potentially biased) news article gives me more faith in OP's tallies and claims. (Aside: it seems _easier_ to lose an election in your own polling unit, where variance plays a larger part, than it is to lose on a wider scale.)
- nivenkos 4y agoThis is a great example of why electronic voting is important and can help secure democracy.
- TazeTSchnitzel 4y agoOn the contrary, eletronic voting doesn't create the paper trail necessary to dig up frauds like this. You can simply program or hack the system to report any vote total you want.
- SkeuomorphicBee 4y agoFirst of all, hacking the electronic system is much much harder than hacking the paper process. In the case at hand the paper tallying process was the one hacked. And second, electronic systems can create a paper trail, just make the electronic machine spit out a paper receipt. Then you have the best of both worlds, you can have instant electronic totals, and then do some random sampling recounts of the receipts to validate the result.
- marcosdumay 4y agoScaling an attack against paper is incredibly difficult, and requires coordination in a level that is almost sure to trigger the law enforcement much before it can change some national-level numbers. Scaling an attack against a computer system is almost the same as doing an attack against a computer system. Few attacks don't scale. But yeah, if you just print the vote and push it into an urn (while the voter can read it), you'll get the best of both worlds.
- TazeTSchnitzel 4y ago> just make the electronic machine spit out a paper receipt Now the person making the vote has to check the receipt matches their input, and they probably don't have a practical form of redress if it doesn't.
- logifail 4y ago> This is a great example of why electronic voting is important and can help secure democracy. If those in power are against change, I wouldn't want to have to put my trust in electronic voting if I was hoping for change. I was left with the impression that it is the paper records in this story that led to the unravelling of an attempt to forge the results. Long live paper ballots.
- churchill 4y agoOh, and Mark didn't mention that Bola Ahmed Tinubu was indicted for heroin charges in the US in 2003, forfeited $460k & is just too old to run a democracy this size. Atiku Abubakar (second candidate) was a former VP and the president he served under (Obasanjo) still insists the dude remains a monument to corruption. There's been a coordinated campaign at all levels to rig this election massively and we saw voter intimidation, manipulation in broad daylight, and the acquiescence of foreign governments to it all.
- churchill 4y agoI meant heroin trafficking
- mmmuhd 4y ago[flagged]
- churchill 4y agoWhy not debunk everything I just wrote instead of attacking me personally? Google is your friend and you can verify everything I said about: Tinubu's drug trafficking indictment: https://www.bbc.com/news/world-africa-61732548 https://www.bbc.com/news/world-africa-61732548
- nimajneb 4y ago[dead]
- klooney 4y agoAlso, this is ridiculous > he became an "instant millionaire" while working as an auditor at Deloitte and Touche.
- churchill 4y agoDeloitte denies having a record of ever employing him, like you can see here [0]. [0]: https://pbs.twimg.com/media/FhhvN-fXEAAOTOK?format=jpg&name=medium https://pbs.twimg.com/media/FhhvN-fXEAAOTOK?format=jpg&name=... Tinubu claimed to be making $850k in annual pre-tax bonuses working for Deloitte. Today, Directors at Deloitte make 340k total comp annually, according to Glassdoor, and that's before you factor in inflation. What type of joke is this?
- londons_explore 4y agoThe votes surprise me... In many regions one party gets 90+% of the vote. Assuming the numbers are correct, then it suggests that most people are easily swayed by their local peers. Is that common in say the USA?
- mmmuhd 4y agoExactly! and this mostly happened in the regions where the OP's preferred candidate won. This is clear scam.
- muyuu 4y agoIt happens in the US too. Tribalism and ideological clustering are so similar, they are being used interchangeably these days. But in some traditional countries there are literal clans and tribes voting in blocks.
- anigbrowl 4y agoYep, bloc voting can be habitual or strategic. There's a town in Northern California where the majority of the seats on the council is held by people who all happen to attend the same megachurch.
- crazygringo 4y ago> it suggests that most people are easily swayed by their local peers. That feels like a particularly uncharitable interpretation to me. I think it's more along the lines of that parties and their policies have very different impacts on different regions. So it makes sense to vote on what is beneficial to your region, and a lot of people will agree on that. So it's not about susceptibility to being "swayed", but genuine policy affecting regions differently.
- churchill 4y ago-
- churchill 4y ago-
- mmmuhd 4y agoDavid Hundeyin is a deceitful, lying criminal, so don't bring his "Content" as any kind of evidence. https://www.icirnigeria.org/controversy-as-oxford-terminates-association-with-onyeka-nwelue-david-hundeyin/ https://www.icirnigeria.org/controversy-as-oxford-terminates...
- deleted 4y ago[deleted]
- pxc 4y ago@dang are these '-' comments an attempt to evade showdead?
- roschdal 4y agoThe people who cast the votes don't decide an election, the people who count the votes do. - Stalin.
- pxc 4y agoIn case the downright cartoonish character of this quotation made anyone else wonder if it were fake... that quotation is, indeed, fake: https://web.archive.org/web/20220128105324/https://www.politifact.com/factchecks/2019/mar/27/viral-image/no-joseph-stalin-didnt-say-statement-about-electio/ https://web.archive.org/web/20220128105324/https://www.polit...
- mtrovo 4y agoIs the access to the original photos open? It might be fit for a good Kaggle competition, although maybe a little too late for this current election.
- jasonjayr 4y agoFrom the article, it seems like the rush was to collect enough evidence to file a challenge within the legal timeframe. With a challenge filed, it seems like there is a bit more time to verify claims + other evidence. (I know nothing of the system of government there, but) -- it seems like the prudent thing to do would be for the courts to mandate a neutral verification of each of those paper sheets. (ie, 10 trusted representatives from each party re-key the figures manually).
- olabyne 4y agoIf you want, you have exactly the same issue to solve with Kenya last year. The pictures of all of the voting sites are available, but the country went to chaos to pick a winner. It is crazy , because on the lower level (in voting offices), the vote process was respected and the numbers are trustworthy, but the higher you go and the more corruption happens, as each aggregation of data removes trust to the system.
- seventytwo 4y agoWow, this was a fantastic read! I have no idea what’s going on in Nigeria, but I hope the truth (whatever it is) will prevail!
- hardlianotion 4y agoThat is a great job - well done from a grateful Nigerian.
- bundie 4y agoI did not know that Nigerians used Hacker News :-D Most people I encounter on this site are oyinbos.
- hardlianotion 4y agoWe are everywhere and cannot be avoided.
- salawat 4y agoOn the Internet, no one knows you aren't a Nigerian Prince. Which is disappointing. Every time I meet an actual Nigerian, I always expect ruffles and platform shoes. I'm not sure I haven't horribly misunderstood something somewhere...
- redman25 4y agoThis might be a sensitive question but I wonder if something like this would work in the United States? With all of the fears of election interference why not trust but verify?
- charles_f 4y agoWould you trust the recount? I mean, the only way to engage the number of people you need to do that kind of recount is by having them very pissed, so most likely feeling like their party was wronged and therefore the thing is partisan by essence. If you're on the winning party you wouldn't trust the numbers the others give you anyhow, so what's the point
- pjc50 4y agoGenuinely the US would do better if it had paper elections with a handcount with observers. The system works in the UK just fine. Unfortunately, there's a category of people in both the US and Nigeria who use "election interference" to mean "accurately counting the votes".
- bunabhucan 4y agoThe distrust is not based on evidence. Actual election fraud is incredibly rare in the US, typically things like someone owning property in two states and voting the state ballot in one and federal in the other. Getting two ballots is legal but using both is jail time. Typical solo offender is a conservative white male.
- rqtwteye 4y agoI still don't understand how we ended up with PDF as sort of standard to archive data. PDF is already pretty bad for things like manuals but for things like spreadsheets we basically collect the data, then we destroy all the structure by putting it in into POF, and later on we painstakingly try to restore the data from PDF which is often almost impossible to do with accuracy. It just shows that bad solutions often win.
- layer8 4y ago> how we ended up with PDF as sort of standard to archive data. I don’t think we really did. They are a standard for archiving typeset page-based documents. Of course, paper documents used to be standard for archiving data, and some continue to do so in the form of PDF. In principle, it is possible to integrate all the structure you want in a PDF (using Marked Content, Structure Attributes and User Properties), but for data (as opposed to document structure) you’d need custom software to generate and interpret that.
- codeulike 4y agothese are just photos embedded in a PDF, which actually isn't that bad an idea, because it lets you scan multiple pages and join them together as a 'document' (not sure if the documents in OP had several pages, but if you've scanned/photographed a multi-page document, PDF is not that bad of a solution)
- SilverCode 4y agoA better option would be to use the TIFF format. You can use it as a container format to store lossless and lossy image formats, and handles multiple images in a single container. It was the standard for scanners until PDF seemed to dominate the scene.
- hunter2_ 4y ago> It was the standard for scanners until PDF seemed to dominate the scene. Probably because it's much easier (for average users with few tools and skills) to print a PDF than to print any sort of non-page-based (e.g., image) file format and have the resulting sheet of paper match the scanned sheet of paper in terms of scale, orientation, position -- assuming both sheets are the same dimensions. Essentially using the file as an intermediary for physical copying of standard paper documents.
- snvzz 4y agoNot providing CSV is at the level of criminal negligence.
- OoTheNigerian 4y agoNice read. It's important to note 1.The 2020 protesters did not begin vandalizing property, but government infiltrated the protests by burning cars and maiming people. 2. The Obidient movement encompassed multiple sub movements of which a part of the #EndSARS was one of them. A vast majority of Peter Obi's supporters were not #EndSARS activists. 3. Elections in Nigeria are fraught with treacherous behavior so everyone suspects everything. It's important to be very careful with your communication. There is a lot of desperation in the land and so if in a position of information leverage, the responsible thing is to handle the privilege with care and transparency.
- harvey9 4y agoThis is some compelling writing. I know this has real life implications for real people so I hope it's not in poor taste to say it would make a good movie.
- cwkoss 4y agoI agree, but still needs an ending! Will this be a story of triumph or tragedy?
- crazygringo 4y agoFirst of all, what a fantastic and inspiring read. But, I'm left greatly confused -- the article never states whether this changed the result. It says that halfway through counting Obi was in the lead, but nothing about when finished counting. And when I look at the spreadsheet, the last row (#3380) appears to be the totals, which lists: APC LP PDP NNPP 149014 85748 329030 8305 Which shows LP (Obi) in third place, just like the official results. So what point is the article trying to make at the end of the day? Or have I misunderstood the numbers?
- sd9 4y agoThose are the results for just one state, Adamawa. However, like you I don't know what the overall results are; I agree that the article could make this clearer.
- crazygringo 4y agoOh thanks for clarifying. Turns out the link to the folder for all the states is here: https://drive.google.com/drive/folders/173oHgms6wYy5WKz_i3Lhl5mXcmobCWHz?usp=sharing https://drive.google.com/drive/folders/173oHgms6wYy5WKz_i3Lh... But there doesn't appear to be any file that calculates the nationwide totals. It just seems like such a strange omission but I'm on mobile and can't add up the numbers from across a ton of different files myself.
- didgetmaster 4y agoI downloaded all the .CSV files from that site and quickly loaded them into a table. It just took a couple minutes, but I didn't stop to verify that there were not duplicate rows across the various files. When I added up the totals, I got: APC - 7,225,399 LP - 5,286,181 PDP - 5,285,900 NNPP - 1,529,575 Note: I was using a beta version of a new database tool I created to do this.
- jxramos 4y agoshould be something quick to whip up in a few minutes in pandas I'd think assuming the column headers are identical and in the same order. It would translate into a bunch of pandas concat call and with the merged table a value_counts for the column where the vote is retained.
- neves 4y agoIs it true that USA does not have a open data law to make everybody publish in CSV?
- JumpCrisscross 4y ago> Is it true that USA does not have an open data law to make everybody publish in CSV? American elections are de-centralised. Each state comes up with its methods. In some, each county. (I'm not sure how publishing a CSV of vote totals would help.)
- pxc 4y agoI'm impressed by the courage of the protesters here, and the tenacity of the youth voters. I hope they get a clear answer and a fair count, and whether they win this time or not, a real shot at cracking up their corrupt, two-party system.
- tr33house 4y agoI'd tried something like this with the Kenyan election but our setup was to use OCR (google cloud) -> text -> parse -> sqlite We started late so the results were out when we finished but I think it'll be a good idea to develop software that can parse the PDF results and display them faster than the electoral bodies can. In Kenya, and Nigeria, the delays cause a lot of anxiety
- pjc50 4y agoStriking reminder of how big the world is that while I had heard of #EndSARS, I hadn't realised the scale of the political violence in Nigeria nor that it had its own Bloody Sunday-scale massacre.
- throwaway81523 4y agoI've done stuff like this semi manually. Use pdftotext to get the text tables out of the pdf, eyeball it and massage with emacs keyboard macros, and in some cases python scripts. It's not that big a deal but it is somewhat ad hoc. I know that OCR software is able to read stuff like magazine articles and figure out column layout, embedded charts, etc. It's weird if is nothing to do that with a pdf. Maybe I'll look around or see if I can hack up something.
- infinityio 4y agounfortunately in this case the text content was handwritten, not computer-generated
- YeGoblynQueenne 4y ago>> We had a brainstorming meeting, and decided to try a new approach. We would simply ask the Obidients to help us do the conversion. If hundreds of Obidients did the transcription, it would go fast. What would guarantee that the Obidients would not, in turn, try to inflate the score of the Labor candidate?
- munchler 4y agoThey planned to transcribe each PDF multiple times in order to validate the results.
- dejongh 4y agoWow. Wild story. Thanks for sharing. Cool twist that a bug ended up identifying the bad guys.
- jxramos 4y ago> Then ominously, on the 20th of October of 2020 some people drove there in unmarked cars and removed all the Cameras installed at the tollgate. They at least capture some photos of the equipment. I wonder if anyone communicated with the individuals.