7 ms·
Company using Mechanical Turk botches U.S. Senate campaign finance records
- vitorbaptistaa 8y agoA bit off-topic, but as an example of what could be done with this kind of information, the Operation Serenata de Amor from Brazil created a robot that analyses the reimbursement claims from Brazilian politicians looking for outliers, filing complaints and tweeting the politicians to ask for clarifications. There have been some pretty funny conversations between politicians and twitter bots because of this :) https://serenata.ai/en/ https://serenata.ai/en/
- jessaustin 8y agoThey argue about reimbursements over Twitter? That seems like a topic for private communication methods. Email bots are also a thing.
- mmt 8y ago> That seems like a topic for private communication methods. Since the topic is public officials acting in their official capacity, I disagree.
- Dowwie 8y agoThis is a political issue more than a technological one.
- komali2 8y agoAgreed, especially considering >Reform advocates say this is in large part because of opposition by a small group of Senate Republicans, most notably Senate Majority Leader Mitch McConnell which is exactly what I'd expect from the dude, but not something I feel like screaming about on HN.
- danso 8y agoI’d say it’s both. The availability of purportedly cheap labor and tech make it easier for the Senate to push off a real solution. It’s also an interesting bit of bureaucratic trivia, in how the House and Senate approach this issue (and other data problems) in completely different and independent ways.
- cdoxsey 8y agoShouldn't they be running each scan through the mechanical turk multiple times to remove errors? I guess that would double or triple their costs.
- danpalmer 8y agoThis is the dirty secret of Mechanical Turk – you're supposed to run it through multiple times according to much of the documentation, but people don't because then it becomes expensive enough that it's no longer an attractive option. I work in a company who have considered it many times for various things, but between the Mechanical Turk API/tech being pretty terrible, and it all being expensive and low quality, we always either end up getting a temp in for a day or two to sit in front of Excel and tidy up data, or if it's a bigger process we outsource it to a data processing company in Bangladesh where we can have dedicated people on our account who sit in a shared Slack channel and who we can train.
- Kagerjay 8y agoHow expensive has mechanical turk gotten though over the years? I thought this was standard practice as well, running it multiple times to reduce human error and do random spot checking. What is the current cost per HIT over years previously?
- danpalmer 8y agoThe last time I looked at this, a few years ago, it was ~$0.10 per HIT, and 2-3 would be needed, and that was for very simple data processing. We have quite complex data processing requirements with multiple interdependent fields, and UI, which would have increased the processing time, so I'd have guessed $1 per item processed total, plus extensive integration time. Our outsourcing gives is far better communication and the ability to train staff doing the processing over time, feedback on their performance, and help them get better. I don't know the figures, but I suspect it's a similar price but with far better accuracy, but we do have enough consistent work for this to make sense - if we were more spikey in our demand then it might not.
- 8y ago
- slics 8y agoThere are so many factors in play here. Government issues a contract for scanning and loading all the paperwork. Contractor wins contract (prime), he then hires sub-contractors to do the their work, which in return can hire some other sub-contractor. If you think about it from a requirement perspective, government asks for apples, by the time those requirements get triaged three levels deep, they get oranges. Government attitude for a shitty product delivered, you can still make juice out of the fruit we got. They still keep pumping yearly, millions of dollars for that same minimal viable product.
- mirimir 8y agoChecking data for amount=ID seems pretty trivial. Also looking for outliers. And of course, for records with nothing in name fields.
- esseti 8y agoa did part of my PhD on crowdsourcing and the poor quality results still remain a problem ;)
- mygo 8y agoI remember back when Yahoo Answers was the go-to for crowd-sourced Q&A. Nobody was getting payed and most answers were absolutely terrible. Then Quora came out. Nobody was getting paid but suddenly experts were answering questions in their field. Wonder if there could be a Quora for MTurk
- booleandilemma 8y agoI think this is the first positive thing I’ve ever read about Quora anywhere.
- JoelTheSuperior 8y agoI mean, it's not perfect - I have seen my fair share of "experts" but I would say as a general rule the answers are fairly high quality.
- ianhawes 8y agoMy understanding is that Senate campaigns can file electronically, but are not legally required and therefore do not.
- eli 8y agoBecause they want their records to be hard to read and analyze. This is a policy/political problem not an OCR one.
- cabaalis 8y agoAlso, it directly benefits the senators for the company hired to process the data to fail in doing so. Any issues raised from the dataset can immediately be refuted.
- txcwpalpha 8y ago> Captricity turns these images into machine-readable text through what its website calls a “groundbreaking collaboration between humans and computers.” > Captricity administers this kind of work through Mechanical Turk, an Amazon-owned online labor marketplace. Ah yes, "groundbreaking collaboration between humans and computers", which in this case actually means "pay a human below minimum wage to type numbers on a keyboard". Gotta love corporate marketing speak. The only thing that would make this even more ridiculous is if Captricity claimed they were using the mystical "machine learning" too. Hmm, let's check their website. [1] >Captricity then uses sophisticated machine learning to package up these fields (we call them “shreds”) into quickly identifiable packets. ...lol 1: https://support.captricity.com/blog/captricitys-secret-weapon-crowdsourcing/ https://support.captricity.com/blog/captricitys-secret-weapo...
- pavel_lishin 8y agoBut are the results stored in the blockchain?
- redleggedfrog 8y agoIn the cloud?
- kbenson 8y agoNo, no, no! They print the blockchain to paper so it can be duplicated and stored in multiple secure physical locations. Blockchain in the cloud is so last year.
- SaltyBackendGuy 8y agoA plant based media storage ledger, with redundancy. I smell a crypto startup.
- nannal 8y agoI wonder which generation the software is ... from?
- kokey 8y agoOh this reminded me immediately of this story about a data migration project https://thedailywtf.com/articles/Importing-Data-the-WTF-Way https://thedailywtf.com/articles/Importing-Data-the-WTF-Way
- ohashi 8y agoCaptricity is terrible on MTurk. They reject and don't pay. A lot of people who regularly work on MTurk avoid them because they are a scummy company. Read their TurkOpticon reviews or on Reddit or other MTurk communities. I hope they disappear.
- seveibar 8y agoMechanical Turk doesn't have built-in quality assurance so unless you're very skilled in QA systems or have a lot of money to burn on validation it's going to give poor results. Btw it's relatively hard to build a QA system that validates work, establishes trusted workers and optimizes for cost without significant redundancy. I do the technology at an MTurk competitor that automates the quality assurance/training process and pays fair wages to refugees to do the work (workaround.online) if anyone is interested in an alternative to MTurk that is still relatively cheap I would highly recommend it.
- stuaxo 8y agoMechanical Turk is basically laundering of grossly underpaid labour.
- logfromblammo 8y agoI once did campaign finance data entry as a child laborer for a newspaper reporter. I entered all the contribution reports for all the the state house and senate races. It was crates upon boxes of public-record paper documents. According to the reporter, I probably only made one mistake in the dollar amounts, due to a single missing contribution report. Not bad for a kid. But it took forever, and it was relatively expensive for the newspaper. This was way back in the early 1990s. Since then, paper filings tapered off, and electronic filings replaced them. It really is a far better way to do it. Clearly, these records need to be digitized to create public transparency, and that need is apparently being met by bottom-feeding tax-eaters doing a minimum-effort job at a top-shelf price. I am not surprised these records are being botched, but I am surprised this story is coming out only now, rather than back in 2001.
- apendleton 8y agoFor a bit more background: in general, House and Senate campaigns use the same software to manage their campaign contribution data, and this software is capable of doing electronic submission (because the House requires it), but the Senate instead prints it all out and hands it in paper, because they're the worst, and then it all gets typed back in again at a cost to the taxpayer of about $250k per filing deadline (how many of those per year there are varies -- they get more frequent closer to elections). The main motivation for the unwillingness to change things here isn't that they like the inaccuracies so much as that they like the delay -- it effectively means the last couple of pre-election filings aren't public until after the election, rather than, you know, instantly.
- everybodyknows 8y agoArticle says McConnell is the blocker of electronic filing. Anyone know about his own campaign finances in the last few months before elections?
- apendleton 8y agohttps://www.opensecrets.org/ https://www.opensecrets.org/ is a good resource, but honestly (having worked on money on politics stuff for several years) none of it is that juicy in isolation. Everybody gets money from everybody. For the most part they're not scared of any particular story coming out, they'd just rather not have a "In the final days of the campaign, X for $Y from Z" story come out the day before the election at all, for any Y or Z (it looks bad pretty much regardless of who it is: banks, pharmaceuticals, defense, lobbyists, energy, whatever)