6 ms·

I've been working on my own mturk-based proofreading system for OCR scans, modeled after Ryan Tate's typingpool. This is very close to a web-based version. Yo

I've been working on my own mturk-based proofreading system for OCR scans, modeled after Ryan Tate's typingpool. This is very close to a web-based version. Your illustrations suggest you chunk content to mturk at the paragraph level, is that correct? Did you try it at other levels of granularity, by the sentence, word, or N number of words? If so, what were the results? If not, why not? How did you settle on 4¢/word? Are you considering a mobile app/FB version of Proofville? What's the back-end? Is it your own system, or is it an existing crowdsourcing infrastructure like PyBossa? How many reviewers compare a passage? 1? 2? Best of N until there's consistency?

lmeyerov 13y ago
A housemate (linked below) runs http://captricity.com/ http://captricity.com/ -- crowdsourcing OCR is a really good idea. The collaborator for the main site used to work @ cap, and we've been evolving a REST API to simplify calling into mturk. For quality, we don't use arbitrary mturkers. Getting good results on mturk takes a lot of experimentation: I now think about it is as a combination of HR agency, traffic conductor, and street vendor. The challenges of building a useful & sustainable app on it are a bit different from what's addressed by crowdsourcing research papers, though that's not so surprising in retrospect. Sounds like this deserves a blogpost. It's been a fun summer project. Basically, nodejs with roll-our-own crowdsourcing algorithms, APIs, and UI widgets. A lot more to do, but it's finally good enough to address my proofreading needs, and I'd like to help others who also need to get some writing done asap :) Btw, we're $0.004 / word, not $0.04 ;-) That means we already cost 60% less than shops with comparable quality and slower turnarounds.
- vitovito 13y ago
  Yes, please do! I can only begin imagine some of the issues, having fed a couple thousand minutes through typingpool, but there isn't a lot of writing about using it at scale. I'll look forward to it.

lmeyerov 13y ago

A housemate (linked below) runs http://captricity.com/ http://captricity.com/ -- crowdsourcing OCR is a really good idea. The collaborator for the main site used to work @ cap, and we've been evolving a REST API to simplify calling into mturk. For quality, we don't use arbitrary mturkers. Getting good results on mturk takes a lot of experimentation: I now think about it is as a combination of HR agency, traffic conductor, and street vendor. The challenges of building a useful & sustainable app on it are a bit different from what's addressed by crowdsourcing research papers, though that's not so surprising in retrospect. Sounds like this deserves a blogpost. It's been a fun summer project. Basically, nodejs with roll-our-own crowdsourcing algorithms, APIs, and UI widgets. A lot more to do, but it's finally good enough to address my proofreading needs, and I'd like to help others who also need to get some writing done asap :) Btw, we're $0.004 / word, not $0.04 ;-) That means we already cost 60% less than shops with comparable quality and slower turnarounds.

vitovito 13y ago
Yes, please do! I can only begin imagine some of the issues, having fed a couple thousand minutes through typingpool, but there isn't a lot of writing about using it at scale. I'll look forward to it.