4 ms·

Show HN: Proofville, A Competitive Editing Game – Crowdsource Your Proofreading

36 points by lmeyerov 13y ago

ams6110 13y ago
Spell and grammar check have been part of any decent word processor for quite a while now.
- thatswrong0 13y ago
  But they can't deeply check style, clarity, or other such qualitative aspects. Word won't tell you when your sentence is difficult to read or confusing.
  - lmeyerov 13y ago
    We've caught a lot that word processors haven't, should make a gallery :)
svetha 13y ago
I absolutely love this. This is about way more than spell and grammar check. It is not just proofreading. Now the crowd can be a test audience to help make content more readable and digestible. I hope this works out!
ivan_ah 13y ago
This is very interesting. Proofreading as a service could be used by so many content companies out there. I guess for something long like a book it would make more sense to have the same person proofread the entire thing, but for blog posts and other text, parallelism could be very powerful. I'm not sure how the pricing will work out though: $0.004/w = $1.4/page = $350 for my book this is 1/6 of what I paid to my copy editor. Who will work for such a low wage? Last but not least, does it all have to be through-the-web. I find I catch a lot more mistakes on paper. Either way, I wish you good luck with your business and I will be following the developments closely.
- lsb 13y ago
  Sometimes copyeditors fix really subtle problems, sometimes copyeditors fix really obvious problems. This should help fix the obvious problems, and make their time better-spent!
  - lmeyerov 13y ago
    Right, looking at crowdsourcing (the main proofist site) was born out of my frustration that most copyediting comments are for simple mistakes that anyone could have caught. I want to make ours cheap enough that you can get 10-20 pairs of eyes per piece of text and without the usual wait. This game is an experiment to make it free, and if we get mass, can get the desired level of matching. (In just one hour, we're already a hit with educators in math/science!)
gojomo 13y ago
Related: the Soylent Word Processor ("It's got humans inside!") from the MIT CS & AI Laboratory: http://projects.csail.mit.edu/soylent/ http://projects.csail.mit.edu/soylent/
- lmeyerov 13y ago
  Definitely! Also, there are a couple of wonderful followup projects that think about how to make crowdsourcing affordable (each task still costs money): http://www.cs.berkeley.edu/~kuangc/ http://www.cs.berkeley.edu/~kuangc/
cdcarter 13y ago
Is there a way to proofread something without having to upload your own text? Now I'm in the mood to edit some copy!
- lmeyerov 13y ago
  There is for the main site. I like the idea of different matchmaking modes for the game. A player suggested allowing more than two reviewers working together in a session, which may be one fun way to do it!
vitovito 13y ago
I've been working on my own mturk-based proofreading system for OCR scans, modeled after Ryan Tate's typingpool. This is very close to a web-based version. Your illustrations suggest you chunk content to mturk at the paragraph level, is that correct? Did you try it at other levels of granularity, by the sentence, word, or N number of words? If so, what were the results? If not, why not? How did you settle on 4¢/word? Are you considering a mobile app/FB version of Proofville? What's the back-end? Is it your own system, or is it an existing crowdsourcing infrastructure like PyBossa? How many reviewers compare a passage? 1? 2? Best of N until there's consistency?
- lmeyerov 13y ago
  A housemate (linked below) runs http://captricity.com/ http://captricity.com/ -- crowdsourcing OCR is a really good idea. The collaborator for the main site used to work @ cap, and we've been evolving a REST API to simplify calling into mturk. For quality, we don't use arbitrary mturkers. Getting good results on mturk takes a lot of experimentation: I now think about it is as a combination of HR agency, traffic conductor, and street vendor. The challenges of building a useful & sustainable app on it are a bit different from what's addressed by crowdsourcing research papers, though that's not so surprising in retrospect. Sounds like this deserves a blogpost. It's been a fun summer project. Basically, nodejs with roll-our-own crowdsourcing algorithms, APIs, and UI widgets. A lot more to do, but it's finally good enough to address my proofreading needs, and I'd like to help others who also need to get some writing done asap :) Btw, we're $0.004 / word, not $0.04 ;-) That means we already cost 60% less than shops with comparable quality and slower turnarounds.
  - vitovito 13y ago
    Yes, please do! I can only begin imagine some of the issues, having fed a couple thousand minutes through typingpool, but there isn't a lot of writing about using it at scale. I'll look forward to it.
relate 13y ago
Tried it out, works well!
jnbiche 13y ago
If this takes off, an API would be great, allowing people to integrate this into WordPress plugins, etc.

36 points by lmeyerov 13y ago

ams6110 13y ago

Spell and grammar check have been part of any decent word processor for quite a while now.

thatswrong0 13y ago
But they can't deeply check style, clarity, or other such qualitative aspects. Word won't tell you when your sentence is difficult to read or confusing.
- lmeyerov 13y ago
  We've caught a lot that word processors haven't, should make a gallery :)

svetha 13y ago

I absolutely love this. This is about way more than spell and grammar check. It is not just proofreading. Now the crowd can be a test audience to help make content more readable and digestible. I hope this works out!

ivan_ah 13y ago

This is very interesting. Proofreading as a service could be used by so many content companies out there. I guess for something long like a book it would make more sense to have the same person proofread the entire thing, but for blog posts and other text, parallelism could be very powerful. I'm not sure how the pricing will work out though: $0.004/w = $1.4/page = $350 for my book this is 1/6 of what I paid to my copy editor. Who will work for such a low wage? Last but not least, does it all have to be through-the-web. I find I catch a lot more mistakes on paper. Either way, I wish you good luck with your business and I will be following the developments closely.

lsb 13y ago
Sometimes copyeditors fix really subtle problems, sometimes copyeditors fix really obvious problems. This should help fix the obvious problems, and make their time better-spent!
- lmeyerov 13y ago
  Right, looking at crowdsourcing (the main proofist site) was born out of my frustration that most copyediting comments are for simple mistakes that anyone could have caught. I want to make ours cheap enough that you can get 10-20 pairs of eyes per piece of text and without the usual wait. This game is an experiment to make it free, and if we get mass, can get the desired level of matching. (In just one hour, we're already a hit with educators in math/science!)

gojomo 13y ago

Related: the Soylent Word Processor ("It's got humans inside!") from the MIT CS & AI Laboratory: http://projects.csail.mit.edu/soylent/ http://projects.csail.mit.edu/soylent/

lmeyerov 13y ago
Definitely! Also, there are a couple of wonderful followup projects that think about how to make crowdsourcing affordable (each task still costs money): http://www.cs.berkeley.edu/~kuangc/ http://www.cs.berkeley.edu/~kuangc/

cdcarter 13y ago

Is there a way to proofread something without having to upload your own text? Now I'm in the mood to edit some copy!

lmeyerov 13y ago
There is for the main site. I like the idea of different matchmaking modes for the game. A player suggested allowing more than two reviewers working together in a session, which may be one fun way to do it!

vitovito 13y ago

I've been working on my own mturk-based proofreading system for OCR scans, modeled after Ryan Tate's typingpool. This is very close to a web-based version. Your illustrations suggest you chunk content to mturk at the paragraph level, is that correct? Did you try it at other levels of granularity, by the sentence, word, or N number of words? If so, what were the results? If not, why not? How did you settle on 4¢/word? Are you considering a mobile app/FB version of Proofville? What's the back-end? Is it your own system, or is it an existing crowdsourcing infrastructure like PyBossa? How many reviewers compare a passage? 1? 2? Best of N until there's consistency?

lmeyerov 13y ago
A housemate (linked below) runs http://captricity.com/ http://captricity.com/ -- crowdsourcing OCR is a really good idea. The collaborator for the main site used to work @ cap, and we've been evolving a REST API to simplify calling into mturk. For quality, we don't use arbitrary mturkers. Getting good results on mturk takes a lot of experimentation: I now think about it is as a combination of HR agency, traffic conductor, and street vendor. The challenges of building a useful & sustainable app on it are a bit different from what's addressed by crowdsourcing research papers, though that's not so surprising in retrospect. Sounds like this deserves a blogpost. It's been a fun summer project. Basically, nodejs with roll-our-own crowdsourcing algorithms, APIs, and UI widgets. A lot more to do, but it's finally good enough to address my proofreading needs, and I'd like to help others who also need to get some writing done asap :) Btw, we're $0.004 / word, not $0.04 ;-) That means we already cost 60% less than shops with comparable quality and slower turnarounds.
- vitovito 13y ago
  Yes, please do! I can only begin imagine some of the issues, having fed a couple thousand minutes through typingpool, but there isn't a lot of writing about using it at scale. I'll look forward to it.

relate 13y ago

Tried it out, works well!

jnbiche 13y ago

If this takes off, an API would be great, allowing people to integrate this into WordPress plugins, etc.