The issue of crowd sourcing answers to questions is a very interesting one. I've recently been reading research in this area with a colleague, and it is far mor

The issue of crowd sourcing answers to questions is a very interesting one. I've recently been reading research in this area with a colleague, and it is far more involved than the article makes it sound. If you are interested, one of the first people to work on the problem were Dawid & Skene [0] back in 1979. They were trying to solve the issue of clinicians talking to patients and asking how they are. Each time they asked, the patient may give a slightly different answer. The goal was to estimate the "true" answer from the number of various "noisy" answers (sound familiar after reading the OP?) The general intuition for most future research into the topic is that if you tend to agree with others more often than not, your score goes up. If your score is higher, your answers are trusted more. Then there are a whole bunch of interesting additions to this principle, such as factoring in question difficulty [1] (e.g. questions which have the most disagreement can be considered more difficult, and hence you may want to ask more people about these questions to get a clearer picture). It is also helpful to seed certain questions for which the answer is "known" (may or may not be possible, depending on the project). Then you can use the answers to these questions to help score people even more accurately. An even more interesting approach is to take into account people who answer adversarially, that is, they always answer incorrectly (either due to malicious intent, or they consistently misinterpret the questions in a backward fasion). If you have somebody who answers incorrectly 90% of the time, then they provide almost exactly the same information as someone who answers correctly 90% of the time. All you need to do is flip their vote once you've identified this. The whole field is very facinating, and there seems to be more and more publications coming out with interesting approaches to the problem. [0] http://scholar.google.com/scholar?cites=15953840037809938294&as_sdt=5,39&sciodt=0,39&hl=en http://scholar.google.com/scholar?cites=15953840037809938294... [1] http://scholar.google.com/scholar?cluster=14409115865258228521&hl=en&as_sdt=0,39 http://scholar.google.com/scholar?cluster=144091158652582285...