3 ms·
On "Wisdom of the Crowds": Group average on its own is no good, but some creative aggregation can be better than either member on its own (and of course better
by eurg 8y ago
On "Wisdom of the Crowds": Group average on its own is no good, but some creative aggregation can be better than either member on its own (and of course better than naive averaging of any sorts). As a practical example, see for instance Metaculus. Excerpt from one comment[1]:
"Since Dec. 1 2018 (sic), the MP has been on the "right" side of 50% every single time, and (in a more meaningful measure) has a mean Brier of 0.048. (Though those are both about to be spoiled by the NASA-LISA question, probably.)" [2]
You can compare metaculus vs. community predictions at [2]. The corresponding community score would be 0.119 during the same time.
[1]: https://www.metaculus.com/questions/666/will-this-question-resolve-positive/ https://www.metaculus.com/questions/666/will-this-question-r...
[2]: https://www.metaculus.com/questions/track-record/ https://www.metaculus.com/questions/track-record/
- barbegal 8y agoI'm not sure this is proving anything except that by developing an algorithm that filters out some predictions you can improve upon those predictions. And that is probably quite obvious given there are some users (presumably) on the site who are just trolling and fairly easy to identify and filter out. Unfortunately, I can't see the data on the Brier score of individuals.
- eurg 8y agoOf course it does not prove anything, but identifying "wisdom of the crowds" only and exactly with "group average" isn't particularly considerate. While public mass media thrives on simplifications, I had hoped that we could iterate on the basic premise.