5 ms·

This sort of proves that there is no such thing as "Wisdom of the Crowds". There seems to be a bit of a cult myth [1][2] about groups being able to "on average"

This sort of proves that there is no such thing as "Wisdom of the Crowds". There seems to be a bit of a cult myth [1][2] about groups being able to "on average" outperform even the best individuals but that seems to be based on some rather flimsy studies whose methodology has not been published. It seems more likely that in reality, data is cherry picked, where it correlates with the "Wisdom of the Crowds" myth nit gets published, if not it gets discarded. The most rigorous experiment I've seen performed, tested a group of reddit users on estimating lines in an image [3]. Some expert individuals were far more accurate than the group average. [1] http://feiss.be/blog/post/169 http://feiss.be/blog/post/169 [2] https://www.cfapubs.org/doi/abs/10.2469/faj.v43.n3.50 https://www.cfapubs.org/doi/abs/10.2469/faj.v43.n3.50 [3] http://feiss.be/blog/post/169 http://feiss.be/blog/post/169

fasj82 9y ago
Checkmate, democracy.
- 36bydesign 9y ago
  Trump was the checkmate
- mattnewton 9y ago
  I tend to think democracy is less about finding some optimal solution and more about finding a solution enough people will be happy with that there isn’t revolution.
  - opportune 9y ago
    Yeah, democracy may have a limited best case scenario compared to other forms of decision making, but in many senses it has one of the best worst case scenarios. More centralized power may be able to get more done, but historically we've seen it often results in decision making that is either just-plain-bad (when leadership is clueless) or which only benefits those in power, to the detriment of everyone else (when leadership is malevolent). Whereas democracy is slower and less efficient, but does ensure that most people are in support of the decisions being made
    - thomasz 9y ago
      Democracy isn't about getting the best policy each and every time, its primary value lies in the fact that it creates framework that facilitates the peaceful transition of power and avoids the otherwise frequent wars of succession, purges, coups and whatnot.
- danbruc 9y ago
  Democracy does not try to make good decisions, it tries to reflected the opinions of the people in the decision.
- api 9y ago
  I don't think democracy is about increasing intelligence so much as resilience and escaping from failure modes. It provides a way to get rid of bad leadership, and it makes certain that huge issues are hard to ignore. In a pure autocracy a bad autocrat is fatal and there is no way to do anything about it but via extralegal means.
- wangii 9y ago
  democracy is not about decision, but decision making. during the process consensus is born and grown. it then generates mandate and enough political capital and willingness for consequent execution. that being said, democracy often leads to wishes that nobody could execute. after all, it's about what participants want, not what they really need.
eurg 9y ago
On "Wisdom of the Crowds": Group average on its own is no good, but some creative aggregation can be better than either member on its own (and of course better than naive averaging of any sorts). As a practical example, see for instance Metaculus. Excerpt from one comment[1]: "Since Dec. 1 2018 (sic), the MP has been on the "right" side of 50% every single time, and (in a more meaningful measure) has a mean Brier of 0.048. (Though those are both about to be spoiled by the NASA-LISA question, probably.)" [2] You can compare metaculus vs. community predictions at [2]. The corresponding community score would be 0.119 during the same time. [1]: https://www.metaculus.com/questions/666/will-this-question-resolve-positive/ https://www.metaculus.com/questions/666/will-this-question-r... [2]: https://www.metaculus.com/questions/track-record/ https://www.metaculus.com/questions/track-record/
- barbegal 9y ago
  I'm not sure this is proving anything except that by developing an algorithm that filters out some predictions you can improve upon those predictions. And that is probably quite obvious given there are some users (presumably) on the site who are just trolling and fairly easy to identify and filter out. Unfortunately, I can't see the data on the Brier score of individuals.
  - eurg 9y ago
    Of course it does not prove anything, but identifying "wisdom of the crowds" only and exactly with "group average" isn't particularly considerate. While public mass media thrives on simplifications, I had hoped that we could iterate on the basic premise.
avip 9y ago
Or maybe it means you have to curate your crowd rather carefully if you're into outperforming arguably the greatest chess player ever.
- deleted 9y ago
  [deleted]
akkartik 9y ago
Proves?! It isn't even an anecdote in favor of your thesis. The World Team stayed with the world champion until move 54 (something proved 13 years later). Avoid over-generalizing.
ordu 9y ago
I cannot agree. This sort doesn't prove nothing about "Wisdom of the Crowds" because the wording is too broad. But it is evidence that crowd can outperform the best individual at least theoretically. The worst mistake the crowd had made was openness of dicussion allowing Kasparov to read this discussion. It gave him the advantage and by his own words it was cruicial advantage to win. Chess is the good candidate for hive-mind, because it is computational task which can be parallelized. You can hire tons of minds, and made them to solve different parts of the common task. There was some issues with management with mind-resources, the worst at the end-game, when many people lost their faith and started to discuss not a next move, but what is wrong with hive-mind. > The most rigorous experiment I've seen performed, tested a group of reddit users on estimating lines in an image [3]. Some expert individuals were far more accurate than the group average. I'm not convinced by this. There are few objections. 1. There was no discussion between elements of hive-mind. So the abilities of hive-mind remains unused. 2. Suspicious statistics. a. The articles compares estimates of individuals with estimates of hive-mind. Hive-mind have one set of answers, while answers of individuals are combined with knowledge of true answer. There are no example of individual outperforming hive-mind on all four data points at once. b. There are no attempt to estimate probability to get top individual answers by chance. Maybe those selected individuals are no better than average person, but just lucky ones? 3. There are different tasks. Different tasks needs different data processing, different data processing needs different processors. For example, there are tasks that benefits from parallelizing, and there are tasks that do not. Here we come close to (1): if you want hive-mind to outperform individual, you need to find a way how to use superpowers of hive-mind with this task.
- thret 9y ago
  Perhaps Deep Blue (or current equivalent) vs the World would be an interesting match, where the AI did not have access to the discussion.
- CuriouslyC 9y ago
  Wisdom of the crowds works when you're able to pool a large number of independent, unbiased estimates to produce a distribution. If the crowd communicates, it is no longer independent. This produces a distribution weighted towards the answers of loudest or most persuasive fraction, which is probably going to be less accurate. In fact, communication between crowd members in many cases significantly reduces the quality of the answers. This has been clearly demonstrated in studies of group brainstorming, where the number and quality of ideas is significantly better when group members come up with ideas independently and pool them at the end than when they come up with ideas together.
cdoxsey 9y ago
To this point, consider the failed startup Trada. (I worked there for a bit) They had a marketplace for ad campaigns with Google. Instead of running ad campaigns or using artificial intelligence, a pool of optimizers would work on a campaign and Trada worked as the middle man between optimizers and advertisers. The Trada logo was a jar of jelly beans: (http://wisdomofcrowds.blogspot.nl/2009/12/jelly-bean-experiment.html http://wisdomofcrowds.blogspot.nl/2009/12/jelly-bean-experim...) > A classic demonstration of group intelligence is the jelly-beans-in-the-jar experiment, in which invariably the group’s estimate is superior to the vast majority of the individual guesses. When finance professor Jack Treynor ran the experiment in his class with a jar that held 850 beans, the group estimate was 871. Only one of the fifty-six people in the class made a better guess. When advertising on Google competition leads to higher prices. For example the keywords "dui lawyer" cost a fortune because the rate of return is so high for each click. Theoretically a creative individual could come up with a set of untapped, low price keywords that could lead to an effective ad campaign at a lower price. In practice the company had major problems. Incompetent optimizers would chew through a whole budget, leading to advertiser churn, and a lot of intervention. Eventually everything becomes micro-managed which isn't a sustainable business model. (There's a reason agencies target big ad campaigns... its the only way to make it worth it because of the amount of resources you have to invest) So there's a single data point where the wisdom of the crowds failed.
- kbenson 9y ago
  I'm not sure I understand how incompetent optimizers leads to an example of the wisdom of the crowds failing. It sounds like the failing you describe is the result of individuals. Is there something I'm not getting?
  - cdoxsey 9y ago
    The optimizers were the crowd in this business model. Some of them made good decisions, others made bad. But since the amount of money available was limited, those bad decisions brought down the performance of the campaign and ultimately the advertiser gave up. An expert would've made better decisions. I think the wisdom of the crowd can sometimes be effective, but there are a lot of ways it can go wrong. Some things I was thinking of: 1. the crowd wasn't big enough 2. the time frame was too short to be effective 3. the heuristic was too complex or poorly tuned to lead to a good outcome 4. creative endeavors with high rates of failure lack enough signal to properly optimize (a strategy might look terrible till it suddenly works) 5. misaligned incentives undermine the goal To illustrate the last case, in high school I had a teacher tell us that he would scale the test by the highest score. For example a 95 would give 100 and +5 to every other score in the class. With incentives like that, if everyone in the class answered no questions, everyone would get 100. (but then all it would take would be one individual to screw the whole class) Crowds are complex, difficult to understand and hard to predict.
xfs 9y ago
Yet wisdom of the crowd becomes a successful machine learning paradigm as ensemble learning where a family of weak learners are combined to produce a stronger learner.
jnordwick 9y ago
The two most rigorous experiments in the wisdom of crowds are: - the stock market - democracy They beat technocratic efforts every time.
- WillReplyfFood 9y ago
  Eh,both of these are technocratic efforts- they where many constitutions and market systems trialed and errored to get where we are today. Nothing of that dropped from the sky, as a law of nature.
empath75 9y ago
Quite the opposite, actually. The world team played better than anybody from that team participating individually could have.
QML 9y ago
It's important to keep in context where the term "Wisdom of the Crowd" came from and where it succeeded. "The classic wisdom-of-the-crowds finding involves point estimation of a continuous quantity. This has contributed to the insight in cognitive science that a crowd's individual judgments can be modeled as a probability distribution of responses with the median centered near the true value of the quantity to be estimated." [1] Notice that the term said nothing about adversarial games and or games which have moves. A better model for those types of games is the experts' problem / multiplicative weights algorithm -- where the solution consists of a weighted plurality which is updated every turn. [2] [1] https://en.wikipedia.org/wiki/Wisdom_of_the_crowd https://en.wikipedia.org/wiki/Wisdom_of_the_crowd [2] https://en.wikipedia.org/wiki/Multiplicative_weight_update_method https://en.wikipedia.org/wiki/Multiplicative_weight_update_m...
Tomminn 9y ago
Read the comment by "Someone". The rest of the world was playing on both sides of the table, since Kasporov was reading their analysis during the game.
BurningFrog 9y ago
"Wisdom of the Crowds" works really well for some problems, and not at all for others.
userbinator 9y ago
Relevant quote: "In questions of science, the authority of a thousand is not worth the humble reasoning of a single individual."
eric_h 9y ago
Wisdom of the crowds works well for counting jelly beans in a jar, not so much for finding answers to much more complicated questions.

fasj82 9y ago

Checkmate, democracy.

36bydesign 9y ago
Trump was the checkmate
mattnewton 9y ago
I tend to think democracy is less about finding some optimal solution and more about finding a solution enough people will be happy with that there isn’t revolution.
- opportune 9y ago
  Yeah, democracy may have a limited best case scenario compared to other forms of decision making, but in many senses it has one of the best worst case scenarios. More centralized power may be able to get more done, but historically we've seen it often results in decision making that is either just-plain-bad (when leadership is clueless) or which only benefits those in power, to the detriment of everyone else (when leadership is malevolent). Whereas democracy is slower and less efficient, but does ensure that most people are in support of the decisions being made
  - thomasz 9y ago
    Democracy isn't about getting the best policy each and every time, its primary value lies in the fact that it creates framework that facilitates the peaceful transition of power and avoids the otherwise frequent wars of succession, purges, coups and whatnot.
danbruc 9y ago
Democracy does not try to make good decisions, it tries to reflected the opinions of the people in the decision.
api 9y ago
I don't think democracy is about increasing intelligence so much as resilience and escaping from failure modes. It provides a way to get rid of bad leadership, and it makes certain that huge issues are hard to ignore. In a pure autocracy a bad autocrat is fatal and there is no way to do anything about it but via extralegal means.
wangii 9y ago
democracy is not about decision, but decision making. during the process consensus is born and grown. it then generates mandate and enough political capital and willingness for consequent execution. that being said, democracy often leads to wishes that nobody could execute. after all, it's about what participants want, not what they really need.

eurg 9y ago

On "Wisdom of the Crowds": Group average on its own is no good, but some creative aggregation can be better than either member on its own (and of course better than naive averaging of any sorts). As a practical example, see for instance Metaculus. Excerpt from one comment[1]: "Since Dec. 1 2018 (sic), the MP has been on the "right" side of 50% every single time, and (in a more meaningful measure) has a mean Brier of 0.048. (Though those are both about to be spoiled by the NASA-LISA question, probably.)" [2] You can compare metaculus vs. community predictions at [2]. The corresponding community score would be 0.119 during the same time. [1]: https://www.metaculus.com/questions/666/will-this-question-resolve-positive/ https://www.metaculus.com/questions/666/will-this-question-r... [2]: https://www.metaculus.com/questions/track-record/ https://www.metaculus.com/questions/track-record/

barbegal 9y ago
I'm not sure this is proving anything except that by developing an algorithm that filters out some predictions you can improve upon those predictions. And that is probably quite obvious given there are some users (presumably) on the site who are just trolling and fairly easy to identify and filter out. Unfortunately, I can't see the data on the Brier score of individuals.
- eurg 9y ago
  Of course it does not prove anything, but identifying "wisdom of the crowds" only and exactly with "group average" isn't particularly considerate. While public mass media thrives on simplifications, I had hoped that we could iterate on the basic premise.

avip 9y ago

Or maybe it means you have to curate your crowd rather carefully if you're into outperforming arguably the greatest chess player ever.

deleted 9y ago
[deleted]

akkartik 9y ago

Proves?! It isn't even an anecdote in favor of your thesis. The World Team stayed with the world champion until move 54 (something proved 13 years later). Avoid over-generalizing.

ordu 9y ago

I cannot agree. This sort doesn't prove nothing about "Wisdom of the Crowds" because the wording is too broad. But it is evidence that crowd can outperform the best individual at least theoretically. The worst mistake the crowd had made was openness of dicussion allowing Kasparov to read this discussion. It gave him the advantage and by his own words it was cruicial advantage to win. Chess is the good candidate for hive-mind, because it is computational task which can be parallelized. You can hire tons of minds, and made them to solve different parts of the common task. There was some issues with management with mind-resources, the worst at the end-game, when many people lost their faith and started to discuss not a next move, but what is wrong with hive-mind. > The most rigorous experiment I've seen performed, tested a group of reddit users on estimating lines in an image [3]. Some expert individuals were far more accurate than the group average. I'm not convinced by this. There are few objections. 1. There was no discussion between elements of hive-mind. So the abilities of hive-mind remains unused. 2. Suspicious statistics. a. The articles compares estimates of individuals with estimates of hive-mind. Hive-mind have one set of answers, while answers of individuals are combined with knowledge of true answer. There are no example of individual outperforming hive-mind on all four data points at once. b. There are no attempt to estimate probability to get top individual answers by chance. Maybe those selected individuals are no better than average person, but just lucky ones? 3. There are different tasks. Different tasks needs different data processing, different data processing needs different processors. For example, there are tasks that benefits from parallelizing, and there are tasks that do not. Here we come close to (1): if you want hive-mind to outperform individual, you need to find a way how to use superpowers of hive-mind with this task.

thret 9y ago
Perhaps Deep Blue (or current equivalent) vs the World would be an interesting match, where the AI did not have access to the discussion.
CuriouslyC 9y ago
Wisdom of the crowds works when you're able to pool a large number of independent, unbiased estimates to produce a distribution. If the crowd communicates, it is no longer independent. This produces a distribution weighted towards the answers of loudest or most persuasive fraction, which is probably going to be less accurate. In fact, communication between crowd members in many cases significantly reduces the quality of the answers. This has been clearly demonstrated in studies of group brainstorming, where the number and quality of ideas is significantly better when group members come up with ideas independently and pool them at the end than when they come up with ideas together.

cdoxsey 9y ago

To this point, consider the failed startup Trada. (I worked there for a bit) They had a marketplace for ad campaigns with Google. Instead of running ad campaigns or using artificial intelligence, a pool of optimizers would work on a campaign and Trada worked as the middle man between optimizers and advertisers. The Trada logo was a jar of jelly beans: (http://wisdomofcrowds.blogspot.nl/2009/12/jelly-bean-experiment.html http://wisdomofcrowds.blogspot.nl/2009/12/jelly-bean-experim...) > A classic demonstration of group intelligence is the jelly-beans-in-the-jar experiment, in which invariably the group’s estimate is superior to the vast majority of the individual guesses. When finance professor Jack Treynor ran the experiment in his class with a jar that held 850 beans, the group estimate was 871. Only one of the fifty-six people in the class made a better guess. When advertising on Google competition leads to higher prices. For example the keywords "dui lawyer" cost a fortune because the rate of return is so high for each click. Theoretically a creative individual could come up with a set of untapped, low price keywords that could lead to an effective ad campaign at a lower price. In practice the company had major problems. Incompetent optimizers would chew through a whole budget, leading to advertiser churn, and a lot of intervention. Eventually everything becomes micro-managed which isn't a sustainable business model. (There's a reason agencies target big ad campaigns... its the only way to make it worth it because of the amount of resources you have to invest) So there's a single data point where the wisdom of the crowds failed.

kbenson 9y ago
I'm not sure I understand how incompetent optimizers leads to an example of the wisdom of the crowds failing. It sounds like the failing you describe is the result of individuals. Is there something I'm not getting?
- cdoxsey 9y ago
  The optimizers were the crowd in this business model. Some of them made good decisions, others made bad. But since the amount of money available was limited, those bad decisions brought down the performance of the campaign and ultimately the advertiser gave up. An expert would've made better decisions. I think the wisdom of the crowd can sometimes be effective, but there are a lot of ways it can go wrong. Some things I was thinking of: 1. the crowd wasn't big enough 2. the time frame was too short to be effective 3. the heuristic was too complex or poorly tuned to lead to a good outcome 4. creative endeavors with high rates of failure lack enough signal to properly optimize (a strategy might look terrible till it suddenly works) 5. misaligned incentives undermine the goal To illustrate the last case, in high school I had a teacher tell us that he would scale the test by the highest score. For example a 95 would give 100 and +5 to every other score in the class. With incentives like that, if everyone in the class answered no questions, everyone would get 100. (but then all it would take would be one individual to screw the whole class) Crowds are complex, difficult to understand and hard to predict.

xfs 9y ago

Yet wisdom of the crowd becomes a successful machine learning paradigm as ensemble learning where a family of weak learners are combined to produce a stronger learner.

jnordwick 9y ago

The two most rigorous experiments in the wisdom of crowds are: - the stock market - democracy They beat technocratic efforts every time.

WillReplyfFood 9y ago
Eh,both of these are technocratic efforts- they where many constitutions and market systems trialed and errored to get where we are today. Nothing of that dropped from the sky, as a law of nature.

empath75 9y ago

Quite the opposite, actually. The world team played better than anybody from that team participating individually could have.

QML 9y ago

It's important to keep in context where the term "Wisdom of the Crowd" came from and where it succeeded. "The classic wisdom-of-the-crowds finding involves point estimation of a continuous quantity. This has contributed to the insight in cognitive science that a crowd's individual judgments can be modeled as a probability distribution of responses with the median centered near the true value of the quantity to be estimated." [1] Notice that the term said nothing about adversarial games and or games which have moves. A better model for those types of games is the experts' problem / multiplicative weights algorithm -- where the solution consists of a weighted plurality which is updated every turn. [2] [1] https://en.wikipedia.org/wiki/Wisdom_of_the_crowd https://en.wikipedia.org/wiki/Wisdom_of_the_crowd [2] https://en.wikipedia.org/wiki/Multiplicative_weight_update_method https://en.wikipedia.org/wiki/Multiplicative_weight_update_m...

Tomminn 9y ago

Read the comment by "Someone". The rest of the world was playing on both sides of the table, since Kasporov was reading their analysis during the game.

BurningFrog 9y ago

"Wisdom of the Crowds" works really well for some problems, and not at all for others.

userbinator 9y ago

Relevant quote: "In questions of science, the authority of a thousand is not worth the humble reasoning of a single individual."

eric_h 9y ago

Wisdom of the crowds works well for counting jelly beans in a jar, not so much for finding answers to much more complicated questions.