5 ms·

The wisdom of smaller crowds

66 points by mazsa 10y ago

kennon42 10y ago
The primary thing that determines overall group performance (assuming no interpersonal biases like peer pressure or coercion, etc) would be the probability that any individual chooses the "right" answer. If this is greater than 0.5, then according to the Condorcet Jury Theorem, the limit of the group as a whole choosing the right answer is 1 as the group size increases. Perhaps having a smaller size allows the "noise" in the group to still produce the "right" answer more often than it would in a larger group?
- thaumasiotes 10y ago
  Well, assuming a standard wisdom-of-the-crowds problem like "how much does that cow weigh?", the answer is drawn from a continuous interval and the probability of choosing the right answer is necessarily zero. Yet, the fact that that's a standard example of a wisdom-of-the-crowds problem suggests that the zero probability of being correct isn't really a big issue. For a slightly different example like "how many jellybeans are there in this jar?", the answer is drawn from a discretized interval, but I feel pretty safe in assuming that the odds of any one crowd member getting it right are well below 50%.
- tedsanders 10y ago
  No, that's incorrect in two ways. First, you must also assume that the individual opinions are independent. Second, the 0.5 probability threshold is only correct for binary situations; in real life where they may be a range of opinions, the threshold is much lower.
  - wutbrodo 10y ago
    > Second, the 0.5 probability threshold is only correct for binary situations; in real life where they may be a range of opinions, the threshold is much lower. The study from the article did focus on binary situations. From the article: > Where previous research on collective intelligence deals mainly with decisions of ‘how much’ or ‘how many,’ the current study applies to ‘this or that’ decisions under a majority vote.
geoelectric 10y ago
The article touches on it, but the clear differentiator I see here is whether the "crowd" has expert knowledge of the domain. If it does, then the noise of experts clashing (resolving conflicts between equally informed ideas) starts outweighing the benefits of them contributing the ideas to begin with. Basically you get a too many chiefs issue. If it doesn't, you're seeking to average variably-educated wild ass guesses and taking advantage of the fact that WAGs have a tendency to form a bell curve around the actual answer if there's any ability to estimate whatsoever. Since there's otherwise no special knowledge, that ends up being the best you can get but requires a lot of people to get right. Without having read up on it, my guess is the latter is related to central limit, with each person's WAG forming the first term of the mean of means; a WAG is itself sampling your own mental model and boiling it to a number. Given enough of those you get a normal that you can average.
- vegabook 10y ago
  this may be why the prediction markets were so terrible at guessing brexit. Each participant was a kind of expert in said domain - not impassionate. Not random. Moreover, it's possible that the crowd view was self re-inforcing.
  - eutectic 10y ago
    The chance of Brexit was hovering around 25% just before the vote; I wouldn't describe that as 'so terrible'.
    - vegabook 10y ago
      as of 10pm in London last night, when one could reasonably expect it to be getting more accurate, it was at 11%. And 25% was in any case a very poor prediction considering these markets are supposed to be more accurate than polls.
      - smeyer 10y ago
        >And 25% was in any case a very poor prediction considering these markets are supposed to be more accurate than polls If the markets predicted 75% A vs 25% B and A always turned up, that would make them a poor prediction. Predicting 25% B may mean they're not making much of a prediction (since they're not so far off from 50/50), but it doesn't mean that B being the outcome is indicative of a the prediction being deeply wrong.
nickff 10y ago
Crowds are wise only if their errors are at least pseudo-stochastic; if they are skewed in some direction by a pervasive bias (high/low, left/right, etc.), their predictions or choices will be driven by that bias.
- Hermel 10y ago
  Even if biased, crowds can be the best option if all others are more biased.
  - geoelectric 10y ago
    Depends on whether you agree with their bias, really.
    - sverige 10y ago
      Not if it's a binary decision, as in the article.
- geoelectric 10y ago
  This is one reason you need to be careful about disqualifying people from the decision if you're relying on the large-crowd effect--for example, an educational requirement to vote or only letting certain engineers participate in planning poker type estimations. Unless you successfully boil it down to a small-crowd expert decision you'll spoil the process with skew. The uninformed people have a place in a larger-crowd process.
hristov 10y ago
After the whole brexit fiasco i do not want to hear about the wisdom of crowds.
rxm 10y ago
The timing of this post is uncanny given how some large crowds voted.
- ktRolster 10y ago
  Maybe they were right? I personally think the brexit was a mistake, but now that it's over, I hope the best for them, and hope they manage to prove me wrong. I certainly don't think that my opinion is necessarily the correct one.
dredmorbius 10y ago
There are a few elements here, one of which geoelectric addresses in an earlier comment: does the crowd have expert knowledge. Another is whether or not the question at hand requires expert knowledge. There are times when you're better off handing off the helm or pilot's seat to a qualified pilot than trying to average inputs of a large crowd, or to allow an electrician or plumber to address a problem within their skill scope. A third aspect though might concern what the negative cost functions of a crowd might be. The wisdom of crowds concept generally assumes the larger the crowd, the better it will be at arriving at some truth. In reality, various biases, distortions, and manipulations can emerge, to the point that the crowd's view is far worse than other options. Aristotle drinks hemlock. Trump is presumptive nominee. Brexit. An element of networked systems, including decisionmaking systems, is what their cost functions are, in the sense of imposing negative results on those participating. I've been arguing for a year or two now that there is such a cost function, and that you can estimate that by noting the maximum size an effective network can grow to. Conversely, you can increase (or decrease) the effective size of a network by addressing that cost function. Increase it and you'll make large-scale aggregation less viable. Decrease it, and you can increase the size of effective aggregation. As examples, a village is constrained in total size not only by its ability to secure necessary inputs (especially food and water), but in its ability to dispose of wastes and noxious emissions. London of the late 18th century had a mortality rate above its natural birthrate, and the only way the city could maintain its population was through net in-migration from the countryside (or foreign lands). This wasn't materially addressed until revolutions in water provision and sanitation, including the first modern sewerage system around 1850, addressed such concerns as cholera epidemics which were killing as many as 50,000 people a year. In programming, Fred Brooks' The Mythical Man Month notes that few programming teams scale well beyond about 6-12 developers. The inter-personal communications costs make larger groups not only inefficient, but less effective, net than smaller ones. To produce larger teams, you've effectively got to split them into smaller ones. That's among the things that a highly modularised development process as is common in Free Software projects achieves -- see Apache, the Linux Kernel, or the Debian Project as examples (Gabriella Coleman, now of McGill University, wrote her dissertation on this topic, it's fascinating reading). Computer chip design essentially removes the space and resistance costs of crowding high densities of electronic gates in small spaces. Again, the cost function is reduced. In email and traditional (POTS and mobile) phone service, increasing amounts of spam are increasing cost functions, reducing the appeal and utility of the network to all involved. POTS has been shedding subscribers for some time, my expectation is that mobile phone service itself will be as well, more especially if interconnects, and filtering of VOIP alternatives (including iChat, Google voice chat, Skype, etc.) are further developed. Those networks, as a Long Island friend of mine some time back said, "gotta leahn to tawk to each othah!

66 points by mazsa 10y ago

The primary thing that determines overall group performance (assuming no interpersonal biases like peer pressure or coercion, etc) would be the probability that any individual chooses the "right" answer. If this is greater than 0.5, then according to the Condorcet Jury Theorem, the limit of the group as a whole choosing the right answer is 1 as the group size increases. Perhaps having a smaller size allows the "noise" in the group to still produce the "right" answer more often than it would in a larger group?

thaumasiotes 10y ago
Well, assuming a standard wisdom-of-the-crowds problem like "how much does that cow weigh?", the answer is drawn from a continuous interval and the probability of choosing the right answer is necessarily zero. Yet, the fact that that's a standard example of a wisdom-of-the-crowds problem suggests that the zero probability of being correct isn't really a big issue. For a slightly different example like "how many jellybeans are there in this jar?", the answer is drawn from a discretized interval, but I feel pretty safe in assuming that the odds of any one crowd member getting it right are well below 50%.
tedsanders 10y ago
No, that's incorrect in two ways. First, you must also assume that the individual opinions are independent. Second, the 0.5 probability threshold is only correct for binary situations; in real life where they may be a range of opinions, the threshold is much lower.
- wutbrodo 10y ago
  > Second, the 0.5 probability threshold is only correct for binary situations; in real life where they may be a range of opinions, the threshold is much lower. The study from the article did focus on binary situations. From the article: > Where previous research on collective intelligence deals mainly with decisions of ‘how much’ or ‘how many,’ the current study applies to ‘this or that’ decisions under a majority vote.

geoelectric 10y ago

The article touches on it, but the clear differentiator I see here is whether the "crowd" has expert knowledge of the domain. If it does, then the noise of experts clashing (resolving conflicts between equally informed ideas) starts outweighing the benefits of them contributing the ideas to begin with. Basically you get a too many chiefs issue. If it doesn't, you're seeking to average variably-educated wild ass guesses and taking advantage of the fact that WAGs have a tendency to form a bell curve around the actual answer if there's any ability to estimate whatsoever. Since there's otherwise no special knowledge, that ends up being the best you can get but requires a lot of people to get right. Without having read up on it, my guess is the latter is related to central limit, with each person's WAG forming the first term of the mean of means; a WAG is itself sampling your own mental model and boiling it to a number. Given enough of those you get a normal that you can average.

vegabook 10y ago
this may be why the prediction markets were so terrible at guessing brexit. Each participant was a kind of expert in said domain - not impassionate. Not random. Moreover, it's possible that the crowd view was self re-inforcing.
- eutectic 10y ago
  The chance of Brexit was hovering around 25% just before the vote; I wouldn't describe that as 'so terrible'.
  - vegabook 10y ago
    as of 10pm in London last night, when one could reasonably expect it to be getting more accurate, it was at 11%. And 25% was in any case a very poor prediction considering these markets are supposed to be more accurate than polls.
    - smeyer 10y ago
      >And 25% was in any case a very poor prediction considering these markets are supposed to be more accurate than polls If the markets predicted 75% A vs 25% B and A always turned up, that would make them a poor prediction. Predicting 25% B may mean they're not making much of a prediction (since they're not so far off from 50/50), but it doesn't mean that B being the outcome is indicative of a the prediction being deeply wrong.

nickff 10y ago

Crowds are wise only if their errors are at least pseudo-stochastic; if they are skewed in some direction by a pervasive bias (high/low, left/right, etc.), their predictions or choices will be driven by that bias.

Hermel 10y ago
Even if biased, crowds can be the best option if all others are more biased.
- geoelectric 10y ago
  Depends on whether you agree with their bias, really.
  - sverige 10y ago
    Not if it's a binary decision, as in the article.
geoelectric 10y ago
This is one reason you need to be careful about disqualifying people from the decision if you're relying on the large-crowd effect--for example, an educational requirement to vote or only letting certain engineers participate in planning poker type estimations. Unless you successfully boil it down to a small-crowd expert decision you'll spoil the process with skew. The uninformed people have a place in a larger-crowd process.

hristov 10y ago

After the whole brexit fiasco i do not want to hear about the wisdom of crowds.

rxm 10y ago

The timing of this post is uncanny given how some large crowds voted.

ktRolster 10y ago
Maybe they were right? I personally think the brexit was a mistake, but now that it's over, I hope the best for them, and hope they manage to prove me wrong. I certainly don't think that my opinion is necessarily the correct one.

dredmorbius 10y ago

There are a few elements here, one of which geoelectric addresses in an earlier comment: does the crowd have expert knowledge. Another is whether or not the question at hand requires expert knowledge. There are times when you're better off handing off the helm or pilot's seat to a qualified pilot than trying to average inputs of a large crowd, or to allow an electrician or plumber to address a problem within their skill scope. A third aspect though might concern what the negative cost functions of a crowd might be. The wisdom of crowds concept generally assumes the larger the crowd, the better it will be at arriving at some truth. In reality, various biases, distortions, and manipulations can emerge, to the point that the crowd's view is far worse than other options. Aristotle drinks hemlock. Trump is presumptive nominee. Brexit. An element of networked systems, including decisionmaking systems, is what their cost functions are, in the sense of imposing negative results on those participating. I've been arguing for a year or two now that there is such a cost function, and that you can estimate that by noting the maximum size an effective network can grow to. Conversely, you can increase (or decrease) the effective size of a network by addressing that cost function. Increase it and you'll make large-scale aggregation less viable. Decrease it, and you can increase the size of effective aggregation. As examples, a village is constrained in total size not only by its ability to secure necessary inputs (especially food and water), but in its ability to dispose of wastes and noxious emissions. London of the late 18th century had a mortality rate above its natural birthrate, and the only way the city could maintain its population was through net in-migration from the countryside (or foreign lands). This wasn't materially addressed until revolutions in water provision and sanitation, including the first modern sewerage system around 1850, addressed such concerns as cholera epidemics which were killing as many as 50,000 people a year. In programming, Fred Brooks' The Mythical Man Month notes that few programming teams scale well beyond about 6-12 developers. The inter-personal communications costs make larger groups not only inefficient, but less effective, net than smaller ones. To produce larger teams, you've effectively got to split them into smaller ones. That's among the things that a highly modularised development process as is common in Free Software projects achieves -- see Apache, the Linux Kernel, or the Debian Project as examples (Gabriella Coleman, now of McGill University, wrote her dissertation on this topic, it's fascinating reading). Computer chip design essentially removes the space and resistance costs of crowding high densities of electronic gates in small spaces. Again, the cost function is reduced. In email and traditional (POTS and mobile) phone service, increasing amounts of spam are increasing cost functions, reducing the appeal and utility of the network to all involved. POTS has been shedding subscribers for some time, my expectation is that mobile phone service itself will be as well, more especially if interconnects, and filtering of VOIP alternatives (including iChat, Google voice chat, Skype, etc.) are further developed. Those networks, as a Long Island friend of mine some time back said, "gotta leahn to tawk to each othah!