6 ms·

Ignoring the Wisdom of Crowds

49 points by joeyespo 14y ago

smoyer 14y ago
As programmers we should be well aware that our customers' "needs" are bounded by what they a) know and b) believe is possible. Guiding them to technological enlightenment can and does provide rewards (financial and otherwise).
- fleitz 14y ago
  Are you essentially saying that if you listen to your customers you'll build a faster horse, instead of a car?
  - smoyer 14y ago
    No ... I'm saying that sometimes we are evangelists and can make them realize they want a flying car.
wisty 14y ago
The other problem with focus groups - they aren't independent. There will always be one person who sounds convincing, and the rest will be biased to whatever that person thinks. The advantage of focus groups is, you can identify themes which resonate. For example, the group might say "integrated" sounds good. That doesn't mean you want an integrated product, it just means that you need a sticker which says "integrated", and some justification (We integrate with Facebook, via our "Like" buttons!)
- hammock 14y ago
  >The other problem with focus groups - they aren't independent. There will always be one person who sounds convincing, and the rest will be biased to whatever that person thinks. Experienced group moderators are aware of this take steps to mitigate and reflect in their notes. There are other forms of market research beyond focus groups. Aggregated quantitative and qualitative one-on-ones also fit the OP's bill of "crowd wisdom"
freshhawk 14y ago
I stopped taking this seriously when they compared the accuracy of the "ask the audience" and "phone a friend" bits on Who Wants to be a Millionaire without considering that everyone used the ask the audience lifeline on an early question and saved the phone a friend for later questions. Without controlling for when they are used the comparison of accuracy is literally meaningless. The same stats show that, rightly or wrongly, the contestants didn't have as much faith in the audience for later stage, much harder questions. I guess I should give the benefit of the doubt and assume the writer didn't notice this and the error is accidental but that's only slightly better than just ignoring it because you wanted a sexy anecdote to support your point.
- hammock 14y ago
  You right that the two lifelines are used at differing stages in the game. I wonder though, what the accuracy rates look like after controlling for what stage of the game you are in. Maybe people SHOULD save their ask the audience for later in the game.
  - T-hawk 14y ago
    Another variable is the subject matter of the question. Anecdotally, it feels that the audience almost invariably nails the answer by a wide margin on pop culture questions. On questions of a scientific or technical nature, the audience opinion is much less clear and more often actually wrong. Phone those questions to a domain knowledge expert. About the worst thing you can do, but contestants love doing this, is to burn the valuable audience lifeline early on a question when you already have a good sense of the answer but just want to "make sure". What a waste! Save that for a question when you have no clue, which WILL happen before you reach the million level.
    - nisse72 14y ago
      Something else they often do is say something like "I'm thinking it's B, but I'm not sure so I'll ask the audience". Well guess what the audience guesses after hearing that?
  - 14y ago
billswift 14y ago
His further reading list includes Surowiecki's The Wisdom of Crowds, but his post doesn't look like he read it himself. Surowiecki has a long list of requirements for the "wisdom of crowds" to actually be helpful, for one example the individual judgments need to be independent, that the post ignores. Without the restrictions you get different failure modes, like bubbles, runs, and traffic jams.
- chmike 14y ago
  Or election mistakes. Democracy assumes people's choice is independent, though it is not. We are still very exposed to manipulation.
- smartbear 14y ago
  The essay was intended to be a light-hearted and readable intro to the concept, not an exhaustive definition that takes an entire book!
  - billswift 14y ago
    Almost everyone has at least heard the phrase "the wisdom of crowds" by now. The most important part of the idea, and the one most people are not familiar with, are the (fairly severe) limits on its usefulness. I wrote a review of the book for Amazon, and one thing I pointed out is that it was a much better balanced treatment of the subject than I had expected from all the hype about the idea that I had previously read. The hype is already all over the place, and has been for years, there is no need to add to it.
antithesis 14y ago
He isn't being very fair: where the goal at first was to be as right as possible, it later became "to piss off as few of the 300 guests as possible". I'm not sure what exact procedure he had in mind for businesses, but in a situation where "votes eliminate the interesting edges, leaving only the boring residue that no one hated enough to vote off the island" applies, you're just not talking about the wisdom of crowds anymore, this wasn't possible in his earlier examples either. An important factor to the wisdom of crowds, and a key difference between his earlier examples and his later examples, is that people vote independently, without even talking about what they're going to vote. For demonstrational purposes, I will now take his holiday meal situation and apply actual wisdom of crowds to them. This isn't a very usual situation, but this is about properly applying the wisdom of crowds. Instead of people stating what they dislike, people may nominate a meal, and afterwards, everyone votes for the meal they want to go with. Like in the Wants to be a Millionaire examples, people cannot talk in this process - they cannot state what they dislike. Instead, the fact that some people can't take spicy food will show in the results, but beside that, every individual will pick a meal that isn't boring, and the end result will be just that. "Allergic to garlic" and "doesn’t eat anything green" are rather specific complaints and were not taken into account, so let's count those 3 people as unsatisfied, but the success here is that most people will in fact be satisfied with the meal.
- smartbear 14y ago
  You can't use your technique for meals because everyone will suggest a different meal, and then you have no basis for a decision. That technique works when you're voting on one variable which has one CORRECT value which you're trying to seek. Creative work is not that.
neuroelectronic 14y ago
iPhone is chicken and rice to me.
ozim 14y ago
This essey is weak, I mean author is comparing apples to oranges. First problem with jellies is Fermi problem, it is approximation. If you have more people giving they answers mean of approximation will be closer to actual value. But second part is about concrete thing it is not estimating or guessing. First part is totally irrelevant. Second part could be used alone. But you just know that you can't please everyone so it is common sense I think. Like going for trip with people who are working in different jobs on different schedules, somebody will have to be left out.

49 points by joeyespo 14y ago

As programmers we should be well aware that our customers' "needs" are bounded by what they a) know and b) believe is possible. Guiding them to technological enlightenment can and does provide rewards (financial and otherwise).

fleitz 14y ago
Are you essentially saying that if you listen to your customers you'll build a faster horse, instead of a car?
- smoyer 14y ago
  No ... I'm saying that sometimes we are evangelists and can make them realize they want a flying car.

wisty 14y ago

The other problem with focus groups - they aren't independent. There will always be one person who sounds convincing, and the rest will be biased to whatever that person thinks. The advantage of focus groups is, you can identify themes which resonate. For example, the group might say "integrated" sounds good. That doesn't mean you want an integrated product, it just means that you need a sticker which says "integrated", and some justification (We integrate with Facebook, via our "Like" buttons!)

hammock 14y ago
>The other problem with focus groups - they aren't independent. There will always be one person who sounds convincing, and the rest will be biased to whatever that person thinks. Experienced group moderators are aware of this take steps to mitigate and reflect in their notes. There are other forms of market research beyond focus groups. Aggregated quantitative and qualitative one-on-ones also fit the OP's bill of "crowd wisdom"

freshhawk 14y ago

I stopped taking this seriously when they compared the accuracy of the "ask the audience" and "phone a friend" bits on Who Wants to be a Millionaire without considering that everyone used the ask the audience lifeline on an early question and saved the phone a friend for later questions. Without controlling for when they are used the comparison of accuracy is literally meaningless. The same stats show that, rightly or wrongly, the contestants didn't have as much faith in the audience for later stage, much harder questions. I guess I should give the benefit of the doubt and assume the writer didn't notice this and the error is accidental but that's only slightly better than just ignoring it because you wanted a sexy anecdote to support your point.

hammock 14y ago
You right that the two lifelines are used at differing stages in the game. I wonder though, what the accuracy rates look like after controlling for what stage of the game you are in. Maybe people SHOULD save their ask the audience for later in the game.
- T-hawk 14y ago
  Another variable is the subject matter of the question. Anecdotally, it feels that the audience almost invariably nails the answer by a wide margin on pop culture questions. On questions of a scientific or technical nature, the audience opinion is much less clear and more often actually wrong. Phone those questions to a domain knowledge expert. About the worst thing you can do, but contestants love doing this, is to burn the valuable audience lifeline early on a question when you already have a good sense of the answer but just want to "make sure". What a waste! Save that for a question when you have no clue, which WILL happen before you reach the million level.
  - nisse72 14y ago
    Something else they often do is say something like "I'm thinking it's B, but I'm not sure so I'll ask the audience". Well guess what the audience guesses after hearing that?
- 14y ago

billswift 14y ago

His further reading list includes Surowiecki's The Wisdom of Crowds, but his post doesn't look like he read it himself. Surowiecki has a long list of requirements for the "wisdom of crowds" to actually be helpful, for one example the individual judgments need to be independent, that the post ignores. Without the restrictions you get different failure modes, like bubbles, runs, and traffic jams.

chmike 14y ago
Or election mistakes. Democracy assumes people's choice is independent, though it is not. We are still very exposed to manipulation.
smartbear 14y ago
The essay was intended to be a light-hearted and readable intro to the concept, not an exhaustive definition that takes an entire book!
- billswift 14y ago
  Almost everyone has at least heard the phrase "the wisdom of crowds" by now. The most important part of the idea, and the one most people are not familiar with, are the (fairly severe) limits on its usefulness. I wrote a review of the book for Amazon, and one thing I pointed out is that it was a much better balanced treatment of the subject than I had expected from all the hype about the idea that I had previously read. The hype is already all over the place, and has been for years, there is no need to add to it.

antithesis 14y ago

He isn't being very fair: where the goal at first was to be as right as possible, it later became "to piss off as few of the 300 guests as possible". I'm not sure what exact procedure he had in mind for businesses, but in a situation where "votes eliminate the interesting edges, leaving only the boring residue that no one hated enough to vote off the island" applies, you're just not talking about the wisdom of crowds anymore, this wasn't possible in his earlier examples either. An important factor to the wisdom of crowds, and a key difference between his earlier examples and his later examples, is that people vote independently, without even talking about what they're going to vote. For demonstrational purposes, I will now take his holiday meal situation and apply actual wisdom of crowds to them. This isn't a very usual situation, but this is about properly applying the wisdom of crowds. Instead of people stating what they dislike, people may nominate a meal, and afterwards, everyone votes for the meal they want to go with. Like in the Wants to be a Millionaire examples, people cannot talk in this process - they cannot state what they dislike. Instead, the fact that some people can't take spicy food will show in the results, but beside that, every individual will pick a meal that isn't boring, and the end result will be just that. "Allergic to garlic" and "doesn’t eat anything green" are rather specific complaints and were not taken into account, so let's count those 3 people as unsatisfied, but the success here is that most people will in fact be satisfied with the meal.

smartbear 14y ago
You can't use your technique for meals because everyone will suggest a different meal, and then you have no basis for a decision. That technique works when you're voting on one variable which has one CORRECT value which you're trying to seek. Creative work is not that.

neuroelectronic 14y ago

iPhone is chicken and rice to me.

ozim 14y ago

This essey is weak, I mean author is comparing apples to oranges. First problem with jellies is Fermi problem, it is approximation. If you have more people giving they answers mean of approximation will be closer to actual value. But second part is about concrete thing it is not estimating or guessing. First part is totally irrelevant. Second part could be used alone. But you just know that you can't please everyone so it is common sense I think. Like going for trip with people who are working in different jobs on different schedules, somebody will have to be left out.