6 ms·

How to Avoid Idea Averaging

43 points by andrewxhill 11y ago

p4wnc6 11y ago
Idea averaging is a wonderful thing, and you should probably do it way more than you already do, and it should probably be your default mode when looking to choose between ideas. A lot of this is spelled out in the literature about why ensemble methods tend to be superior to individual methods in statistical inference, which has a lot of ties to why averages of non-expert opinions tend to be as accurate or even more accurate than small pools of expert opinions. In the limit, when you reallocate credibility to a set of ideas in a manner that is fully derived from your prior beliefs and the evidence you have, it becomes exactly Bayesian reasoning. A posterior distribution is exactly an "idea average" where each idea (each potential outcome) gets as much (and no more) credibility as it deserves, according to the prior and the probability model at hand. Here's somewhat of a popular account of it: < https://www.washingtonpost.com/business/is-it-better-to-trust-the-best-expert-or-the-average-of-a-group-of-experts/2013/11/15/59cc716e-4b01-11e3-be6b-d3d28122e6d4_story.html https://www.washingtonpost.com/business/is-it-better-to-trus... > There is also the recent stuff under the marketing buzzword "Wisdom of Crowds" e.g. < http://onlinelibrary.wiley.com/doi/10.1111/j.1551-6709.2011.01223.x/abstract;jsessionid=58C236728B81FFF49A4E84DF1E3B65CB.f02t04 http://onlinelibrary.wiley.com/doi/10.1111/j.1551-6709.2011.... > or the Wikipedia article on it too (and many sources that point out potential problems and corner cases that are also important). In general though, I think I have to disagree with the article's premise. Unless two distinct ideas truly are mutually exclusive (most things aren't), then it's better to have a diversified portfolio over the space of all the ideas than to put all your eggs in one basket, and averaging ideas is sort of a humble heuristic that seems to work in a lot of areas.
- gimlids 11y ago
  Averaging different economists' predictions makes sense. They each provide an x-y curve. Those curves average together in a mathematical way. Product design ideas don't average together in this way. If one person wants to design a knife and another wants to design a spoon, and you average them together, you get a spoon with a sharp blade on one side, or maybe the blade is on the handle of the spoon, but either way, bad idea!! Or best case maybe you end up with a swiss army knife, in which case it's harder to clean, etc. Adding the knife feature sacrifices the simplicity of the spoon, which is extremely valuable.
  - p4wnc6 11y ago
    First, I did mention that when two things truly are mutually exclusive, it can be a reason for picking only one. Two designs that are entirely non-functional when combined could count as mutually exclusive. Even two designs that could be successful each on their own, but such that neither one could be created with 1/2 of the total budget, could be considered mutually exclusive. I think you're pointing out a straw-man though. There are lots of cases where two ideas are not totally mutually exclusive, especially in design. And there are lots of times when the budget will allow you to explore building both designs in parallel and "investing in both" (thus diversifying your investment by not allocating the whole budget to only one project). I agree when there is real mutual exclusivity you have to confidently choose one way to invest. I just think this is rarer than stated, and that lots of times you aren't actually forced to make that type of bifurcating decision, and so in general you should be eager to compromise more and more humble about not knowing which of the ideas will work, thus more willing to average over them, or divide up the budget and try multiple things.
- purplerabbit 11y ago
  It's hard for me to understand exactly what it is that you're saying. You're trying to layer too many insights into your sentences, which which makes your comment confusing
- davmre 11y ago
  This is a very confused post. A posterior is not an "average", it's a probability distribution. That distribution may have a mean value, but that's not a meaningful quantity in general. For example, you're chasing someone and you come to a fork in the road. You know they must have taken the left or the right branch, so your posterior distribution on their location has two modes. It also has a mean value, in the middle of the forest between the two branches, but unless you have specific evidence they've abandoned the road, this is a very low-probability state. The type of averaging prescribed by Bayesian decision theory is averaging of utilities, not of beliefs or actions. You take the action having the highest expected utility, where the expectation is taken over your posterior distribution. Assuming you actually want to catch your suspect, the expected utility of following either the left or the right fork will be much higher than the utility of dropping the pursuit to poke around in the middle of the forest.
  - p4wnc6 11y ago
    I think you are mistaken. Yes, a posterior is a probability distribution, which is exactly taking each possible outcome in accordance with its probability. I didn't say anything about collapsing the distribution down to a single point estimate like the posterior mean, or the MAP estimate, or any other single statistic. I am saying that I hold in my head a bunch of beliefs all at the same time (thus they are "blended" or "averaged" together), each in accordance with the amount of credibility it deserves. I think it's totally fine to speak about this as a type of uncollapsed "averaging" process, and indeed when you do something like hierarchical models where you supply a metaprior distinguishing between two alternative models, we widely talk about such things as model averaging even in the statistics literature. It seems like a rather misguided nitpick to me to insist that the English word "average" cannot be invoked unless it specifically coincides with the statistics word "mean". Further, you're simply wrong about Bayesianism being about averages of utilities. The best account of Bayesian probability as a mapping of plausibilities to the unit interval is in Jaynes' The Logic of Science, but there is also a brief account of it in David Mumford's essay The Dawning of the Age of Stochasticity and e.g. in the introduction of Bayesian estimation supersedes the t-test by Kruschke [1] (where he explicitly describes it in terms of reallocation of credibility). Further, a la Jaynes, I think the right way to understand probability at all is in a mind-projection fallacy sense of the term: it describes your state of ignorance about the uncertain item. It's not about utilities of actions -- utilities can be modeled with probabilities if the utility of certain actions is uncertain, but that is different than the base concept of probability being about which outcomes are more valued. [1] http://www.indiana.edu/~kruschke/BEST/BEST.pdf http://www.indiana.edu/~kruschke/BEST/BEST.pdf
mizzao 11y ago
Idea averaging is not an example of "the wisdom of crowds". It's a manifestation of groupthink. More specifically, it's an example of when a group of people will tend to agree on something that is the least objectionable on average. This means that it's less likely that an unconventional idea will be accepted. This concept has been studied for a long time in the social psychology literature. For a survey of related topics: http://chicagounbound.uchicago.edu/cgi/viewcontent.cgi?article=1165&context=law_and_economics http://chicagounbound.uchicago.edu/cgi/viewcontent.cgi?artic... (see specifically "Hidden Profiles and Common Knowledge").
- p4wnc6 11y ago
  Since most unconventional ideas tend to fail (because people have thought for a long time about what would be good, and over time those things have often become conventional), it is a useful heuristic to go with conservative, conventional ideas. I think this totally fits in with "wisdom of crowds". Of course, this also means you won't generate that once-in-a-million super great unconventional idea. But then again, I think that's the point. If you try to generate the great unconventional idea, the base rate of failure is so high that you should not have so much hubris as to believe you actually can do it, and most often you should admit that what you think is a great unconventional idea really probably can't do any better than whatever conventional wisdom would choose. I think my view on this is not popular because everyone wants to believe in the idea underdog -- the idea that was spurned by conventional wisdom but proved itself regardless and made someone famous for executing upon it. That's great and all, but in the same way that putting all of your money on a single stock is a bad strategy, failing to average across ideas embraced by conventional wisdom is also (often, but not always) a bad strategy. On a different note though, there is a lot of value in your comment even connecting it back to the "wisdom of crowds" stuff. One of the issues with "wisdom of crowds" is that the aggregated average tends only to be better whenever there is a sufficient variance of opinions in the crowd. This way, there is not systematic bias in everyone's thinking, meaning that the average too will have that bias. The more variation, the more that each participant's particular way of being wrong will contribute to "canceling out" the errors, leaving the average in a good position. To me it suggests that when unconventional ideas do hit it big, it is because they are highly related to some source of demonstrable bias in conventional wisdom, and by removing that bias, such ideas can actually improve upon whatever is the current state of conventional wisdom. But most of the time, it seems we only hit upon these bias-based improvements by accident because tons and tons of people keep trying (and failing, often to their personal detriment) to commit to unconventional ideas that aren't articulated in terms of any demonstrable bias, and it ends up basically being an idea lottery to see who happens to hit the "jackpot" idea. That's what makes conventional wisdom appealing as an alternative, steady expected return with low risk, but it's also what means it will never lead you to the next big thing.
cheetos 11y ago
Also known as design by committee: https://en.wikipedia.org/wiki/Design_by_committee https://en.wikipedia.org/wiki/Design_by_committee
erikrothoff 11y ago
What would you call this topic or category of theory? I'm interested in learning more. Any books or literature that's recommended? I've become the "product" guy in the startup I'm in, and would love to learn more about helping a group innovate and push forward.
- andrewxhill 11y ago
  Not sure... what part of it interests you? Teamwork? Innovation? Decision making? I was putting together a "best read of 2015" type post but never finished it. There were a couple on my list that I think are related and maybe you'd find interesting. This one from HN a few weeks back, http://justinkan.com/three-stories http://justinkan.com/three-stories Higher level company organization stuff I thought these two books gave at times contradictory and at times very complimentary thoughts on what made teams work. In reality, neither were exactly about making teams work though :) Playing to Win http://www.amazon.com/Playing-Win-Strategy-Really-Works/dp/1491528796 http://www.amazon.com/Playing-Win-Strategy-Really-Works/dp/1... is on creating strategies Reinventing organizations http://www.amazon.com/Reinventing-Organizations-Frederic-Laloux/dp/2960133501 http://www.amazon.com/Reinventing-Organizations-Frederic-Lal... is about holacracy but talks a lot to people's motivations in a company. Hopefully I have a couple more interesting things to say on andrewxhill.com too, so keep an eye on it!
sharemywin 11y ago
I think something the article missed is you should probably test your ideas(cheaply if possible) because facts are better than opinions(apparently drunken' ones in the authors case).
snowwrestler 11y ago
The rotating image carousel seems like an example of idea averaging. - "What should we put on our homepage?" - A bunch of people have different ideas. - "I know, we can put them all up with a carousel!" Even Apple has succumbed to his, which is perhaps not surprising. Carousels often come out of collaborative cultures (everyone wins!), which has been a point of emphasis for Tim Cook. Avoiding carousels requires that a single person is willing to make a bunch of colleagues unhappy by picking only one thing at a time to feature.

43 points by andrewxhill 11y ago

Idea averaging is a wonderful thing, and you should probably do it way more than you already do, and it should probably be your default mode when looking to choose between ideas. A lot of this is spelled out in the literature about why ensemble methods tend to be superior to individual methods in statistical inference, which has a lot of ties to why averages of non-expert opinions tend to be as accurate or even more accurate than small pools of expert opinions. In the limit, when you reallocate credibility to a set of ideas in a manner that is fully derived from your prior beliefs and the evidence you have, it becomes exactly Bayesian reasoning. A posterior distribution is exactly an "idea average" where each idea (each potential outcome) gets as much (and no more) credibility as it deserves, according to the prior and the probability model at hand. Here's somewhat of a popular account of it: < https://www.washingtonpost.com/business/is-it-better-to-trust-the-best-expert-or-the-average-of-a-group-of-experts/2013/11/15/59cc716e-4b01-11e3-be6b-d3d28122e6d4_story.html https://www.washingtonpost.com/business/is-it-better-to-trus... > There is also the recent stuff under the marketing buzzword "Wisdom of Crowds" e.g. < http://onlinelibrary.wiley.com/doi/10.1111/j.1551-6709.2011.01223.x/abstract;jsessionid=58C236728B81FFF49A4E84DF1E3B65CB.f02t04 http://onlinelibrary.wiley.com/doi/10.1111/j.1551-6709.2011.... > or the Wikipedia article on it too (and many sources that point out potential problems and corner cases that are also important). In general though, I think I have to disagree with the article's premise. Unless two distinct ideas truly are mutually exclusive (most things aren't), then it's better to have a diversified portfolio over the space of all the ideas than to put all your eggs in one basket, and averaging ideas is sort of a humble heuristic that seems to work in a lot of areas.

gimlids 11y ago
Averaging different economists' predictions makes sense. They each provide an x-y curve. Those curves average together in a mathematical way. Product design ideas don't average together in this way. If one person wants to design a knife and another wants to design a spoon, and you average them together, you get a spoon with a sharp blade on one side, or maybe the blade is on the handle of the spoon, but either way, bad idea!! Or best case maybe you end up with a swiss army knife, in which case it's harder to clean, etc. Adding the knife feature sacrifices the simplicity of the spoon, which is extremely valuable.
- p4wnc6 11y ago
  First, I did mention that when two things truly are mutually exclusive, it can be a reason for picking only one. Two designs that are entirely non-functional when combined could count as mutually exclusive. Even two designs that could be successful each on their own, but such that neither one could be created with 1/2 of the total budget, could be considered mutually exclusive. I think you're pointing out a straw-man though. There are lots of cases where two ideas are not totally mutually exclusive, especially in design. And there are lots of times when the budget will allow you to explore building both designs in parallel and "investing in both" (thus diversifying your investment by not allocating the whole budget to only one project). I agree when there is real mutual exclusivity you have to confidently choose one way to invest. I just think this is rarer than stated, and that lots of times you aren't actually forced to make that type of bifurcating decision, and so in general you should be eager to compromise more and more humble about not knowing which of the ideas will work, thus more willing to average over them, or divide up the budget and try multiple things.
purplerabbit 11y ago
It's hard for me to understand exactly what it is that you're saying. You're trying to layer too many insights into your sentences, which which makes your comment confusing
davmre 11y ago
This is a very confused post. A posterior is not an "average", it's a probability distribution. That distribution may have a mean value, but that's not a meaningful quantity in general. For example, you're chasing someone and you come to a fork in the road. You know they must have taken the left or the right branch, so your posterior distribution on their location has two modes. It also has a mean value, in the middle of the forest between the two branches, but unless you have specific evidence they've abandoned the road, this is a very low-probability state. The type of averaging prescribed by Bayesian decision theory is averaging of utilities, not of beliefs or actions. You take the action having the highest expected utility, where the expectation is taken over your posterior distribution. Assuming you actually want to catch your suspect, the expected utility of following either the left or the right fork will be much higher than the utility of dropping the pursuit to poke around in the middle of the forest.
- p4wnc6 11y ago
  I think you are mistaken. Yes, a posterior is a probability distribution, which is exactly taking each possible outcome in accordance with its probability. I didn't say anything about collapsing the distribution down to a single point estimate like the posterior mean, or the MAP estimate, or any other single statistic. I am saying that I hold in my head a bunch of beliefs all at the same time (thus they are "blended" or "averaged" together), each in accordance with the amount of credibility it deserves. I think it's totally fine to speak about this as a type of uncollapsed "averaging" process, and indeed when you do something like hierarchical models where you supply a metaprior distinguishing between two alternative models, we widely talk about such things as model averaging even in the statistics literature. It seems like a rather misguided nitpick to me to insist that the English word "average" cannot be invoked unless it specifically coincides with the statistics word "mean". Further, you're simply wrong about Bayesianism being about averages of utilities. The best account of Bayesian probability as a mapping of plausibilities to the unit interval is in Jaynes' The Logic of Science, but there is also a brief account of it in David Mumford's essay The Dawning of the Age of Stochasticity and e.g. in the introduction of Bayesian estimation supersedes the t-test by Kruschke [1] (where he explicitly describes it in terms of reallocation of credibility). Further, a la Jaynes, I think the right way to understand probability at all is in a mind-projection fallacy sense of the term: it describes your state of ignorance about the uncertain item. It's not about utilities of actions -- utilities can be modeled with probabilities if the utility of certain actions is uncertain, but that is different than the base concept of probability being about which outcomes are more valued. [1] http://www.indiana.edu/~kruschke/BEST/BEST.pdf http://www.indiana.edu/~kruschke/BEST/BEST.pdf

mizzao 11y ago

Idea averaging is not an example of "the wisdom of crowds". It's a manifestation of groupthink. More specifically, it's an example of when a group of people will tend to agree on something that is the least objectionable on average. This means that it's less likely that an unconventional idea will be accepted. This concept has been studied for a long time in the social psychology literature. For a survey of related topics: http://chicagounbound.uchicago.edu/cgi/viewcontent.cgi?article=1165&context=law_and_economics http://chicagounbound.uchicago.edu/cgi/viewcontent.cgi?artic... (see specifically "Hidden Profiles and Common Knowledge").

p4wnc6 11y ago
Since most unconventional ideas tend to fail (because people have thought for a long time about what would be good, and over time those things have often become conventional), it is a useful heuristic to go with conservative, conventional ideas. I think this totally fits in with "wisdom of crowds". Of course, this also means you won't generate that once-in-a-million super great unconventional idea. But then again, I think that's the point. If you try to generate the great unconventional idea, the base rate of failure is so high that you should not have so much hubris as to believe you actually can do it, and most often you should admit that what you think is a great unconventional idea really probably can't do any better than whatever conventional wisdom would choose. I think my view on this is not popular because everyone wants to believe in the idea underdog -- the idea that was spurned by conventional wisdom but proved itself regardless and made someone famous for executing upon it. That's great and all, but in the same way that putting all of your money on a single stock is a bad strategy, failing to average across ideas embraced by conventional wisdom is also (often, but not always) a bad strategy. On a different note though, there is a lot of value in your comment even connecting it back to the "wisdom of crowds" stuff. One of the issues with "wisdom of crowds" is that the aggregated average tends only to be better whenever there is a sufficient variance of opinions in the crowd. This way, there is not systematic bias in everyone's thinking, meaning that the average too will have that bias. The more variation, the more that each participant's particular way of being wrong will contribute to "canceling out" the errors, leaving the average in a good position. To me it suggests that when unconventional ideas do hit it big, it is because they are highly related to some source of demonstrable bias in conventional wisdom, and by removing that bias, such ideas can actually improve upon whatever is the current state of conventional wisdom. But most of the time, it seems we only hit upon these bias-based improvements by accident because tons and tons of people keep trying (and failing, often to their personal detriment) to commit to unconventional ideas that aren't articulated in terms of any demonstrable bias, and it ends up basically being an idea lottery to see who happens to hit the "jackpot" idea. That's what makes conventional wisdom appealing as an alternative, steady expected return with low risk, but it's also what means it will never lead you to the next big thing.

cheetos 11y ago

Also known as design by committee: https://en.wikipedia.org/wiki/Design_by_committee https://en.wikipedia.org/wiki/Design_by_committee

erikrothoff 11y ago

What would you call this topic or category of theory? I'm interested in learning more. Any books or literature that's recommended? I've become the "product" guy in the startup I'm in, and would love to learn more about helping a group innovate and push forward.

andrewxhill 11y ago
Not sure... what part of it interests you? Teamwork? Innovation? Decision making? I was putting together a "best read of 2015" type post but never finished it. There were a couple on my list that I think are related and maybe you'd find interesting. This one from HN a few weeks back, http://justinkan.com/three-stories http://justinkan.com/three-stories Higher level company organization stuff I thought these two books gave at times contradictory and at times very complimentary thoughts on what made teams work. In reality, neither were exactly about making teams work though :) Playing to Win http://www.amazon.com/Playing-Win-Strategy-Really-Works/dp/1491528796 http://www.amazon.com/Playing-Win-Strategy-Really-Works/dp/1... is on creating strategies Reinventing organizations http://www.amazon.com/Reinventing-Organizations-Frederic-Laloux/dp/2960133501 http://www.amazon.com/Reinventing-Organizations-Frederic-Lal... is about holacracy but talks a lot to people's motivations in a company. Hopefully I have a couple more interesting things to say on andrewxhill.com too, so keep an eye on it!

sharemywin 11y ago

I think something the article missed is you should probably test your ideas(cheaply if possible) because facts are better than opinions(apparently drunken' ones in the authors case).

snowwrestler 11y ago

The rotating image carousel seems like an example of idea averaging. - "What should we put on our homepage?" - A bunch of people have different ideas. - "I know, we can put them all up with a carousel!" Even Apple has succumbed to his, which is perhaps not surprising. Carousels often come out of collaborative cultures (everyone wins!), which has been a point of emphasis for Tim Cook. Avoiding carousels requires that a single person is willing to make a bunch of colleagues unhappy by picking only one thing at a time to feature.