6 ms·

Idea averaging is a wonderful thing, and you should probably do it way more than you already do, and it should probably be your default mode when looking to cho

by p4wnc6 11y ago

Idea averaging is a wonderful thing, and you should probably do it way more than you already do, and it should probably be your default mode when looking to choose between ideas. A lot of this is spelled out in the literature about why ensemble methods tend to be superior to individual methods in statistical inference, which has a lot of ties to why averages of non-expert opinions tend to be as accurate or even more accurate than small pools of expert opinions. In the limit, when you reallocate credibility to a set of ideas in a manner that is fully derived from your prior beliefs and the evidence you have, it becomes exactly Bayesian reasoning. A posterior distribution is exactly an "idea average" where each idea (each potential outcome) gets as much (and no more) credibility as it deserves, according to the prior and the probability model at hand. Here's somewhat of a popular account of it: < https://www.washingtonpost.com/business/is-it-better-to-trust-the-best-expert-or-the-average-of-a-group-of-experts/2013/11/15/59cc716e-4b01-11e3-be6b-d3d28122e6d4_story.html https://www.washingtonpost.com/business/is-it-better-to-trus... > There is also the recent stuff under the marketing buzzword "Wisdom of Crowds" e.g. < http://onlinelibrary.wiley.com/doi/10.1111/j.1551-6709.2011.01223.x/abstract;jsessionid=58C236728B81FFF49A4E84DF1E3B65CB.f02t04 http://onlinelibrary.wiley.com/doi/10.1111/j.1551-6709.2011.... > or the Wikipedia article on it too (and many sources that point out potential problems and corner cases that are also important). In general though, I think I have to disagree with the article's premise. Unless two distinct ideas truly are mutually exclusive (most things aren't), then it's better to have a diversified portfolio over the space of all the ideas than to put all your eggs in one basket, and averaging ideas is sort of a humble heuristic that seems to work in a lot of areas.

gimlids 11y ago
Averaging different economists' predictions makes sense. They each provide an x-y curve. Those curves average together in a mathematical way. Product design ideas don't average together in this way. If one person wants to design a knife and another wants to design a spoon, and you average them together, you get a spoon with a sharp blade on one side, or maybe the blade is on the handle of the spoon, but either way, bad idea!! Or best case maybe you end up with a swiss army knife, in which case it's harder to clean, etc. Adding the knife feature sacrifices the simplicity of the spoon, which is extremely valuable.
- p4wnc6 11y ago
  First, I did mention that when two things truly are mutually exclusive, it can be a reason for picking only one. Two designs that are entirely non-functional when combined could count as mutually exclusive. Even two designs that could be successful each on their own, but such that neither one could be created with 1/2 of the total budget, could be considered mutually exclusive. I think you're pointing out a straw-man though. There are lots of cases where two ideas are not totally mutually exclusive, especially in design. And there are lots of times when the budget will allow you to explore building both designs in parallel and "investing in both" (thus diversifying your investment by not allocating the whole budget to only one project). I agree when there is real mutual exclusivity you have to confidently choose one way to invest. I just think this is rarer than stated, and that lots of times you aren't actually forced to make that type of bifurcating decision, and so in general you should be eager to compromise more and more humble about not knowing which of the ideas will work, thus more willing to average over them, or divide up the budget and try multiple things.
purplerabbit 11y ago
It's hard for me to understand exactly what it is that you're saying. You're trying to layer too many insights into your sentences, which which makes your comment confusing
davmre 11y ago
This is a very confused post. A posterior is not an "average", it's a probability distribution. That distribution may have a mean value, but that's not a meaningful quantity in general. For example, you're chasing someone and you come to a fork in the road. You know they must have taken the left or the right branch, so your posterior distribution on their location has two modes. It also has a mean value, in the middle of the forest between the two branches, but unless you have specific evidence they've abandoned the road, this is a very low-probability state. The type of averaging prescribed by Bayesian decision theory is averaging of utilities, not of beliefs or actions. You take the action having the highest expected utility, where the expectation is taken over your posterior distribution. Assuming you actually want to catch your suspect, the expected utility of following either the left or the right fork will be much higher than the utility of dropping the pursuit to poke around in the middle of the forest.
- p4wnc6 11y ago
  I think you are mistaken. Yes, a posterior is a probability distribution, which is exactly taking each possible outcome in accordance with its probability. I didn't say anything about collapsing the distribution down to a single point estimate like the posterior mean, or the MAP estimate, or any other single statistic. I am saying that I hold in my head a bunch of beliefs all at the same time (thus they are "blended" or "averaged" together), each in accordance with the amount of credibility it deserves. I think it's totally fine to speak about this as a type of uncollapsed "averaging" process, and indeed when you do something like hierarchical models where you supply a metaprior distinguishing between two alternative models, we widely talk about such things as model averaging even in the statistics literature. It seems like a rather misguided nitpick to me to insist that the English word "average" cannot be invoked unless it specifically coincides with the statistics word "mean". Further, you're simply wrong about Bayesianism being about averages of utilities. The best account of Bayesian probability as a mapping of plausibilities to the unit interval is in Jaynes' The Logic of Science, but there is also a brief account of it in David Mumford's essay The Dawning of the Age of Stochasticity and e.g. in the introduction of Bayesian estimation supersedes the t-test by Kruschke [1] (where he explicitly describes it in terms of reallocation of credibility). Further, a la Jaynes, I think the right way to understand probability at all is in a mind-projection fallacy sense of the term: it describes your state of ignorance about the uncertain item. It's not about utilities of actions -- utilities can be modeled with probabilities if the utility of certain actions is uncertain, but that is different than the base concept of probability being about which outcomes are more valued. [1] http://www.indiana.edu/~kruschke/BEST/BEST.pdf http://www.indiana.edu/~kruschke/BEST/BEST.pdf

by p4wnc6 11y ago

gimlids 11y ago

Averaging different economists' predictions makes sense. They each provide an x-y curve. Those curves average together in a mathematical way. Product design ideas don't average together in this way. If one person wants to design a knife and another wants to design a spoon, and you average them together, you get a spoon with a sharp blade on one side, or maybe the blade is on the handle of the spoon, but either way, bad idea!! Or best case maybe you end up with a swiss army knife, in which case it's harder to clean, etc. Adding the knife feature sacrifices the simplicity of the spoon, which is extremely valuable.

p4wnc6 11y ago
First, I did mention that when two things truly are mutually exclusive, it can be a reason for picking only one. Two designs that are entirely non-functional when combined could count as mutually exclusive. Even two designs that could be successful each on their own, but such that neither one could be created with 1/2 of the total budget, could be considered mutually exclusive. I think you're pointing out a straw-man though. There are lots of cases where two ideas are not totally mutually exclusive, especially in design. And there are lots of times when the budget will allow you to explore building both designs in parallel and "investing in both" (thus diversifying your investment by not allocating the whole budget to only one project). I agree when there is real mutual exclusivity you have to confidently choose one way to invest. I just think this is rarer than stated, and that lots of times you aren't actually forced to make that type of bifurcating decision, and so in general you should be eager to compromise more and more humble about not knowing which of the ideas will work, thus more willing to average over them, or divide up the budget and try multiple things.

purplerabbit 11y ago

It's hard for me to understand exactly what it is that you're saying. You're trying to layer too many insights into your sentences, which which makes your comment confusing

davmre 11y ago

This is a very confused post. A posterior is not an "average", it's a probability distribution. That distribution may have a mean value, but that's not a meaningful quantity in general. For example, you're chasing someone and you come to a fork in the road. You know they must have taken the left or the right branch, so your posterior distribution on their location has two modes. It also has a mean value, in the middle of the forest between the two branches, but unless you have specific evidence they've abandoned the road, this is a very low-probability state. The type of averaging prescribed by Bayesian decision theory is averaging of utilities, not of beliefs or actions. You take the action having the highest expected utility, where the expectation is taken over your posterior distribution. Assuming you actually want to catch your suspect, the expected utility of following either the left or the right fork will be much higher than the utility of dropping the pursuit to poke around in the middle of the forest.

p4wnc6 11y ago
I think you are mistaken. Yes, a posterior is a probability distribution, which is exactly taking each possible outcome in accordance with its probability. I didn't say anything about collapsing the distribution down to a single point estimate like the posterior mean, or the MAP estimate, or any other single statistic. I am saying that I hold in my head a bunch of beliefs all at the same time (thus they are "blended" or "averaged" together), each in accordance with the amount of credibility it deserves. I think it's totally fine to speak about this as a type of uncollapsed "averaging" process, and indeed when you do something like hierarchical models where you supply a metaprior distinguishing between two alternative models, we widely talk about such things as model averaging even in the statistics literature. It seems like a rather misguided nitpick to me to insist that the English word "average" cannot be invoked unless it specifically coincides with the statistics word "mean". Further, you're simply wrong about Bayesianism being about averages of utilities. The best account of Bayesian probability as a mapping of plausibilities to the unit interval is in Jaynes' The Logic of Science, but there is also a brief account of it in David Mumford's essay The Dawning of the Age of Stochasticity and e.g. in the introduction of Bayesian estimation supersedes the t-test by Kruschke [1] (where he explicitly describes it in terms of reallocation of credibility). Further, a la Jaynes, I think the right way to understand probability at all is in a mind-projection fallacy sense of the term: it describes your state of ignorance about the uncertain item. It's not about utilities of actions -- utilities can be modeled with probabilities if the utility of certain actions is uncertain, but that is different than the base concept of probability being about which outcomes are more valued. [1] http://www.indiana.edu/~kruschke/BEST/BEST.pdf http://www.indiana.edu/~kruschke/BEST/BEST.pdf