4 ms·

This is a very confused post. A posterior is not an "average", it's a probability distribution. That distribution may have a mean value, but that's not a meanin

by davmre 11y ago

This is a very confused post. A posterior is not an "average", it's a probability distribution. That distribution may have a mean value, but that's not a meaningful quantity in general. For example, you're chasing someone and you come to a fork in the road. You know they must have taken the left or the right branch, so your posterior distribution on their location has two modes. It also has a mean value, in the middle of the forest between the two branches, but unless you have specific evidence they've abandoned the road, this is a very low-probability state. The type of averaging prescribed by Bayesian decision theory is averaging of utilities, not of beliefs or actions. You take the action having the highest expected utility, where the expectation is taken over your posterior distribution. Assuming you actually want to catch your suspect, the expected utility of following either the left or the right fork will be much higher than the utility of dropping the pursuit to poke around in the middle of the forest.

p4wnc6 11y ago
I think you are mistaken. Yes, a posterior is a probability distribution, which is exactly taking each possible outcome in accordance with its probability. I didn't say anything about collapsing the distribution down to a single point estimate like the posterior mean, or the MAP estimate, or any other single statistic. I am saying that I hold in my head a bunch of beliefs all at the same time (thus they are "blended" or "averaged" together), each in accordance with the amount of credibility it deserves. I think it's totally fine to speak about this as a type of uncollapsed "averaging" process, and indeed when you do something like hierarchical models where you supply a metaprior distinguishing between two alternative models, we widely talk about such things as model averaging even in the statistics literature. It seems like a rather misguided nitpick to me to insist that the English word "average" cannot be invoked unless it specifically coincides with the statistics word "mean". Further, you're simply wrong about Bayesianism being about averages of utilities. The best account of Bayesian probability as a mapping of plausibilities to the unit interval is in Jaynes' The Logic of Science, but there is also a brief account of it in David Mumford's essay The Dawning of the Age of Stochasticity and e.g. in the introduction of Bayesian estimation supersedes the t-test by Kruschke [1] (where he explicitly describes it in terms of reallocation of credibility). Further, a la Jaynes, I think the right way to understand probability at all is in a mind-projection fallacy sense of the term: it describes your state of ignorance about the uncertain item. It's not about utilities of actions -- utilities can be modeled with probabilities if the utility of certain actions is uncertain, but that is different than the base concept of probability being about which outcomes are more valued. [1] http://www.indiana.edu/~kruschke/BEST/BEST.pdf http://www.indiana.edu/~kruschke/BEST/BEST.pdf
- yummyfajitas 11y ago
  No, it's you who are mistaken. The posterior is not remotely like idea averaging. Idea averaging is about taking multiple ideas and merging them into one. The Bayesian posterior is merely assigning probabilities to each individual idea. I.e., idea averaging is "knife + spoon = spoon with sharp edge". Bayesian posterior is "50% knife is best, 50% spoon is best => evidence => 5% knife is best, 95% spoon is best". The latter is what the article is saying we should do; don't build a new idea incorporating everyone's contributions, instead just figure out which individual idea is the best and do it.
  - p4wnc6 11y ago
    I still think the other comment was mistaken. At a given moment just before drawing a sample from some random variable X, that random variable can have a lot of different possible outcomes. What do I believe X is? I believe X is every different possibility in X's sample space. I simultaneously believe X is a lot of different things. But the degree of belief for each thing is governed by the probability for that outcome being the one I draw. In this sense, my cumulative belief about X is an average of all those different beliefs. That's not the same thing as saying that I have to carry around just one belief that is equal to the posterior mean (or any other point estimate). > idea averaging is "knife + spoon = spoon with sharp edge". Bayesian posterior is "50% knife is best, 50% spoon is best => evidence => 5% knife is best, 95% spoon is best". There's a lot wrong with the above quote from your comment. First, you're confounding two very different things. One thing is whether two ideas are mutually exclusive (e.g. the product is fundamentally non-functional if it is a spoon with sharp edge, so blending the two ideas is not in the set of feasible solutions). I already granted in my first comment that genuine mutual exclusivity can be a reason to avoid idea averaging -- but it's just overstated: true mutual exclusivity like that doesn't happen as much as people argue it does. Secondly, and more importantly, you could break up your budget to pursue the knife solution and the spoon solution independently. What fraction of your budget should you allocate to the knife part? That is, what if 'knife' and 'spoon' are two distinct models that you are considering, and instead of doing crappy model selection to choose merely one of them, you want to produce a model that is a superposition of the two -- what metaprior probability should you choose that the 'knife' model will be the one that succeeds, versus the 'spoon' model? Maybe something more concrete would be helpful, just think of a Gaussian Mixture Model. You have many different models and you're not choosing just one. You are blending them (and people call this model averaging) by using whatever each different component model's prior probability is to govern what the overall outcome is. Maybe we are just talking past one another, but I really think the way you keep emphasizing the idea "posterior is a distribution" makes me think you continue to misunderstand me. Another way to put it is that you are averaging over a space of distributions -- sort of like the Dirichlet distribution. The thing you end up with, the thing that is the result of computing an average, is itself a probability distribution, that comes as an average over a bunch of other probability distributions.

by davmre 11y ago

p4wnc6 11y ago

I think you are mistaken. Yes, a posterior is a probability distribution, which is exactly taking each possible outcome in accordance with its probability. I didn't say anything about collapsing the distribution down to a single point estimate like the posterior mean, or the MAP estimate, or any other single statistic. I am saying that I hold in my head a bunch of beliefs all at the same time (thus they are "blended" or "averaged" together), each in accordance with the amount of credibility it deserves. I think it's totally fine to speak about this as a type of uncollapsed "averaging" process, and indeed when you do something like hierarchical models where you supply a metaprior distinguishing between two alternative models, we widely talk about such things as model averaging even in the statistics literature. It seems like a rather misguided nitpick to me to insist that the English word "average" cannot be invoked unless it specifically coincides with the statistics word "mean". Further, you're simply wrong about Bayesianism being about averages of utilities. The best account of Bayesian probability as a mapping of plausibilities to the unit interval is in Jaynes' The Logic of Science, but there is also a brief account of it in David Mumford's essay The Dawning of the Age of Stochasticity and e.g. in the introduction of Bayesian estimation supersedes the t-test by Kruschke [1] (where he explicitly describes it in terms of reallocation of credibility). Further, a la Jaynes, I think the right way to understand probability at all is in a mind-projection fallacy sense of the term: it describes your state of ignorance about the uncertain item. It's not about utilities of actions -- utilities can be modeled with probabilities if the utility of certain actions is uncertain, but that is different than the base concept of probability being about which outcomes are more valued. [1] http://www.indiana.edu/~kruschke/BEST/BEST.pdf http://www.indiana.edu/~kruschke/BEST/BEST.pdf

yummyfajitas 11y ago
No, it's you who are mistaken. The posterior is not remotely like idea averaging. Idea averaging is about taking multiple ideas and merging them into one. The Bayesian posterior is merely assigning probabilities to each individual idea. I.e., idea averaging is "knife + spoon = spoon with sharp edge". Bayesian posterior is "50% knife is best, 50% spoon is best => evidence => 5% knife is best, 95% spoon is best". The latter is what the article is saying we should do; don't build a new idea incorporating everyone's contributions, instead just figure out which individual idea is the best and do it.
- p4wnc6 11y ago
  I still think the other comment was mistaken. At a given moment just before drawing a sample from some random variable X, that random variable can have a lot of different possible outcomes. What do I believe X is? I believe X is every different possibility in X's sample space. I simultaneously believe X is a lot of different things. But the degree of belief for each thing is governed by the probability for that outcome being the one I draw. In this sense, my cumulative belief about X is an average of all those different beliefs. That's not the same thing as saying that I have to carry around just one belief that is equal to the posterior mean (or any other point estimate). > idea averaging is "knife + spoon = spoon with sharp edge". Bayesian posterior is "50% knife is best, 50% spoon is best => evidence => 5% knife is best, 95% spoon is best". There's a lot wrong with the above quote from your comment. First, you're confounding two very different things. One thing is whether two ideas are mutually exclusive (e.g. the product is fundamentally non-functional if it is a spoon with sharp edge, so blending the two ideas is not in the set of feasible solutions). I already granted in my first comment that genuine mutual exclusivity can be a reason to avoid idea averaging -- but it's just overstated: true mutual exclusivity like that doesn't happen as much as people argue it does. Secondly, and more importantly, you could break up your budget to pursue the knife solution and the spoon solution independently. What fraction of your budget should you allocate to the knife part? That is, what if 'knife' and 'spoon' are two distinct models that you are considering, and instead of doing crappy model selection to choose merely one of them, you want to produce a model that is a superposition of the two -- what metaprior probability should you choose that the 'knife' model will be the one that succeeds, versus the 'spoon' model? Maybe something more concrete would be helpful, just think of a Gaussian Mixture Model. You have many different models and you're not choosing just one. You are blending them (and people call this model averaging) by using whatever each different component model's prior probability is to govern what the overall outcome is. Maybe we are just talking past one another, but I really think the way you keep emphasizing the idea "posterior is a distribution" makes me think you continue to misunderstand me. Another way to put it is that you are averaging over a space of distributions -- sort of like the Dirichlet distribution. The thing you end up with, the thing that is the result of computing an average, is itself a probability distribution, that comes as an average over a bunch of other probability distributions.