5 ms·
I don't want to discount the usefulness of log probabilities. But this post gets off to a poor start. > Probabilities are very rarely added together, and proba
by abeppu 6y ago
I don't want to discount the usefulness of log probabilities. But this post gets off to a poor start.
> Probabilities are very rarely added together, and probability distributions even more rarely.
Umm, we _often_ add probabilities when we want to know the probability that any of a number of finite but mutually exclusive outcomes will happen.
We often add distributions (perhaps with weights) when we think about mixture models, or when there are multiple mechanisms which can both give rise to the same outcomes.
A point the author didn't make about log probabilities which likely relevant to this audience is that we often think about the product of many probability terms when modeling a complex outcome of many steps (e.g. a language model where words are 'drawn' from a distribution, and so the probability of a sentence is a product of many terms corresponding to many generative steps). In these cases log probabilities are often also important for computations to avoid underflow issues. I.e. when you have to reason about large, complex, composed structures generated from stochastic processes, _everything_ can be astoundingly unlikely in absolute terms, and log probabilities let us keep track of the important differences between astoundingly unlikely things.
- moultano 6y agoSure, but mixture models and mutually exclusive events are less common to work with than just "models." Obviously there are times when you add probabilities together, but usually you don't. Even when you're writing a mixture model, or doing bayes nets, most of the work is done with log probabilities in the various branches, and only retransformed into linear space when the branches are combined.
- srean 6y agoMixture models are surprisingly common and addition/subtraction are not limited to mutually exclusive events. In such cases one goes the route of inclusion/exclusion formula. Summation is so fundamental to probabilities that it is how probabilities are even defined axiomatically in the Kolmogorov system. https://en.wikipedia.org/wiki/Sigma_additivity https://en.wikipedia.org/wiki/Sigma_additivity I think what you meant was that in computation one often needs to multiply probabilities and in such cases it is helpful to work in the log semiring https://en.wikipedia.org/wiki/Log_semiring https://en.wikipedia.org/wiki/Log_semiring
- kgwgk 6y ago> Obviously there are times when you add probabilities together, but usually you don't. Maybe you don’t but it’s hardly unusual. Integrating out nuisance parameters from a posterior distribution is adding probabilities. Monte Carlo methods are based on adding probabilities.
- mb7733 6y ago>Sure, but mixture models and mutually exclusive events are less common to work with than just "models." What do you mean by this? A mixture model is just a model. Also, how can you do absolutely anything without "working with mutually exclusive events"? You need mutual exclusivity to support the concept of a decision or a hypothesis.
- joppy 6y agoLog-probabilities are added so much that there are often special-purpose functions for adding them, Eg https://numpy.org/doc/stable/reference/generated/numpy.logaddexp.html https://numpy.org/doc/stable/reference/generated/numpy.logad...