The soft maximum often comes up when dealing with probabilistic models and in neural networks. I'm surprised that a blog that has been highlighting numerical is

The soft maximum often comes up when dealing with probabilistic models and in neural networks. I'm surprised that a blog that has been highlighting numerical issues hasn't pointed out how the soft maximum should be computed. If x is vector of values then the naive code is: log(sum(exp(x))) where exp operates elementwise. However, if even a single item of x is large (1000 say) this will return Inf. If x are log probabilities, where you want the log of the sum of the probabilities (common), the elements of x might all be less than -1000 (also common) and then the function will return -Inf. Many people have a function called logsumexp() kicking about to compute the softmax robustly. It will do something like: y = max(x); return y + log(sum(exp(x-y))); One example, slightly more elaborate, Matlab implementation is in: http://research.microsoft.com/en-us/um/people/minka/software/lightspeed/ http://research.microsoft.com/en-us/um/people/minka/software...