As a side note, KL divergence is actually symmetric to the second order: If you have a distribution "p" parameterized by x, the divergence between p|x_0 from a

As a side note, KL divergence is actually symmetric to the second order: If you have a distribution "p" parameterized by x, the divergence between p|x_0 from a nearby value p|x_1 is approximately symmetric. This is useful because the Hessian of the KL divergence (if it exists) with respect to the parameters of P defines a Riemannian metric called the Fisher information metric. This provides a good distance measure that takes into account how much the information content, or entropy, changes as you move around in parameter space. This is really useful for fast online variational Bayesian methods. Gradient descent in the Euclidean space of the parameters can be pretty lousy, but using the "natural gradient" using the Fisher information metric gives a more natural definition of distance. The Fisher information can be derived more generally if the Hessian doesn't exist: it is also the variance of the gradient of the log-likelihood.