4 ms·

To summarize succinctly, KL(q||p) quantifies how badly you screw up if the true distribution is “q” and you instead think it is “p”. Note that KL divergence is

To summarize succinctly, KL(q||p) quantifies how badly you screw up if the true distribution is “q” and you instead think it is “p”. Note that KL divergence is not symmetric! Eg: If the true distribution of coin tosses is 100% heads and your model has 50/50, you won’t mess up big — compared with when the true coin is 50/50 and your model is 100 percent heads (and you would have been willing to bet a LOT of money that there will be no tails in the outcome). In this technical sense, it is preferable to be conservative than overly confident.

ASpring 8y ago
Good point on KL divergence not being symmetric. For anyone who wants to quantify KL divergence in a way that can be used as a metric, look into Jensen-Shannon distance which is based on KL-divergence.
cultus 8y ago
As a side note, KL divergence is actually symmetric to the second order: If you have a distribution "p" parameterized by x, the divergence between p|x_0 from a nearby value p|x_1 is approximately symmetric. This is useful because the Hessian of the KL divergence (if it exists) with respect to the parameters of P defines a Riemannian metric called the Fisher information metric. This provides a good distance measure that takes into account how much the information content, or entropy, changes as you move around in parameter space. This is really useful for fast online variational Bayesian methods. Gradient descent in the Euclidean space of the parameters can be pretty lousy, but using the "natural gradient" using the Fisher information metric gives a more natural definition of distance. The Fisher information can be derived more generally if the Hessian doesn't exist: it is also the variance of the gradient of the log-likelihood.
- shmageggy 8y ago
  Very interesting and useful. Got any good references for further reading?
- srean 8y ago
  > As a side note, KL divergence is actually symmetric to the second order: This is backwards. Its a tautology. Any function no matter how egregiously asymmetric is locally symmetric if it is twice differentiable at that point. This is so by construction, you are approximating it locally by the best possible quadratic [hence locally symmetric] curve [surface]. Hence, the claim about symmetry is not false, but it is vacuous. Much like the claim that the equation of French curve is such that no matter how you turn it at its highest point it makes a tangent with the horizontal. That said Fisher information metric does have many uses.

ASpring 8y ago

Good point on KL divergence not being symmetric. For anyone who wants to quantify KL divergence in a way that can be used as a metric, look into Jensen-Shannon distance which is based on KL-divergence.

cultus 8y ago

As a side note, KL divergence is actually symmetric to the second order: If you have a distribution "p" parameterized by x, the divergence between p|x_0 from a nearby value p|x_1 is approximately symmetric. This is useful because the Hessian of the KL divergence (if it exists) with respect to the parameters of P defines a Riemannian metric called the Fisher information metric. This provides a good distance measure that takes into account how much the information content, or entropy, changes as you move around in parameter space. This is really useful for fast online variational Bayesian methods. Gradient descent in the Euclidean space of the parameters can be pretty lousy, but using the "natural gradient" using the Fisher information metric gives a more natural definition of distance. The Fisher information can be derived more generally if the Hessian doesn't exist: it is also the variance of the gradient of the log-likelihood.

shmageggy 8y ago
Very interesting and useful. Got any good references for further reading?
srean 8y ago
> As a side note, KL divergence is actually symmetric to the second order: This is backwards. Its a tautology. Any function no matter how egregiously asymmetric is locally symmetric if it is twice differentiable at that point. This is so by construction, you are approximating it locally by the best possible quadratic [hence locally symmetric] curve [surface]. Hence, the claim about symmetry is not false, but it is vacuous. Much like the claim that the equation of French curve is such that no matter how you turn it at its highest point it makes a tangent with the horizontal. That said Fisher information metric does have many uses.