The KL divergence is also called relative entropy. Unlike the ordinary entropy, relative entropy is invariant under parameter transformations. The maximum relat

The KL divergence is also called relative entropy. Unlike the ordinary entropy, relative entropy is invariant under parameter transformations. The maximum relative entropy principle generalises Bayesian inference. The distribution relative to which you're computing the entropy plays the role of the prior. By the way, I find the following way to rewrite the entropy easier to understand because all quantities are positive: sum(-p_i log(p_i)) = sum(p_i log(1/p_i)) = E[log(1/p_i)] log(1/p_i) tells you how many bits you need to encode an event with probability p_i. The more unlikely the event, the more bits you need. The entropy is the expected number of bits you need.