I think KL divergence is best understood through the lens of variational inference. Inference in the Bayesian regime is a balancing act between best explaining

I think KL divergence is best understood through the lens of variational inference. Inference in the Bayesian regime is a balancing act between best explaining the data and “keeping it simple”, by staying close to the prior. KL divergence is the (only) divergence measure that places just the right amount of weight on the prior to make variational inference proper Bayesian inference. I’ve written about it here: http://www.openias.org/variational-coin-toss http://www.openias.org/variational-coin-toss