Could you explain KL divergence and/or cross entropy to me? from wikipedia: "In mathematical statistics, the Kullback–Leibler divergence is a measure of how on

Could you explain KL divergence and/or cross entropy to me? from wikipedia: "In mathematical statistics, the Kullback–Leibler divergence is a measure of how one probability distribution diverges from a second expected probability distribution" Kullback-Leibler divergence is used a lot in the reinforcement learning setting "In information theory, the cross entropy between two probability distributions p and q over the same underlying set of events measures the average number of bits needed to identify an event drawn from the set" Cross Entropy is used as a cost function in neural networks rather than least squares http://neuralnetworksanddeeplearning.com/chap3.html#the_cross-entropy_cost_function http://neuralnetworksanddeeplearning.com/chap3.html#the_cros...