3 ms·
> The entropy is the expected number of bits when this allocation is done using the incorrect distribution Is there any source that would derive and/or explain
by tziki 8y ago
> The entropy is the expected number of bits when this allocation is done using the incorrect distribution
Is there any source that would derive and/or explain this more in-depth? I've been trying to develop an intuition for this, but haven't come across a good explanation.
- pizza 8y agothe OG http://math.harvard.edu/~ctm/home/text/others/shannon/entropy/entropy.pdf http://math.harvard.edu/~ctm/home/text/others/shannon/entrop... also lookup kullback-leibler divergence
- banachtarski 8y agoThe other reply mentioning "kullback-leibler divergence" (aka KL divergence) is what you need to understand as this is the fundamental concept. Minimizing this quantity is equivalent to minimizing the given "cross-entropy loss" expression. More generally to understand where this comes from, you'll want to read about information theory.
- banachtarski 8y agohttps://rdipietro.github.io/friendly-intro-to-cross-entropy-loss/ https://rdipietro.github.io/friendly-intro-to-cross-entropy-... I found this which seems much better