I actually sympathize with the theoretician (disclaimer: I work in information-theoretic areas). Information theory is easy to motivate at a first cut, but if y

I actually sympathize with the theoretician (disclaimer: I work in information-theoretic areas). Information theory is easy to motivate at a first cut, but if you want to really understand it, then there are some hairy issues. There is a lot of slip between the cup and the lip when it comes to information theory (Shannon himself made several serious errors in his original 1948 paper which took decades to fully work out). Many seemingly "obvious" facts in information theory are tricky to show. Some examples: (1) From the article: Cross entropy is always greater than or equal to Entropy since we are coding the wrong distribution. How do you show this? For any two probability vectors (p,q), can we say H(p) >= H(p,q)? Any proof I know involves some delicate usage of Jensen's inequality. (By the way, I feel that the notation used by the author is non-standard. H(p,q) usually stands for the joint entropy, which is quite different.) (2) Another famous fact about entropy : conditioning always reduces entropy - for any two random variables X, Y, we have H(X|Y) <= H(X) and H(Y|X) <= H(Y). This is called Shannon's inequality, and the proof involves a subtle trick. (3) You can easily show that if p=q, then KL(p||q)=0. But it is also true that if KL(p||q)=0, then p=q. The second fact is quite tricky, and used to appear as a question in Ph.D qualifying exams.