Chapter 2 of that excellent book by Cover and Thomas is the roundup of all the definitions of entropy, relative entropy, mutual information, etc. I think it's

Chapter 2 of that excellent book by Cover and Thomas is the roundup of all the definitions of entropy, relative entropy, mutual information, etc. I think it's the first place I ever saw the "Information Diagram" (https://en.wikipedia.org/wiki/Information_diagram https://en.wikipedia.org/wiki/Information_diagram) which is a Venn-like diagram that really helps to remember all these relationships.