Compression is a good entry to Shannon entropy. What was also eye-opening for me was that the metric was motivated by fulfilling a specification which matches

Compression is a good entry to Shannon entropy. What was also eye-opening for me was that the metric was motivated by fulfilling a specification which matches an intuitive notion of information [1], much like how the Kolmogorov axioms were characterized to capture the intuitive notion of probability, or how the Church-Turing thesis defines computation. [1] https://en.wikipedia.org/wiki/Entropy_(information_theory)#Characterization https://en.wikipedia.org/wiki/Entropy_(information_theory)#C...