Well written! I was just looking for the Shannon Minimum Coding Rate definition the other day, so it was nice to have a quick refresher on the basics. This als

Well written! I was just looking for the Shannon Minimum Coding Rate definition the other day, so it was nice to have a quick refresher on the basics. This also highlights a fundamental issue: the coding scheme is tailored to the expected data input. That's why there's no one-size-fits-all compression algorithm, and you can achieve far greater efficiency if you have a good model of the shape of the input data. (The same applies to lossy) The MCR is also a nice way to demonstrate how random data can't be compressed. Let's say you have 256 symbols, each with probability 1/256 (independent etc). The Shannon Entropy = -sum(P(x) * log2(P(x) for x in 0..255) = -2^8 * 2^-8 * -8 = 8 bits. No compression possible.