Perhaps the best way I began to understand entropy is when reading about tail inequalities and divergences. In particular, we can view the entropy as a sort of

Perhaps the best way I began to understand entropy is when reading about tail inequalities and divergences. In particular, we can view the entropy as a sort of “distance to a uniform distribution.” (Specifically an additive factor away from the KL divergence between the uniform distribution and the distribution under measurement.) Studying the distributions from this perspective yields a bunch of important consequences like, how many samples do you need to tell one distribution from another? Or, how many bits are needed to approximately describe the distribution, from a more general family? Surprisingly, being able to answer these two questions in a precise sense gives you a whole load of tools to fun things like: can an algorithm decide a question with k bits of information? Or can two physical systems diverge in behavior in a meaningful way after T time? For the (fairly) mathematically inclined reader, I very highly recommend Massart’s Concentration Inequalities and Model Selection.