Hi! Entropy unfortunately much more complicated than that in practice, mainly as actually finding the real underlying entropy of a variable is quite difficult i

Hi! Entropy unfortunately much more complicated than that in practice, mainly as actually finding the real underlying entropy of a variable is quite difficult in practice! However, we can define it as a quantity with respect to different values. But the entropy of a variable as estimated by the model is generally not the actual entropy of the variable, and this gets worse for sequences -- we can maybe upper bound the entropy of a sequence when measuring it, but this is not always a useful or important quantity for us to have. For more info, please see https://people.math.harvard.edu/~ctm/home/text/others/shannon/entropy/entropy.pdf https://people.math.harvard.edu/~ctm/home/text/others/shanno...