3 ms·
The definition of entropy (from Wolfram Alpha): > The (Shannon) entropy of a variable X is defined as > H(X)=-sum_(x)P(x)log_2[P(x)] > bits, where P(x) is the
by zby 2y ago
The definition of entropy (from Wolfram Alpha):
> The (Shannon) entropy of a variable X is defined as
> H(X)=-sum_(x)P(x)log_2[P(x)]
> bits, where P(x) is the probability that X is in the state x, and Plog_2P is defined as 0 if P=0.
The X they input into that formula is a function that chooses one of the tokens according to the probability in that step. Isn't that a good definition of a random variable?
- tbalsam 2y agoHi! Entropy unfortunately much more complicated than that in practice, mainly as actually finding the real underlying entropy of a variable is quite difficult in practice! However, we can define it as a quantity with respect to different values. But the entropy of a variable as estimated by the model is generally not the actual entropy of the variable, and this gets worse for sequences -- we can maybe upper bound the entropy of a sequence when measuring it, but this is not always a useful or important quantity for us to have. For more info, please see https://people.math.harvard.edu/~ctm/home/text/others/shannon/entropy/entropy.pdf https://people.math.harvard.edu/~ctm/home/text/others/shanno...