The weighting with probability turns it in to an expected value. Remember that if x is a random variable, then its expected value is E[x] = Sum_{All x} x p(

The weighting with probability turns it in to an expected value. Remember that if x is a random variable, then its expected value is E[x] = Sum_{All x} x p(x) The interpretation is that E[x] is the average value of x. In particular, if we observe a bunch of x's one after the other, call them x_i, then the sample mean S_n = (1/n) Sum_i x_i which for a given "n" is random, converges to the deterministic constant E[x] as "n" gets large. In this sense, the above formula for E[x] is "inevitable". In the above case, the thing being averaged is "x", but the same holds true for x^2 ("what is the average value of x^2 ?"), or, in general, f(x) for any function f(.). In this case, we're using the rather unusual function f(x) = log_2(1/p(x)), but the same intuition holds. It's the average number of bits needed to encode x, for instance.