4 ms·

"And if we weigh this by the probability of that particular event happening, we get info ∝ p ⋅ log2(1/p)" I fail to see the motivation of this step, and I thin

"And if we weigh this by the probability of that particular event happening, we get info ∝ p ⋅ log2(1/p)" I fail to see the motivation of this step, and I think that's preventing me to see the argument as "intuitive". Could somebody explain? The two steps back (info ∝ 1/p) it still makes sense to me: the more rare the event is, the bigger the resulting number is, so in the case the event happens, the more "surprised" we are, and more information is gained. However, what do we achieve by weighing the bitcount of the information with the probability?

mturmon 6y ago
The weighting with probability turns it in to an expected value. Remember that if x is a random variable, then its expected value is E[x] = Sum_{All x} x p(x) The interpretation is that E[x] is the average value of x. In particular, if we observe a bunch of x's one after the other, call them x_i, then the sample mean S_n = (1/n) Sum_i x_i which for a given "n" is random, converges to the deterministic constant E[x] as "n" gets large. In this sense, the above formula for E[x] is "inevitable". In the above case, the thing being averaged is "x", but the same holds true for x^2 ("what is the average value of x^2 ?"), or, in general, f(x) for any function f(.). In this case, we're using the rather unusual function f(x) = log_2(1/p(x)), but the same intuition holds. It's the average number of bits needed to encode x, for instance.
- GolDDranks 6y ago
  Thanks. I was stuck with the idea that the whole argument was trying to formulate just the "amount of information gained | event happens", when it actually formulated the expected entropy gain.
GolDDranks 6y ago
Ah, I think I got it. The point of the exercise is not to formulate the concept of "amount of suprise (∝ amount of information gained) IN CASE the event happens" but the "EXPECTED amount of entropy gain", for us to know before it happens. That's why we need to take a middle ground between very common events that aren't surprising, and gain us hardly anything, and rare events that gain us a lot of information, but happen so rarely that they don't matter a whole lot. The formula derived here manages to find the balance between these two extremes.
- mturmon 6y ago
  I would agree w/ your statement here. The entropy is the on-average (or, expected) amount of information gained from seeing one "x".

mturmon 6y ago

The weighting with probability turns it in to an expected value. Remember that if x is a random variable, then its expected value is E[x] = Sum_{All x} x p(x) The interpretation is that E[x] is the average value of x. In particular, if we observe a bunch of x's one after the other, call them x_i, then the sample mean S_n = (1/n) Sum_i x_i which for a given "n" is random, converges to the deterministic constant E[x] as "n" gets large. In this sense, the above formula for E[x] is "inevitable". In the above case, the thing being averaged is "x", but the same holds true for x^2 ("what is the average value of x^2 ?"), or, in general, f(x) for any function f(.). In this case, we're using the rather unusual function f(x) = log_2(1/p(x)), but the same intuition holds. It's the average number of bits needed to encode x, for instance.

GolDDranks 6y ago
Thanks. I was stuck with the idea that the whole argument was trying to formulate just the "amount of information gained | event happens", when it actually formulated the expected entropy gain.

GolDDranks 6y ago

Ah, I think I got it. The point of the exercise is not to formulate the concept of "amount of suprise (∝ amount of information gained) IN CASE the event happens" but the "EXPECTED amount of entropy gain", for us to know before it happens. That's why we need to take a middle ground between very common events that aren't surprising, and gain us hardly anything, and rare events that gain us a lot of information, but happen so rarely that they don't matter a whole lot. The formula derived here manages to find the balance between these two extremes.

mturmon 6y ago
I would agree w/ your statement here. The entropy is the on-average (or, expected) amount of information gained from seeing one "x".