2 ms·
Ah, I think I got it. The point of the exercise is not to formulate the concept of "amount of suprise (∝ amount of information gained) IN CASE the event happens
by GolDDranks 6y ago
Ah, I think I got it. The point of the exercise is not to formulate the concept of "amount of suprise (∝ amount of information gained) IN CASE the event happens" but the "EXPECTED amount of entropy gain", for us to know before it happens.
That's why we need to take a middle ground between very common events that aren't surprising, and gain us hardly anything, and rare events that gain us a lot of information, but happen so rarely that they don't matter a whole lot.
The formula derived here manages to find the balance between these two extremes.
- mturmon 6y agoI would agree w/ your statement here. The entropy is the on-average (or, expected) amount of information gained from seeing one "x".