This is pedantic, but I don't like the formulation of entropy as sum of p log(1/p). I think of log(p) as information of a single event, for which log base 1/2 g

This is pedantic, but I don't like the formulation of entropy as sum of p log(1/p). I think of log(p) as information of a single event, for which log base 1/2 gives the answer in bits. This makes the negative sign unnecessary, and technically all these formulas should specify the base of log > 1. Everything is cleaner with log base 1/2 (instead of e.g. using the equivalent negative log base 2). This comes up in log likelihood all the time too. I guess it's a prejudice against fractional bases.