There's a bit of a conceptual problem with trying to measure entropy. That's because entropy is based on probabilities, and probabilities are subjective. What a

There's a bit of a conceptual problem with trying to measure entropy. That's because entropy is based on probabilities, and probabilities are subjective. What are "the" odds that my brother will pass his SAT exam? My brother can make the most confident prediction (lowest entropy[1]). But the reader of this comment can do no better [2] than take the generic SAT success rate (higher entropy). And some peasant in Tuvalu might as well say 1/2 (maximum entropy). So entropy is subjective. It's a measure of how much new information you expect to get from a message. The more you know already, the lower the entropy. EDIT: An awesome quote from Jaynes. "Probability assignments are not assertions of any physical property; they are a description of a state of knowledge. It is therefore illogical to speak of "verifying" the probabilities by performing experiments on the physical world. That would be like trying to verify a boy's love for his dog by performing experiments on the dog" [1] Not necessarily true, see Wiki or David MacKay's wonderful book for details. http://www.inference.phy.cam.ac.uk/mackay/itila/ http://www.inference.phy.cam.ac.uk/mackay/itila/ [2] Also not strictly true! You might argue that his passing the SAT should correlate with my passing it, and estimate what my odds would be/were from my comment history.