3 ms·
Shannon's information is a measure of the size of a set that an element is selected from, is that right? So a letter conveys log2(26) bits of information. Tha
by yters 18y ago
Shannon's information is a measure of the size of a set that an element is selected from, is that right? So a letter conveys log2(26) bits of information. That alone doesn't allow me to discriminate between the information content of a random string and an English sentence, since both supposedly contain the same amount of information by this bare metric.
If I look at the occurrence of subsets of the string, then that would be a better discriminator: the random string's subsets should follow a normal distribution while the English string's subsets will be highly skewed.
However, that doesn't work when I try to discriminate between an English string and a string generated by a simple algorithm, since the latter's subset distribution will also be highly skewed. What kind of metric discriminates the English sentence from either case?
Am I making sense here? I haven't had any formal training in information theory, and my brain is kind of fried right now.