4 ms·

This article describes higher probability letter combinations as higher entropy and lower probability combinations as lower entropy. For example, "yuzz-a-ma-tu

This article describes higher probability letter combinations as higher entropy and lower probability combinations as lower entropy. For example, "yuzz-a-ma-tuzz" is low entropy because z's generally appear with lower probability in English. I thought entropy worked the other way round - high probability means low entropy and low probability means high entropy. Which is it?

refactor_master 6y ago
It does indeed sound like it’s written the wrong way around. Random information = high entropy. A common word is therefore low entropy. Here’s a good analogy I found: > Informally, the amount of information in an email is proportional to the amount of “surprise” its reading causes. For example, if an email is simply a repeat of an earlier email, then it is not informative at all. On the other hand, if say the email reveals the outcome of a cliff-hanger election, then it is highly informative. Similarly, the information in a variable is tied to the amount of surprise that value of the variable causes when revealed. Shannon’s entropy quantifies the amount of information in a variable, thus providing the foundation for a theory around the notion of information. https://arxiv.org/pdf/1405.2061.pdf https://arxiv.org/pdf/1405.2061.pdf