3 ms·
I'm pretty sure the theoretical floor on storage size required is the minimum number of distinct characters in the charset, otherwise you no longer have a uniqu
by munchbunny 8y ago
I'm pretty sure the theoretical floor on storage size required is the minimum number of distinct characters in the charset, otherwise you no longer have a unique identifier for each character. Based on that I'd imagine that it's literally impossible to fit a charset of 64+ characters into less than 7 bits.
- jmiserez 8y agoNot quite. If they're all equally likely, then yes you need 6 bits for 64, and 7 bits for 128 chars. But you can get that number down significantly if your characters are not all equally likely, then you don't need as many bits. The actual theoretical lower bound is the Shannon entropy, for English text you typically need only 0.6-1.3 bits per character, see https://en.wikipedia.org/wiki/Entropy_(information_theory) https://en.wikipedia.org/wiki/Entropy_(information_theory)
- munchbunny 8y agoSorry, I wasn't clear about the statement I was making. You are of course right about being able to store text consisting of multiple characters from an N-bit charset at much less than N bits per character. What I meant was that the single-character representation isn't going to go below N bits per character if there are at least 2^N unique characters.
- jmiserez 8y agoIf you want them all to have the same length, no. But even in practice this is not the case anymore, e.g. the most common encoding UTF-8 uses 8 bits for the ASCII characters and then 16/24/32 bits for the lesser used code points. And there are some exotic UTF-5/6/7 variants as well: https://en.wikipedia.org/wiki/Comparison_of_Unicode_encodings https://en.wikipedia.org/wiki/Comparison_of_Unicode_encoding...