3 ms·
> using UTF-8 to examine the informational properties of Chinese text is absurd. By the same reasoning, using 8-bit ASCII to examine the informational properti
by kijin 12y ago
> using UTF-8 to examine the informational properties of Chinese text is absurd.
By the same reasoning, using 8-bit ASCII to examine the informational properties of English text is also absurd. English can be comfortably expressed with only 6 bits per character, after all.
Anyway, I did include a GB-encoded version in my calculations. And once you compress the text, the charset-related difference becomes much smaller anyway.
- thaumasiotes 12y agoAgreed. In fact, I'll go farther and say I misspoke, raising the objection you rightly criticize when it would make more sense to object to using UTF-8 to compare english text to chinese text. I have no issues with your comment, though, and I agree that under compression is the correct place to make the comparison. It's faintly aggravating to me that the same text compresses differently according to the intermediate encoding, but that's between me and the universe. English can be comfortably expressed in 5 bits per character, using 26 letters and the five symbols [. ',"] . Capitalization and numeric digits are nice-to-haves. ;)