3 ms·
However, this is only the case if you have just a big blob of text and UTF-8 might be better if the east asian text is interspersed with ASCII characters. For e
by ufo 8y ago
However, this is only the case if you have just a big blob of text and UTF-8 might be better if the east asian text is interspersed with ASCII characters. For example, HTML with east asian text often is smaller when encoded with UTF-8, because a significant portion of the content is HTML markup.
- camgunz 8y agoRight well again, if your average East Asian codepoint is 3 bytes in UTF-8, and your average markup codepoint is one byte, then as the percentage of markup in your corpus rises you'll grow asymptotically towards 1. But not all text contains markup ;) Consider any database storage for example, text in a game, e-books, visual novels. I guess anything that isn't the web or JSON is what I'm saying -- which is a lot.
- a1369209993 8y agoAlso bear in mind that gzip (or any non-toy compression algorithm) will do much better than either on any non-contrived input.