3 ms·
For Asian languages, UTF-8 is basically the same size as any other encoding when compressed[0] (and you should be using compression if you care about space) so
by gary_0 2y ago
For Asian languages, UTF-8 is basically the same size as any other encoding when compressed[0] (and you should be using compression if you care about space) so in practice there is no data size advantage to using non-standard encodings.
[0] https://utf8everywhere.org/#asian https://utf8everywhere.org/#asian
- neild 2y agoIn addition, Chinese characters encode more information than English letters, so a text written in Chinese will generally consume fewer bytes than the same text in English even when using UTF-8. (Consider: Horse is five letters, but 馬 is one character. Even at three bytes per character, Chinese wins.)
- kstrauser 2y agoTrue, but even then you wouldn’t want to store it egregiously badly.
- Panzer04 2y agoPresumably that derives from the overhead of encoding an english character as a full byte? Given there's only 26 characters normally, you could fit that into 5 bits instead, which funnily enough does actually line up with the chinese character encoding (5x5 vs 1x24).
- kps 2y agoYes. It's the non-Latin alphabets that lose with either UTF-8 or UTF-16, compared with stateful ISO 2022 page switching.