4 ms·
> I wonder if there is ever going to be an encoding that replaces UTF-8? Possibly if CJK keeps gaining importance there may be a new encoding which provides sm
by masklinn 9y ago
> I wonder if there is ever going to be an encoding that replaces UTF-8?
Possibly if CJK keeps gaining importance there may be a new encoding which provides smaller encoding of these characters rather than the current primacy of western/european character sets.
- userbinator 9y agoThat is already served well by UTF-16. 50% smaller than UTF-8 for CJK, yet still covers the full Unicode range.
- rlanday 9y agoMost commonly-used CJK characters are in the Basic Multilingual Plane, and take 2 bytes to represent in UTF-16 and 3 in UTF-8. So you're only really saving 33%, not 50%. Plus, if you're storing e.g. HTML, all the HTML markup characters go from one byte to two bytes, which is going to offset the CJK advantage to some degree (or maybe even outweigh it). I think most web documents are served compressed anyway, which makes the difference even smaller.
- jack1243star 9y agoNo, Unicode fares terribly for CJK languages, see Han Unification. I can't stress enough how it is a lie that Unicode can represent all languages in use, when it cannot even distinguish Japanese from Chinese.
- jcranmer 9y agoIt can't distinguish between English, French, German, Swedish, Norwegian, Spanish, etc. either. And there's no charset that does that. Do you find that a problem? If so, you're very much in the minority; if not, why distinguish between Japanese and Chinese but not English and French?
- kalleboo 9y agoBecause the characters are different enough that it causes actual day-to-day problems (I see text all the time on printouts/signs where some incorrect font substitution has taken place and you get Chinese instead of Japanese characters). You can (probably correctly) argue that these are really just deficiencies in text editing/markup tools where you can't mark the language of text but the fact that that's required for correct presentation only in CJK languages seems to indicate that it's a problem with this unification in particular. How would you like if there was Latin/Greek/Cyrillic unification? I mean they're all just alphabets that make the same sounds they're all basically the same right?
- mixmastamyk 9y agoWouldn't bother me personally, why should it? However, as there are only a few tens of characters between them, it wouldn't save much.
- jack1243star 9y agoSwedish ä and German ä are the same character. 令 is a different character (of the same origin) in Japanese, Traditional Chinese and Simplied Chinese (and it depends on your browser setup which you get on screen, which is absolutely insane). It is like saying we should use English p as Cyrillic р, which only look similar, or as п, because they have the same origin or sound? And no, CJK language users is not a minority, although many of us have become used to the font inconsistensies (because while wrong, they are legible to us).
- jcranmer 9y agoThe question comes down to if Chinese hanzi and Japanese kanji are the same alphabet or not. It's not clear what the answer should be, but for Unicode, it was absolutely necessary to treat them as the same alphabet, if they were not to break the 64K character limit (it's dubious Unicode would have seen its modern universality if it weren't a compact fixed 16-bit encoding, especially since the genius of UTF-8 encoding was a rather later addition). Another point that is commonly missed is that Han unification was first proposed by East Asians (not ignorant Westerners), specifically the Chinese (which is why the Chinese don't really object to Unicode but the Japanese do). A side note: it's interesting that you bring up ä as an example, because ä is actually a typographic unification of two very distinct letters. In languages like English and French, the diacritic is a diaeresis: a mark, derived from Ancient Greek indicating that the vowel is to be pronounced separately rather than as a diphthong. In languages like German, it's an umlaut, where it's a typographic reduction of a superscript e. They are in fact very different characters that merely look the same.