5 ms·

I would be more receptive to endless emojis if Unicode bothered to accept archaic and historical forms of characters as well as deprecating Han unification. It’

I would be more receptive to endless emojis if Unicode bothered to accept archaic and historical forms of characters as well as deprecating Han unification. It’s rather odd that they reject actual useful things while accepting endless objects that have never been found in any text prior.

rmunn 1y ago
I know several linguists who also know more than a little about computers. The number of times I've heard rants about the Unicode committee rejecting a perfectly valid historical character, yet adding more "modern hieroglyphics" (emojis)... well, let's just say that it's happened more than once.
lifthrasiir 1y ago
Eh, Han unification was an one-off decision. Now many (but not all) characters have been disunified as needed, like the infamous Biang character [1] which received two different code points. Of course common characters are much less likely to be disunified, because at this point many decades have been passed after the initial encoding and any disunification would cause compatibility issues. [1] https://en.wikipedia.org/wiki/Biangbiang_noodles#Unicode https://en.wikipedia.org/wiki/Biangbiang_noodles#Unicode
- numpad0 1y ago
  It's an upheld decision. The unification is not about reducing character counts overall, but to co-mingle CJKV languages. Adding more characters is not un-mingling existing characters. One thing I feared might happen and do seem to be happening is, Chinese LLMs and AI projects seem to be moving towards Chinese-English bilingual models away from regular omni-lingual models, which, I think is, because LLMs would become confused with Chinese-invalid syntaxes and dictionary definitions, and/or generally perform worse, if substantial non-Chinese CJKV data was included in the dataset. At the polar opposite of computing, Hollow Knight: Sliksong released just days prior is having Han Unification font problem as well: as you might know, thanks to Han Unification, CJKV languages each require its own font, of which no two cannot be active at the same time, and characters become mangled if application developer spends substantial cost implementing such non-standard feature. The developers was not aware of that, and did not spend extra cost doing so, and is getting review bombed in China. It just needs to be reversed. It's a real problem. Adding more obscure characters and obscure features is tangential and not a solution. Different isolated clusters of characters uses need to be separated, not overlapped into one same area, like there are no "GermanFrench-English dictionary".
  - lifthrasiir 1y ago
    The unification, implemented in Unicode 1.1, is definitely a character count reduction mechanism. I'm very sure that if the decision to abandon 16-bit character set was done earlier then the unification wouldn't have happened. And I'm saying this as a CJKV person and past gamedev: CJKV languages each require its own font no matter whether the Han unification is implemented or not. There are simply too many glyphs there; not just unified characters, but also common characters that are not considered unified are also often varying across countries. If you account for all those glyph variations in a single font, you just can't cope up because OpenType only supports at most 65,536 glyphs in a single typeface. In the alternative universe OpenType may have been extended to allow more glyphs in a single typeface, I don't know, but CJKV characters are simply complex enough to require multiple font files in general. Han unification is of less concern when you have too many glyphs.
    - numpad0 1y ago
      > not just unified characters, but also common characters that are not considered unified are also often varying across countries. That's the unification, the issues stemming from CJKVs each not having own code points. The issue is not that CJKVs need multiple font files and it's cumbersome, the issue is that no two CJKV fonts may be loaded at the same time because there are conflicting glyphs. Conflicting glyphs. That's just wrong.
      - lifthrasiir 1y ago
        If you somehow want to display, say, both Japanese and Chinese texts at the same time, there is no technical obstacle that prevents you to do so. Pan-Unicode fonts come with differently named files for CJKV characters so that is not even difficult. Yes, your assets will have multiple multi-megabyte font files. Is that a problem for modern games? I don't think so. There is a single circumstance where this is not generally doable: a user name in globally serviced online games. (Guess why I know of this case...) Unless there is a hint that a particular user prefers one's user name to be displayed in a certain way, it is difficult to decide which font to use (or even which set of fonts to use). But it's a very niche problem and otherwise you know which language of the text you are showing and can pick the correct font from your assets.
BeFlatXIII 1y ago
Also, they ought to accept the Tengwar script.
zahlman 1y ago
I still don't understand what's wrong with Han unification now that we have variation selectors, as well as... those characters that describe ways of laying out CJK character components relative to each other (I know I've seen these, but I have absolutely no idea what they're called and can't for the life of me figure out how to search for them). (I think they should have unified Latin/Greek/Cyrillic, too, with variation selectors to disambiguate the overlaps. Yes, including special cases like the Greek question mark and Cyrillic multiocular O.)

rmunn 1y ago

I know several linguists who also know more than a little about computers. The number of times I've heard rants about the Unicode committee rejecting a perfectly valid historical character, yet adding more "modern hieroglyphics" (emojis)... well, let's just say that it's happened more than once.

lifthrasiir 1y ago

Eh, Han unification was an one-off decision. Now many (but not all) characters have been disunified as needed, like the infamous Biang character [1] which received two different code points. Of course common characters are much less likely to be disunified, because at this point many decades have been passed after the initial encoding and any disunification would cause compatibility issues. [1] https://en.wikipedia.org/wiki/Biangbiang_noodles#Unicode https://en.wikipedia.org/wiki/Biangbiang_noodles#Unicode

numpad0 1y ago
It's an upheld decision. The unification is not about reducing character counts overall, but to co-mingle CJKV languages. Adding more characters is not un-mingling existing characters. One thing I feared might happen and do seem to be happening is, Chinese LLMs and AI projects seem to be moving towards Chinese-English bilingual models away from regular omni-lingual models, which, I think is, because LLMs would become confused with Chinese-invalid syntaxes and dictionary definitions, and/or generally perform worse, if substantial non-Chinese CJKV data was included in the dataset. At the polar opposite of computing, Hollow Knight: Sliksong released just days prior is having Han Unification font problem as well: as you might know, thanks to Han Unification, CJKV languages each require its own font, of which no two cannot be active at the same time, and characters become mangled if application developer spends substantial cost implementing such non-standard feature. The developers was not aware of that, and did not spend extra cost doing so, and is getting review bombed in China. It just needs to be reversed. It's a real problem. Adding more obscure characters and obscure features is tangential and not a solution. Different isolated clusters of characters uses need to be separated, not overlapped into one same area, like there are no "GermanFrench-English dictionary".
- lifthrasiir 1y ago
  The unification, implemented in Unicode 1.1, is definitely a character count reduction mechanism. I'm very sure that if the decision to abandon 16-bit character set was done earlier then the unification wouldn't have happened. And I'm saying this as a CJKV person and past gamedev: CJKV languages each require its own font no matter whether the Han unification is implemented or not. There are simply too many glyphs there; not just unified characters, but also common characters that are not considered unified are also often varying across countries. If you account for all those glyph variations in a single font, you just can't cope up because OpenType only supports at most 65,536 glyphs in a single typeface. In the alternative universe OpenType may have been extended to allow more glyphs in a single typeface, I don't know, but CJKV characters are simply complex enough to require multiple font files in general. Han unification is of less concern when you have too many glyphs.
  - numpad0 1y ago
    > not just unified characters, but also common characters that are not considered unified are also often varying across countries. That's the unification, the issues stemming from CJKVs each not having own code points. The issue is not that CJKVs need multiple font files and it's cumbersome, the issue is that no two CJKV fonts may be loaded at the same time because there are conflicting glyphs. Conflicting glyphs. That's just wrong.
    - lifthrasiir 1y ago
      If you somehow want to display, say, both Japanese and Chinese texts at the same time, there is no technical obstacle that prevents you to do so. Pan-Unicode fonts come with differently named files for CJKV characters so that is not even difficult. Yes, your assets will have multiple multi-megabyte font files. Is that a problem for modern games? I don't think so. There is a single circumstance where this is not generally doable: a user name in globally serviced online games. (Guess why I know of this case...) Unless there is a hint that a particular user prefers one's user name to be displayed in a certain way, it is difficult to decide which font to use (or even which set of fonts to use). But it's a very niche problem and otherwise you know which language of the text you are showing and can pick the correct font from your assets.

BeFlatXIII 1y ago

Also, they ought to accept the Tengwar script.

zahlman 1y ago

I still don't understand what's wrong with Han unification now that we have variation selectors, as well as... those characters that describe ways of laying out CJK character components relative to each other (I know I've seen these, but I have absolutely no idea what they're called and can't for the life of me figure out how to search for them). (I think they should have unified Latin/Greek/Cyrillic, too, with variation selectors to disambiguate the overlaps. Yes, including special cases like the Greek question mark and Cyrillic multiocular O.)