4 ms·

Wasn't the whole point of Unicode to have a single encoding that could represent all languages unambiguously so that you don't need any meta-information to disp

Wasn't the whole point of Unicode to have a single encoding that could represent all languages unambiguously so that you don't need any meta-information to display a string ? Is there a reason why they chose to represent characters that are obviously different with the same code point ? Everyone would find it outrageous if they decided to have a single character for the russian м and the english m just because they have the same greek origin...

Sniffnoy 5y ago
The reason why is that Unicode was originally 16-bit, and there was no way they could fit everything into 16 bits without CJK unification. Of course later it turned out there was no way they could fit everything into 16 bits anyway, and so they were forced to expand it, and so we now both have a larger Unicode (with all the messes that's caused) but also still have CJK unification...
- lovasoa 5y ago
  Is it too late to add separate code points for the chinese, japanese, and korean versions of the han characters ?
  - lifthrasiir 5y ago
    In principle there is a designated "disunificiation" procedure when it's desirable. More accurately speaking, each CJK character is thought to represent not a single or a few glyphs listed in the code chart but rather a glyphic subset, and the disunification splits that set into partitions. But this is generally applied to a few selected characters and only when it's safe to do so. Massive disunification was to my knowledge never suggested or proposed, and that would surely prompt a large scale disruption throughout CJK users (say, how about existing texts?).
    - rrobukef 5y ago
      So it's possible to modify the skin tone of emoji but impossible to disunify CJK characters? There are RTL modifiers for Arabic languages, it's impossible for CJK? It shouldn't be harder than existing unicode handling.
      - lifthrasiir 5y ago
        Everything boils down to the interoperability and compatibility. Emoji was added because Apple and Google had to deal with (then-)Japanese emails, and skin tones were not specified. It were implementations that impose certain skin tones (that do not even match the original Japanese emojis) and as a result Unicode had to introduce a mechanism to change skin tones and mandate the default emoji without that mechanism to be neutral. RTL "modifiers" are actually formatting characters closely tied with the Unicode Bidirectional Algorithm [1]. Until then texts with both RTL and LTR fragments were handled incoherently, for example legacy character sets were still struggling with logical vs. visual order issues. So they are indeed Unicode inventions, but necessary ones that do not alter existing texts. For CJK characters Unicode now provides ideographic variation selectors that select the exact glyph (or more accurately, a restricted glyphic subset of the base character). They do not disunify characters but they do provide a strong hint to display those characters in a specified way. In this way they do not cause an additional issue to existing Unicode systems (as they should already do normalization and collation in the Unicode way). The disunification by comparison would almost instantly break existing texts. [1] https://unicode.org/reports/tr9/ https://unicode.org/reports/tr9/
        account42 5y ago
        > Unicode had to introduce a mechanism to change skin tones and mandate the default emoji without that mechanism to be neutral. Adding skin tones was a choice, there was no immediate need for it.
        musicale 5y ago
        The default seems to be yellow though, like original Lego minifigs (which I assume was one of the original plastic brick colors, and a nod to the bright yellow makeup that was popular at the time in Denmark.)
dustintrex 5y ago
It's a really complicated issue, because whether the character is different or not is debatable. In English writing, we consider a serif A to be the same character as a sans-serif A, even though the glyph is obviously different, and neither do we distinguish between a "French" A and a "German" A. So what do we do with 国 and 國? The first of those is always used in simplified Chinese and usually in Japanese, while the second is used in traditional Chinese and sometimes in Japanese (eg. names). Is this one, two or three characters? More on the topic: https://en.wikipedia.org/wiki/Han_unification https://en.wikipedia.org/wiki/Han_unification
- SpicyLemonZest 5y ago
  It’s worth noting this kind of distinction did used to exist in the Latin alphabet as well. For much of the 19th and 20th centuries, German letters were different, as part of a debate about whether German text should be written in blackletter or Latin/Antiqua/what-other-Europeans-use script.
  - dotancohen 5y ago
    We still have these issues in e.g. Hebrew and Yiddish, Arabic and Persian, and tons of adapted Cyrillic scripts from ex-Soviet states. Not to mention Northern-European accented vowels, and cedilla letters such as Ç. I'm personally of the belief that the accented and cedilla characters should be exclusively stored as combining character pairs, even if modern keyboard mappings require only a single keypress. My own language stores every character as two bytes (at a minimum), so the storage aspect is a solved problem.
- afiori 5y ago
  as an armchair linguist it is clearly two (not one, not three) distinct "characters", by going how I am guessing native speakers think about them. If all three had different codepoints and you replaced 国 with 國 a lot of people would realize, less so if you replaced 國 with 國. To my understanding the only argument in favour for han unification was that it would have taken-up a lot of codepoints otherwise.
josefx 5y ago
> Everyone would find it outrageous if they decided to have a single character for the russian м and the english m just because they have the same greek origin... Did you know that some languages distinguish dot less Iı and dotted İi ? English mixes them Ii and unicode needs to know exactly based on what language you might want to upper/lower case because it can't tell an english I appart from a Turkish dotless I.
zczc 5y ago
There are some 8-bit and 7-bit encodings which have unified Latin and Greek, like 7-bit SMS encoding [1] [1] https://en.wikipedia.org/wiki/GSM_03.38 https://en.wikipedia.org/wiki/GSM_03.38
jhanschoo 5y ago
Even in Latin script languages you have issues if you don't specify a lang tag. e.g. a font's 'fi' ligature may omit the tittle on the 'i', but it is necessary in Turkish. Or you are using a font without coverage for French and your browser renders œ in another font. The CJK variant issues under not specifying a lang are indeed present in Latin but to a smaller extent.

Sniffnoy 5y ago

The reason why is that Unicode was originally 16-bit, and there was no way they could fit everything into 16 bits without CJK unification. Of course later it turned out there was no way they could fit everything into 16 bits anyway, and so they were forced to expand it, and so we now both have a larger Unicode (with all the messes that's caused) but also still have CJK unification...

lovasoa 5y ago
Is it too late to add separate code points for the chinese, japanese, and korean versions of the han characters ?
- lifthrasiir 5y ago
  In principle there is a designated "disunificiation" procedure when it's desirable. More accurately speaking, each CJK character is thought to represent not a single or a few glyphs listed in the code chart but rather a glyphic subset, and the disunification splits that set into partitions. But this is generally applied to a few selected characters and only when it's safe to do so. Massive disunification was to my knowledge never suggested or proposed, and that would surely prompt a large scale disruption throughout CJK users (say, how about existing texts?).
  - rrobukef 5y ago
    So it's possible to modify the skin tone of emoji but impossible to disunify CJK characters? There are RTL modifiers for Arabic languages, it's impossible for CJK? It shouldn't be harder than existing unicode handling.
    - lifthrasiir 5y ago
      Everything boils down to the interoperability and compatibility. Emoji was added because Apple and Google had to deal with (then-)Japanese emails, and skin tones were not specified. It were implementations that impose certain skin tones (that do not even match the original Japanese emojis) and as a result Unicode had to introduce a mechanism to change skin tones and mandate the default emoji without that mechanism to be neutral. RTL "modifiers" are actually formatting characters closely tied with the Unicode Bidirectional Algorithm [1]. Until then texts with both RTL and LTR fragments were handled incoherently, for example legacy character sets were still struggling with logical vs. visual order issues. So they are indeed Unicode inventions, but necessary ones that do not alter existing texts. For CJK characters Unicode now provides ideographic variation selectors that select the exact glyph (or more accurately, a restricted glyphic subset of the base character). They do not disunify characters but they do provide a strong hint to display those characters in a specified way. In this way they do not cause an additional issue to existing Unicode systems (as they should already do normalization and collation in the Unicode way). The disunification by comparison would almost instantly break existing texts. [1] https://unicode.org/reports/tr9/ https://unicode.org/reports/tr9/
      - account42 5y ago
        > Unicode had to introduce a mechanism to change skin tones and mandate the default emoji without that mechanism to be neutral. Adding skin tones was a choice, there was no immediate need for it.
        musicale 5y ago
        The default seems to be yellow though, like original Lego minifigs (which I assume was one of the original plastic brick colors, and a nod to the bright yellow makeup that was popular at the time in Denmark.)

dustintrex 5y ago

It's a really complicated issue, because whether the character is different or not is debatable. In English writing, we consider a serif A to be the same character as a sans-serif A, even though the glyph is obviously different, and neither do we distinguish between a "French" A and a "German" A. So what do we do with 国 and 國? The first of those is always used in simplified Chinese and usually in Japanese, while the second is used in traditional Chinese and sometimes in Japanese (eg. names). Is this one, two or three characters? More on the topic: https://en.wikipedia.org/wiki/Han_unification https://en.wikipedia.org/wiki/Han_unification

SpicyLemonZest 5y ago
It’s worth noting this kind of distinction did used to exist in the Latin alphabet as well. For much of the 19th and 20th centuries, German letters were different, as part of a debate about whether German text should be written in blackletter or Latin/Antiqua/what-other-Europeans-use script.
- dotancohen 5y ago
  We still have these issues in e.g. Hebrew and Yiddish, Arabic and Persian, and tons of adapted Cyrillic scripts from ex-Soviet states. Not to mention Northern-European accented vowels, and cedilla letters such as Ç. I'm personally of the belief that the accented and cedilla characters should be exclusively stored as combining character pairs, even if modern keyboard mappings require only a single keypress. My own language stores every character as two bytes (at a minimum), so the storage aspect is a solved problem.
afiori 5y ago
as an armchair linguist it is clearly two (not one, not three) distinct "characters", by going how I am guessing native speakers think about them. If all three had different codepoints and you replaced 国 with 國 a lot of people would realize, less so if you replaced 國 with 國. To my understanding the only argument in favour for han unification was that it would have taken-up a lot of codepoints otherwise.

josefx 5y ago

> Everyone would find it outrageous if they decided to have a single character for the russian м and the english m just because they have the same greek origin... Did you know that some languages distinguish dot less Iı and dotted İi ? English mixes them Ii and unicode needs to know exactly based on what language you might want to upper/lower case because it can't tell an english I appart from a Turkish dotless I.

zczc 5y ago

There are some 8-bit and 7-bit encodings which have unified Latin and Greek, like 7-bit SMS encoding [1] [1] https://en.wikipedia.org/wiki/GSM_03.38 https://en.wikipedia.org/wiki/GSM_03.38

jhanschoo 5y ago

Even in Latin script languages you have issues if you don't specify a lang tag. e.g. a font's 'fi' ligature may omit the tittle on the 'i', but it is necessary in Turkish. Or you are using a font without coverage for French and your browser renders œ in another font. The CJK variant issues under not specifying a lang are indeed present in Latin but to a smaller extent.