3 ms·

Several years ago I read a post calling Han Unification essentially “Western Elitism” and made the argument that “no one” would merge the Latin alphabet and the

by donatj 3y ago

Several years ago I read a post calling Han Unification essentially “Western Elitism” and made the argument that “no one” would merge the Latin alphabet and their Greek homoglphs. I disagree and am still mad about it. I ABSOLUTELY would combine them. Not doing so was an huge mistake. Every European character of the same general design should be merged. The lack of this causes so many problems and makes so many things that could be easy so much more difficult. Why do English, Spanish, German and Italian all share the same homogylhs, even though German has for instance has ß among others, Spanish has ñ, and Italian completely lacks 5e letters JKWXY. Greek or Russian though? Completely different character sets despite containing many of the same glyphs. That to me seems like “Western European Elitism” There are of course special cases where for instance the upper and lower characters don’t match across languages, but they can be resolved individually rather than as a full set.

klausa 3y ago
Out of curiosity, what languages do you speak and what is your native language?
vore 3y ago
I don't think Han unification is as evil as people make it out to be: the characters that got unified do share the same semantic meaning across languages, whereas Latin vs Cyrillic there is no semantic equivalence since letters are purely orthographic glyphs. The thing that sucks is that there's no real inline way (variation selectors, kind of) to select the language for the CJK glyph.
- sirn 3y ago
  > the characters that got unified do share the same semantic meaning across languages It doesn't in practice. For example: 机 in Japanese means "table", while the same codepoint in Simplified Chinese means "machine" (i.e. 機 in Japanese).
  - vore 3y ago
    For that example, in Chinese 机 is an alternative orthography of 几 (https://en.wiktionary.org/wiki/%E6%9C%BA#Etymology_2 https://en.wiktionary.org/wiki/%E6%9C%BA#Etymology_2) as in e.g. 茶几 – it's not perfect, but language is messy and I don't think Unicode did the worst job of a messy situation.
deleted 3y ago
[deleted]
thayne 3y ago
I think there are tradeoffs to each approach, and if it had gone the other way, there would be different problems For example characters for languages would be split up between blocks. Among other things this would make collation even more complicated and locale dependent than it already is. And people might complain that it is "western european elitism" that A is in the latin block instead of the greek or cyrrilic block.
kmeisthax 3y ago
"UniGreek" didn't happen because character encodings already existed that encoded Latin, Greek, and/or Cyrillic separately. Round-trip conversion is a non-negotiable requirement of Unicode, so merging any of those characters was impossible. There were no encodings that treated Japanese and Chinese simplifications of the same characters differently, hence why UniHan was allowed to happen. For similar reasons, we have twelve meaningless kanji in Unicode that exist purely as typographical errors during the standardization of Shift-JIS[0]. Furthermore, the primary motivation for UniHan wasn't to 'clean up' similar orthographies to avoid encoding homoglyphs. It was to stay inside 16 bits of coding space. Unicode was fighting a civil war against UCS, which proposed a 32-bit codepoint space, which would mean 32-bit characters, which the entire industry NOPE'd out of. Turns out, UCS was right, 16 bits was not enough, and the result is that every system that jumped on the Unicode train early[1] is now permanently cursed with improperly handling less-common kanji and most emoji. For the record, I consider both UniHan and the hypothetical "UniGreek" a mistake. What characters get encoded in Unicode should match what speakers of a given language would consider distinct characters, not what we can arbitrarily merge to fit under a given coding bitrate. The only viable long-term solution for internationalized text was 32-bit codepoints encoded using a variable length encoding compatible with ASCII. We wound up with 20-bit codepoints, but I suspect at some point we'll need to break UTF-16 some more. [0] This is known as "yurei moji" in Japanese [1] Windows, Java, and JavaScript[2] all mishandle codepoints outside the 16-bit basic multilingual plane, which were encoded as pairs of 16-bit surrogate values in a special range of non-codepoints. Modern UTF-16 handling is supposed to treat these as single astral characters and reject broken surrogates, but the systems in question retain Unicode 1.0 era quirks for backwards compatibility. A few years after the Unicode/UCS wars, Ken Thompson and Rob Pike would propose UTF-8, implementing it in Plan 9. This encoding was and is superior to UTF-16 in every possible way - including support for 31-bit code points, which UTF-16 surrogates can't do. UTF-8 isn't mangled by anything except the above programs... and MySQL, which had to add a second "no seriously it's UTF-8 for real" encoding. [2] ActionScript inclusive. Yes, Ruffle has its own wide string library because of this.
arp242 3y ago
> Every European character of the same general design should be merged. Won't combining Latin and Greek alphabets be horrible? Many formulas for starters, but also things like "β-radiation" (although I suppose b-radiation works too). I don't really have strong opinions on Han unification as I don't speak those languages, but "Euro Unification" would seem like a right pain, and in general I feel Unicode is often too clever by half.
deleted 3y ago
[deleted]

by donatj 3y ago

klausa 3y ago

Out of curiosity, what languages do you speak and what is your native language?

vore 3y ago

I don't think Han unification is as evil as people make it out to be: the characters that got unified do share the same semantic meaning across languages, whereas Latin vs Cyrillic there is no semantic equivalence since letters are purely orthographic glyphs. The thing that sucks is that there's no real inline way (variation selectors, kind of) to select the language for the CJK glyph.

sirn 3y ago
> the characters that got unified do share the same semantic meaning across languages It doesn't in practice. For example: 机 in Japanese means "table", while the same codepoint in Simplified Chinese means "machine" (i.e. 機 in Japanese).
- vore 3y ago
  For that example, in Chinese 机 is an alternative orthography of 几 (https://en.wiktionary.org/wiki/%E6%9C%BA#Etymology_2 https://en.wiktionary.org/wiki/%E6%9C%BA#Etymology_2) as in e.g. 茶几 – it's not perfect, but language is messy and I don't think Unicode did the worst job of a messy situation.

deleted 3y ago

[deleted]

thayne 3y ago

I think there are tradeoffs to each approach, and if it had gone the other way, there would be different problems For example characters for languages would be split up between blocks. Among other things this would make collation even more complicated and locale dependent than it already is. And people might complain that it is "western european elitism" that A is in the latin block instead of the greek or cyrrilic block.

kmeisthax 3y ago

"UniGreek" didn't happen because character encodings already existed that encoded Latin, Greek, and/or Cyrillic separately. Round-trip conversion is a non-negotiable requirement of Unicode, so merging any of those characters was impossible. There were no encodings that treated Japanese and Chinese simplifications of the same characters differently, hence why UniHan was allowed to happen. For similar reasons, we have twelve meaningless kanji in Unicode that exist purely as typographical errors during the standardization of Shift-JIS[0]. Furthermore, the primary motivation for UniHan wasn't to 'clean up' similar orthographies to avoid encoding homoglyphs. It was to stay inside 16 bits of coding space. Unicode was fighting a civil war against UCS, which proposed a 32-bit codepoint space, which would mean 32-bit characters, which the entire industry NOPE'd out of. Turns out, UCS was right, 16 bits was not enough, and the result is that every system that jumped on the Unicode train early[1] is now permanently cursed with improperly handling less-common kanji and most emoji. For the record, I consider both UniHan and the hypothetical "UniGreek" a mistake. What characters get encoded in Unicode should match what speakers of a given language would consider distinct characters, not what we can arbitrarily merge to fit under a given coding bitrate. The only viable long-term solution for internationalized text was 32-bit codepoints encoded using a variable length encoding compatible with ASCII. We wound up with 20-bit codepoints, but I suspect at some point we'll need to break UTF-16 some more. [0] This is known as "yurei moji" in Japanese [1] Windows, Java, and JavaScript[2] all mishandle codepoints outside the 16-bit basic multilingual plane, which were encoded as pairs of 16-bit surrogate values in a special range of non-codepoints. Modern UTF-16 handling is supposed to treat these as single astral characters and reject broken surrogates, but the systems in question retain Unicode 1.0 era quirks for backwards compatibility. A few years after the Unicode/UCS wars, Ken Thompson and Rob Pike would propose UTF-8, implementing it in Plan 9. This encoding was and is superior to UTF-16 in every possible way - including support for 31-bit code points, which UTF-16 surrogates can't do. UTF-8 isn't mangled by anything except the above programs... and MySQL, which had to add a second "no seriously it's UTF-8 for real" encoding. [2] ActionScript inclusive. Yes, Ruffle has its own wide string library because of this.

arp242 3y ago

> Every European character of the same general design should be merged. Won't combining Latin and Greek alphabets be horrible? Many formulas for starters, but also things like "β-radiation" (although I suppose b-radiation works too). I don't really have strong opinions on Han unification as I don't speak those languages, but "Euro Unification" would seem like a right pain, and in general I feel Unicode is often too clever by half.

deleted 3y ago

[deleted]