5 ms·

Ideally, the document format should specify which parts are in what language. Both MS Word and HTML can do that.

by TorKlingberg 6y ago

lucio 6y ago
The idea of UNICODE was to avoid that.
- numpad0 6y ago
  Was. Whoever thought it a good idea to merge letters with similar shapes to same code points must’ve been drinking way too much ancient Chinese civilization kool-aid.
  - jerf 6y ago
    I think they were trying to keep Unicode fitting into 16 bits. We later gave that up, but now it's kinda too late to go back and de-unify.
    - dhosek 6y ago
      IIRC, ISO-10646 initially preserved each of the 16-bit national codings for Han characters (including three separate Chinese encodings). I don't know if that has been retained at all in later versions of the standard and a cursory reading of the relevant wikipedia pages is not informative.
  - jcranmer 6y ago
    That would be the Chinese, Japanese, and Koreans. The original motivation for the efforts that resulted in Han unification was to help with library and bibliography management (some of these efforts were by non-CJK speakers). One of the original design goals of Unicode was to be able to represent all of the characters in existing character sets uniquely (so two distinct characters in some charset requires two distinct characters in Unicode), and another design goal was to be able to facilitate conversion between different character sets representing the same script. This dovetailed nicely with the existing efforts to unify CJK scripts for bibliographies, hence Han unification.
- matvore 6y ago
  I believe the plan behind the Chinese characters in Unicode was always that different hanzi-using languages had different renderings of certain characters, and that rendering it properly would be up to the font.
  - corty 6y ago
    Yes, but that was botched from the start. Latin alphabets include various "font variants" in unicode, you can write C, ℂ, ℭ, 𝒞, ⠉, 𝐂, Ⅽ, 𝙲 and maybe a few more… I guess the accusations of Unicode being somewhat centered on western languages do have a point.
enriquto 6y ago
> Ideally, the document format should specify which parts are in what language. But this is "off-band" information and it kind of defeats the purpose of unicode. Moreover it is extremely annoying in practice: imagine that you write a text (e.g., a comment on HN) where you want to explain the character differences between japanese and chinese. How do you get to do that? There are no "language selector characters" in unicode!
- hibbelig 6y ago
  It would be nice if it was possible to write mixed-language Chinese and Japanese here in HN comments. It would also be nice if it was possible to write math here. But neither are possible...
  - enriquto 6y ago
    Yes, it's a bit surprising that things like α, β, γ can be written, but not other letters. Yet, you can go a long way using unicode math symbols. EDIT: just to try, it seems that it works 平仮名ひらがな漢字汉字 한자
- arm 6y ago
  Yes, it would really be great if the Unicode Consortium designated some Unicode codepoints to be combining characters specifying the type of glyph to be used for the preceding Han character… and in fact, it seems like they have (for some at least)! The Variation Selectors Unicode block¹ has three variation selectors designated for this purpose: • U+FE00 VARIATION SELECTOR-1 • U+FE01 VARIATION SELECTOR-2 • U+FE02 VARIATION SELECTOR-3 I assume they work like U+FE0E VARIATION SELECTOR-15 (for 'text presentation' of the preceding character) and U+FE0F VARIATION SELECTOR-16 (for 'emoji presentation' of the preceding character). You can see examples of that in action here². ―――――― ¹ — https://en.wikipedia.org/wiki/Variation_Selectors_%28Unicode_block%29 https://en.wikipedia.org/wiki/Variation_Selectors_%28Unicode... ² — https://apple.stackexchange.com/questions/180400/how-to-turn-off-emojis-entirely-on-ios/240450#240450 https://apple.stackexchange.com/questions/180400/how-to-turn...

Ideally, the document format should specify which parts are in what language. Both MS Word and HTML can do that.

lucio 6y ago

The idea of UNICODE was to avoid that.

numpad0 6y ago
Was. Whoever thought it a good idea to merge letters with similar shapes to same code points must’ve been drinking way too much ancient Chinese civilization kool-aid.
- jerf 6y ago
  I think they were trying to keep Unicode fitting into 16 bits. We later gave that up, but now it's kinda too late to go back and de-unify.
  - dhosek 6y ago
    IIRC, ISO-10646 initially preserved each of the 16-bit national codings for Han characters (including three separate Chinese encodings). I don't know if that has been retained at all in later versions of the standard and a cursory reading of the relevant wikipedia pages is not informative.
- jcranmer 6y ago
  That would be the Chinese, Japanese, and Koreans. The original motivation for the efforts that resulted in Han unification was to help with library and bibliography management (some of these efforts were by non-CJK speakers). One of the original design goals of Unicode was to be able to represent all of the characters in existing character sets uniquely (so two distinct characters in some charset requires two distinct characters in Unicode), and another design goal was to be able to facilitate conversion between different character sets representing the same script. This dovetailed nicely with the existing efforts to unify CJK scripts for bibliographies, hence Han unification.
matvore 6y ago
I believe the plan behind the Chinese characters in Unicode was always that different hanzi-using languages had different renderings of certain characters, and that rendering it properly would be up to the font.
- corty 6y ago
  Yes, but that was botched from the start. Latin alphabets include various "font variants" in unicode, you can write C, ℂ, ℭ, 𝒞, ⠉, 𝐂, Ⅽ, 𝙲 and maybe a few more… I guess the accusations of Unicode being somewhat centered on western languages do have a point.

enriquto 6y ago

> Ideally, the document format should specify which parts are in what language. But this is "off-band" information and it kind of defeats the purpose of unicode. Moreover it is extremely annoying in practice: imagine that you write a text (e.g., a comment on HN) where you want to explain the character differences between japanese and chinese. How do you get to do that? There are no "language selector characters" in unicode!

hibbelig 6y ago
It would be nice if it was possible to write mixed-language Chinese and Japanese here in HN comments. It would also be nice if it was possible to write math here. But neither are possible...
- enriquto 6y ago
  Yes, it's a bit surprising that things like α, β, γ can be written, but not other letters. Yet, you can go a long way using unicode math symbols. EDIT: just to try, it seems that it works 平仮名ひらがな漢字汉字 한자
arm 6y ago
Yes, it would really be great if the Unicode Consortium designated some Unicode codepoints to be combining characters specifying the type of glyph to be used for the preceding Han character… and in fact, it seems like they have (for some at least)! The Variation Selectors Unicode block¹ has three variation selectors designated for this purpose: • U+FE00 VARIATION SELECTOR-1 • U+FE01 VARIATION SELECTOR-2 • U+FE02 VARIATION SELECTOR-3 I assume they work like U+FE0E VARIATION SELECTOR-15 (for 'text presentation' of the preceding character) and U+FE0F VARIATION SELECTOR-16 (for 'emoji presentation' of the preceding character). You can see examples of that in action here². ―――――― ¹ — https://en.wikipedia.org/wiki/Variation_Selectors_%28Unicode_block%29 https://en.wikipedia.org/wiki/Variation_Selectors_%28Unicode... ² — https://apple.stackexchange.com/questions/180400/how-to-turn-off-emojis-entirely-on-ios/240450#240450 https://apple.stackexchange.com/questions/180400/how-to-turn...