4 ms·

Unicode did kind of dumb thing with CJK, the unified Chinese and Japanese kanjis makes displaying the CJK text much harder problem than it should be, as it now

Unicode did kind of dumb thing with CJK, the unified Chinese and Japanese kanjis makes displaying the CJK text much harder problem than it should be, as it now relies also on a language specific font to be displayed correctly[0]. I guess this could be bandaided by some sort of language marker in the UTF8 bytestring which then a text shaping engine would have to understand and switch the font accordingly.. 0: https://heistak.github.io/your-code-displays-japanese-wrong/ https://heistak.github.io/your-code-displays-japanese-wrong/

mnau 2y ago
Wasn't it solved by https://en.wikipedia.org/wiki/Han_unification#Ideographic_Variation_Database_(IVD) https://en.wikipedia.org/wiki/Han_unification#Ideographic_Va... ? Kind of a band-aid (it's necessary to stuff a variant selector after a CJK codepoint), but should work. These decisions were made back in 1992 and codepoint in 16-bit was one of desired goals. Non-unified CJK wouldn't fit. In hindsight, it looks like a rather unfortunate decision, but having more codepoints that would fit to 16 bits could seriously hamper adoption and different standard would win (compute resources were far more limiting back then). In either case, it's like 4 byte addressing in IPv4, in hindsight, 6+ bytes would be better, but what's done is done. Edit: Even in 2000s, when C# was released, string was just a sequence of 16-bit code units (not codepoints), so they could deal with BMP without problems and astral planes were ... mostly DIY. They added Rune support (32-bit codepoint) only in .NET Core 3.0 (2019).
- Cloudef 2y ago
  Seems so, I wonder when the whole stack starts supporting it (IMEs, fonts, text shaping engines) Adobe / Google seems to have a font https://ken-lunde.medium.com/improving-font-information-processing-infrastructure-pan-cjk-typeface-development-465c2bef1235 https://ken-lunde.medium.com/improving-font-information-proc... EDIT: Seems IVS is really old, but its still a problem so not holding my breath

mnau 2y ago

Wasn't it solved by https://en.wikipedia.org/wiki/Han_unification#Ideographic_Variation_Database_(IVD) https://en.wikipedia.org/wiki/Han_unification#Ideographic_Va... ? Kind of a band-aid (it's necessary to stuff a variant selector after a CJK codepoint), but should work. These decisions were made back in 1992 and codepoint in 16-bit was one of desired goals. Non-unified CJK wouldn't fit. In hindsight, it looks like a rather unfortunate decision, but having more codepoints that would fit to 16 bits could seriously hamper adoption and different standard would win (compute resources were far more limiting back then). In either case, it's like 4 byte addressing in IPv4, in hindsight, 6+ bytes would be better, but what's done is done. Edit: Even in 2000s, when C# was released, string was just a sequence of 16-bit code units (not codepoints), so they could deal with BMP without problems and astral planes were ... mostly DIY. They added Rune support (32-bit codepoint) only in .NET Core 3.0 (2019).

Cloudef 2y ago
Seems so, I wonder when the whole stack starts supporting it (IMEs, fonts, text shaping engines) Adobe / Google seems to have a font https://ken-lunde.medium.com/improving-font-information-processing-infrastructure-pan-cjk-typeface-development-465c2bef1235 https://ken-lunde.medium.com/improving-font-information-proc... EDIT: Seems IVS is really old, but its still a problem so not holding my breath