3 ms·

Here's another fun Unicode pitfall: does any terminal provide a way to display Chinese and Japanese text simultaneously, using the appropriate versions of the g

Here's another fun Unicode pitfall: does any terminal provide a way to display Chinese and Japanese text simultaneously, using the appropriate versions of the glyphs for each language's characters?

mananaysiempre 4y ago
As far as existing terminals are concerned, I don’t know. FWIW, there are similar problems (though only to the point of looking wrong, not of misunderstanding) in other scripts: Cyrillic as used in Bulgarian and a number of other languages[1] and even Latin as used in Polish[2]. Even the Han version, though, does not seem to me to be the sort of “what does it even mean?” problem like those I listed above; more like what you want the input to be. You can make your terminal keep language state, e.g. using the deprecated language tags. Pro: some form of this likely already needs to happen for bidi support; similar to what HTML does. Con: no text file or program ever did this; your nice UTF-8-only terminal is now stateful and goes mad after `head /dev/urandom`. Alternatively, you can require the driving program to emit variation selectors for each Han character. Pro: the state and the ensuing madness is now limited; you can still pretend you’re looking at a stream of characters. Con: no text file or program ever did this; neither does HTML although it theorerically could. [1] https://commons.wikimedia.org/wiki/File:Cyrillic_alternates.svg https://commons.wikimedia.org/wiki/File:Cyrillic_alternates.... [2] https://www.twardoch.com/download/polishhowto/kreska.html https://www.twardoch.com/download/polishhowto/kreska.html
a1369209993 4y ago
> does any terminal provide a way to display Chinese and Japanese text simultaneously, using the appropriate versions of the glyphs Does any terminal provide a way to display (say) two consecutive lines in different fonts? Because according to Unicode (and for once I don't particularly disagree), Chinese versus Japanese ideographs are the same characters, drawn in two different fonts.
- hnfong 4y ago
  https://www.facebook.com/appliedphilosophy/photos/a.828540073892609/891825587564057 https://www.facebook.com/appliedphilosophy/photos/a.82854007... Many years ago I thought the highlighted text was a light hearted joke. Now I see what it is actually quite serious. (And now I understand why, among all people, a Japanese philosopher brought up the issue) I personally think Unicode didn't go essentialist enough though. There's lots of obscure CJK characters in unicode that, when you look up a dictionary, just says it's another form of a common character. What I heard from people in the know was that the CJK committee basically made an exception for the Japanese because they were threatening to withdraw from the Unicode effort unless their demands were met, and for the most part they got what they wanted. The "Chinese versus Japanese ideographs are the same characters, drawn in two different fonts" is actually a minority of those cases, which I personally interpret as clerical errors instead of a philosophical disagreement. For example if Unicode were actually thorough with the essentialist philosophy, 關 and 関 would have the same code point.
apaprocki 4y ago
No, because it's not actually possible without extra information. The codepoints in Unicode are the same if the languages use the same character (Han unification). With no extra information, if a string is only comprised of unified codepoints, it isn't inherently any one language, so you'd have to resort to a dumb external setting that lets users pick one font in that situation. Next best, you'd have to scan the entire string and find at least one non-unified codepoint and then use that to guess what language it is and then map the correct font for that language. Or, if you have external data such as an encoding string, file metadata, text source, you can simply obey what the metadata says. Unification was done originally because early versions of Unicode were bit-constrained and there simply was not enough space to have separate codepoints for each language. This would need to be "undone" in some future version of Unicode and text encoders would need to make the backward incompatible change (in terms of rendering in unknown downstream software) of using new codepoints if this were to be done. Then the "proper" font would be deterministic.

Here's another fun Unicode pitfall: does any terminal provide a way to display Chinese and Japanese text simultaneously, using the appropriate versions of the glyphs for each language's characters?

mananaysiempre 4y ago

As far as existing terminals are concerned, I don’t know. FWIW, there are similar problems (though only to the point of looking wrong, not of misunderstanding) in other scripts: Cyrillic as used in Bulgarian and a number of other languages[1] and even Latin as used in Polish[2]. Even the Han version, though, does not seem to me to be the sort of “what does it even mean?” problem like those I listed above; more like what you want the input to be. You can make your terminal keep language state, e.g. using the deprecated language tags. Pro: some form of this likely already needs to happen for bidi support; similar to what HTML does. Con: no text file or program ever did this; your nice UTF-8-only terminal is now stateful and goes mad after `head /dev/urandom`. Alternatively, you can require the driving program to emit variation selectors for each Han character. Pro: the state and the ensuing madness is now limited; you can still pretend you’re looking at a stream of characters. Con: no text file or program ever did this; neither does HTML although it theorerically could. [1] https://commons.wikimedia.org/wiki/File:Cyrillic_alternates.svg https://commons.wikimedia.org/wiki/File:Cyrillic_alternates.... [2] https://www.twardoch.com/download/polishhowto/kreska.html https://www.twardoch.com/download/polishhowto/kreska.html

a1369209993 4y ago

> does any terminal provide a way to display Chinese and Japanese text simultaneously, using the appropriate versions of the glyphs Does any terminal provide a way to display (say) two consecutive lines in different fonts? Because according to Unicode (and for once I don't particularly disagree), Chinese versus Japanese ideographs are the same characters, drawn in two different fonts.

hnfong 4y ago
https://www.facebook.com/appliedphilosophy/photos/a.828540073892609/891825587564057 https://www.facebook.com/appliedphilosophy/photos/a.82854007... Many years ago I thought the highlighted text was a light hearted joke. Now I see what it is actually quite serious. (And now I understand why, among all people, a Japanese philosopher brought up the issue) I personally think Unicode didn't go essentialist enough though. There's lots of obscure CJK characters in unicode that, when you look up a dictionary, just says it's another form of a common character. What I heard from people in the know was that the CJK committee basically made an exception for the Japanese because they were threatening to withdraw from the Unicode effort unless their demands were met, and for the most part they got what they wanted. The "Chinese versus Japanese ideographs are the same characters, drawn in two different fonts" is actually a minority of those cases, which I personally interpret as clerical errors instead of a philosophical disagreement. For example if Unicode were actually thorough with the essentialist philosophy, 關 and 関 would have the same code point.

apaprocki 4y ago

No, because it's not actually possible without extra information. The codepoints in Unicode are the same if the languages use the same character (Han unification). With no extra information, if a string is only comprised of unified codepoints, it isn't inherently any one language, so you'd have to resort to a dumb external setting that lets users pick one font in that situation. Next best, you'd have to scan the entire string and find at least one non-unified codepoint and then use that to guess what language it is and then map the correct font for that language. Or, if you have external data such as an encoding string, file metadata, text source, you can simply obey what the metadata says. Unification was done originally because early versions of Unicode were bit-constrained and there simply was not enough space to have separate codepoints for each language. This would need to be "undone" in some future version of Unicode and text encoders would need to make the backward incompatible change (in terms of rendering in unknown downstream software) of using new codepoints if this were to be done. Then the "proper" font would be deterministic.