4 ms·

Can we please talk about Unicode without the myth of Han Unification being bad somehow? The problem here is exactly the lack of unification in Roman alphabets!

lmm 3y ago
> Can we please talk about Unicode without the myth of Han Unification being bad somehow? It's not a myth, as anyone living in Japan knows, and the "just use Unicode, all you need is Unicode" dogma is really harmful; a lot of "international" software has become significantly worse for Japanese users since it took hold. > The problem here is exactly the lack of unification in Roman alphabets! Problems caused by failing to unify characters that look the same do not mean it was a good idea to unify characters that look different!
- earthboundkid 3y ago
  > It's not a myth, as anyone living in Japan knows I lived in Japan. It is a myth. :-¥
- eviks 3y ago
  Would it still be harmful if language tag were used?
  - lmm 3y ago
    If the tag mechanism was used consistently and handled by all software, no. But in practice the only way that would happen is if the tag mechanism was required for many languages. Unicode is, in practice, a system that works the same way for ~every human language except Japanese, which makes it much worse than the previous "byte stream + encoding" system where any program written to support anything more than just US English would naturally work correctly for every other language, including Japanese.
    - renhanxue 3y ago
      > Unicode is, in practice, a system that works the same way for ~every human language except Japanese This is simply not true. As I've pointed out in a sibling comment, Unicode has a lot of surprising and frustrating behaviors with many European languages as well if you use it without locale data. The characters will look right, but e.g. searching, sorting and case-insensitive comparisons will not work as expected if the application is not locale aware.
      - lmm 3y ago
        > The characters will look right, but e.g. searching, sorting and case-insensitive comparisons will not work as expected This is quite a different situation from Japan. A lot of applications don't do searching, sorting, or case-insensitive comparisons, but virtually every application displays text.
- jhanschoo 3y ago
  > "just use Unicode, all you need is Unicode" dogma is really harmful; a lot of "international" software has become significantly worse for Japanese users since it took hold. The alternative would be that the software used Shift_JIS with a Japanese font. If the software used a Japanese font for Japanese it wouldn't need metadata anyway. There really isn't a problem with Han unification as long as you always switch to a font appropriate for your language; you don't need to configure metadata. If you don't you are always going to run into missing codepoint problems. In cases where the system or user configures the font, properly using Unicode is still easier than configuring alternate encodings for multiple languages.
  - lmm 3y ago
    > The alternative would be that the software used Shift_JIS with a Japanese font. As far as I know all Shift_JIS fonts are Japanese; you would have to be wilfully perverse to make one that wasn't. > If the software used a Japanese font for Japanese it wouldn't need metadata anyway. If it just uses the system default font for that encoding, as almost all software does, then it will also behave correctly. > There really isn't a problem with Han unification as long as you always switch to a font appropriate for your language Right. But approximately no software does that, because if you don't do it then your software will work fine everywhere other than Japan, and even in Japan it will kind-of-sort-of work to the point that a non-native probably won't notice a problem. > In cases where the system or user configures the font, properly using Unicode is still easier than configuring alternate encodings for multiple languages. I'm not convinced it is. Configuring your software to use the right font on a Unicode system is, as far as I can see, at least as hard as configuring your software to use the right encoding on a non-Unicode system. It just fails less obviously when you don't, particularly outside Japan.
    - GoblinSlayer 3y ago
      Japanese programs aren't globalized and already rely on the system being fine tuned for Japanese, so default font is already correct.
      - 3y ago
renhanxue 3y ago
Both problems are missing the point: you cannot handle Unicode correctly without locale information (which needs to be carried alongside as metadata outside of the string itself). To a Swede or a Finn, o and ö are different letters, as distinct as a and b (ö sorts at the very end at the alphabet). A search function that mixes them up would be very frustrating. On the other hand, to an American, a search function that doesn't find "coöperation" when you search for "cooperation" is also very frustrating. Back in Sweden, v and w are basically the same letter, especially when it comes to people's last names, and should probably be treated the same. Further south, if you try to lowercase an I and the text is in Turkish (or in certain other Turkic languages), you want a dotless i (ı), not a regular lowercase i. This is extremely spooky if you try to do case insensitive equality comparisons and aren't paying attention, because if you do it wrong and end up with a regular lowercase i, you've lost information and uppercasing again will not restore the original string. There are tons and tons of problems like this in European languages. The root cause is exactly the same as the Han unification gripes: Unicode without locale information is not enough to handle natural languages in the way users expect.
- earthboundkid 3y ago
  Exactly! Thank you for giving a good explanation of why this whole post is founded on a fundamental misunderstanding.
- eviks 3y ago
  > which needs to be carried alongside as metadata outside of the string itself Why not as data tagged with the appropriate language? https://www.unicode.org/faq/languagetagging.html https://www.unicode.org/faq/languagetagging.html
  - renhanxue 3y ago
    If you mean in-band language tagging inside the string itself, the page you're linking to points out that this is deprecated. The tag characters are now mostly used for emoji stuff. If you only need to be compatible with yourself you can of course do whatever you like, but otherwise, I agree with what the linked page says: > Users who need to tag text with the language identity should be using standard markup mechanisms, such as those provided by HTML, XML, or other rich text mechanisms. In other contexts, such as databases or internet protocols, language should generally be indicated by appropriate data fields, rather than by embedded language tags or markup.

lmm 3y ago

> Can we please talk about Unicode without the myth of Han Unification being bad somehow? It's not a myth, as anyone living in Japan knows, and the "just use Unicode, all you need is Unicode" dogma is really harmful; a lot of "international" software has become significantly worse for Japanese users since it took hold. > The problem here is exactly the lack of unification in Roman alphabets! Problems caused by failing to unify characters that look the same do not mean it was a good idea to unify characters that look different!

earthboundkid 3y ago
> It's not a myth, as anyone living in Japan knows I lived in Japan. It is a myth. :-¥
eviks 3y ago
Would it still be harmful if language tag were used?
- lmm 3y ago
  If the tag mechanism was used consistently and handled by all software, no. But in practice the only way that would happen is if the tag mechanism was required for many languages. Unicode is, in practice, a system that works the same way for ~every human language except Japanese, which makes it much worse than the previous "byte stream + encoding" system where any program written to support anything more than just US English would naturally work correctly for every other language, including Japanese.
  - renhanxue 3y ago
    > Unicode is, in practice, a system that works the same way for ~every human language except Japanese This is simply not true. As I've pointed out in a sibling comment, Unicode has a lot of surprising and frustrating behaviors with many European languages as well if you use it without locale data. The characters will look right, but e.g. searching, sorting and case-insensitive comparisons will not work as expected if the application is not locale aware.
    - lmm 3y ago
      > The characters will look right, but e.g. searching, sorting and case-insensitive comparisons will not work as expected This is quite a different situation from Japan. A lot of applications don't do searching, sorting, or case-insensitive comparisons, but virtually every application displays text.
jhanschoo 3y ago
> "just use Unicode, all you need is Unicode" dogma is really harmful; a lot of "international" software has become significantly worse for Japanese users since it took hold. The alternative would be that the software used Shift_JIS with a Japanese font. If the software used a Japanese font for Japanese it wouldn't need metadata anyway. There really isn't a problem with Han unification as long as you always switch to a font appropriate for your language; you don't need to configure metadata. If you don't you are always going to run into missing codepoint problems. In cases where the system or user configures the font, properly using Unicode is still easier than configuring alternate encodings for multiple languages.
- lmm 3y ago
  > The alternative would be that the software used Shift_JIS with a Japanese font. As far as I know all Shift_JIS fonts are Japanese; you would have to be wilfully perverse to make one that wasn't. > If the software used a Japanese font for Japanese it wouldn't need metadata anyway. If it just uses the system default font for that encoding, as almost all software does, then it will also behave correctly. > There really isn't a problem with Han unification as long as you always switch to a font appropriate for your language Right. But approximately no software does that, because if you don't do it then your software will work fine everywhere other than Japan, and even in Japan it will kind-of-sort-of work to the point that a non-native probably won't notice a problem. > In cases where the system or user configures the font, properly using Unicode is still easier than configuring alternate encodings for multiple languages. I'm not convinced it is. Configuring your software to use the right font on a Unicode system is, as far as I can see, at least as hard as configuring your software to use the right encoding on a non-Unicode system. It just fails less obviously when you don't, particularly outside Japan.
  - GoblinSlayer 3y ago
    Japanese programs aren't globalized and already rely on the system being fine tuned for Japanese, so default font is already correct.
    - 3y ago

renhanxue 3y ago

Both problems are missing the point: you cannot handle Unicode correctly without locale information (which needs to be carried alongside as metadata outside of the string itself). To a Swede or a Finn, o and ö are different letters, as distinct as a and b (ö sorts at the very end at the alphabet). A search function that mixes them up would be very frustrating. On the other hand, to an American, a search function that doesn't find "coöperation" when you search for "cooperation" is also very frustrating. Back in Sweden, v and w are basically the same letter, especially when it comes to people's last names, and should probably be treated the same. Further south, if you try to lowercase an I and the text is in Turkish (or in certain other Turkic languages), you want a dotless i (ı), not a regular lowercase i. This is extremely spooky if you try to do case insensitive equality comparisons and aren't paying attention, because if you do it wrong and end up with a regular lowercase i, you've lost information and uppercasing again will not restore the original string. There are tons and tons of problems like this in European languages. The root cause is exactly the same as the Han unification gripes: Unicode without locale information is not enough to handle natural languages in the way users expect.

earthboundkid 3y ago
Exactly! Thank you for giving a good explanation of why this whole post is founded on a fundamental misunderstanding.
eviks 3y ago
> which needs to be carried alongside as metadata outside of the string itself Why not as data tagged with the appropriate language? https://www.unicode.org/faq/languagetagging.html https://www.unicode.org/faq/languagetagging.html
- renhanxue 3y ago
  If you mean in-band language tagging inside the string itself, the page you're linking to points out that this is deprecated. The tag characters are now mostly used for emoji stuff. If you only need to be compatible with yourself you can of course do whatever you like, but otherwise, I agree with what the linked page says: > Users who need to tag text with the language identity should be using standard markup mechanisms, such as those provided by HTML, XML, or other rich text mechanisms. In other contexts, such as databases or internet protocols, language should generally be indicated by appropriate data fields, rather than by embedded language tags or markup.