7 ms·

Yes, please! This aligns with my rants on the subject [1] because I want all software to use UTF-8 by default. In my experience as an application developer ne

by bhauer 9y ago

Yes, please! This aligns with my rants on the subject [1] because I want all software to use UTF-8 by default. In my experience as an application developer nearly all of my character encoding grief has been caused by an application or library somewhere in my overall system architecture not using UTF-8 by default. This is followed by me needing to find how to configure it. I am by no means a character encoding expert, but I want everything to use UTF-8 unless there's a very particular need for something else (I'm still waiting to find out what that might be). [1] http://tiamat.tsotech.com/utf8-everything http://tiamat.tsotech.com/utf8-everything

lmm 9y ago
Software that needs to display Japanese names (particularly if it may also need to display Chinese names) tends to use traditional encodings to avoid Unicode's Han unification. (Some characters are considered by Unicode to be the same character despite the Japanese and Chinese ways of writing that character being visually distinguishable; Unicode regards this as the same character being written in different fonts (just as in our Latin alphabet there are multiple visually distinct ways to write the lower-case "a" or the number 4, one can choose to cross "7"s or not, etc.) but customers sometimes don't find that acceptable; in particular many Japanese people are very unhappy to see their names written with the Chinese versions of those characters, which tend to be how international fonts render them)
- mey 9y ago
  So it is the same symbol, recognized as that symbol, meaning the same thing but showing up as what some people consider comic sans? So the problem is the font?
  - cryptonector 9y ago
    The problem is not the font. Users of CJK actually want this distinction in codepoint assignments, not font choice. Non-CJK users might think it silly, and maybe it is, but whatever the case, unification has been abandoned.
    - unscaled 9y ago
      Yes, theoretically speaking this distinction might have been useful, but so would have been combining hieroglyphs and hangul jamo and the ability to define an entirely new CJK character by specifying components and placements would have been even nicer. None of this would be solved by devolving back to popular pre-unicode standards like EUC-JP or Big5.
  - lmm 9y ago
    Yes except that there's no way to choose a font that all users will be happy with - if you switch your application to a font that uses the Japanese way of writing the character, some Chinese people with that character in their name will be equally unhappy. (It's hard to create an exact analogy because there's no country that the US has the same kind of tense relationship with, and we don't attach quite the same personal meaning to the exact reading of our name - we're used to seeing our names written in capitals and lower-case and being the same name, we don't tell people the meanings of the characters used in our names as we spell them, but I guess imagine having a 7 in your name that you care about quite personally and then a program you use renders it as a French 7 even though you hate the French - you can tell that it's the same character, but it's still wrong and feels very rude. It's the sort of thing that seems trivial in the abstract but ends up being felt surprisingly personally - I'm always a lot more pissed off than I expected to be when people call me "Imm"). You could do something like switching the font based on the system locale, but at that point you're no better off than when you were using the system's character encoding, and following a less well-trodden path.
    - unscaled 9y ago
      Well, there is really no way to solve this other than using a language-specific font or markup to indicate the language of the text. The first solution is probably the more common one, since there are very few international fonts that actually do CJK languages well to begin with. The number of characters you need to cover is hard enough to make it a daunting task, and if Han Unification didn't exist it could have been 3 times harder... The TRON Multilingual System (which was championed by the Han Unification opponents in the 90s) basically does second solution. It did avoid character unification between Chinese, Japanese and Korean, but it still chose to specify the language of the text to support different overall processing based on the language (e.g. different ligatures may be activated for different European languages). If you're talking about Japanese users being unhappy that their name characters change when its rendered _within Chinese Text_ - well, that's how it's always been in newspapers, books and official documents. Signs all over Japan (made by Japanese for Chinese tourists) show the Chinese rendition of Japanese place names such as 涩谷 (Shibuya) and nobody seems to bother. It goes the other way too, of course.
- cryptonector 9y ago
  Unicode abandoned unification.
  - lmm 9y ago
    That doesn't fix the problem for existing characters though, right? If I run a file through "iconv -f SHIFT-JIS -t UTF-8" then it's going to come out using codepoints that are rendered as the "wrong" characters in most international fonts, right?
    - cryptonector 9y ago
      Is that conversion standardized and stable?
      - lmm 9y ago
        I would hope and expect so - the current implementation may be "wrong" but having it behave differently in a newer version would be even worse than the current situation.
        cryptonector 9y ago
        For many scripts a conversion from non-Unicode to Unicode can't really be standard without also specifying a normalization form for output. I don't know of any standards in this space.
        lmm 9y ago
        I can't imagine defaulting to anything other than NFC.
        cryptonector 9y ago
        Sure, but that's not responsive.
- unscaled 9y ago
  I think I've yet to met a Japanese person who was even mildly unhappy to see their name rendered incorrectly due to Han Unification. It seems like when Unicode came out, the TRON guys and some other Japanese engineer were pretty vocal about the evil of Han Unification, but the reality is that the average Japanese user probably doesn't care. They get their name butchered at least once a month in their bank account statement (in half-width Katakana), and if they have an unusual character or a slightly unusual reading in their name, their fellow compatriots would butcher their name as well. Compared to that, the number of times people from Japan browsing at a site configured to display Japanese language with Chinese Fonts? That must happen once in Super Blue Blood Moon, but not that often. And "many" is definitely exaggeration, considering the number of Japanese name which use characters subject to heavily modified by unification. In other words, "bone" is not a popular character in Japanese names, who would have figured. Han Unification is not without its pains, of course, and I still can't tell for sure whether it was a good idea, but using EUC-JP or Shift-JIS is not going to get you around it, since you still need a Japanese font to display the characters correctly as they should look in Japanese. And I would gladly take UTF-8 over EUC-JP and Shift-JIS every day.
cryptonector 9y ago
Using UTF-16 is silly because it doesn't mean that length in code units == length in codepoints -- that was only ever true of UCS-2. Not that length in codepoints is all that useful anyways (because of decomposition: you can't just split on the Nth codepoint, only on the Nth character). But because that was true of UCS-2, some systems (e.g., Windows) selected UCS-2 and then UTF-16. Sad, but true. UTF-8 is clearly superior.
- ridiculous_fish 9y ago
  It's not obvious that UTF-8 is superior. UTF-8 has subtleties such as non-shortest forms and invalid code units. These issues have lead to security exploits [1] but are impossible in UTF-16 by design. The internal encoding of strings attracts too much attention. A proper string type abstracts over these details. When doing arithmetic in Haskell, Python, JavaScript, etc. you don't worry about how the numbers are encoded; strings should be the same. [1] https://unicode.org/reports/tr36/#UTF-8_Exploit https://unicode.org/reports/tr36/#UTF-8_Exploit
  - cryptonector 9y ago
    A brief search shows for "UTF-16 vulnerability" finds relevant results. UTF-8 overlong sequences haven't been a problem for a while. The choice of encoding is definitely relevant for all external file formats, protocols, APIs. We can't just let the choice of encoding be purely internal, though I agree that programmers should not need to care about what their runtimes use internally for strings, as long as it is Unicode (UTF-8, 16, 32, or some other UTF say). But no UTF actually gives you random access by character index because a) no runtime is going to be ensuring normalization of all strings to precomposed sequences (not even NFC because NFC is closed to new compositions!), b) not every valid character will have pre-composed codepoints therefore (a) isn't even possible. Given this, UTF-16 and UTF-32 have no real advantages over UTF-8, and the false sense of simplicity they convey is dangerous anyways.

by bhauer 9y ago

lmm 9y ago

Software that needs to display Japanese names (particularly if it may also need to display Chinese names) tends to use traditional encodings to avoid Unicode's Han unification. (Some characters are considered by Unicode to be the same character despite the Japanese and Chinese ways of writing that character being visually distinguishable; Unicode regards this as the same character being written in different fonts (just as in our Latin alphabet there are multiple visually distinct ways to write the lower-case "a" or the number 4, one can choose to cross "7"s or not, etc.) but customers sometimes don't find that acceptable; in particular many Japanese people are very unhappy to see their names written with the Chinese versions of those characters, which tend to be how international fonts render them)

mey 9y ago
So it is the same symbol, recognized as that symbol, meaning the same thing but showing up as what some people consider comic sans? So the problem is the font?
- cryptonector 9y ago
  The problem is not the font. Users of CJK actually want this distinction in codepoint assignments, not font choice. Non-CJK users might think it silly, and maybe it is, but whatever the case, unification has been abandoned.
  - unscaled 9y ago
    Yes, theoretically speaking this distinction might have been useful, but so would have been combining hieroglyphs and hangul jamo and the ability to define an entirely new CJK character by specifying components and placements would have been even nicer. None of this would be solved by devolving back to popular pre-unicode standards like EUC-JP or Big5.
- lmm 9y ago
  Yes except that there's no way to choose a font that all users will be happy with - if you switch your application to a font that uses the Japanese way of writing the character, some Chinese people with that character in their name will be equally unhappy. (It's hard to create an exact analogy because there's no country that the US has the same kind of tense relationship with, and we don't attach quite the same personal meaning to the exact reading of our name - we're used to seeing our names written in capitals and lower-case and being the same name, we don't tell people the meanings of the characters used in our names as we spell them, but I guess imagine having a 7 in your name that you care about quite personally and then a program you use renders it as a French 7 even though you hate the French - you can tell that it's the same character, but it's still wrong and feels very rude. It's the sort of thing that seems trivial in the abstract but ends up being felt surprisingly personally - I'm always a lot more pissed off than I expected to be when people call me "Imm"). You could do something like switching the font based on the system locale, but at that point you're no better off than when you were using the system's character encoding, and following a less well-trodden path.
  - unscaled 9y ago
    Well, there is really no way to solve this other than using a language-specific font or markup to indicate the language of the text. The first solution is probably the more common one, since there are very few international fonts that actually do CJK languages well to begin with. The number of characters you need to cover is hard enough to make it a daunting task, and if Han Unification didn't exist it could have been 3 times harder... The TRON Multilingual System (which was championed by the Han Unification opponents in the 90s) basically does second solution. It did avoid character unification between Chinese, Japanese and Korean, but it still chose to specify the language of the text to support different overall processing based on the language (e.g. different ligatures may be activated for different European languages). If you're talking about Japanese users being unhappy that their name characters change when its rendered _within Chinese Text_ - well, that's how it's always been in newspapers, books and official documents. Signs all over Japan (made by Japanese for Chinese tourists) show the Chinese rendition of Japanese place names such as 涩谷 (Shibuya) and nobody seems to bother. It goes the other way too, of course.
cryptonector 9y ago
Unicode abandoned unification.
- lmm 9y ago
  That doesn't fix the problem for existing characters though, right? If I run a file through "iconv -f SHIFT-JIS -t UTF-8" then it's going to come out using codepoints that are rendered as the "wrong" characters in most international fonts, right?
  - cryptonector 9y ago
    Is that conversion standardized and stable?
    - lmm 9y ago
      I would hope and expect so - the current implementation may be "wrong" but having it behave differently in a newer version would be even worse than the current situation.
      - cryptonector 9y ago
        For many scripts a conversion from non-Unicode to Unicode can't really be standard without also specifying a normalization form for output. I don't know of any standards in this space.
        lmm 9y ago
        I can't imagine defaulting to anything other than NFC.
        cryptonector 9y ago
        Sure, but that's not responsive.
unscaled 9y ago
I think I've yet to met a Japanese person who was even mildly unhappy to see their name rendered incorrectly due to Han Unification. It seems like when Unicode came out, the TRON guys and some other Japanese engineer were pretty vocal about the evil of Han Unification, but the reality is that the average Japanese user probably doesn't care. They get their name butchered at least once a month in their bank account statement (in half-width Katakana), and if they have an unusual character or a slightly unusual reading in their name, their fellow compatriots would butcher their name as well. Compared to that, the number of times people from Japan browsing at a site configured to display Japanese language with Chinese Fonts? That must happen once in Super Blue Blood Moon, but not that often. And "many" is definitely exaggeration, considering the number of Japanese name which use characters subject to heavily modified by unification. In other words, "bone" is not a popular character in Japanese names, who would have figured. Han Unification is not without its pains, of course, and I still can't tell for sure whether it was a good idea, but using EUC-JP or Shift-JIS is not going to get you around it, since you still need a Japanese font to display the characters correctly as they should look in Japanese. And I would gladly take UTF-8 over EUC-JP and Shift-JIS every day.

cryptonector 9y ago

Using UTF-16 is silly because it doesn't mean that length in code units == length in codepoints -- that was only ever true of UCS-2. Not that length in codepoints is all that useful anyways (because of decomposition: you can't just split on the Nth codepoint, only on the Nth character). But because that was true of UCS-2, some systems (e.g., Windows) selected UCS-2 and then UTF-16. Sad, but true. UTF-8 is clearly superior.

ridiculous_fish 9y ago
It's not obvious that UTF-8 is superior. UTF-8 has subtleties such as non-shortest forms and invalid code units. These issues have lead to security exploits [1] but are impossible in UTF-16 by design. The internal encoding of strings attracts too much attention. A proper string type abstracts over these details. When doing arithmetic in Haskell, Python, JavaScript, etc. you don't worry about how the numbers are encoded; strings should be the same. [1] https://unicode.org/reports/tr36/#UTF-8_Exploit https://unicode.org/reports/tr36/#UTF-8_Exploit
- cryptonector 9y ago
  A brief search shows for "UTF-16 vulnerability" finds relevant results. UTF-8 overlong sequences haven't been a problem for a while. The choice of encoding is definitely relevant for all external file formats, protocols, APIs. We can't just let the choice of encoding be purely internal, though I agree that programmers should not need to care about what their runtimes use internally for strings, as long as it is Unicode (UTF-8, 16, 32, or some other UTF say). But no UTF actually gives you random access by character index because a) no runtime is going to be ensuring normalization of all strings to precomposed sequences (not even NFC because NFC is closed to new compositions!), b) not every valid character will have pre-composed codepoints therefore (a) isn't even possible. Given this, UTF-16 and UTF-32 have no real advantages over UTF-8, and the false sense of simplicity they convey is dangerous anyways.