3 ms·

The(/a) rationale for UTF-16 is that it's more space-efficient for languages that use high code points, like East Asian languages.

udp 7y ago
Sure, but the majority of the text computers have to parse has a syntax consisting of low codepoint ASCII characters (think XML tags vs. data). The fragments of text using a high codepoint language are usually only interesting at render time.
- mehrdadn 7y ago
  I'm not supporting the position; I'm just explaining what the argument for it is, since you said you never understood the rationale behind it and felt it gave you the worst of both worlds. But I will say that the concern you have does not take into account the concerns other people have who make this argument.
  - SAI_Peregrinus 7y ago
    The other aspect of importance is that languages with Chinese characters have higher information density per character, so they tend to even out in terms of actual storage requirements.
jandrese 7y ago
IIRC the real rationale is back in the early days people bet that 16 bits would be enough for a fixed length encoding, but the bet didn't pay off and now they're stuck with the worst of both worlds.
- mehrdadn 7y ago
  Depends what you mean by "rationale" I guess. I was trying to explain why one might still want to use it, not why one might be stuck using it.
- lokedhs 7y ago
  Your explanation is correct. UTF-16 is a hack on top of a hack, and no one uses it in new software. However, Java and Javascript are stuck with it for legacy reasons. It wouldn't be so bad if people just understood that there are (almost) no valid uses-cases for measuring the size of a string. As someone else mentioned, it's a binary blob for all intents and purposes, and if processing of its content needs to be done (such as displaying it on the screen, or performing, say, word-wrap) then it should be handed over to libraries that have been designed for this purpose, because these things are very complicated. Simply by having a single non-ASCII character in my name, I'm seeing software fail on this on a regular basis.
nwallin 7y ago
East Asian HTML and XML documents are typically smaller in utf8 than utf16 because the of the markup. (on mobile so u can't prove it right now. maybe when I get home tonight) utf16 is a bad default for almost everybody. The actual reason utf16 is used is because it was easier to port to utf16 from ucs16. Much the way it's often acceptable to assume utf8 is ASCII it's often acceptable to assume utf16 is utf8. Even today, I imagine there are a lot of applications which assume utf16 is actually ucs16.
- edflsafoiewq 7y ago
  That depends on what kind documents of course. It isn't usually true for text-centric content. Here's Dazai Osamu's Hashire Melos: https://www.aozora.gr.jp/cards/000035/files/1567_14913.html https://www.aozora.gr.jp/cards/000035/files/1567_14913.html. Despite having an unusually large amount of markup (most Japanese text wouldn't add the ruby characters), it's smaller as UTF-16 than UTF-8.

udp 7y ago

Sure, but the majority of the text computers have to parse has a syntax consisting of low codepoint ASCII characters (think XML tags vs. data). The fragments of text using a high codepoint language are usually only interesting at render time.

mehrdadn 7y ago
I'm not supporting the position; I'm just explaining what the argument for it is, since you said you never understood the rationale behind it and felt it gave you the worst of both worlds. But I will say that the concern you have does not take into account the concerns other people have who make this argument.
- SAI_Peregrinus 7y ago
  The other aspect of importance is that languages with Chinese characters have higher information density per character, so they tend to even out in terms of actual storage requirements.

jandrese 7y ago

IIRC the real rationale is back in the early days people bet that 16 bits would be enough for a fixed length encoding, but the bet didn't pay off and now they're stuck with the worst of both worlds.

mehrdadn 7y ago
Depends what you mean by "rationale" I guess. I was trying to explain why one might still want to use it, not why one might be stuck using it.
lokedhs 7y ago
Your explanation is correct. UTF-16 is a hack on top of a hack, and no one uses it in new software. However, Java and Javascript are stuck with it for legacy reasons. It wouldn't be so bad if people just understood that there are (almost) no valid uses-cases for measuring the size of a string. As someone else mentioned, it's a binary blob for all intents and purposes, and if processing of its content needs to be done (such as displaying it on the screen, or performing, say, word-wrap) then it should be handed over to libraries that have been designed for this purpose, because these things are very complicated. Simply by having a single non-ASCII character in my name, I'm seeing software fail on this on a regular basis.

nwallin 7y ago

East Asian HTML and XML documents are typically smaller in utf8 than utf16 because the of the markup. (on mobile so u can't prove it right now. maybe when I get home tonight) utf16 is a bad default for almost everybody. The actual reason utf16 is used is because it was easier to port to utf16 from ucs16. Much the way it's often acceptable to assume utf8 is ASCII it's often acceptable to assume utf16 is utf8. Even today, I imagine there are a lot of applications which assume utf16 is actually ucs16.

edflsafoiewq 7y ago
That depends on what kind documents of course. It isn't usually true for text-centric content. Here's Dazai Osamu's Hashire Melos: https://www.aozora.gr.jp/cards/000035/files/1567_14913.html https://www.aozora.gr.jp/cards/000035/files/1567_14913.html. Despite having an unusually large amount of markup (most Japanese text wouldn't add the ruby characters), it's smaller as UTF-16 than UTF-8.