3 ms·

> AFAIK the only advantage of UTF-16 is that it's a little more efficient when encoding text written in certain scripts. and IIRC east Asian languages (which b

> AFAIK the only advantage of UTF-16 is that it's a little more efficient when encoding text written in certain scripts. and IIRC east Asian languages (which benefit the most) almost only use UTF-8

gsnedders 8y ago
And almost all actual uses are cases where markup (be it HTML, OOXML, ODF) are sufficiently common that the total file size ends up being larger using UTF-16 than UTF-8.
jcranmer 8y ago
I don't know if this is still the case, but as of a few years ago, there was clear evidence that the Japanese would rather have their text corrupted via mojibake than silently use UTF-8. Of course, that's about avoiding Unicode entirely rather than preferring UTF-16 over UTF-8.
- pmarreck 8y ago
  TIL about "mojibake" (I visited Japan for 2 weeks for the first time about a month ago; had a great time)
- simias 8y ago
  Do you know why that is? Why don't they like Unicode?
  - rspeer 8y ago
    A couple reasons I'm aware of: - Han unification. Chinese and Japanese characters that have the same etymology but are written differently are sometimes assigned to the same codepoint, and it's up to the font to distinguish. And application designers... tend not to detect the language and change the font accordingly. This leads to Japanese text being rendered in a Chinese font. - Some Han de-unification has happened in Unicode now. Great. Which version should fonts and encodings support? - Nobody actually knows the complete de facto mapping between Shift-JIS and Unicode. Yes, there are standards and ICU and Python modules and stuff; they're incomplete. This leads to data loss surrounding rare characters. (Tell me you've got such a mapping and I'll give you some strings I found in the wild to decode with it.)
    - ksec 8y ago
      >Han unification. I am still hoping some day down the road we can fix this without overstepping on each other's culture and fonts / glyph.
  - jcranmer 8y ago
    One legitimate concern I know of for email is that you could still get Japanese mobile phones as late as 2008 that didn't support UTF-8. People setting their mail filters to autotrash UTF-8 has also definitely happened (pro tip: never attempt to filter by charset; if you really want to, filter by script instead). I believe another major issue is that a fair amount of software sets language environment for things such as fonts based on charset. Charsets such as ISO-2022-JP, GBK, and Big5 give you a clear clue as to what the underlying language is, but UTF-8 just tells you "here is CJK ideographs, good luck!" On the other hand, I've never really heard of any opposition to Unicode from Chinese or Korean locales, so I suspect a large part of it is just stubborn tech entrenchment in Japan as opposed to strong technical reasons.
thaumasiotes 8y ago
> east Asian languages (which benefit the most) almost only use UTF-8 China, a major user of East Asian languages, uses GB2312, certainly not UTF-8. UTF-8 assigns 3 bytes to Chinese characters so it can use the 2-byte space for European characters; that's not exactly a big selling point when you're using East Asian languages.

> AFAIK the only advantage of UTF-16 is that it's a little more efficient when encoding text written in certain scripts. and IIRC east Asian languages (which benefit the most) almost only use UTF-8

gsnedders 8y ago

And almost all actual uses are cases where markup (be it HTML, OOXML, ODF) are sufficiently common that the total file size ends up being larger using UTF-16 than UTF-8.

jcranmer 8y ago

I don't know if this is still the case, but as of a few years ago, there was clear evidence that the Japanese would rather have their text corrupted via mojibake than silently use UTF-8. Of course, that's about avoiding Unicode entirely rather than preferring UTF-16 over UTF-8.

pmarreck 8y ago
TIL about "mojibake" (I visited Japan for 2 weeks for the first time about a month ago; had a great time)
simias 8y ago
Do you know why that is? Why don't they like Unicode?
- rspeer 8y ago
  A couple reasons I'm aware of: - Han unification. Chinese and Japanese characters that have the same etymology but are written differently are sometimes assigned to the same codepoint, and it's up to the font to distinguish. And application designers... tend not to detect the language and change the font accordingly. This leads to Japanese text being rendered in a Chinese font. - Some Han de-unification has happened in Unicode now. Great. Which version should fonts and encodings support? - Nobody actually knows the complete de facto mapping between Shift-JIS and Unicode. Yes, there are standards and ICU and Python modules and stuff; they're incomplete. This leads to data loss surrounding rare characters. (Tell me you've got such a mapping and I'll give you some strings I found in the wild to decode with it.)
  - ksec 8y ago
    >Han unification. I am still hoping some day down the road we can fix this without overstepping on each other's culture and fonts / glyph.
- jcranmer 8y ago
  One legitimate concern I know of for email is that you could still get Japanese mobile phones as late as 2008 that didn't support UTF-8. People setting their mail filters to autotrash UTF-8 has also definitely happened (pro tip: never attempt to filter by charset; if you really want to, filter by script instead). I believe another major issue is that a fair amount of software sets language environment for things such as fonts based on charset. Charsets such as ISO-2022-JP, GBK, and Big5 give you a clear clue as to what the underlying language is, but UTF-8 just tells you "here is CJK ideographs, good luck!" On the other hand, I've never really heard of any opposition to Unicode from Chinese or Korean locales, so I suspect a large part of it is just stubborn tech entrenchment in Japan as opposed to strong technical reasons.

thaumasiotes 8y ago

> east Asian languages (which benefit the most) almost only use UTF-8 China, a major user of East Asian languages, uses GB2312, certainly not UTF-8. UTF-8 assigns 3 bytes to Chinese characters so it can use the 2-byte space for European characters; that's not exactly a big selling point when you're using East Asian languages.