3 ms·

> UTF-16 is great for lots of East Asian languages, which billions of people use. In UTF-8, most of those languages require 3 bytes to encode a 32-bit codepoint

by hnfong 5y ago

> UTF-16 is great for lots of East Asian languages, which billions of people use. In UTF-8, most of those languages require 3 bytes to encode a 32-bit codepoint, in UTF-16 they only ever need 2. This ends up being a huge savings. "Only ever" is an exaggeration. The vast majority of standard Chinese (aka. Putonghua) characters are in the BMP, but occasionally there are characters outside of it. In particular a lot of Hong Kong characters are outside of the BMP, and before emoji forced software to fix things, systems claiming to support UTF-16 (but were instead UCS-2) kept failing for say 1% of the characters. I've had much better luck with UTF-8 systems than purported UTF-16 ones. The problem with UTF-16 is not necessarily in its inherent technical issues, but rather, it's how often systems with UCS-2 support just claim they support UTF-16, and leave users/developers with a crappy experience when working with characters outside the BMP. There's also the encoding agnosticism thing -- with UTF-16 you run into issues about byte order marks (with or without), endian problems, and generally incompatibility with ASCII. As I said, the situation probably improved quite a bit after emoji became popular. But if I had the choice, I'd choose UTF-8 over UTF-16 any time. Also, although it's generally true that you "can't store UTF-8 in something expecting ASCII", in practice a lot of systems are somewhat more graceful than that. For example, if I grep a UTF-8 text file, grep doesn't need to know the text encoding if all you're trying to find is an ASCII string. Similarly it's possible to edit a UTF-8 text file as an ASCII file if the editor preserves the "binary" bits. In conclusion, "different encodings are good at different things" is only true when considered in isolation. The historical baggage of UCS-2, and separately, of ASCII-compatibility tilts the balance strongly in favor of UTF-8 IMHO. The alleged 33% saving in Chinese text doesn't really matter that much in the grand scheme of things -- which is perhaps why people use UTF-8 in spite of the "advantages" you mentioned.

camgunz 5y ago
Yeah I mean, I broadly agree with what you're saying. Text encoding is a mess, and pragmatic solutions are needed. And when jamming text in various places, jamming UTF-8 is typically more successful than jamming in UTF-16, because of NULL bytes and the shortsightedness of UCS-2. But I think that 99% of the things you have to do for UTF-8 (validate, encode/decode, no indexing) you have to do for UTF-16, and if we're arguing for systems that are UTF-aware, UTF-16 does great. If we're arguing for trying to jam encoded bytes into systems, well it's a crapshoot. UTF-8 being better at winning that crapshoot doesn't feel like a recipe for robustness to me, and certainly doesn't strike me as a perfect solution. OOB configuration does, though. > For example, if I grep a UTF-8 text file, grep doesn't need to know the text encoding if all you're trying to find is an ASCII string. Well, this is a weird example. grep uses locales, so yeah you're probably fine if you're grepping for ASCII using an ASCII-compatible locale. But if you're not, then what you feed to grep has to be encoded in your locale's encoding (I think). So I think this is actually an example in favor of "this should be OOB configurable", and an example of "ASCII-compatibility lulls into a false sense of security".

by hnfong 5y ago

camgunz 5y ago

Yeah I mean, I broadly agree with what you're saying. Text encoding is a mess, and pragmatic solutions are needed. And when jamming text in various places, jamming UTF-8 is typically more successful than jamming in UTF-16, because of NULL bytes and the shortsightedness of UCS-2. But I think that 99% of the things you have to do for UTF-8 (validate, encode/decode, no indexing) you have to do for UTF-16, and if we're arguing for systems that are UTF-aware, UTF-16 does great. If we're arguing for trying to jam encoded bytes into systems, well it's a crapshoot. UTF-8 being better at winning that crapshoot doesn't feel like a recipe for robustness to me, and certainly doesn't strike me as a perfect solution. OOB configuration does, though. > For example, if I grep a UTF-8 text file, grep doesn't need to know the text encoding if all you're trying to find is an ASCII string. Well, this is a weird example. grep uses locales, so yeah you're probably fine if you're grepping for ASCII using an ASCII-compatible locale. But if you're not, then what you feed to grep has to be encoded in your locale's encoding (I think). So I think this is actually an example in favor of "this should be OOB configurable", and an example of "ASCII-compatibility lulls into a false sense of security".