5 ms·

Broad backwards compatibility with ASCII is a strong reason to prefer UTF-8 in most applications, however I find the issues with UTF-16 are overstated and the a

by chalst 16y ago

Broad backwards compatibility with ASCII is a strong reason to prefer UTF-8 in most applications, however I find the issues with UTF-16 are overstated and the advantages of it ignored. "UTF-16 has no practical advantages over UTF-8" - UTF-16 is better - i.e., quite a bit more compact, since, e.g., Chinese characters always take 3 characters in UTF-8, for asian languages. "A related drawback is the loss of self-synchronization at the byte level." - Maybe this is a problem, maybe not. Maybe the failure of UTF-8 to be self-synchronising at the 4-bit level is a problem is some circumstances. I don't mean to be flippant, but the wider point is that with UTF-16, you really need to commit to 16-bit char width. "The encoding is inflexible" - I think the author has confused the fixed-width UCS-2 and the variable-width UTF-16. "We lose backwards compatibility with code treating NUL as a string terminator." - Not true. NUL is a 16-bit character in UTF-16. Use a C compiler that supports 16-bit char width.

yuhong 16y ago
>"We lose backwards compatibility with code treating NUL as a string terminator." - Not true. NUL is a 16-bit character in UTF-16. Use a C compiler that supports 16-bit char width. Yea, it really should be "We lose backwards compatibility with code treating a zero 8-bit byte as a string terminator." >"UTF-16 has no practical advantages over UTF-8" - UTF-16 is better - i.e., quite a bit more compact, since, e.g., Chinese characters always take 3 characters in UTF-8, for asian languages. For pure CJK text files, yes.
udoprog 16y ago
"...Unicode has been standardising CJK character sets" - CJK characters sets are complex, and will always face opposition, besides, this is less related to how the characters are encoded, and more how to choose and composite your code points. "I think the author has confused the fixed-length UCS-2 and the variable length UTF-16." - No, in the same sentence you are quoting he mentions that utf-16 is limited to 0x110000 codepoints, contra utf-8 that is specified to expand up to 6 bytes. "Not true. NUL is a 16-bit character in UTF-16. Use a C compiler that supports 16-bit char width." - I won't be converting my source code into utf-16 any time soon. Besides, C is not a good example where this is an actual problem, the "unicode strings" will still just be represented as binary blobs - different depending on which encoding you choose. It's more of a problem in e.g. Python, where the native string is an array of characters, not just a null terminated blob. I wouldn't encode my source code in utf-8, however, if my editor my default happened to support utf-8, this wouldn't be an issue unless I enter an non-ascii compatible code point. I see this as a big plus.
- chalst 16y ago
  "confused ... UCS-2 and ... UTF-16" - You are quite right, he is clear on the difference. It's presumably possible to apply the same trick to UTF-16 as was applied to UCS-2, to expand further, but this is ad-hoc and not built in. "supports 16-bit char width" - I mean the compiler has CHAR_BIT == 16; this is independent of the character encoding of the source.
othermaciej 16y ago
> "UTF-16 has no practical advantages over UTF-8" - UTF-16 is better - i.e., quite a bit more compact, since, e.g., Chinese characters always take 3 characters in UTF-8, for asian languages. In some contexts that may matter. In other cases, you can expect enough ASCII mixed in to outweigh that effect. For example, on the Web though, if you're sending HTML, the savings from using 8 bits for the ASCII tags will nearly outweigh the cost of extra bytes for content text. Gzip shrinks that further to essentially no difference.
- fnl 16y ago
  As I posted there, too, he also missed one very important plus in UTF-16: character counting. If you ensure there are no Surrogate Chars (which is a simple byte-mask comparison using every second byte (only!) as in: "0xD800 & byte == 0xD800" [using the entire range to avoid having to think about byte order]) in your array, you can be 100% sure about the character length just by dividing the byte-length in two. With UTF-8, you have to look at each and every byte one by one and count the number of characters to establish the actual char-length. Finally, given how rare the Supplementary Range characters are, the Surrogate Range check can be safely skipped in many scenarios, making it significantly faster to establish char lengths in UTF-16 than UTF-8. EDIT: oh, and before any "semantic-nerd" comes along: I am fully aware that 0xXXXX are two bytes, so, if you want, read "two-byte" for every time I mention "byte" above... (doh ;))
  - wladimir 16y ago
    Right, if you ensure that only ASCII characters are used in UTF-8, which you can check using 0x80 & byte == 0x00, counting the number of characters is easy. But to check that this condition holds, you need to iterate over the whole string anyway. I don't see how this gives any advantage to either of the encodings. Both for UTF-8 and UTF-16 you have to implement some decoding to reliably count the number of characters.
    - fnl 16y ago
      The important fact lies in the last sentence; in all of my apps, I could so far safely and soundly ignore the fact that Surrogates are two characters long to establish character lengths. So by using UTF-16, all those apps are significantly faster than one that would have been based on UTF-8. And, to add to this, by the same nature, it is far easier to take char slices of a UTF-16 encoded byte array than of a UTF-8 one.
lysium 16y ago
You don't mention issues and your advantages are only marginal for specific use cases such as Asian languages. "UTF-16 is better": but 'only' for Asian languages; most code is still written in ASCII "need to commit to 16-bit char width": the OP's point was that many many tools don't operate on 16-bit char width and with UTF-8 you can continue to use them. The other two points are no advantages for UTF-16.

by chalst 16y ago

yuhong 16y ago

>"We lose backwards compatibility with code treating NUL as a string terminator." - Not true. NUL is a 16-bit character in UTF-16. Use a C compiler that supports 16-bit char width. Yea, it really should be "We lose backwards compatibility with code treating a zero 8-bit byte as a string terminator." >"UTF-16 has no practical advantages over UTF-8" - UTF-16 is better - i.e., quite a bit more compact, since, e.g., Chinese characters always take 3 characters in UTF-8, for asian languages. For pure CJK text files, yes.

udoprog 16y ago

"...Unicode has been standardising CJK character sets" - CJK characters sets are complex, and will always face opposition, besides, this is less related to how the characters are encoded, and more how to choose and composite your code points. "I think the author has confused the fixed-length UCS-2 and the variable length UTF-16." - No, in the same sentence you are quoting he mentions that utf-16 is limited to 0x110000 codepoints, contra utf-8 that is specified to expand up to 6 bytes. "Not true. NUL is a 16-bit character in UTF-16. Use a C compiler that supports 16-bit char width." - I won't be converting my source code into utf-16 any time soon. Besides, C is not a good example where this is an actual problem, the "unicode strings" will still just be represented as binary blobs - different depending on which encoding you choose. It's more of a problem in e.g. Python, where the native string is an array of characters, not just a null terminated blob. I wouldn't encode my source code in utf-8, however, if my editor my default happened to support utf-8, this wouldn't be an issue unless I enter an non-ascii compatible code point. I see this as a big plus.

chalst 16y ago
"confused ... UCS-2 and ... UTF-16" - You are quite right, he is clear on the difference. It's presumably possible to apply the same trick to UTF-16 as was applied to UCS-2, to expand further, but this is ad-hoc and not built in. "supports 16-bit char width" - I mean the compiler has CHAR_BIT == 16; this is independent of the character encoding of the source.

othermaciej 16y ago

> "UTF-16 has no practical advantages over UTF-8" - UTF-16 is better - i.e., quite a bit more compact, since, e.g., Chinese characters always take 3 characters in UTF-8, for asian languages. In some contexts that may matter. In other cases, you can expect enough ASCII mixed in to outweigh that effect. For example, on the Web though, if you're sending HTML, the savings from using 8 bits for the ASCII tags will nearly outweigh the cost of extra bytes for content text. Gzip shrinks that further to essentially no difference.

fnl 16y ago
As I posted there, too, he also missed one very important plus in UTF-16: character counting. If you ensure there are no Surrogate Chars (which is a simple byte-mask comparison using every second byte (only!) as in: "0xD800 & byte == 0xD800" [using the entire range to avoid having to think about byte order]) in your array, you can be 100% sure about the character length just by dividing the byte-length in two. With UTF-8, you have to look at each and every byte one by one and count the number of characters to establish the actual char-length. Finally, given how rare the Supplementary Range characters are, the Surrogate Range check can be safely skipped in many scenarios, making it significantly faster to establish char lengths in UTF-16 than UTF-8. EDIT: oh, and before any "semantic-nerd" comes along: I am fully aware that 0xXXXX are two bytes, so, if you want, read "two-byte" for every time I mention "byte" above... (doh ;))
- wladimir 16y ago
  Right, if you ensure that only ASCII characters are used in UTF-8, which you can check using 0x80 & byte == 0x00, counting the number of characters is easy. But to check that this condition holds, you need to iterate over the whole string anyway. I don't see how this gives any advantage to either of the encodings. Both for UTF-8 and UTF-16 you have to implement some decoding to reliably count the number of characters.
  - fnl 16y ago
    The important fact lies in the last sentence; in all of my apps, I could so far safely and soundly ignore the fact that Surrogates are two characters long to establish character lengths. So by using UTF-16, all those apps are significantly faster than one that would have been based on UTF-8. And, to add to this, by the same nature, it is far easier to take char slices of a UTF-16 encoded byte array than of a UTF-8 one.

lysium 16y ago

You don't mention issues and your advantages are only marginal for specific use cases such as Asian languages. "UTF-16 is better": but 'only' for Asian languages; most code is still written in ASCII "need to commit to 16-bit char width": the OP's point was that many many tools don't operate on 16-bit char width and with UTF-8 you can continue to use them. The other two points are no advantages for UTF-16.