3 ms·

tl:dr; Use UTF-8 when you need to use unicode with legacy APIs, never anywhere else. UNIX isn't UTF-8 because UTF-8 is better, UNIX is UTF-8 because you can pa

by fleitz 15y ago

tl:dr; Use UTF-8 when you need to use unicode with legacy APIs, never anywhere else. UNIX isn't UTF-8 because UTF-8 is better, UNIX is UTF-8 because you can pass UTF-8 strings to functions that expect ASCII and it kinda works. This is really the only thing you need to know about UTF-8 and why it's better. There are few pieces of software that don't have to talk to legacy APIs that store strings natively in UTF-8. C# and Java are probably the best examples of software that was engineered from the ground up and thus uses UTF-16 internally because it's much less likely to run into issues like String.length returning 32 yet only containing 31 characters. If you use UTF-8 expect this result anytime a string contains a real genuine apostrophe. "UTF-8 and UTF-32 result the same order when sorted lexicographically. UTF-16 does not." This is complete and utter bullshit, to sort a string lexicographically you need to decode it, if you've decoded the string into UNICODE then they sort the exact same way. There are lots of gotchas for sorting UNICODE strings including normalization because you can write the semantically equivalent strings in unicode multiple ways. eg. ligatures. If you're sorting bit strings that happen to contain UTF-8/32 then you're not sorting lexicographically and your results will be screwed up anyway.

gwillen 15y ago
> decoded the string into UNICODE I think you are quite confused. 1) Unicode is not an acroynm. 2) You cannot "decode into Unicode". I think you mean "decode into codepoints". 3) If that is what you mean, then you are wrong about sorting: Sorting UTF-8 and UTF-32 bytestrings will indeed sort them lexicographically by code point, which was the author's point. No, that will not generally be the sort you _want_; but no amount of 'decoding' will give you the sort you want. For that you need to first normalize, and then follow the collation rules, which don't sort by raw code points at all.
- deleted 15y ago
  [deleted]
Tloewald 15y ago
Your tl;dr is misleading, doesn't represent the thrust of the article, cherry picks nits, and makes assertions that are contradicted with evidence in the article (e.g. UTF-16 is not fixed length).
dfox 15y ago
UTF-8 is explicitly designed in such way that unix API dont have to care about it and such that lexicographic ordering of utf-8 encoded byte streams is same as lexicographic ordering of unicode code point vectors (which arguably almost never is what you care about when sorting text strings). Both of these features are result of conscious design and not some random coincidence. As for sorting text strings of any kind (which has mostly nothing to do with unicode), you have to care about what user expects, which is dependent on locale and sometimes even on user's preference. Algorithms to do text sorting are for each separate locale mostly non-trivial, but fortunately can be generalized into one relatively straight-forward algorithm (similar to unicode normalization) involving pretty large database and some special cases. When you are sorting bit strings you are sorting lexicographically, but that is usually not what you should be doing.
- fleitz 15y ago
  "When you are sorting bit strings you are sorting lexicographically, but that is usually not what you should be doing." That's my point about it being bullshit, it sounds like a feature you might want, but honestly it's useful to very few people.
  - ybungalobill 15y ago
    Man, the article doesn't try to argue for or against UTF-8 based on lexicographical order. It's in the "facts" section, and it's a correct fact. So it's just as neutral as "Widechar is 2 bytes [...] 4 on others." or "In both [...] characters may take up to 4 bytes".
jsprinkles 15y ago
> UNIX isn't UTF-8 These are the only three words that are correct in your comment. You're spreading a lot of FUD about UTF-8, and in many cases, you are completely incorrect about many things technical. I hate to make this personal, but, you really need to investigate what you're talking about before jumping on HN and talking shit about UTF-8. UTF-8 has some criticisms, but none of what you have written so far is even remotely valid. I hope people realize this instead of get scared away which is, interestingly, what the point of the manifesto linked above is all about.
mjg59 15y ago
"it's much less likely to run into issues like String.length returning 32 yet only containing 31 characters" This is exactly the problem with UTF-16. Most APIs that use it will have support for string operations that return the number of codepoints rather than the number of bytes, and as a result people think that it's a solved problem. But in fact you've only solved half the problem, because the number of codepoints is almost certainly not what you want - you want the number of characters, and the only way your library functions can know that is to know which Unicode codepoints are combining characters. And that set potentially gets larger with every new Unicode release. In other words, if you're relying on languages that have native UTF-16 support to tell you the number of printable characters, your application is inevitably going to be broken the first time someone uses a newly-defined combining character. UTF-16 buys you absolutely nothing useful in this respect. (Example: How many characters is "é"? "é"? "é"? Does UTF-16 give you a more useful answer to that question?)
- pjscott 15y ago
  > (Example: How many characters is "é"? "é"? "é"? Does UTF-16 give you a more useful answer to that question?) In case anybody was wondering: 1. The first "é" is an "e" followed by a combining acute accent. 2. The second "é" is a single code point for a lowercase-e-with-acute. 3. The third "é" is the same as the second "é", but with a zero-width non-breaking space in front of it. All of these are, of course, the same letter.
vorg 15y ago
> "UTF-8 and UTF-32 result the same order when sorted lexicographically. UTF-16 does not." This is complete and utter bullshit, to sort a string lexicographically you need to decode it, if you've decoded the string into UNICODE then they sort the exact same way. One original purpose of Unicode, still mentioned in the published for v6.0, is to assist the 100-odd other encodings, not replace them. Each other encoding only needs a conversion process to and from Unicode, i.e. 200 conversion processes overall, instead of into every other encoding, i.e. 9900 processes overall. For sorting, Unicode is order-invariant. The text should be converted into a relevant country-specific encoding, sorted within that encoding, then converted back to Unicode.
blasdel 15y ago
Java does NOT implement UTF-16, nor do most systems that claim to. It's UCS-2, which is its own even worse pile of bullshit.

by fleitz 15y ago

gwillen 15y ago

> decoded the string into UNICODE I think you are quite confused. 1) Unicode is not an acroynm. 2) You cannot "decode into Unicode". I think you mean "decode into codepoints". 3) If that is what you mean, then you are wrong about sorting: Sorting UTF-8 and UTF-32 bytestrings will indeed sort them lexicographically by code point, which was the author's point. No, that will not generally be the sort you _want_; but no amount of 'decoding' will give you the sort you want. For that you need to first normalize, and then follow the collation rules, which don't sort by raw code points at all.

deleted 15y ago
[deleted]

Tloewald 15y ago

Your tl;dr is misleading, doesn't represent the thrust of the article, cherry picks nits, and makes assertions that are contradicted with evidence in the article (e.g. UTF-16 is not fixed length).

dfox 15y ago

UTF-8 is explicitly designed in such way that unix API dont have to care about it and such that lexicographic ordering of utf-8 encoded byte streams is same as lexicographic ordering of unicode code point vectors (which arguably almost never is what you care about when sorting text strings). Both of these features are result of conscious design and not some random coincidence. As for sorting text strings of any kind (which has mostly nothing to do with unicode), you have to care about what user expects, which is dependent on locale and sometimes even on user's preference. Algorithms to do text sorting are for each separate locale mostly non-trivial, but fortunately can be generalized into one relatively straight-forward algorithm (similar to unicode normalization) involving pretty large database and some special cases. When you are sorting bit strings you are sorting lexicographically, but that is usually not what you should be doing.

fleitz 15y ago
"When you are sorting bit strings you are sorting lexicographically, but that is usually not what you should be doing." That's my point about it being bullshit, it sounds like a feature you might want, but honestly it's useful to very few people.
- ybungalobill 15y ago
  Man, the article doesn't try to argue for or against UTF-8 based on lexicographical order. It's in the "facts" section, and it's a correct fact. So it's just as neutral as "Widechar is 2 bytes [...] 4 on others." or "In both [...] characters may take up to 4 bytes".

jsprinkles 15y ago

> UNIX isn't UTF-8 These are the only three words that are correct in your comment. You're spreading a lot of FUD about UTF-8, and in many cases, you are completely incorrect about many things technical. I hate to make this personal, but, you really need to investigate what you're talking about before jumping on HN and talking shit about UTF-8. UTF-8 has some criticisms, but none of what you have written so far is even remotely valid. I hope people realize this instead of get scared away which is, interestingly, what the point of the manifesto linked above is all about.

mjg59 15y ago

"it's much less likely to run into issues like String.length returning 32 yet only containing 31 characters" This is exactly the problem with UTF-16. Most APIs that use it will have support for string operations that return the number of codepoints rather than the number of bytes, and as a result people think that it's a solved problem. But in fact you've only solved half the problem, because the number of codepoints is almost certainly not what you want - you want the number of characters, and the only way your library functions can know that is to know which Unicode codepoints are combining characters. And that set potentially gets larger with every new Unicode release. In other words, if you're relying on languages that have native UTF-16 support to tell you the number of printable characters, your application is inevitably going to be broken the first time someone uses a newly-defined combining character. UTF-16 buys you absolutely nothing useful in this respect. (Example: How many characters is "é"? "é"? "é"? Does UTF-16 give you a more useful answer to that question?)

pjscott 15y ago
> (Example: How many characters is "é"? "é"? "é"? Does UTF-16 give you a more useful answer to that question?) In case anybody was wondering: 1. The first "é" is an "e" followed by a combining acute accent. 2. The second "é" is a single code point for a lowercase-e-with-acute. 3. The third "é" is the same as the second "é", but with a zero-width non-breaking space in front of it. All of these are, of course, the same letter.

vorg 15y ago

> "UTF-8 and UTF-32 result the same order when sorted lexicographically. UTF-16 does not." This is complete and utter bullshit, to sort a string lexicographically you need to decode it, if you've decoded the string into UNICODE then they sort the exact same way. One original purpose of Unicode, still mentioned in the published for v6.0, is to assist the 100-odd other encodings, not replace them. Each other encoding only needs a conversion process to and from Unicode, i.e. 200 conversion processes overall, instead of into every other encoding, i.e. 9900 processes overall. For sorting, Unicode is order-invariant. The text should be converted into a relevant country-specific encoding, sorted within that encoding, then converted back to Unicode.

blasdel 15y ago

Java does NOT implement UTF-16, nor do most systems that claim to. It's UCS-2, which is its own even worse pile of bullshit.