3 ms·
Using UTF-16 is silly because it doesn't mean that length in code units == length in codepoints -- that was only ever true of UCS-2. Not that length in codepoi
by cryptonector 9y ago
Using UTF-16 is silly because it doesn't mean that length in code units == length in codepoints -- that was only ever true of UCS-2. Not that length in codepoints is all that useful anyways (because of decomposition: you can't just split on the Nth codepoint, only on the Nth character). But because that was true of UCS-2, some systems (e.g., Windows) selected UCS-2 and then UTF-16. Sad, but true.
UTF-8 is clearly superior.
- ridiculous_fish 9y agoIt's not obvious that UTF-8 is superior. UTF-8 has subtleties such as non-shortest forms and invalid code units. These issues have lead to security exploits [1] but are impossible in UTF-16 by design. The internal encoding of strings attracts too much attention. A proper string type abstracts over these details. When doing arithmetic in Haskell, Python, JavaScript, etc. you don't worry about how the numbers are encoded; strings should be the same. [1] https://unicode.org/reports/tr36/#UTF-8_Exploit https://unicode.org/reports/tr36/#UTF-8_Exploit
- cryptonector 9y agoA brief search shows for "UTF-16 vulnerability" finds relevant results. UTF-8 overlong sequences haven't been a problem for a while. The choice of encoding is definitely relevant for all external file formats, protocols, APIs. We can't just let the choice of encoding be purely internal, though I agree that programmers should not need to care about what their runtimes use internally for strings, as long as it is Unicode (UTF-8, 16, 32, or some other UTF say). But no UTF actually gives you random access by character index because a) no runtime is going to be ensuring normalization of all strings to precomposed sequences (not even NFC because NFC is closed to new compositions!), b) not every valid character will have pre-composed codepoints therefore (a) isn't even possible. Given this, UTF-16 and UTF-32 have no real advantages over UTF-8, and the false sense of simplicity they convey is dangerous anyways.