3 ms·
It's not obvious that UTF-8 is superior. UTF-8 has subtleties such as non-shortest forms and invalid code units. These issues have lead to security exploits [1]
by ridiculous_fish 9y ago
It's not obvious that UTF-8 is superior. UTF-8 has subtleties such as non-shortest forms and invalid code units. These issues have lead to security exploits [1] but are impossible in UTF-16 by design.
The internal encoding of strings attracts too much attention. A proper string type abstracts over these details. When doing arithmetic in Haskell, Python, JavaScript, etc. you don't worry about how the numbers are encoded; strings should be the same.
[1] https://unicode.org/reports/tr36/#UTF-8_Exploit https://unicode.org/reports/tr36/#UTF-8_Exploit
- cryptonector 9y agoA brief search shows for "UTF-16 vulnerability" finds relevant results. UTF-8 overlong sequences haven't been a problem for a while. The choice of encoding is definitely relevant for all external file formats, protocols, APIs. We can't just let the choice of encoding be purely internal, though I agree that programmers should not need to care about what their runtimes use internally for strings, as long as it is Unicode (UTF-8, 16, 32, or some other UTF say). But no UTF actually gives you random access by character index because a) no runtime is going to be ensuring normalization of all strings to precomposed sequences (not even NFC because NFC is closed to new compositions!), b) not every valid character will have pre-composed codepoints therefore (a) isn't even possible. Given this, UTF-16 and UTF-32 have no real advantages over UTF-8, and the false sense of simplicity they convey is dangerous anyways.