3 ms·
> If new code isn't working naturally in UTF-8 in 2021 then it's wrong, period. UTF-8 is a nuisance as an in-memory representation because the characters are v
by grishka 6y ago
> If new code isn't working naturally in UTF-8 in 2021 then it's wrong, period.
UTF-8 is a nuisance as an in-memory representation because the characters are variable size. You can't get the length of a string without parsing it start to end, and you can't get a character by index without parsing and counting all the previous ones. 16-bit characters (wchar_t, Java char, whatever NSStrings are made of, etc) work fine in 99% of the cases.
UTF-8 is indisputably a good encoding for when you're sending something over the network or putting it into a file or a database.
- ChrisSD 6y agoUTF-16 is a variable length encoding. As is UTF-32 for that matter. A single character can be made of multiple code points.
- TwoBit 6y agoTrue, but text processing is still easier with UTF-32, because UTF-8 and UTF-16 strings need to be converted to UTF-32 before you can do anything with them.
- dwheeler 6y ago> 16-bit characters... work fine in 99% of the cases. In other words, they don't work :-). UTF-16 is also variable-length. Sometimes a character fits in 16 bits, and sometimes it doesn't. From a practical view it's worse than UTF-8, because tests are less likely to detect bugs before shipping. Even UTF-32 is, in reality, variable-length. Many code points are combining characters, so you need multiple code points to get a single grapheme. If your language or API requires you to do something, then you'll need to do that. But unless there's an API requirement, in most situations UTF-8 is the best choice for network, storage, and processing. There are exceptions, but they're just that... exceptions.