3 ms·

UTF-16 allows to quickly index a char in a string covering the most languages that way. UTF-8 is a different story, as nearly every byte of UTF-8 string is a s

UTF-16 allows to quickly index a char in a string covering the most languages that way. UTF-8 is a different story, as nearly every byte of UTF-8 string is a surrogate for most languages except English/ASCII. Now, how can a char be indexed then? Only by slow iteration from the beginning of a string? UTF-8 is a good choice for storage, true. However, it seems to be a not so good choice for string processing from the algorithmic point of view. Or maybe there is a catch that makes UTF-8 strings well-suited for string processing as well?

WorldMaker 8y ago
> UTF-16 allows to quickly index a char in a string covering the most languages that way. Nope. UTF-16 is not UCS-2 anymore for multiple reasons. You cannot assume you can index a UTF-16 string as if it was UCS-2 anymore. UCS-4 (32-bit) is a possible option, though not generally recommended either. There are multiple tricks for getting great performance in UTF-8 string processing such as codepoint maps of various sorts. It's quite well suited for string processing, and it's better to use UTF-8 at this point simply because it also helps from falling into the UTF-16 is not (and has not been for some time) UCS-2 trap, and UCS-2 cannot represent a lot of Unicode today (including many emoji).
- garganzol 8y ago
  Never made a claim that UTF-16 is UCS-2. "Most languages" does not mean "all languages". Still, many popular platforms treat it like that in terms of char indexing. They are able to get away with it since a surrogate char is a rare guest in UTF-16. P.S. It's surprising how easily HN crowd downvotes something when it falls out of a whimsical "popular contemporary view on things". Having a lot of years of text processing behind my shoulders, I raised an important question and got an immediate down-vote attack. But never mind, I'll survive.
  - ygra 8y ago
    All those languages make it quite clear that you only get access to a code unit that way and other APIs usually exist to actually get access to code points, which then usually take a string and an index. That the most visible mechanism of accessing parts of a string is based on code units along with UTF-16 masking bad code unless Emoji are involved (for most developers at least), is a problem, though. UTF-8 makes the failure case pop up much faster as pretty much every language doesn't only use ASCII.
  - WorldMaker 8y ago
    You definitely can't claim "most languages" today with the commonality of emoji and emoji-based sub-languages/communication forms/media. Surrogate characters are no longer rare in UTF-16 in average usage on the internet and that's impacting everything. Even if you are restricting "most languages" to mean "most programming languages", there's an increasing rise of emoji in comments alone for languages that support UTF-8 and some languages are picking up increasing emoji usage in places like identifier names. Platforms that treat UTF-16 like UCS-2 and allow raw char indexing rather than codepoint-oriented traversal are wrong in 2019. (It's been wrong since the very definition of UTF-16, such as RFC 2781 in 2000.) UTF-16 is a disappointing hack because it allowed UCS-2 platforms the luxury of pretending that the UCS-2 era never ended. UTF-8 at least leaves you very aware that the UCS-2 emperor has no clothes.

WorldMaker 8y ago

> UTF-16 allows to quickly index a char in a string covering the most languages that way. Nope. UTF-16 is not UCS-2 anymore for multiple reasons. You cannot assume you can index a UTF-16 string as if it was UCS-2 anymore. UCS-4 (32-bit) is a possible option, though not generally recommended either. There are multiple tricks for getting great performance in UTF-8 string processing such as codepoint maps of various sorts. It's quite well suited for string processing, and it's better to use UTF-8 at this point simply because it also helps from falling into the UTF-16 is not (and has not been for some time) UCS-2 trap, and UCS-2 cannot represent a lot of Unicode today (including many emoji).

garganzol 8y ago
Never made a claim that UTF-16 is UCS-2. "Most languages" does not mean "all languages". Still, many popular platforms treat it like that in terms of char indexing. They are able to get away with it since a surrogate char is a rare guest in UTF-16. P.S. It's surprising how easily HN crowd downvotes something when it falls out of a whimsical "popular contemporary view on things". Having a lot of years of text processing behind my shoulders, I raised an important question and got an immediate down-vote attack. But never mind, I'll survive.
- ygra 8y ago
  All those languages make it quite clear that you only get access to a code unit that way and other APIs usually exist to actually get access to code points, which then usually take a string and an index. That the most visible mechanism of accessing parts of a string is based on code units along with UTF-16 masking bad code unless Emoji are involved (for most developers at least), is a problem, though. UTF-8 makes the failure case pop up much faster as pretty much every language doesn't only use ASCII.
- WorldMaker 8y ago
  You definitely can't claim "most languages" today with the commonality of emoji and emoji-based sub-languages/communication forms/media. Surrogate characters are no longer rare in UTF-16 in average usage on the internet and that's impacting everything. Even if you are restricting "most languages" to mean "most programming languages", there's an increasing rise of emoji in comments alone for languages that support UTF-8 and some languages are picking up increasing emoji usage in places like identifier names. Platforms that treat UTF-16 like UCS-2 and allow raw char indexing rather than codepoint-oriented traversal are wrong in 2019. (It's been wrong since the very definition of UTF-16, such as RFC 2781 in 2000.) UTF-16 is a disappointing hack because it allowed UCS-2 platforms the luxury of pretending that the UCS-2 era never ended. UTF-8 at least leaves you very aware that the UCS-2 emperor has no clothes.