3 ms·

Never made a claim that UTF-16 is UCS-2. "Most languages" does not mean "all languages". Still, many popular platforms treat it like that in terms of char inde

Never made a claim that UTF-16 is UCS-2. "Most languages" does not mean "all languages". Still, many popular platforms treat it like that in terms of char indexing. They are able to get away with it since a surrogate char is a rare guest in UTF-16. P.S. It's surprising how easily HN crowd downvotes something when it falls out of a whimsical "popular contemporary view on things". Having a lot of years of text processing behind my shoulders, I raised an important question and got an immediate down-vote attack. But never mind, I'll survive.

ygra 8y ago
All those languages make it quite clear that you only get access to a code unit that way and other APIs usually exist to actually get access to code points, which then usually take a string and an index. That the most visible mechanism of accessing parts of a string is based on code units along with UTF-16 masking bad code unless Emoji are involved (for most developers at least), is a problem, though. UTF-8 makes the failure case pop up much faster as pretty much every language doesn't only use ASCII.
WorldMaker 8y ago
You definitely can't claim "most languages" today with the commonality of emoji and emoji-based sub-languages/communication forms/media. Surrogate characters are no longer rare in UTF-16 in average usage on the internet and that's impacting everything. Even if you are restricting "most languages" to mean "most programming languages", there's an increasing rise of emoji in comments alone for languages that support UTF-8 and some languages are picking up increasing emoji usage in places like identifier names. Platforms that treat UTF-16 like UCS-2 and allow raw char indexing rather than codepoint-oriented traversal are wrong in 2019. (It's been wrong since the very definition of UTF-16, such as RFC 2781 in 2000.) UTF-16 is a disappointing hack because it allowed UCS-2 platforms the luxury of pretending that the UCS-2 era never ended. UTF-8 at least leaves you very aware that the UCS-2 emperor has no clothes.

ygra 8y ago

All those languages make it quite clear that you only get access to a code unit that way and other APIs usually exist to actually get access to code points, which then usually take a string and an index. That the most visible mechanism of accessing parts of a string is based on code units along with UTF-16 masking bad code unless Emoji are involved (for most developers at least), is a problem, though. UTF-8 makes the failure case pop up much faster as pretty much every language doesn't only use ASCII.

WorldMaker 8y ago

You definitely can't claim "most languages" today with the commonality of emoji and emoji-based sub-languages/communication forms/media. Surrogate characters are no longer rare in UTF-16 in average usage on the internet and that's impacting everything. Even if you are restricting "most languages" to mean "most programming languages", there's an increasing rise of emoji in comments alone for languages that support UTF-8 and some languages are picking up increasing emoji usage in places like identifier names. Platforms that treat UTF-16 like UCS-2 and allow raw char indexing rather than codepoint-oriented traversal are wrong in 2019. (It's been wrong since the very definition of UTF-16, such as RFC 2781 in 2000.) UTF-16 is a disappointing hack because it allowed UCS-2 platforms the luxury of pretending that the UCS-2 era never ended. UTF-8 at least leaves you very aware that the UCS-2 emperor has no clothes.