3 ms·
The article explains this. To summarize: the "codepoint" is a broken metric for what a grapheme "is" in basically any context. Edge cases would be truncating
by samatman 6y ago
The article explains this.
To summarize: the "codepoint" is a broken metric for what a grapheme "is" in basically any context. Edge cases would be truncating with a guarantee of a valid encoding on the substring? But really you want to truncate at extended grapheme cluster boundaries. Truncating a country flag between two of the regional indicator symbols might not throw errors in your code, but no user is going to consider that a valid string. The same is true of all manner of composed characters, and there are a lot of them.
So the only advantage of using the very-often-longer UTF-16 encoding is that it's an attractive nuisance! This makes it easier to write code which will do the wrong thing, constantly, but at a low enough rate that developers will put off fixing it.
Unicode is variable width, and what width you need is application-specific. That's the whole point of the article! UTF-8 doesn't try to hide any of this from you.