25 ms·

This guide omits an IMPORTANT issue: handling characters outside the Basic Multilingual Plane (BMP). JavaScript, like some other languages, suffers from the UCS

This guide omits an IMPORTANT issue: handling characters outside the Basic Multilingual Plane (BMP). JavaScript, like some other languages, suffers from the UCS-2 curse - that is, it assumes that all characters fit inside 16 bits, even though that is no longer true. For example, the cited text says: "You can return the UTF-16 code for a character in a string using charCodeAt()...". Not true. This only works if the UTF-16 code fits in 16 bits; if it's more than 16 bits, charCodeAt will only return a part of the character. There are lots of discussions about this, here's one: https://stackoverflow.com/questions/3744721/javascript-strings-outside-of-the-bmp https://stackoverflow.com/questions/3744721/javascript-strin... JavaScript can handle characters outside the BMP, but you sometimes have to aware of the problem & carefully code around it when such characters are possible.

crazygringo 6y ago
Exactly, and emoji are outside the BMP, so it's not exactly an edge case, but the norm where two code units (UTF-16 double-bytes) are used to make one code point (Unicode character) [1]. And it gets even worse, when you consider that for many purposes you're not even interested in code points but in graphemes which can be sequences of code points -- e.g. a single visible emoji might actually be a sequence of 5 code points, represented by 8 UTF-16 code units, taking up 16 bytes [2]. Similarly a single accented character will often be two code points (letter plus combining diacritic). If you want to split a string by graphemes -- e.g. to count its visible length, or delete its last visible character -- you can either use a library for it [3], or the relatively new Intl.Segmenter [4] which is in Chrome and Safari, but hasn't made it to Firefox [5]. Kind of amazing it's 2021 and you still can't calculate the number of visible characters (graphemes) in a string using native functions across all modern browsers. [1] https://blog.jonnew.com/posts/poo-dot-length-equals-two https://blog.jonnew.com/posts/poo-dot-length-equals-two [2] https://www.contentful.com/blog/2016/12/06/unicode-javascript-and-the-emoji-family/ https://www.contentful.com/blog/2016/12/06/unicode-javascrip... [3] https://github.com/orling/grapheme-splitter https://github.com/orling/grapheme-splitter [4] https://github.com/tc39/proposal-intl-segmenter https://github.com/tc39/proposal-intl-segmenter [5] https://bugzilla.mozilla.org/show_bug.cgi?id=1423593 https://bugzilla.mozilla.org/show_bug.cgi?id=1423593
- davethedevguy 6y ago
  Thank you for these comments. I didn't know about this at all! I'll read up on it until I understand it, and then add something to the article that covers it.
  - crazygringo 6y ago
    This might also help you: https://stackoverflow.com/questions/4547609/how-to-get-character-array-from-a-string/34717402#34717402 https://stackoverflow.com/questions/4547609/how-to-get-chara... e.g. use Array.from() to at least process code points rather than code units, though that's still not graphemes.
- dwheeler 6y ago
  You're right about the graphemes. If you need it, you need it, but I recommend writing code that does not need to count graphemes if you can avoid them. In many cases strings are best considered units that can be concatenated at will, but it's best if you avoid splitting them, and if you must split them, generally only split them on ASCII character boundaries. Don't consider "lengths" as something that has a meaning to humans (it doesn't), and don't assume that a "character" is a single JavaScript character (it isn't). If you normally just work strings as opaque sequences of "characters" that can be later displayed, you can avoid many complications (though obviously NOT all of them).

crazygringo 6y ago

Exactly, and emoji are outside the BMP, so it's not exactly an edge case, but the norm where two code units (UTF-16 double-bytes) are used to make one code point (Unicode character) [1]. And it gets even worse, when you consider that for many purposes you're not even interested in code points but in graphemes which can be sequences of code points -- e.g. a single visible emoji might actually be a sequence of 5 code points, represented by 8 UTF-16 code units, taking up 16 bytes [2]. Similarly a single accented character will often be two code points (letter plus combining diacritic). If you want to split a string by graphemes -- e.g. to count its visible length, or delete its last visible character -- you can either use a library for it [3], or the relatively new Intl.Segmenter [4] which is in Chrome and Safari, but hasn't made it to Firefox [5]. Kind of amazing it's 2021 and you still can't calculate the number of visible characters (graphemes) in a string using native functions across all modern browsers. [1] https://blog.jonnew.com/posts/poo-dot-length-equals-two https://blog.jonnew.com/posts/poo-dot-length-equals-two [2] https://www.contentful.com/blog/2016/12/06/unicode-javascript-and-the-emoji-family/ https://www.contentful.com/blog/2016/12/06/unicode-javascrip... [3] https://github.com/orling/grapheme-splitter https://github.com/orling/grapheme-splitter [4] https://github.com/tc39/proposal-intl-segmenter https://github.com/tc39/proposal-intl-segmenter [5] https://bugzilla.mozilla.org/show_bug.cgi?id=1423593 https://bugzilla.mozilla.org/show_bug.cgi?id=1423593

davethedevguy 6y ago
Thank you for these comments. I didn't know about this at all! I'll read up on it until I understand it, and then add something to the article that covers it.
- crazygringo 6y ago
  This might also help you: https://stackoverflow.com/questions/4547609/how-to-get-character-array-from-a-string/34717402#34717402 https://stackoverflow.com/questions/4547609/how-to-get-chara... e.g. use Array.from() to at least process code points rather than code units, though that's still not graphemes.
dwheeler 6y ago
You're right about the graphemes. If you need it, you need it, but I recommend writing code that does not need to count graphemes if you can avoid them. In many cases strings are best considered units that can be concatenated at will, but it's best if you avoid splitting them, and if you must split them, generally only split them on ASCII character boundaries. Don't consider "lengths" as something that has a meaning to humans (it doesn't), and don't assume that a "character" is a single JavaScript character (it isn't). If you normally just work strings as opaque sequences of "characters" that can be later displayed, you can avoid many complications (though obviously NOT all of them).