4 ms·
> 2. Length in terms of number of visible characters is the number of grapheme clusters. There's a fun subtlety in this case too: a single grapheme cluster nee
by _rend 5y ago
> 2. Length in terms of number of visible characters is the number of grapheme clusters.
There's a fun subtlety in this case too: a single grapheme cluster need not draw as a single "visible character" (or glyph) on-screen. The visual representation of a grapheme cluster is entirely dependent on the text system which draws this, and the meaning that system applies to the cluster itself. This is especially true for multi-element emoji clusters, whose recommended meanings[1] change with evolving versions of Unicode.
To add to this, Unicode 12 simplified the definition of grapheme clusters by actually generalizing them so that they can be matched effectively by a regex. (See the "extended grapheme cluster" definition in TR29[2].) This reduced the overall number of special cases and hard-coded lists of combinations in the definitions of grapheme clusters (particularly around emoji), but it also means that there are now infinitely more valid grapheme clusters that don't necessarily map to a single glyph.
One really simple example of this is, e.g.
©
(Edit: it appears that HN is actually stripping out the ZWJ text from this example and leaving just the Copyright symbol. See below for how to reproduce this text on your machine.)
That is,
U+1F434 <horse head> + U+200D <zero width joiner> + U+00A9 <copyright sign>
This cluster trivially matches the definition of
extended grapheme cluster := crlf | Control | precore* _core_ postcore*
core := hangul-syllable | ri-sequence | xpicto-sequence | [^Control CR LF]
xpicto-sequence := \p{Extended_Pictographic} (Extend* ZWJ \p{Extended_Pictographic})*
where
U+1F434: Extended_Pictographic
U+200D: ZWJ
U+00A9: Extended_Pictographic
(I picked this combination somewhat randomly, but ideally, this is an example that should hopefully last as it feels unlikely that "horse copyright" would have a meaningful glyph definition in the future. As of posting this, the above text shows up as two side-by-side glyphs on my machine (macOS Monterey 21A559): a horse, followed by the copyright sign. This may look similar on your machine, or it may not.)
Importantly, you can tell this is actually treated as a real grapheme cluster by the text system on macOS because if you copy that string into a Cocoa text view (e.g., TextEdit), you will only be able to place your cursor on either side of the cluster, but not split it in the middle. A nice interactive way to see this in action is inserting U+1F434 into the document, followed by U+00A9. Then, move your cursor in between those two glyphs and insert U+200D: your cursor should then bounce out from the middle of the newly-formed cluster to the beginning.
This was a pretty short example, but this is arbitrarily extensible: (Edit: Originally I had posted U+2705 <check mark symbol> + U+200D + U+1F434 <horse head> + U+200D + U+1F50B <battery> + U+200D + U+1F9F7 <safety pin> [sorry, no staple emoji] but HN stripped that out too. It does appear correctly in the text area while typing, but HN replaces the sequence with spaces after posting.)
As linked above, Unicode does offer a list of sequences like this that are considered to be "meaningful"[1], which you can largely expect vendors which offer emoji representations to respect (and some vendors may offer glyphs for sequences beyond what is suggested here). If you've ever run into this: additions to this list over time explains why transporting a Unicode sequence which appears as a single glyph on one OS can appear as multiple glyphs on an older one (each individual glyph may be supported, but their combination may or may not have a meaning).
In general, if this is interesting to you, you may enjoy trawling through the Unicode emoji data files [3]. You may discover something new!
[1] https://www.unicode.org/Public/emoji/14.0/emoji-zwj-sequences.txt https://www.unicode.org/Public/emoji/14.0/emoji-zwj-sequence...
[2] https://www.unicode.org/reports/tr29/tr29-35.html#Table_Combining_Char_Sequences_and_Grapheme_Clusters https://www.unicode.org/reports/tr29/tr29-35.html#Table_Comb...
[3] https://www.unicode.org/reports/tr51/#emoji_data https://www.unicode.org/reports/tr51/#emoji_data