25 ms·

The issue is that the concept of “length” without additional context no longer applies to Unicode strings. There’s not one length there’s four or five - at leas

The issue is that the concept of “length” without additional context no longer applies to Unicode strings. There’s not one length there’s four or five - at least. 1. Length in the context of storage or transmission is the number of bytes of its UTF-8 representation - and not just that, one of its decomposed forms too. 2. Length in terms of number of visible characters is the number of grapheme clusters. 3. Length in terms of visible space on screen is the number of pixels wide and tall when rendered in a given font. 4. Length when parsing is based on the number of code points. If you look at it that way it makes perfect sense that the “length” of a string could change when any operation is performed on it - because there’s no such thing as a canonical length. tl;dr bug or not, the concept of a canonical string length is an anachronism from the pre-Unicode days when all those were the same thing. There’s no such thing as a string length anymore.

darrenf 5y ago
Raku provides explicit methods for #1, #2 and #4 on strings: $string.encode('UTF-8').bytes; #1 $string.chars; #2 $string.codes; #4 https://course.raku.org/essentials/strings/string-length/ https://course.raku.org/essentials/strings/string-length/
- goldenkey 5y ago
  For those that don't know, Raku is just Perl 6 renamed.
  - lizmat 5y ago
    > is just Perl 6 renamed Though this was technically correct in 2019, meanwhile the Raku Programming Language has continued to evolve, de-emphasizing its heritage more and more. For example, Covid restrictions allowing, 2022 will see the first European Raku Conference.
    - goldenkey 5y ago
      Is it a fork..or just a rename for a breaking version tree? I understand that backwards compatibility is not a required goal. So isn't it more like Python 3 rather than an entirely different programming language?
      - elcaro 5y ago
        > isn't it more like Python 3 Not in the slightest. Python 3 made breaking changes to the CPython code base (yes that's a simplification). There's enough backwards incompatibility to consider them different languages, but I'd guess that there might still be bits and pieces of code that is identical (or largely unchanged) between the Python 2 and 3 implementations. Raku - on the other hand - does not use any of the Perl code base; it's a completely different runtime written from the ground up.
      - raiph 5y ago
        It's not a fork any more than C# is a fork of C. It's not a rename for a breaking version tree any more than a C# compiler is of GCC. It is backwards compatible. The first strong demonstration of this was using the Catalyst framework, a major Perl package, in Raku programs, 6 years ago. It's not remotely like Python 3 vs Python 2. Python 3 is a tweak of Python 2 that utterly failed to confront hard problems that desperately needed -- and still need -- to be confronted. For example, Raku sits alongside Swift and Elixir as the only two mainstream PLs yet to properly address character processing in the Unicode era. In contrast, Python 3's doc doesn't even mention graphemes. Rakudo has no GIL; Python is stuck with CPython's. And so on for a long list of truly huge PL problems. Raku is an entirely different programming language. (This is so despite Raku(do) being able to support backwards compatibility via the same mechanism that allows it to use Python modules as if they were Raku modules. And C libraries, and Ruby ones, and so on.)
        goldenkey 5y ago
        I see, thank you for your comprehensive explanation. I am intrigued and excited to try it out. I used to use Perl and CPAN all the time :-)
pvg 5y ago
This particular capitalization would have generated a longer string without Unicode - it's a language convention that a capitalization routine could apply to just about any encoding that has an ß.
zionic 5y ago
Except almost everyone always means #2. No one asked for strings to be ruined in this way, and this kind of pedantry has caused untold frustration from developers who just want their strings to work properly. If you must expose the underlying byte array do so via a sane access function that returns a typed array. As for “string length in pixels”, that has absolutely nothing to do with the string itself as that’s determined in the UI layer that ingests the string.
- garmaine 5y ago
  > Except almost everyone always means #2. I don't think this is true. Certainly there is no string-length library I'm aware of that handles it that way. The usual default these days (correct or not) is #4 -- length is the number of unicode code points.
  - wolf550e 5y ago
    Some languages that had the misfortune to be developed while people thought UCS-2 would be enough define string length as number of UTF-16 code units, which is not even on the list because it's not a useful number.
    - garmaine 5y ago
      Yeah, and most languages/libraries which predate Unicode define string length as the number of bytes (#1). That's probably the most common interpretation, actually. But most new implementations count code point, in my experience. I believe this is also the Unicode recommendation when context doesn't determine a different algorithm to read. Been a while since I read those recommendations though, so I could be wrong.
      - zinekeller 5y ago
        > I believe this is also the Unicode recommendation when context doesn't determine a different algorithm to read. Except that emojis are universally two "characters", even those that are encoded as several codepoints. Also, non-composite Korean jamo versus composited jamo.
vadfa 5y ago
>3. Length in terms of visible space on screen is the number of pixels wide and tall when rendered in a given font. Grasping at straws eh?
- arcticbull 5y ago
  Nah this came out of personal experience working at a company where a designer wanted to limit a block of text to a certain “length” but in reality they wanted as many characters as would fit in a box without scrolling, scaling or wrapping. This has to be calculated by bounding box.
quietbritishjim 5y ago
That is an issue, for sure. But that is not "the issue" that they're describing. They are clearly talking about (2), so confusion about other meanings is irrelevant. Their point is that they would expect number of visble grapheme clusters to be the same when converting between cases. But in some languages that's not true, as their example demonstrates. (An upper case ß does exist in Unicode but culturally it's less correct than SS.)
_rend 5y ago
> 2. Length in terms of number of visible characters is the number of grapheme clusters. There's a fun subtlety in this case too: a single grapheme cluster need not draw as a single "visible character" (or glyph) on-screen. The visual representation of a grapheme cluster is entirely dependent on the text system which draws this, and the meaning that system applies to the cluster itself. This is especially true for multi-element emoji clusters, whose recommended meanings[1] change with evolving versions of Unicode. To add to this, Unicode 12 simplified the definition of grapheme clusters by actually generalizing them so that they can be matched effectively by a regex. (See the "extended grapheme cluster" definition in TR29[2].) This reduced the overall number of special cases and hard-coded lists of combinations in the definitions of grapheme clusters (particularly around emoji), but it also means that there are now infinitely more valid grapheme clusters that don't necessarily map to a single glyph. One really simple example of this is, e.g. © (Edit: it appears that HN is actually stripping out the ZWJ text from this example and leaving just the Copyright symbol. See below for how to reproduce this text on your machine.) That is, U+1F434 <horse head> + U+200D <zero width joiner> + U+00A9 <copyright sign> This cluster trivially matches the definition of extended grapheme cluster := crlf | Control | precore* _core_ postcore* core := hangul-syllable | ri-sequence | xpicto-sequence | [^Control CR LF] xpicto-sequence := \p{Extended_Pictographic} (Extend* ZWJ \p{Extended_Pictographic})* where U+1F434: Extended_Pictographic U+200D: ZWJ U+00A9: Extended_Pictographic (I picked this combination somewhat randomly, but ideally, this is an example that should hopefully last as it feels unlikely that "horse copyright" would have a meaningful glyph definition in the future. As of posting this, the above text shows up as two side-by-side glyphs on my machine (macOS Monterey 21A559): a horse, followed by the copyright sign. This may look similar on your machine, or it may not.) Importantly, you can tell this is actually treated as a real grapheme cluster by the text system on macOS because if you copy that string into a Cocoa text view (e.g., TextEdit), you will only be able to place your cursor on either side of the cluster, but not split it in the middle. A nice interactive way to see this in action is inserting U+1F434 into the document, followed by U+00A9. Then, move your cursor in between those two glyphs and insert U+200D: your cursor should then bounce out from the middle of the newly-formed cluster to the beginning. This was a pretty short example, but this is arbitrarily extensible: (Edit: Originally I had posted U+2705 <check mark symbol> + U+200D + U+1F434 <horse head> + U+200D + U+1F50B <battery> + U+200D + U+1F9F7 <safety pin> [sorry, no staple emoji] but HN stripped that out too. It does appear correctly in the text area while typing, but HN replaces the sequence with spaces after posting.) As linked above, Unicode does offer a list of sequences like this that are considered to be "meaningful"[1], which you can largely expect vendors which offer emoji representations to respect (and some vendors may offer glyphs for sequences beyond what is suggested here). If you've ever run into this: additions to this list over time explains why transporting a Unicode sequence which appears as a single glyph on one OS can appear as multiple glyphs on an older one (each individual glyph may be supported, but their combination may or may not have a meaning). In general, if this is interesting to you, you may enjoy trawling through the Unicode emoji data files [3]. You may discover something new! [1] https://www.unicode.org/Public/emoji/14.0/emoji-zwj-sequences.txt https://www.unicode.org/Public/emoji/14.0/emoji-zwj-sequence... [2] https://www.unicode.org/reports/tr29/tr29-35.html#Table_Combining_Char_Sequences_and_Grapheme_Clusters https://www.unicode.org/reports/tr29/tr29-35.html#Table_Comb... [3] https://www.unicode.org/reports/tr51/#emoji_data https://www.unicode.org/reports/tr51/#emoji_data
ASalazarMX 5y ago
Hey, let's keep inventing lengths. 5. Length in terms of time until complete display when rendered in a given architecture. 6. Length in terms of Braille characters needed to display it. 7. Length in terms of time of lecture at 300 wpm. String length is overwhelmingly discussed as character length, meaning #2. Length in bytes should only be an issue for data transmission or storage, and people who still work with ASCII. Rendering dimensions are not relevant to the published article.
- rtkwe 5y ago
  #2 is variable in arabic where some letters are combined which caused the old "effective power" crash in iOS.
- deleted 5y ago
  [deleted]
- belval 5y ago
  I used to say the same thing, but then I was hired at a big company to work on products that needed to handle Arabic, Thai, Devanagari and Bengali. They don't have a proper concept of characters and diacritics count as a character, not as an accent so you have verticality in a string! I've come to respect the unicode spec as something with a lot of inherent complexity that you have to understand to work with. Your inner voice will scream "there has to be a better way" all the way through the project, but there actually isn't, you just have to accept that human language and writing is complex and representing it in software is equally complex.
  - ASalazarMX 5y ago
    I understand that representing human languages is even more complex than I suppose, it's just that the article posted is a tweet surprised that a German word is one character wider in uppercase. It was not about graphic representation but number of characters.

darrenf 5y ago

Raku provides explicit methods for #1, #2 and #4 on strings: $string.encode('UTF-8').bytes; #1 $string.chars; #2 $string.codes; #4 https://course.raku.org/essentials/strings/string-length/ https://course.raku.org/essentials/strings/string-length/

goldenkey 5y ago
For those that don't know, Raku is just Perl 6 renamed.
- lizmat 5y ago
  > is just Perl 6 renamed Though this was technically correct in 2019, meanwhile the Raku Programming Language has continued to evolve, de-emphasizing its heritage more and more. For example, Covid restrictions allowing, 2022 will see the first European Raku Conference.
  - goldenkey 5y ago
    Is it a fork..or just a rename for a breaking version tree? I understand that backwards compatibility is not a required goal. So isn't it more like Python 3 rather than an entirely different programming language?
    - elcaro 5y ago
      > isn't it more like Python 3 Not in the slightest. Python 3 made breaking changes to the CPython code base (yes that's a simplification). There's enough backwards incompatibility to consider them different languages, but I'd guess that there might still be bits and pieces of code that is identical (or largely unchanged) between the Python 2 and 3 implementations. Raku - on the other hand - does not use any of the Perl code base; it's a completely different runtime written from the ground up.
    - raiph 5y ago
      It's not a fork any more than C# is a fork of C. It's not a rename for a breaking version tree any more than a C# compiler is of GCC. It is backwards compatible. The first strong demonstration of this was using the Catalyst framework, a major Perl package, in Raku programs, 6 years ago. It's not remotely like Python 3 vs Python 2. Python 3 is a tweak of Python 2 that utterly failed to confront hard problems that desperately needed -- and still need -- to be confronted. For example, Raku sits alongside Swift and Elixir as the only two mainstream PLs yet to properly address character processing in the Unicode era. In contrast, Python 3's doc doesn't even mention graphemes. Rakudo has no GIL; Python is stuck with CPython's. And so on for a long list of truly huge PL problems. Raku is an entirely different programming language. (This is so despite Raku(do) being able to support backwards compatibility via the same mechanism that allows it to use Python modules as if they were Raku modules. And C libraries, and Ruby ones, and so on.)
      - goldenkey 5y ago
        I see, thank you for your comprehensive explanation. I am intrigued and excited to try it out. I used to use Perl and CPAN all the time :-)

pvg 5y ago

This particular capitalization would have generated a longer string without Unicode - it's a language convention that a capitalization routine could apply to just about any encoding that has an ß.

zionic 5y ago

Except almost everyone always means #2. No one asked for strings to be ruined in this way, and this kind of pedantry has caused untold frustration from developers who just want their strings to work properly. If you must expose the underlying byte array do so via a sane access function that returns a typed array. As for “string length in pixels”, that has absolutely nothing to do with the string itself as that’s determined in the UI layer that ingests the string.

garmaine 5y ago
> Except almost everyone always means #2. I don't think this is true. Certainly there is no string-length library I'm aware of that handles it that way. The usual default these days (correct or not) is #4 -- length is the number of unicode code points.
- wolf550e 5y ago
  Some languages that had the misfortune to be developed while people thought UCS-2 would be enough define string length as number of UTF-16 code units, which is not even on the list because it's not a useful number.
  - garmaine 5y ago
    Yeah, and most languages/libraries which predate Unicode define string length as the number of bytes (#1). That's probably the most common interpretation, actually. But most new implementations count code point, in my experience. I believe this is also the Unicode recommendation when context doesn't determine a different algorithm to read. Been a while since I read those recommendations though, so I could be wrong.
    - zinekeller 5y ago
      > I believe this is also the Unicode recommendation when context doesn't determine a different algorithm to read. Except that emojis are universally two "characters", even those that are encoded as several codepoints. Also, non-composite Korean jamo versus composited jamo.

vadfa 5y ago

>3. Length in terms of visible space on screen is the number of pixels wide and tall when rendered in a given font. Grasping at straws eh?

arcticbull 5y ago
Nah this came out of personal experience working at a company where a designer wanted to limit a block of text to a certain “length” but in reality they wanted as many characters as would fit in a box without scrolling, scaling or wrapping. This has to be calculated by bounding box.

quietbritishjim 5y ago

That is an issue, for sure. But that is not "the issue" that they're describing. They are clearly talking about (2), so confusion about other meanings is irrelevant. Their point is that they would expect number of visble grapheme clusters to be the same when converting between cases. But in some languages that's not true, as their example demonstrates. (An upper case ß does exist in Unicode but culturally it's less correct than SS.)

_rend 5y ago

> 2. Length in terms of number of visible characters is the number of grapheme clusters. There's a fun subtlety in this case too: a single grapheme cluster need not draw as a single "visible character" (or glyph) on-screen. The visual representation of a grapheme cluster is entirely dependent on the text system which draws this, and the meaning that system applies to the cluster itself. This is especially true for multi-element emoji clusters, whose recommended meanings[1] change with evolving versions of Unicode. To add to this, Unicode 12 simplified the definition of grapheme clusters by actually generalizing them so that they can be matched effectively by a regex. (See the "extended grapheme cluster" definition in TR29[2].) This reduced the overall number of special cases and hard-coded lists of combinations in the definitions of grapheme clusters (particularly around emoji), but it also means that there are now infinitely more valid grapheme clusters that don't necessarily map to a single glyph. One really simple example of this is, e.g. © (Edit: it appears that HN is actually stripping out the ZWJ text from this example and leaving just the Copyright symbol. See below for how to reproduce this text on your machine.) That is, U+1F434 <horse head> + U+200D <zero width joiner> + U+00A9 <copyright sign> This cluster trivially matches the definition of extended grapheme cluster := crlf | Control | precore* _core_ postcore* core := hangul-syllable | ri-sequence | xpicto-sequence | [^Control CR LF] xpicto-sequence := \p{Extended_Pictographic} (Extend* ZWJ \p{Extended_Pictographic})* where U+1F434: Extended_Pictographic U+200D: ZWJ U+00A9: Extended_Pictographic (I picked this combination somewhat randomly, but ideally, this is an example that should hopefully last as it feels unlikely that "horse copyright" would have a meaningful glyph definition in the future. As of posting this, the above text shows up as two side-by-side glyphs on my machine (macOS Monterey 21A559): a horse, followed by the copyright sign. This may look similar on your machine, or it may not.) Importantly, you can tell this is actually treated as a real grapheme cluster by the text system on macOS because if you copy that string into a Cocoa text view (e.g., TextEdit), you will only be able to place your cursor on either side of the cluster, but not split it in the middle. A nice interactive way to see this in action is inserting U+1F434 into the document, followed by U+00A9. Then, move your cursor in between those two glyphs and insert U+200D: your cursor should then bounce out from the middle of the newly-formed cluster to the beginning. This was a pretty short example, but this is arbitrarily extensible: (Edit: Originally I had posted U+2705 <check mark symbol> + U+200D + U+1F434 <horse head> + U+200D + U+1F50B <battery> + U+200D + U+1F9F7 <safety pin> [sorry, no staple emoji] but HN stripped that out too. It does appear correctly in the text area while typing, but HN replaces the sequence with spaces after posting.) As linked above, Unicode does offer a list of sequences like this that are considered to be "meaningful"[1], which you can largely expect vendors which offer emoji representations to respect (and some vendors may offer glyphs for sequences beyond what is suggested here). If you've ever run into this: additions to this list over time explains why transporting a Unicode sequence which appears as a single glyph on one OS can appear as multiple glyphs on an older one (each individual glyph may be supported, but their combination may or may not have a meaning). In general, if this is interesting to you, you may enjoy trawling through the Unicode emoji data files [3]. You may discover something new! [1] https://www.unicode.org/Public/emoji/14.0/emoji-zwj-sequences.txt https://www.unicode.org/Public/emoji/14.0/emoji-zwj-sequence... [2] https://www.unicode.org/reports/tr29/tr29-35.html#Table_Combining_Char_Sequences_and_Grapheme_Clusters https://www.unicode.org/reports/tr29/tr29-35.html#Table_Comb... [3] https://www.unicode.org/reports/tr51/#emoji_data https://www.unicode.org/reports/tr51/#emoji_data

ASalazarMX 5y ago

Hey, let's keep inventing lengths. 5. Length in terms of time until complete display when rendered in a given architecture. 6. Length in terms of Braille characters needed to display it. 7. Length in terms of time of lecture at 300 wpm. String length is overwhelmingly discussed as character length, meaning #2. Length in bytes should only be an issue for data transmission or storage, and people who still work with ASCII. Rendering dimensions are not relevant to the published article.

rtkwe 5y ago
#2 is variable in arabic where some letters are combined which caused the old "effective power" crash in iOS.
deleted 5y ago
[deleted]
belval 5y ago
I used to say the same thing, but then I was hired at a big company to work on products that needed to handle Arabic, Thai, Devanagari and Bengali. They don't have a proper concept of characters and diacritics count as a character, not as an accent so you have verticality in a string! I've come to respect the unicode spec as something with a lot of inherent complexity that you have to understand to work with. Your inner voice will scream "there has to be a better way" all the way through the project, but there actually isn't, you just have to accept that human language and writing is complex and representing it in software is equally complex.
- ASalazarMX 5y ago
  I understand that representing human languages is even more complex than I suppose, it's just that the article posted is a tweet surprised that a German word is one character wider in uppercase. It was not about graphic representation but number of characters.