5 ms·

What Is the “Length” of a String?

44 points by gk1 7y ago

fireattack 7y ago
I prefer this article: https://hsivonen.fi/string-length/ https://hsivonen.fi/string-length/ HN discussion: https://news.ycombinator.com/item?id=20914184 https://news.ycombinator.com/item?id=20914184)
pg_is_a_butt 7y ago
What is the "Ratio" of Stanley Nickels to Schrute Bucks?
mjevans 7y ago
IIRC from some other article; most JavaScript engines use a UTF-16 Unicode encoding for characters, and it looks like the .length attribute is reporting the memory footprint in terms of 16 bit codepoints. A single grapheme (display 'character', pictograph, or fixed sized space) may be composed of one or more codepoints. 'length' is subjective. It makes sense to want to know how many storage units are required. Knowing the number of printed 'graphemes' (do we count non-printing 'characters'?) might also be useful. The display length (at a given scaling size, taking in to consideration font kerning/etc) can also be useful. That's where the linked article really goes off the rails. Characters don't have printing widths, not outside of extremely specific circumstances. You have to ask the layout engine in use what dimensions are occupied after it solves the very complex problem.
udp 7y ago
It really doesn't matter. A string is a binary blob until you need to parse it, or display it. If you need to parse a string and it's UTF-8, you can pretend it's an ASCII string because the control characters you need for parsing are probably ASCII characters such as {, [, or ". UTF-8 continuation characters set a high bit to ensure that UTF-8 is a superset of ASCII, so you can use all of the standard C library functions. If you need to display a string, you're already using a font library which, in addition to providing the logic to iterate through the string character by character, can work out kerning etc. I never understood the rationale for using encodings such as UTF-16. They seem to be the worst of both worlds: strings for which ASCII would be adequate take 2x the space, and the encoding is still multi-byte. I once worked with a Windows developer who swore blind that UTF-16 was not a multi-byte encoding. When I provided evidence to the contrary, they responded something along the lines of "ok, but who would ever need more than 16 bits worth of characters?". ¯\_(ツ)_/¯
- mehrdadn 7y ago
  The(/a) rationale for UTF-16 is that it's more space-efficient for languages that use high code points, like East Asian languages.
  - udp 7y ago
    Sure, but the majority of the text computers have to parse has a syntax consisting of low codepoint ASCII characters (think XML tags vs. data). The fragments of text using a high codepoint language are usually only interesting at render time.
    - mehrdadn 7y ago
      I'm not supporting the position; I'm just explaining what the argument for it is, since you said you never understood the rationale behind it and felt it gave you the worst of both worlds. But I will say that the concern you have does not take into account the concerns other people have who make this argument.
      - SAI_Peregrinus 7y ago
        The other aspect of importance is that languages with Chinese characters have higher information density per character, so they tend to even out in terms of actual storage requirements.
  - jandrese 7y ago
    IIRC the real rationale is back in the early days people bet that 16 bits would be enough for a fixed length encoding, but the bet didn't pay off and now they're stuck with the worst of both worlds.
    - mehrdadn 7y ago
      Depends what you mean by "rationale" I guess. I was trying to explain why one might still want to use it, not why one might be stuck using it.
wintorez 7y ago
There are two data types that the more you think about them, the weirder they become: String & Date
kevincox 7y ago
Of course measuring each character also doesn't work because of kerning, ligatures and probably other rendering features.
maxdamantus 7y ago
> Instead of rendering all the strings in each column, we can split the strings into their corresponding graphemes and render them individually. This allows us to cache the pixel length of each grapheme we encounter. I wonder if they're aware of the `measureText` method used in canvas: https://developer.mozilla.org/en-US/docs/Web/API/CanvasRenderingContext2D/measureText https://developer.mozilla.org/en-US/docs/Web/API/CanvasRende... Seems a lot simpler than trying to split strings up by graphemes, and probably more reliable. Pairs of letters such as "fi" are rendered as ligatures (single glyphs) in some fonts, and Unicode no longer standardises them (since the combinations are completely arbitrary). Also, grapheme clustering is going to change according to Unicode version, and not everyone's browser is going to be using the same Unicode version. If you want to know the "string length", you basically either want the number of code units to store it (in languages that handle strings sensibly, this is just the number of bytes, since UTF-8 should always be used, except for historical reasons), or you want to pass it to a font rendering library to tell you something about pixels.

44 points by gk1 7y ago

I prefer this article: https://hsivonen.fi/string-length/ https://hsivonen.fi/string-length/ HN discussion: https://news.ycombinator.com/item?id=20914184 https://news.ycombinator.com/item?id=20914184)

pg_is_a_butt 7y ago

What is the "Ratio" of Stanley Nickels to Schrute Bucks?

mjevans 7y ago

IIRC from some other article; most JavaScript engines use a UTF-16 Unicode encoding for characters, and it looks like the .length attribute is reporting the memory footprint in terms of 16 bit codepoints. A single grapheme (display 'character', pictograph, or fixed sized space) may be composed of one or more codepoints. 'length' is subjective. It makes sense to want to know how many storage units are required. Knowing the number of printed 'graphemes' (do we count non-printing 'characters'?) might also be useful. The display length (at a given scaling size, taking in to consideration font kerning/etc) can also be useful. That's where the linked article really goes off the rails. Characters don't have printing widths, not outside of extremely specific circumstances. You have to ask the layout engine in use what dimensions are occupied after it solves the very complex problem.

udp 7y ago

It really doesn't matter. A string is a binary blob until you need to parse it, or display it. If you need to parse a string and it's UTF-8, you can pretend it's an ASCII string because the control characters you need for parsing are probably ASCII characters such as {, [, or ". UTF-8 continuation characters set a high bit to ensure that UTF-8 is a superset of ASCII, so you can use all of the standard C library functions. If you need to display a string, you're already using a font library which, in addition to providing the logic to iterate through the string character by character, can work out kerning etc. I never understood the rationale for using encodings such as UTF-16. They seem to be the worst of both worlds: strings for which ASCII would be adequate take 2x the space, and the encoding is still multi-byte. I once worked with a Windows developer who swore blind that UTF-16 was not a multi-byte encoding. When I provided evidence to the contrary, they responded something along the lines of "ok, but who would ever need more than 16 bits worth of characters?". ¯\_(ツ)_/¯

mehrdadn 7y ago
The(/a) rationale for UTF-16 is that it's more space-efficient for languages that use high code points, like East Asian languages.
- udp 7y ago
  Sure, but the majority of the text computers have to parse has a syntax consisting of low codepoint ASCII characters (think XML tags vs. data). The fragments of text using a high codepoint language are usually only interesting at render time.
  - mehrdadn 7y ago
    I'm not supporting the position; I'm just explaining what the argument for it is, since you said you never understood the rationale behind it and felt it gave you the worst of both worlds. But I will say that the concern you have does not take into account the concerns other people have who make this argument.
    - SAI_Peregrinus 7y ago
      The other aspect of importance is that languages with Chinese characters have higher information density per character, so they tend to even out in terms of actual storage requirements.
- jandrese 7y ago
  IIRC the real rationale is back in the early days people bet that 16 bits would be enough for a fixed length encoding, but the bet didn't pay off and now they're stuck with the worst of both worlds.
  - mehrdadn 7y ago
    Depends what you mean by "rationale" I guess. I was trying to explain why one might still want to use it, not why one might be stuck using it.

wintorez 7y ago

There are two data types that the more you think about them, the weirder they become: String & Date

kevincox 7y ago

Of course measuring each character also doesn't work because of kerning, ligatures and probably other rendering features.

maxdamantus 7y ago

> Instead of rendering all the strings in each column, we can split the strings into their corresponding graphemes and render them individually. This allows us to cache the pixel length of each grapheme we encounter. I wonder if they're aware of the `measureText` method used in canvas: https://developer.mozilla.org/en-US/docs/Web/API/CanvasRenderingContext2D/measureText https://developer.mozilla.org/en-US/docs/Web/API/CanvasRende... Seems a lot simpler than trying to split strings up by graphemes, and probably more reliable. Pairs of letters such as "fi" are rendered as ligatures (single glyphs) in some fonts, and Unicode no longer standardises them (since the combinations are completely arbitrary). Also, grapheme clustering is going to change according to Unicode version, and not everyone's browser is going to be using the same Unicode version. If you want to know the "string length", you basically either want the number of code units to store it (in languages that handle strings sensibly, this is just the number of bytes, since UTF-8 should always be used, except for historical reasons), or you want to pass it to a font rendering library to tell you something about pixels.