4 ms·

Grapheme clusters and no. Let’s see if anyone has a different interpretation or unsupported use-case.

by 3np 5y ago

AstralStorm 5y ago
That makes length operation require full parsing of the text according to Unicode rules. Hope you have an up to date set of these! Codepoints is easy, as are bytes. Graphemes much harder. So, what's the length of a sole non-breaking joiner codepoint? (U+2060, yes it's allowed)
- 3np 5y ago
  I’m really not an expert but is there anything particular with this one? Looking here the answer looks like “0” for all contexts to me. https://codepoints.net/U+2060 https://codepoints.net/U+2060 And well, I don’t think anyone’s arguing that a “length” function needs to be exposed by the standard library of any language (I saw that Swift does btw). Besides from byte count it’s the only interpretation of “length” that I think makes sense for users (like I mentioned above, validating that a name has at least length 1, for example)
  - JohnHaugeland 5y ago
    > And well, I don’t think anyone’s arguing that a “length” function needs to be exposed by the standard library of any language that's literally the core premise of the original post
    - 3np 5y ago
      ...is it?
dotancohen 5y ago
> Grapheme clusters and no. Let’s see if anyone has a different interpretation or unsupported use-case. Here's one. How many characters is this: שלום And how about this: שַלוֹם Notice that the former is lacking the diacritics of the latter. Is that more characters? Maybe a better example: ال vs ﻻ. Those are both Alif-Lamm, and no that is not a ligature. The former is often normalized to the latter, but could be stored separately. And does adding Hamza ء add a character? Does it depend if it's at the end of the word?
- 3np 5y ago
  I completely lack the cultural context to tell if this is trivial or not. Naive heuristic: look at the font rendering on my Mac: ء : 1 (but it sounds like it can be 0 depending on context) ﻻ : 1 ال : 2 שלום : 4 שַלוֹם : 4 Do you see any issues? In Japanese, which I am more familiar with, maybe it’s similar to ゙, which as you can see here has width 1 but is mostly used as a voicing sound mark as in turning か (ka) into が (ga), thus 0 (or more properly, as part of a cluster with length 1) The use-case I’m getting at is answering the question “does this render anything visible (including white space)”? I think it’s a reasonable answer on the whole falsehood-programmers-believe-about-names validation deal (after trimming leading and trailing whitespace). It’s the most intuitive and practically useful interpretation of “length” for arbitrary Unicode as text. The fact that it’s hard, complex, inelegant and arguably inconsistent does not change that.
  - dotancohen 5y ago
    > Do you see any issues? The only real issue that I see is that ﻻ is considered to be two letters, even if it is a single codepoint or grapheme. Regarding שַלוֹם I personally would consider it to be four letters, but the culture associated with that script (my culture) enjoys dissecting even the shape of the lines of the letters to look for meaning, I'm willing to bet that I'll find someone who will argue it to be six for the context he needs it to be. That's why I mentioned it. > In Japanese, which I am more familiar with, maybe it’s similar to ゙, Yes, for the Hebrew example, I believe so. If you're really interested, it would be more akin to our voicing mark dagesh, which e.g. turns V ב into B בּ and F פ into P פּ.‎
    - JohnHaugeland 5y ago
      Wait'll you find out that whether ß is one letter or two varies based on which brand of German you're drinking. Programmers should understand that it doesn't matter if they think the thousands of years of language rules are irrational; Unicode either handles them or it's wrong. Unicode doesn't exist to standardize the world's languages to a representation that programmers enjoy. It exists to encode all the world's languages *as they are*. Whether you are sympathetic to the rules of foreign languages isn't super relevant in practice.
level3 5y ago
One problem with grapheme clusters as length is that you can't do math with it. In other words, string A (of length 1) concatenated with string B (of length 1) might result in string C that also has length 1. Meanwhile, concatenating them in the reverse order might make a string D with length 2. This won't matter for most applications but it's something you may easily overlook when assuming grapheme clusters are enough for everything.
thrashh 5y ago
But who cares? This is like small potatoes. If you give a Japanese writer 140 characters, they can already encode double the amount of information an English writer can. Non-storage “character”-based lengths have always been a poor estimation of information encoding so worrying about some missing graphemes feels like you’re missing the bigger problem.

Grapheme clusters and no. Let’s see if anyone has a different interpretation or unsupported use-case.

by 3np 5y ago

Grapheme clusters and no. Let’s see if anyone has a different interpretation or unsupported use-case.

That makes length operation require full parsing of the text according to Unicode rules. Hope you have an up to date set of these! Codepoints is easy, as are bytes. Graphemes much harder. So, what's the length of a sole non-breaking joiner codepoint? (U+2060, yes it's allowed)

3np 5y ago
I’m really not an expert but is there anything particular with this one? Looking here the answer looks like “0” for all contexts to me. https://codepoints.net/U+2060 https://codepoints.net/U+2060 And well, I don’t think anyone’s arguing that a “length” function needs to be exposed by the standard library of any language (I saw that Swift does btw). Besides from byte count it’s the only interpretation of “length” that I think makes sense for users (like I mentioned above, validating that a name has at least length 1, for example)
- JohnHaugeland 5y ago
  > And well, I don’t think anyone’s arguing that a “length” function needs to be exposed by the standard library of any language that's literally the core premise of the original post
  - 3np 5y ago
    ...is it?

dotancohen 5y ago

> Grapheme clusters and no. Let’s see if anyone has a different interpretation or unsupported use-case. Here's one. How many characters is this: שלום And how about this: שַלוֹם Notice that the former is lacking the diacritics of the latter. Is that more characters? Maybe a better example: ال vs ﻻ. Those are both Alif-Lamm, and no that is not a ligature. The former is often normalized to the latter, but could be stored separately. And does adding Hamza ء add a character? Does it depend if it's at the end of the word?

3np 5y ago
I completely lack the cultural context to tell if this is trivial or not. Naive heuristic: look at the font rendering on my Mac: ء : 1 (but it sounds like it can be 0 depending on context) ﻻ : 1 ال : 2 שלום : 4 שַלוֹם : 4 Do you see any issues? In Japanese, which I am more familiar with, maybe it’s similar to ゙, which as you can see here has width 1 but is mostly used as a voicing sound mark as in turning か (ka) into が (ga), thus 0 (or more properly, as part of a cluster with length 1) The use-case I’m getting at is answering the question “does this render anything visible (including white space)”? I think it’s a reasonable answer on the whole falsehood-programmers-believe-about-names validation deal (after trimming leading and trailing whitespace). It’s the most intuitive and practically useful interpretation of “length” for arbitrary Unicode as text. The fact that it’s hard, complex, inelegant and arguably inconsistent does not change that.
- dotancohen 5y ago
  > Do you see any issues? The only real issue that I see is that ﻻ is considered to be two letters, even if it is a single codepoint or grapheme. Regarding שַלוֹם I personally would consider it to be four letters, but the culture associated with that script (my culture) enjoys dissecting even the shape of the lines of the letters to look for meaning, I'm willing to bet that I'll find someone who will argue it to be six for the context he needs it to be. That's why I mentioned it. > In Japanese, which I am more familiar with, maybe it’s similar to ゙, Yes, for the Hebrew example, I believe so. If you're really interested, it would be more akin to our voicing mark dagesh, which e.g. turns V ב into B בּ and F פ into P פּ.‎
  - JohnHaugeland 5y ago
    Wait'll you find out that whether ß is one letter or two varies based on which brand of German you're drinking. Programmers should understand that it doesn't matter if they think the thousands of years of language rules are irrational; Unicode either handles them or it's wrong. Unicode doesn't exist to standardize the world's languages to a representation that programmers enjoy. It exists to encode all the world's languages *as they are*. Whether you are sympathetic to the rules of foreign languages isn't super relevant in practice.

level3 5y ago

One problem with grapheme clusters as length is that you can't do math with it. In other words, string A (of length 1) concatenated with string B (of length 1) might result in string C that also has length 1. Meanwhile, concatenating them in the reverse order might make a string D with length 2. This won't matter for most applications but it's something you may easily overlook when assuming grapheme clusters are enough for everything.

thrashh 5y ago

But who cares? This is like small potatoes. If you give a Japanese writer 140 characters, they can already encode double the amount of information an English writer can. Non-storage “character”-based lengths have always been a poor estimation of information encoding so worrying about some missing graphemes feels like you’re missing the bigger problem.