12 ms·

Easy enough to normalize the terminology for length to mean character count and size to mean byte capacity.

dotancohen 5y ago
"I don't know or care what a character is. You're a character. Now limit this field to exactly 50 letters. Not 51. And not 49." And that client doesn't consider spaces nor punctuation as letters. Numbers count, though. Thankfully at the time emojis were not a consideration.
arcticbull 5y ago
So you mean grapheme clusters or code points? Do you want to count zero-width characters? More specifically what are you trying to do?
- 3np 5y ago
  Grapheme clusters and no. Let’s see if anyone has a different interpretation or unsupported use-case.
  - AstralStorm 5y ago
    That makes length operation require full parsing of the text according to Unicode rules. Hope you have an up to date set of these! Codepoints is easy, as are bytes. Graphemes much harder. So, what's the length of a sole non-breaking joiner codepoint? (U+2060, yes it's allowed)
    - 3np 5y ago
      I’m really not an expert but is there anything particular with this one? Looking here the answer looks like “0” for all contexts to me. https://codepoints.net/U+2060 https://codepoints.net/U+2060 And well, I don’t think anyone’s arguing that a “length” function needs to be exposed by the standard library of any language (I saw that Swift does btw). Besides from byte count it’s the only interpretation of “length” that I think makes sense for users (like I mentioned above, validating that a name has at least length 1, for example)
      - JohnHaugeland 5y ago
        > And well, I don’t think anyone’s arguing that a “length” function needs to be exposed by the standard library of any language that's literally the core premise of the original post
        3np 5y ago
        ...is it?
  - dotancohen 5y ago
    > Grapheme clusters and no. Let’s see if anyone has a different interpretation or unsupported use-case. Here's one. How many characters is this: שלום And how about this: שַלוֹם Notice that the former is lacking the diacritics of the latter. Is that more characters? Maybe a better example: ال vs ﻻ. Those are both Alif-Lamm, and no that is not a ligature. The former is often normalized to the latter, but could be stored separately. And does adding Hamza ء add a character? Does it depend if it's at the end of the word?
    - 3np 5y ago
      I completely lack the cultural context to tell if this is trivial or not. Naive heuristic: look at the font rendering on my Mac: ء : 1 (but it sounds like it can be 0 depending on context) ﻻ : 1 ال : 2 שלום : 4 שַלוֹם : 4 Do you see any issues? In Japanese, which I am more familiar with, maybe it’s similar to ゙, which as you can see here has width 1 but is mostly used as a voicing sound mark as in turning か (ka) into が (ga), thus 0 (or more properly, as part of a cluster with length 1) The use-case I’m getting at is answering the question “does this render anything visible (including white space)”? I think it’s a reasonable answer on the whole falsehood-programmers-believe-about-names validation deal (after trimming leading and trailing whitespace). It’s the most intuitive and practically useful interpretation of “length” for arbitrary Unicode as text. The fact that it’s hard, complex, inelegant and arguably inconsistent does not change that.
      - dotancohen 5y ago
        > Do you see any issues? The only real issue that I see is that ﻻ is considered to be two letters, even if it is a single codepoint or grapheme. Regarding שַלוֹם I personally would consider it to be four letters, but the culture associated with that script (my culture) enjoys dissecting even the shape of the lines of the letters to look for meaning, I'm willing to bet that I'll find someone who will argue it to be six for the context he needs it to be. That's why I mentioned it. > In Japanese, which I am more familiar with, maybe it’s similar to ゙, Yes, for the Hebrew example, I believe so. If you're really interested, it would be more akin to our voicing mark dagesh, which e.g. turns V ב into B בּ and F פ into P פּ.‎
  - level3 5y ago
    One problem with grapheme clusters as length is that you can't do math with it. In other words, string A (of length 1) concatenated with string B (of length 1) might result in string C that also has length 1. Meanwhile, concatenating them in the reverse order might make a string D with length 2. This won't matter for most applications but it's something you may easily overlook when assuming grapheme clusters are enough for everything.
  - thrashh 5y ago
    But who cares? This is like small potatoes. If you give a Japanese writer 140 characters, they can already encode double the amount of information an English writer can. Non-storage “character”-based lengths have always been a poor estimation of information encoding so worrying about some missing graphemes feels like you’re missing the bigger problem.
- deleted 5y ago
  [deleted]
- JohnHaugeland 5y ago
  > So you mean grapheme clusters or code points? I mean character count. The unicode standard defines that as separate from and meaningfully different than grapheme clusters or codepoints. The confusion you're relying on isn't real. . > Do you want to count zero-width characters? Are they characters? Yes? Then I want to count them. . > More specifically what are you trying to do? (narrows eyes) I want to count characters. You're trying to make that sound confusing, but it really isn't. I don't care if you think it's a "zero width" character. Zero width space is often not actually zero width in programmers' fonts, and almost every font has at least a dozen of these wrong. I don't care about whatever other special moves you think you have, either. This is actually very simple. The unicode standard has something called a "character count." It is more work than the grapheme cluster count. The grapheme cluster count doesn't honor removals, replacements, substitutions, and it does something different in case folding. The people trying super hard to show how technically apt they are at the difficulties in Unicode are just repeating things they've heard other people say. The actual unicode standard makes this straightforward, and has these two terms separated already. If you genuinely want to undersatnd this, read Unicode 14 chapter 2, "general structure." It's about 50 pages. You can probably get away with just reading 2.2.3. It is three pages long. It's called "characters, not glyphs," because 𝐞𝐯𝐞𝐧 𝐭𝐡𝐞 𝐚𝐮𝐭𝐡𝐨𝐫𝐬 𝐨𝐟 𝐭𝐡𝐞 𝐔𝐧𝐢𝐜𝐨𝐝𝐞 𝐬𝐭𝐚𝐧𝐝𝐚𝐫𝐝 𝐰𝐚𝐧𝐭 𝐩𝐞𝐨𝐩𝐥𝐞 𝐭𝐨 𝐬𝐭𝐨𝐩 𝐩𝐫𝐞𝐭𝐞𝐧𝐝𝐢𝐧𝐠 𝐭𝐡𝐚𝐭 "𝐜𝐡𝐚𝐫𝐚𝐜𝐭𝐞𝐫" 𝐦𝐞𝐚𝐧𝐬 𝐚𝐧𝐲𝐭𝐡𝐢𝐧𝐠 𝐨𝐭𝐡𝐞𝐫 𝐭𝐡𝐚𝐧 𝐚 𝐟𝐮𝐥𝐥𝐲 𝐚𝐠𝐠𝐫𝐞𝐠𝐚𝐭𝐞𝐝 𝐬𝐞𝐫𝐢𝐞𝐬. The word "character" is well defined in Unicode. If you think it means anything other than point 2, 𝒚𝒐𝒖 𝒂𝒓𝒆 𝒔𝒊𝒎𝒑𝒍𝒚 𝒊𝒏𝒄𝒐𝒓𝒓𝒆𝒄𝒕, 𝒏𝒐𝒕 𝒑𝒍𝒚𝒊𝒏𝒈 𝒂 𝒅𝒆𝒆𝒑 𝒖𝒏𝒅𝒆𝒓𝒔𝒕𝒂𝒏𝒅𝒊𝒏𝒈 𝒐𝒇 𝒄𝒉𝒂𝒓𝒂𝒄𝒕𝒆𝒓 𝒆𝒏𝒄𝒐𝒅𝒊𝒏𝒈 𝒕𝒐𝒑𝒊𝒄𝒔. All you need is on pages 15 and 16. Go ahead. . I want every choice to be made in accord with the Unicode standard. Every technicality you guys are trying to bring up was handled 20 years ago. These words are actually well defined in the context of Unicode, and they're non-confusing in any other context. If you struggle with this, it is by choice. Size means byte count. Length means character count. No, it doesn't matter if you incorrectly pull technical terminology like "grapheme clusters" and "code points," because I don't mean either of those. I mean 𝐜𝐡𝐚𝐫𝐚𝐜𝐭𝐞𝐫 𝐜𝐨𝐮𝐧𝐭. If you have the sequence `capital a`, `zero width joiner`, `joining umlaut`, `special character`, `emoji face`, `skin color modifier`, you have: 1. Six codepoints 2. Three grapheme clusters 3. Four characters Please wait until you can explain why before continuing to attempt to teach technicalities, friend. It doesn't work the way you claim. This might help. https://www.unicode.org/versions/Unicode14.0.0/UnicodeStandard-14.0.pdf https://www.unicode.org/versions/Unicode14.0.0/UnicodeStanda... Here, let's save you some time on some other technicalities that aren't. 1. If you write a unicode modifying character after something that cannot be modified - like adding an umlaut to a zero width joiner - then the umlaut will be considered a full character, but also discarded. Should it be counted? According to the Unicode standard, yes. 2. If you write a zero width anything, and it's a character, should it be counted? Yes. 3. If you write something that is a grapheme cluster, but your font or renderer doesn't support it, so it renders as two characters (by example, they added couple emoji in Unicode 10 and gay couples in Unicode 11, so phones that were released inbetween would render a two-man couple as two individual men.) Should that be counted as a single character as it's written, or a double character as it's rendered? Single. 4. If you're in a font that includes language variants - by example, Swiss German fancy fonts sometimes separate the S Tset into two distinct S characters that are flowed separately - should that be calculated as one character or two? Two, it turns out. 5. If a character pair is kerned into a single symbol, like an English cursive capital F to lower case E, should that be counted as one character or two? Two. There's a whole chapter of these. If you really enjoy going through them, stop quizzing me and just read it. These questions have been answered for literal decades. Just go read the standard already.
Sharlin 5y ago
"Character count" is not a well-defined concept because "character" is not a well-defined concept in general.
- zionic 5y ago
  It absolutely is, unless you allow xenophilia to overcomplicate the issue. You don't need to completely bork your string implementation to support every language ever concocted by mankind. No one is coding in cuneiform.
simonh 5y ago
A bold decision, to use the term 'easy' in a discussion of unicode.
- simonh 5y ago
  However in this case it seems a justified one, having read your outstanding rebuttal to a sibling comment. Much appreciated.

"I don't know or care what a character is. You're a character. Now limit this field to exactly 50 letters. Not 51. And not 49." And that client doesn't consider spaces nor punctuation as letters. Numbers count, though. Thankfully at the time emojis were not a consideration.

arcticbull 5y ago

So you mean grapheme clusters or code points? Do you want to count zero-width characters? More specifically what are you trying to do?

3np 5y ago
Grapheme clusters and no. Let’s see if anyone has a different interpretation or unsupported use-case.
- AstralStorm 5y ago
  That makes length operation require full parsing of the text according to Unicode rules. Hope you have an up to date set of these! Codepoints is easy, as are bytes. Graphemes much harder. So, what's the length of a sole non-breaking joiner codepoint? (U+2060, yes it's allowed)
  - 3np 5y ago
    I’m really not an expert but is there anything particular with this one? Looking here the answer looks like “0” for all contexts to me. https://codepoints.net/U+2060 https://codepoints.net/U+2060 And well, I don’t think anyone’s arguing that a “length” function needs to be exposed by the standard library of any language (I saw that Swift does btw). Besides from byte count it’s the only interpretation of “length” that I think makes sense for users (like I mentioned above, validating that a name has at least length 1, for example)
    - JohnHaugeland 5y ago
      > And well, I don’t think anyone’s arguing that a “length” function needs to be exposed by the standard library of any language that's literally the core premise of the original post
      - 3np 5y ago
        ...is it?
- dotancohen 5y ago
  > Grapheme clusters and no. Let’s see if anyone has a different interpretation or unsupported use-case. Here's one. How many characters is this: שלום And how about this: שַלוֹם Notice that the former is lacking the diacritics of the latter. Is that more characters? Maybe a better example: ال vs ﻻ. Those are both Alif-Lamm, and no that is not a ligature. The former is often normalized to the latter, but could be stored separately. And does adding Hamza ء add a character? Does it depend if it's at the end of the word?
  - 3np 5y ago
    I completely lack the cultural context to tell if this is trivial or not. Naive heuristic: look at the font rendering on my Mac: ء : 1 (but it sounds like it can be 0 depending on context) ﻻ : 1 ال : 2 שלום : 4 שַלוֹם : 4 Do you see any issues? In Japanese, which I am more familiar with, maybe it’s similar to ゙, which as you can see here has width 1 but is mostly used as a voicing sound mark as in turning か (ka) into が (ga), thus 0 (or more properly, as part of a cluster with length 1) The use-case I’m getting at is answering the question “does this render anything visible (including white space)”? I think it’s a reasonable answer on the whole falsehood-programmers-believe-about-names validation deal (after trimming leading and trailing whitespace). It’s the most intuitive and practically useful interpretation of “length” for arbitrary Unicode as text. The fact that it’s hard, complex, inelegant and arguably inconsistent does not change that.
    - dotancohen 5y ago
      > Do you see any issues? The only real issue that I see is that ﻻ is considered to be two letters, even if it is a single codepoint or grapheme. Regarding שַלוֹם I personally would consider it to be four letters, but the culture associated with that script (my culture) enjoys dissecting even the shape of the lines of the letters to look for meaning, I'm willing to bet that I'll find someone who will argue it to be six for the context he needs it to be. That's why I mentioned it. > In Japanese, which I am more familiar with, maybe it’s similar to ゙, Yes, for the Hebrew example, I believe so. If you're really interested, it would be more akin to our voicing mark dagesh, which e.g. turns V ב into B בּ and F פ into P פּ.‎
- level3 5y ago
  One problem with grapheme clusters as length is that you can't do math with it. In other words, string A (of length 1) concatenated with string B (of length 1) might result in string C that also has length 1. Meanwhile, concatenating them in the reverse order might make a string D with length 2. This won't matter for most applications but it's something you may easily overlook when assuming grapheme clusters are enough for everything.
- thrashh 5y ago
  But who cares? This is like small potatoes. If you give a Japanese writer 140 characters, they can already encode double the amount of information an English writer can. Non-storage “character”-based lengths have always been a poor estimation of information encoding so worrying about some missing graphemes feels like you’re missing the bigger problem.
deleted 5y ago
[deleted]
JohnHaugeland 5y ago
> So you mean grapheme clusters or code points? I mean character count. The unicode standard defines that as separate from and meaningfully different than grapheme clusters or codepoints. The confusion you're relying on isn't real. . > Do you want to count zero-width characters? Are they characters? Yes? Then I want to count them. . > More specifically what are you trying to do? (narrows eyes) I want to count characters. You're trying to make that sound confusing, but it really isn't. I don't care if you think it's a "zero width" character. Zero width space is often not actually zero width in programmers' fonts, and almost every font has at least a dozen of these wrong. I don't care about whatever other special moves you think you have, either. This is actually very simple. The unicode standard has something called a "character count." It is more work than the grapheme cluster count. The grapheme cluster count doesn't honor removals, replacements, substitutions, and it does something different in case folding. The people trying super hard to show how technically apt they are at the difficulties in Unicode are just repeating things they've heard other people say. The actual unicode standard makes this straightforward, and has these two terms separated already. If you genuinely want to undersatnd this, read Unicode 14 chapter 2, "general structure." It's about 50 pages. You can probably get away with just reading 2.2.3. It is three pages long. It's called "characters, not glyphs," because 𝐞𝐯𝐞𝐧 𝐭𝐡𝐞 𝐚𝐮𝐭𝐡𝐨𝐫𝐬 𝐨𝐟 𝐭𝐡𝐞 𝐔𝐧𝐢𝐜𝐨𝐝𝐞 𝐬𝐭𝐚𝐧𝐝𝐚𝐫𝐝 𝐰𝐚𝐧𝐭 𝐩𝐞𝐨𝐩𝐥𝐞 𝐭𝐨 𝐬𝐭𝐨𝐩 𝐩𝐫𝐞𝐭𝐞𝐧𝐝𝐢𝐧𝐠 𝐭𝐡𝐚𝐭 "𝐜𝐡𝐚𝐫𝐚𝐜𝐭𝐞𝐫" 𝐦𝐞𝐚𝐧𝐬 𝐚𝐧𝐲𝐭𝐡𝐢𝐧𝐠 𝐨𝐭𝐡𝐞𝐫 𝐭𝐡𝐚𝐧 𝐚 𝐟𝐮𝐥𝐥𝐲 𝐚𝐠𝐠𝐫𝐞𝐠𝐚𝐭𝐞𝐝 𝐬𝐞𝐫𝐢𝐞𝐬. The word "character" is well defined in Unicode. If you think it means anything other than point 2, 𝒚𝒐𝒖 𝒂𝒓𝒆 𝒔𝒊𝒎𝒑𝒍𝒚 𝒊𝒏𝒄𝒐𝒓𝒓𝒆𝒄𝒕, 𝒏𝒐𝒕 𝒑𝒍𝒚𝒊𝒏𝒈 𝒂 𝒅𝒆𝒆𝒑 𝒖𝒏𝒅𝒆𝒓𝒔𝒕𝒂𝒏𝒅𝒊𝒏𝒈 𝒐𝒇 𝒄𝒉𝒂𝒓𝒂𝒄𝒕𝒆𝒓 𝒆𝒏𝒄𝒐𝒅𝒊𝒏𝒈 𝒕𝒐𝒑𝒊𝒄𝒔. All you need is on pages 15 and 16. Go ahead. . I want every choice to be made in accord with the Unicode standard. Every technicality you guys are trying to bring up was handled 20 years ago. These words are actually well defined in the context of Unicode, and they're non-confusing in any other context. If you struggle with this, it is by choice. Size means byte count. Length means character count. No, it doesn't matter if you incorrectly pull technical terminology like "grapheme clusters" and "code points," because I don't mean either of those. I mean 𝐜𝐡𝐚𝐫𝐚𝐜𝐭𝐞𝐫 𝐜𝐨𝐮𝐧𝐭. If you have the sequence `capital a`, `zero width joiner`, `joining umlaut`, `special character`, `emoji face`, `skin color modifier`, you have: 1. Six codepoints 2. Three grapheme clusters 3. Four characters Please wait until you can explain why before continuing to attempt to teach technicalities, friend. It doesn't work the way you claim. This might help. https://www.unicode.org/versions/Unicode14.0.0/UnicodeStandard-14.0.pdf https://www.unicode.org/versions/Unicode14.0.0/UnicodeStanda... Here, let's save you some time on some other technicalities that aren't. 1. If you write a unicode modifying character after something that cannot be modified - like adding an umlaut to a zero width joiner - then the umlaut will be considered a full character, but also discarded. Should it be counted? According to the Unicode standard, yes. 2. If you write a zero width anything, and it's a character, should it be counted? Yes. 3. If you write something that is a grapheme cluster, but your font or renderer doesn't support it, so it renders as two characters (by example, they added couple emoji in Unicode 10 and gay couples in Unicode 11, so phones that were released inbetween would render a two-man couple as two individual men.) Should that be counted as a single character as it's written, or a double character as it's rendered? Single. 4. If you're in a font that includes language variants - by example, Swiss German fancy fonts sometimes separate the S Tset into two distinct S characters that are flowed separately - should that be calculated as one character or two? Two, it turns out. 5. If a character pair is kerned into a single symbol, like an English cursive capital F to lower case E, should that be counted as one character or two? Two. There's a whole chapter of these. If you really enjoy going through them, stop quizzing me and just read it. These questions have been answered for literal decades. Just go read the standard already.

Sharlin 5y ago

"Character count" is not a well-defined concept because "character" is not a well-defined concept in general.

zionic 5y ago
It absolutely is, unless you allow xenophilia to overcomplicate the issue. You don't need to completely bork your string implementation to support every language ever concocted by mankind. No one is coding in cuneiform.

simonh 5y ago

A bold decision, to use the term 'easy' in a discussion of unicode.

simonh 5y ago
However in this case it seems a justified one, having read your outstanding rebuttal to a sibling comment. Much appreciated.