3 ms·

I completely lack the cultural context to tell if this is trivial or not. Naive heuristic: look at the font rendering on my Mac: ء : 1 (but it sounds like it

by 3np 5y ago

I completely lack the cultural context to tell if this is trivial or not. Naive heuristic: look at the font rendering on my Mac: ء : 1 (but it sounds like it can be 0 depending on context) ﻻ : 1 ال : 2 שלום : 4 שַלוֹם : 4 Do you see any issues? In Japanese, which I am more familiar with, maybe it’s similar to ゙, which as you can see here has width 1 but is mostly used as a voicing sound mark as in turning か (ka) into が (ga), thus 0 (or more properly, as part of a cluster with length 1) The use-case I’m getting at is answering the question “does this render anything visible (including white space)”? I think it’s a reasonable answer on the whole falsehood-programmers-believe-about-names validation deal (after trimming leading and trailing whitespace). It’s the most intuitive and practically useful interpretation of “length” for arbitrary Unicode as text. The fact that it’s hard, complex, inelegant and arguably inconsistent does not change that.

dotancohen 5y ago
> Do you see any issues? The only real issue that I see is that ﻻ is considered to be two letters, even if it is a single codepoint or grapheme. Regarding שַלוֹם I personally would consider it to be four letters, but the culture associated with that script (my culture) enjoys dissecting even the shape of the lines of the letters to look for meaning, I'm willing to bet that I'll find someone who will argue it to be six for the context he needs it to be. That's why I mentioned it. > In Japanese, which I am more familiar with, maybe it’s similar to ゙, Yes, for the Hebrew example, I believe so. If you're really interested, it would be more akin to our voicing mark dagesh, which e.g. turns V ב into B בּ and F פ into P פּ.‎
- JohnHaugeland 5y ago
  Wait'll you find out that whether ß is one letter or two varies based on which brand of German you're drinking. Programmers should understand that it doesn't matter if they think the thousands of years of language rules are irrational; Unicode either handles them or it's wrong. Unicode doesn't exist to standardize the world's languages to a representation that programmers enjoy. It exists to encode all the world's languages *as they are*. Whether you are sympathetic to the rules of foreign languages isn't super relevant in practice.
  - dotancohen 5y ago
    > Unicode doesn't exist to standardize the world's languages to a > representationthat programmers enjoy. It exists to encode all > the world's languages *as they are*. Thank you for this terrific quote that I'm going to have to upset managers and clients with. This is the eloquence that I've been striving to express for a decade.
    - JohnHaugeland 5y ago
      Very kind words. Thank you.
JohnHaugeland 5y ago
> Naive heuristic: Unicode has a fully spelled out and testable algorithm that covers this. There's no need to make anything up on the spot. You don't need cultural context. It's not trivial. The algorithm is six printed pages. No individual part of it is particularly hard, but every language has its own weird stuff to add. I feel strongly that programmers should implement Unicode case folding with tests at least once, the way they should implement sorting and network connections and so on. . > I think it’s a reasonable answer on the whole falsehood-programmers-believe-about-names validation deal All these "false things programmers believe about X" documents? Go through Unicode just one time, you know 80% of all of them. I get that you like the framing you just made up, but there's an official one that the world's language nerds have been arguing over for 20 years, including having most of the population actually using it. I suggest you consider that it might be pretty well developed by now. . > The use-case I’m getting at is answering the question “does this render anything visible (including white space)”? This is not the appropriate approach, as white space can be rendered by non-characters, and characters can render nothing and still be characters under Unicode rules. . > It’s the most intuitive and practically useful interpretation of “length” for arbitrary Unicode as text. It also fails for Mongol, math symbols, music symbols, the zero-width space, the conjoiner system, et cetera. . > The fact that it’s hard, complex, inelegant and arguably inconsistent Fortunately, the real approach is easy, complex, inelegant, and wholly consistant. It's also well documented, and works reliably between programming languages that aren't Perl. At any rate, you're off in another thread instructing me that the linguists are wrong with their centuries of focus because of things you thought up on the spot, and that Unicode is wrong because it counts spaces when counting characters, so, I think I'm going to disconnect now. Thanks for understanding. Have a good day.

by 3np 5y ago

dotancohen 5y ago

> Do you see any issues? The only real issue that I see is that ﻻ is considered to be two letters, even if it is a single codepoint or grapheme. Regarding שַלוֹם I personally would consider it to be four letters, but the culture associated with that script (my culture) enjoys dissecting even the shape of the lines of the letters to look for meaning, I'm willing to bet that I'll find someone who will argue it to be six for the context he needs it to be. That's why I mentioned it. > In Japanese, which I am more familiar with, maybe it’s similar to ゙, Yes, for the Hebrew example, I believe so. If you're really interested, it would be more akin to our voicing mark dagesh, which e.g. turns V ב into B בּ and F פ into P פּ.‎

JohnHaugeland 5y ago
Wait'll you find out that whether ß is one letter or two varies based on which brand of German you're drinking. Programmers should understand that it doesn't matter if they think the thousands of years of language rules are irrational; Unicode either handles them or it's wrong. Unicode doesn't exist to standardize the world's languages to a representation that programmers enjoy. It exists to encode all the world's languages *as they are*. Whether you are sympathetic to the rules of foreign languages isn't super relevant in practice.
- dotancohen 5y ago
  > Unicode doesn't exist to standardize the world's languages to a > representationthat programmers enjoy. It exists to encode all > the world's languages *as they are*. Thank you for this terrific quote that I'm going to have to upset managers and clients with. This is the eloquence that I've been striving to express for a decade.
  - JohnHaugeland 5y ago
    Very kind words. Thank you.

JohnHaugeland 5y ago

> Naive heuristic: Unicode has a fully spelled out and testable algorithm that covers this. There's no need to make anything up on the spot. You don't need cultural context. It's not trivial. The algorithm is six printed pages. No individual part of it is particularly hard, but every language has its own weird stuff to add. I feel strongly that programmers should implement Unicode case folding with tests at least once, the way they should implement sorting and network connections and so on. . > I think it’s a reasonable answer on the whole falsehood-programmers-believe-about-names validation deal All these "false things programmers believe about X" documents? Go through Unicode just one time, you know 80% of all of them. I get that you like the framing you just made up, but there's an official one that the world's language nerds have been arguing over for 20 years, including having most of the population actually using it. I suggest you consider that it might be pretty well developed by now. . > The use-case I’m getting at is answering the question “does this render anything visible (including white space)”? This is not the appropriate approach, as white space can be rendered by non-characters, and characters can render nothing and still be characters under Unicode rules. . > It’s the most intuitive and practically useful interpretation of “length” for arbitrary Unicode as text. It also fails for Mongol, math symbols, music symbols, the zero-width space, the conjoiner system, et cetera. . > The fact that it’s hard, complex, inelegant and arguably inconsistent Fortunately, the real approach is easy, complex, inelegant, and wholly consistant. It's also well documented, and works reliably between programming languages that aren't Perl. At any rate, you're off in another thread instructing me that the linguists are wrong with their centuries of focus because of things you thought up on the spot, and that Unicode is wrong because it counts spaces when counting characters, so, I think I'm going to disconnect now. Thanks for understanding. Have a good day.