3 ms·

> A 'character' meaning what? […] it's ambiguity in the concept of "character" All of this already standardised in detail in Unicode. There is no ambiguity. Re

by bmn__ 5y ago

> A 'character' meaning what? […] it's ambiguity in the concept of "character" All of this already standardised in detail in Unicode. There is no ambiguity. Read it some time to mend your ignorance, it answers all your questions with justification, I am willing to spend time on the answers only. > the second character of sœur (o or the ligature?) the letter œ, both character (C) and grapheme cluster (GC) the letter o does not exist in this text (again, according to the standard) > the second character of حبيبي (the canonical form ب or the contextual form ﺒ ?) the letter ب, both C and GC the letter ﺒ is a presentation form and does not exist in this text > the third character of есть (Cyrillic t with or without soft sign the letter т, both C and GC the letter ь, both C and GC, is the 4th in the text > the first character of 실례합니다 (Korean phoneme or syllabic grapheme?) the letter 실, both C and GC the text does not contain the jamo ᄉ Unicode is not concerned with phonemes, only with writing > the first character of ﷺ is entirely itself, the letter ﷺ, both C and GC > or ﷽ ? is entirely itself, the symbol ﷽, both C and GC

torstenvl 5y ago
- bmn__ 5y ago
  > you are oversimplifying to the point of inaccuracy No, what I wrote is an accurate reflection of Unicode. It does not matter what Jean Dupont and Hong Gildong think, but what we agreed on as a standard. > They do not exist as standalone code points in the binary form of the encoded text This terminology is confused/conceptually wrong. The binary form of encoded text consists of bytes. Decoded text consists of characters/grapheme clusters. A code point is the number corresponding to a character in the repertoire tables. By definition a code point cannot exist in encoded text. The correct way to determine whether sœur contains œ or o is to use a standard compliant implementation. Perl has always been nothing short of excellent, so: $ perl -Mutf8 -E'say "sœur" =~ "œ" ? "yes" : "no"' yes $ perl -Mutf8 -E'say "sœur" =~ "o" ? "yes" : "no"' no > don't let that anger control you I'm not angry. Projecting much?
  - torstenvl 5y ago
    Software written in the way you suggest will fail in the real world, where, to beat a dead horse, "sœur" indisputably does contain the letter "o." You seem to be arguing that the way Unicode treats language is the Right Answer, but that simply isn't the case. Real people speaking real human languages in the real world do not care whether the œ ligature is implemented in a character set or in the font or even what those terms mean. What they do care about is that they searched for "Lettre à ma sœur.docx" and your software failed to show them "Lettre à ma soeur.docx" I choose not to write software that way. The right thing to do when dealing with human language depends on context, which is ambiguous. Unicode is a just a tool - indeed, one that might be helpful in the case of some ligatures by using NFKD - and nothing more. It certainly isn't an arbiter of what constitutes a character in a human language. Have a great night.
    - bmn__ 5y ago
      #!/usr/bin/env perl use utf8; use Unicode::Collate; my $uc = Unicode::Collate->new( normalization => undef, level => 1 ); printf "pos %d len %d\n", $uc->index('Lettre à ma sœur.docx', $_) for 'sœur', 'soeur', 'lettre a ma-soeur _DOCX_'; __END__ pos 12 len 4 pos 12 len 4 pos 0 len 21 You have been wrong three times in a row now, just because you have an insufficient understanding of the standard. Time to do what I said in the beginning and read it, or do you want to continue arguing against straw-men and pretending to be knowledgeable enough about the subject matter to talk about it authoritatively?

by bmn__ 5y ago

torstenvl 5y ago

bmn__ 5y ago
> you are oversimplifying to the point of inaccuracy No, what I wrote is an accurate reflection of Unicode. It does not matter what Jean Dupont and Hong Gildong think, but what we agreed on as a standard. > They do not exist as standalone code points in the binary form of the encoded text This terminology is confused/conceptually wrong. The binary form of encoded text consists of bytes. Decoded text consists of characters/grapheme clusters. A code point is the number corresponding to a character in the repertoire tables. By definition a code point cannot exist in encoded text. The correct way to determine whether sœur contains œ or o is to use a standard compliant implementation. Perl has always been nothing short of excellent, so: $ perl -Mutf8 -E'say "sœur" =~ "œ" ? "yes" : "no"' yes $ perl -Mutf8 -E'say "sœur" =~ "o" ? "yes" : "no"' no > don't let that anger control you I'm not angry. Projecting much?
- torstenvl 5y ago
  Software written in the way you suggest will fail in the real world, where, to beat a dead horse, "sœur" indisputably does contain the letter "o." You seem to be arguing that the way Unicode treats language is the Right Answer, but that simply isn't the case. Real people speaking real human languages in the real world do not care whether the œ ligature is implemented in a character set or in the font or even what those terms mean. What they do care about is that they searched for "Lettre à ma sœur.docx" and your software failed to show them "Lettre à ma soeur.docx" I choose not to write software that way. The right thing to do when dealing with human language depends on context, which is ambiguous. Unicode is a just a tool - indeed, one that might be helpful in the case of some ligatures by using NFKD - and nothing more. It certainly isn't an arbiter of what constitutes a character in a human language. Have a great night.
  - bmn__ 5y ago
    #!/usr/bin/env perl use utf8; use Unicode::Collate; my $uc = Unicode::Collate->new( normalization => undef, level => 1 ); printf "pos %d len %d\n", $uc->index('Lettre à ma sœur.docx', $_) for 'sœur', 'soeur', 'lettre a ma-soeur _DOCX_'; __END__ pos 12 len 4 pos 12 len 4 pos 0 len 21 You have been wrong three times in a row now, just because you have an insufficient understanding of the standard. Time to do what I said in the beginning and read it, or do you want to continue arguing against straw-men and pretending to be knowledgeable enough about the subject matter to talk about it authoritatively?