4 ms·

A 'character' meaning what? The first code point? The first non-combining code point? The first non-combining code point along with all associated combining cod

A 'character' meaning what? The first code point? The first non-combining code point? The first non-combining code point along with all associated combining code points? The first non-combining code point along with all associated combining code points modified to look like it would look in conjunction with surrounding non-combining code points along with their associated combining code points? The first displayable component of a code point? What are the following? the second character of sœur (o or the ligature?) the second character of حبيبي (the canonical form ب or the contextual form ﺒ ?) the third character of есть (Cyrillic t with or without soft sign, which is always a separate code point and always displayed to the right but changes the sound?) the first character of 실례합니다 (Korean phoneme or syllabic grapheme?) the first character of ﷺ or ﷽ ? The main issue isn't programming language support, it's ambiguity in the concept of "character" and conventions about how languages treat them. Imagine how the letter i would be treated if Unicode were invented by a Turk. The fundamental issue here is that human communication is deeply nuanced in a way that does not lend itself well to systematic encoding or fast/naive algorithms on that encoding. Even in the plain ASCII range it's impossible to know how to render a word like "ANSCHLUSS" in lower case (or how many 'characters' such a word would have) without knowledge of the language, country of origin, and time period in which the word was written.

Gigachad 5y ago
Ultimately I think it doesn’t really matter. Not all languages and character sets have orderings. You just order the Latin or closely related characters and numbers and you are ok for almost all situations. If you are a company or product which caters to more languages, you can work out what you do for those. But there is no need to work out how to order every single codepoint in Unicode because it isn’t very useful.
- tialaramex 5y ago
  You've gone from arguing that it isn't complex if people have the right language, to redefining the problem as whatever it was easy for you to do in your preferred language. Just say "I don't have the faintest idea how to do this properly". It would be much less of a problem if people would admit they don't know how and need somebody else's help than it is when they insist it "doesn't really matter" as you have and produce garbage that doesn't work but insist that's just because everybody is holding it wrong.
bmn__ 5y ago
> A 'character' meaning what? […] it's ambiguity in the concept of "character" All of this already standardised in detail in Unicode. There is no ambiguity. Read it some time to mend your ignorance, it answers all your questions with justification, I am willing to spend time on the answers only. > the second character of sœur (o or the ligature?) the letter œ, both character (C) and grapheme cluster (GC) the letter o does not exist in this text (again, according to the standard) > the second character of حبيبي (the canonical form ب or the contextual form ﺒ ?) the letter ب, both C and GC the letter ﺒ is a presentation form and does not exist in this text > the third character of есть (Cyrillic t with or without soft sign the letter т, both C and GC the letter ь, both C and GC, is the 4th in the text > the first character of 실례합니다 (Korean phoneme or syllabic grapheme?) the letter 실, both C and GC the text does not contain the jamo ᄉ Unicode is not concerned with phonemes, only with writing > the first character of ﷺ is entirely itself, the letter ﷺ, both C and GC > or ﷽ ? is entirely itself, the symbol ﷽, both C and GC
- torstenvl 5y ago
  - bmn__ 5y ago
    > you are oversimplifying to the point of inaccuracy No, what I wrote is an accurate reflection of Unicode. It does not matter what Jean Dupont and Hong Gildong think, but what we agreed on as a standard. > They do not exist as standalone code points in the binary form of the encoded text This terminology is confused/conceptually wrong. The binary form of encoded text consists of bytes. Decoded text consists of characters/grapheme clusters. A code point is the number corresponding to a character in the repertoire tables. By definition a code point cannot exist in encoded text. The correct way to determine whether sœur contains œ or o is to use a standard compliant implementation. Perl has always been nothing short of excellent, so: $ perl -Mutf8 -E'say "sœur" =~ "œ" ? "yes" : "no"' yes $ perl -Mutf8 -E'say "sœur" =~ "o" ? "yes" : "no"' no > don't let that anger control you I'm not angry. Projecting much?
    - 5y ago

Gigachad 5y ago

Ultimately I think it doesn’t really matter. Not all languages and character sets have orderings. You just order the Latin or closely related characters and numbers and you are ok for almost all situations. If you are a company or product which caters to more languages, you can work out what you do for those. But there is no need to work out how to order every single codepoint in Unicode because it isn’t very useful.

tialaramex 5y ago
You've gone from arguing that it isn't complex if people have the right language, to redefining the problem as whatever it was easy for you to do in your preferred language. Just say "I don't have the faintest idea how to do this properly". It would be much less of a problem if people would admit they don't know how and need somebody else's help than it is when they insist it "doesn't really matter" as you have and produce garbage that doesn't work but insist that's just because everybody is holding it wrong.

bmn__ 5y ago

> A 'character' meaning what? […] it's ambiguity in the concept of "character" All of this already standardised in detail in Unicode. There is no ambiguity. Read it some time to mend your ignorance, it answers all your questions with justification, I am willing to spend time on the answers only. > the second character of sœur (o or the ligature?) the letter œ, both character (C) and grapheme cluster (GC) the letter o does not exist in this text (again, according to the standard) > the second character of حبيبي (the canonical form ب or the contextual form ﺒ ?) the letter ب, both C and GC the letter ﺒ is a presentation form and does not exist in this text > the third character of есть (Cyrillic t with or without soft sign the letter т, both C and GC the letter ь, both C and GC, is the 4th in the text > the first character of 실례합니다 (Korean phoneme or syllabic grapheme?) the letter 실, both C and GC the text does not contain the jamo ᄉ Unicode is not concerned with phonemes, only with writing > the first character of ﷺ is entirely itself, the letter ﷺ, both C and GC > or ﷽ ? is entirely itself, the symbol ﷽, both C and GC

torstenvl 5y ago
- bmn__ 5y ago
  > you are oversimplifying to the point of inaccuracy No, what I wrote is an accurate reflection of Unicode. It does not matter what Jean Dupont and Hong Gildong think, but what we agreed on as a standard. > They do not exist as standalone code points in the binary form of the encoded text This terminology is confused/conceptually wrong. The binary form of encoded text consists of bytes. Decoded text consists of characters/grapheme clusters. A code point is the number corresponding to a character in the repertoire tables. By definition a code point cannot exist in encoded text. The correct way to determine whether sœur contains œ or o is to use a standard compliant implementation. Perl has always been nothing short of excellent, so: $ perl -Mutf8 -E'say "sœur" =~ "œ" ? "yes" : "no"' yes $ perl -Mutf8 -E'say "sœur" =~ "o" ? "yes" : "no"' no > don't let that anger control you I'm not angry. Projecting much?
  - 5y ago