4 ms·

This is an outcome of the historic process which gave us modern Unicode. The Turkish alphabet ISO 8859-9 was published in 1988, and it doesn't distinguish betw

This is an outcome of the historic process which gave us modern Unicode. The Turkish alphabet ISO 8859-9 was published in 1988, and it doesn't distinguish between I and I, since why would it? It's a byte encoding designed specifically to accommodate Turkish. Unicode has a principle of adopting whatever makes it easy to convert from the preferred legacy encoding to Unicode, and for Turkish that meant using the same code point for I in both Turkish and practically everywhere else, since I is in ASCII and the bulk of Latin-based character encodings are based on ASCII. I point this out because it could indeed have been the case that they ended up with separate encodings, which would have solved your problem. But it wouldn't have solved the actual problem, which is the collation and casing are locale-specific according to Unicode, and indeed they must be. Example: in Dutch the letters 'ij' are one letter for collation purposes, as well as for casing, see https://en.wikipedia.org/wiki/IJsselstein https://en.wikipedia.org/wiki/IJsselstein for an example. This is of course impossible to get correct if the application doesn't know it's dealing with Dutch, which it often can't know. The very same problem you have with Turkish.

schiffern 5y ago
>The very same problem you have with Turkish. Not really the same. The brokenness of more complex text transformations (collation, alphabetization, upper-casing, lower-casing) seems a bit less catastrophic than breaking the fundamental ability to unambiguously draw the character. If you don't think that's an important goal.... then what are "character sets" for anyway?
- bombcar 5y ago
  That brings up the philosophical problem - is the English word taco the same as the Spanish word taco and if they aren’t the same, are the characters the same, or should there be an a for every language?
  - samatman 5y ago
    Which semantic HTML at least attempts to solve with the `span lang` construct, although "taco" is maybe not the ideal example, but saying e.g. <span lang=fr>Jean</span> could in principle tell a screen reader to pronounce this name correctly, rather than the equally-correct English pronunciation which is a homophone of "gene". This is another question which doesn't belong in the Unicode character specification, but should be resolved when necessary at a higher level of abstraction.
    - bombcar 5y ago
      Exactly. Unicode’s “job” is to provide a code point for every letter - which necessary involves some trade offs such as whether a character is duplicated or not depending on the history of the region (which depends on all sorts of historical accidents such as whether keyboards were even available). People shouldn’t expect to know what language a Unicode string is in without external help, even if heuristics work often.
      - nanis 5y ago
        > Exactly. Unicode’s “job” is to provide a code point for every letter And in this case Unicode fails catastrophically by not providing codepoints for the uppercase of "ı" and the lowercase of "İ" while potentially allowing for ethnically sensitive renderings of shit.
- samatman 5y ago
  There is no such breakage at all, the assumption that casing and collation can happen without locale-specific information is invalid, Unicode isn't designed to prevent this and we'd hate it if it was. This may not be a solved problem in a given programming language's standard library, and/or one might see a bug such as the one GP was reporting, but neither of these conditions is inevitable, here's how you do it in Swift: "i".uppercased(with: Locale(identifier: "tr_TR")) // returns "İ" "i".uppercased(with: Locale(identifier: "en_US")) // returns "I" Same deal with "ij".uppercased and "nl_NL" locale. It's not a bug that Unicode just threw its hands up and said that complex string operations are locale-specific. The more you learn about the various scripts it represents, the more you will appreciate that there is no other choice.
  - schiffern 5y ago
    I don't disagree, actually. IMO "the brokenness of more complex text transformations" is fundamental, not a 'bug.' Hopefully this makes what I meant a bit more clear.
  - nanis 5y ago
    That is baloney ... The fact is Unicode could have provided extra codepoints so that at the least the possibility of correctly processing a mainly French document that includes passages in Turkish at least existed. The current state of affairs categorically rules that out it contrary to the situation with any other language.
nanis 5y ago
> This is an outcome of the historic process which gave us modern Unicode. That historic process does not preclude introducing two extra codepoints to at least allow for the possibility of correctly encoding the upper case of "ı" and the lower case of "İ".
- samatman 5y ago
  You're talking about a historic process using the present tense, forgivable in an ESL speaker, except clearly the intention of the sentence requires the present tense! The past isn't something you can argue with, no more than the wind.

schiffern 5y ago

>The very same problem you have with Turkish. Not really the same. The brokenness of more complex text transformations (collation, alphabetization, upper-casing, lower-casing) seems a bit less catastrophic than breaking the fundamental ability to unambiguously draw the character. If you don't think that's an important goal.... then what are "character sets" for anyway?

bombcar 5y ago
That brings up the philosophical problem - is the English word taco the same as the Spanish word taco and if they aren’t the same, are the characters the same, or should there be an a for every language?
- samatman 5y ago
  Which semantic HTML at least attempts to solve with the `span lang` construct, although "taco" is maybe not the ideal example, but saying e.g. <span lang=fr>Jean</span> could in principle tell a screen reader to pronounce this name correctly, rather than the equally-correct English pronunciation which is a homophone of "gene". This is another question which doesn't belong in the Unicode character specification, but should be resolved when necessary at a higher level of abstraction.
  - bombcar 5y ago
    Exactly. Unicode’s “job” is to provide a code point for every letter - which necessary involves some trade offs such as whether a character is duplicated or not depending on the history of the region (which depends on all sorts of historical accidents such as whether keyboards were even available). People shouldn’t expect to know what language a Unicode string is in without external help, even if heuristics work often.
    - nanis 5y ago
      > Exactly. Unicode’s “job” is to provide a code point for every letter And in this case Unicode fails catastrophically by not providing codepoints for the uppercase of "ı" and the lowercase of "İ" while potentially allowing for ethnically sensitive renderings of shit.
samatman 5y ago
There is no such breakage at all, the assumption that casing and collation can happen without locale-specific information is invalid, Unicode isn't designed to prevent this and we'd hate it if it was. This may not be a solved problem in a given programming language's standard library, and/or one might see a bug such as the one GP was reporting, but neither of these conditions is inevitable, here's how you do it in Swift: "i".uppercased(with: Locale(identifier: "tr_TR")) // returns "İ" "i".uppercased(with: Locale(identifier: "en_US")) // returns "I" Same deal with "ij".uppercased and "nl_NL" locale. It's not a bug that Unicode just threw its hands up and said that complex string operations are locale-specific. The more you learn about the various scripts it represents, the more you will appreciate that there is no other choice.
- schiffern 5y ago
  I don't disagree, actually. IMO "the brokenness of more complex text transformations" is fundamental, not a 'bug.' Hopefully this makes what I meant a bit more clear.
- nanis 5y ago
  That is baloney ... The fact is Unicode could have provided extra codepoints so that at the least the possibility of correctly processing a mainly French document that includes passages in Turkish at least existed. The current state of affairs categorically rules that out it contrary to the situation with any other language.

nanis 5y ago

> This is an outcome of the historic process which gave us modern Unicode. That historic process does not preclude introducing two extra codepoints to at least allow for the possibility of correctly encoding the upper case of "ı" and the lower case of "İ".

samatman 5y ago
You're talking about a historic process using the present tense, forgivable in an ESL speaker, except clearly the intention of the sentence requires the present tense! The past isn't something you can argue with, no more than the wind.