3 ms·

>The very same problem you have with Turkish. Not really the same. The brokenness of more complex text transformations (collation, alphabetization, upper-casin

>The very same problem you have with Turkish. Not really the same. The brokenness of more complex text transformations (collation, alphabetization, upper-casing, lower-casing) seems a bit less catastrophic than breaking the fundamental ability to unambiguously draw the character. If you don't think that's an important goal.... then what are "character sets" for anyway?

bombcar 5y ago
That brings up the philosophical problem - is the English word taco the same as the Spanish word taco and if they aren’t the same, are the characters the same, or should there be an a for every language?
- samatman 5y ago
  Which semantic HTML at least attempts to solve with the `span lang` construct, although "taco" is maybe not the ideal example, but saying e.g. <span lang=fr>Jean</span> could in principle tell a screen reader to pronounce this name correctly, rather than the equally-correct English pronunciation which is a homophone of "gene". This is another question which doesn't belong in the Unicode character specification, but should be resolved when necessary at a higher level of abstraction.
  - bombcar 5y ago
    Exactly. Unicode’s “job” is to provide a code point for every letter - which necessary involves some trade offs such as whether a character is duplicated or not depending on the history of the region (which depends on all sorts of historical accidents such as whether keyboards were even available). People shouldn’t expect to know what language a Unicode string is in without external help, even if heuristics work often.
    - nanis 5y ago
      > Exactly. Unicode’s “job” is to provide a code point for every letter And in this case Unicode fails catastrophically by not providing codepoints for the uppercase of "ı" and the lowercase of "İ" while potentially allowing for ethnically sensitive renderings of shit.
samatman 5y ago
There is no such breakage at all, the assumption that casing and collation can happen without locale-specific information is invalid, Unicode isn't designed to prevent this and we'd hate it if it was. This may not be a solved problem in a given programming language's standard library, and/or one might see a bug such as the one GP was reporting, but neither of these conditions is inevitable, here's how you do it in Swift: "i".uppercased(with: Locale(identifier: "tr_TR")) // returns "İ" "i".uppercased(with: Locale(identifier: "en_US")) // returns "I" Same deal with "ij".uppercased and "nl_NL" locale. It's not a bug that Unicode just threw its hands up and said that complex string operations are locale-specific. The more you learn about the various scripts it represents, the more you will appreciate that there is no other choice.
- schiffern 5y ago
  I don't disagree, actually. IMO "the brokenness of more complex text transformations" is fundamental, not a 'bug.' Hopefully this makes what I meant a bit more clear.
- nanis 5y ago
  That is baloney ... The fact is Unicode could have provided extra codepoints so that at the least the possibility of correctly processing a mainly French document that includes passages in Turkish at least existed. The current state of affairs categorically rules that out it contrary to the situation with any other language.

bombcar 5y ago

That brings up the philosophical problem - is the English word taco the same as the Spanish word taco and if they aren’t the same, are the characters the same, or should there be an a for every language?

samatman 5y ago
Which semantic HTML at least attempts to solve with the `span lang` construct, although "taco" is maybe not the ideal example, but saying e.g. <span lang=fr>Jean</span> could in principle tell a screen reader to pronounce this name correctly, rather than the equally-correct English pronunciation which is a homophone of "gene". This is another question which doesn't belong in the Unicode character specification, but should be resolved when necessary at a higher level of abstraction.
- bombcar 5y ago
  Exactly. Unicode’s “job” is to provide a code point for every letter - which necessary involves some trade offs such as whether a character is duplicated or not depending on the history of the region (which depends on all sorts of historical accidents such as whether keyboards were even available). People shouldn’t expect to know what language a Unicode string is in without external help, even if heuristics work often.
  - nanis 5y ago
    > Exactly. Unicode’s “job” is to provide a code point for every letter And in this case Unicode fails catastrophically by not providing codepoints for the uppercase of "ı" and the lowercase of "İ" while potentially allowing for ethnically sensitive renderings of shit.

samatman 5y ago

There is no such breakage at all, the assumption that casing and collation can happen without locale-specific information is invalid, Unicode isn't designed to prevent this and we'd hate it if it was. This may not be a solved problem in a given programming language's standard library, and/or one might see a bug such as the one GP was reporting, but neither of these conditions is inevitable, here's how you do it in Swift: "i".uppercased(with: Locale(identifier: "tr_TR")) // returns "İ" "i".uppercased(with: Locale(identifier: "en_US")) // returns "I" Same deal with "ij".uppercased and "nl_NL" locale. It's not a bug that Unicode just threw its hands up and said that complex string operations are locale-specific. The more you learn about the various scripts it represents, the more you will appreciate that there is no other choice.

schiffern 5y ago
I don't disagree, actually. IMO "the brokenness of more complex text transformations" is fundamental, not a 'bug.' Hopefully this makes what I meant a bit more clear.
nanis 5y ago
That is baloney ... The fact is Unicode could have provided extra codepoints so that at the least the possibility of correctly processing a mainly French document that includes passages in Turkish at least existed. The current state of affairs categorically rules that out it contrary to the situation with any other language.