6 ms·

The basic answer here is that Unicode exists to encode characters, or really, scripts and their characters. Not typefaces or fonts. Consider broadcasting of te

The basic answer here is that Unicode exists to encode characters, or really, scripts and their characters. Not typefaces or fonts. Consider broadcasting of text in Morse code. The Morse for the Cyrillic letter В is International Morse W. In the early years of Unicode, conversion from disparate encodings to Unicode was an urgent priority. Insofar as possible, they wanted to preserve the collation properties of those encodings, so the characters were in the same order as the original encoding whenever they could be. But it's more that Unicode encodes scripts, which have characters, it doesn't encode shapes. With 10,000 caveats to go with that, Unicode is messy and will preserve every mistake until the end of time. But encoding Α and A and А as three different letters, that they did on purpose, because they are three different letters, because they're a part of three different scripts.

schoen 3y ago
It occurs to me (after mentioning collation order, in a different part of this thread, as one reason that we would want to distinguish scripts) that it might be unclear even for collation purposes when scripts are or are not distinct, especially for Cyrillic, Latin, and Arabic scripts which are used to write many different languages which have often added their own extensions. I guess the official answer is "attempt to distinguish everything that any language is known to distinguish, and then use locales to implement different collation orders by language", or something like that? But it's still not totally obvious how one could make a principled decision about, say, whether the encoding of Persian and Urdu writing (obviously including their extensions) should be unified with the encoding of Arabic writing. One could argue that Nastaliq is like a "font"... or not...
- samatman 3y ago
  Characters in Unicode can have more than one script property, so the question "is this text entirely Bengali/Devanagari" can be answered even though they share characters. But Unicode encodes scripts, not languages, and not shapes. Many things we might want to do with strings require a locale property, which Unicode tried allowing as an inline representation, this was later deprecated. I'm not convinced that was the correct decision, but it is what it is. If you want to properly handle Turkish casing or Swedish collation, you have to know that the text you're working with is Turkish or Swedish, no way around it.

schoen 3y ago

It occurs to me (after mentioning collation order, in a different part of this thread, as one reason that we would want to distinguish scripts) that it might be unclear even for collation purposes when scripts are or are not distinct, especially for Cyrillic, Latin, and Arabic scripts which are used to write many different languages which have often added their own extensions. I guess the official answer is "attempt to distinguish everything that any language is known to distinguish, and then use locales to implement different collation orders by language", or something like that? But it's still not totally obvious how one could make a principled decision about, say, whether the encoding of Persian and Urdu writing (obviously including their extensions) should be unified with the encoding of Arabic writing. One could argue that Nastaliq is like a "font"... or not...

samatman 3y ago
Characters in Unicode can have more than one script property, so the question "is this text entirely Bengali/Devanagari" can be answered even though they share characters. But Unicode encodes scripts, not languages, and not shapes. Many things we might want to do with strings require a locale property, which Unicode tried allowing as an inline representation, this was later deprecated. I'm not convinced that was the correct decision, but it is what it is. If you want to properly handle Turkish casing or Swedish collation, you have to know that the text you're working with is Turkish or Swedish, no way around it.