3 ms·
It occurs to me (after mentioning collation order, in a different part of this thread, as one reason that we would want to distinguish scripts) that it might be
by schoen 3y ago
It occurs to me (after mentioning collation order, in a different part of this thread, as one reason that we would want to distinguish scripts) that it might be unclear even for collation purposes when scripts are or are not distinct, especially for Cyrillic, Latin, and Arabic scripts which are used to write many different languages which have often added their own extensions.
I guess the official answer is "attempt to distinguish everything that any language is known to distinguish, and then use locales to implement different collation orders by language", or something like that?
But it's still not totally obvious how one could make a principled decision about, say, whether the encoding of Persian and Urdu writing (obviously including their extensions) should be unified with the encoding of Arabic writing. One could argue that Nastaliq is like a "font"... or not...
- samatman 3y agoCharacters in Unicode can have more than one script property, so the question "is this text entirely Bengali/Devanagari" can be answered even though they share characters. But Unicode encodes scripts, not languages, and not shapes. Many things we might want to do with strings require a locale property, which Unicode tried allowing as an inline representation, this was later deprecated. I'm not convinced that was the correct decision, but it is what it is. If you want to properly handle Turkish casing or Swedish collation, you have to know that the text you're working with is Turkish or Swedish, no way around it.