7 ms·
I agree with much of what you said here, but 'character', and therefore indexing by character, isn't a well behaved concept in the unicode world. Normalization
by bradleyjg 8y ago
I agree with much of what you said here, but 'character', and therefore indexing by character, isn't a well behaved concept in the unicode world. Normalization helps somewhat, though it's another expense at string creation time. However, even after normalization there are plenty of graphemes that don't have a precomposed characters.
Our fundamental ideas about how a string should work and what kinds of things we might want to do with one are deeply influenced by the languages spoken by the pioneers of the computer age. Short of someone from a very different language tradition not being exposed to these implicit assumptions in their formative programming years I don't think we are ever going to know what a different path would look like.
- jfk13 8y ago> Normalization helps somewhat, though it's another expense at string creation time And needs to be handled with care...there are edge cases where it can bite you. For example, if Unicode is being used internally to process data in a legacy CJK encoding, normalisation may lose distinctions that are needed for accurate round-trip conversion. Another surprise "gotcha" is that simply concatenating two already-normalised strings may give you a result that is not normalised.