4 ms·
My point was simply that the fact that there are multiple representations of characters that look the same is just a tiny part of the complexity involved in mak
by renhanxue 3y ago
My point was simply that the fact that there are multiple representations of characters that look the same is just a tiny part of the complexity involved in making text behave like users want. It's not that uncommon for people to think that "oh I'll just normalize the string and that'll solve my problems", but normalization is just a small part of quote-unquote "proper" Unicode handling.
The "proper" way of sorting and comparing Unicode strings is part of the standard; it's called the Unicode Collation Algorithm (https://unicode.org/reports/tr10/ https://unicode.org/reports/tr10/). It is unwieldy to say the least, but it is tuneable (see the "Tailoring" part) and can be used to implement o/ö equivalence if desired. I think it's great that this algorithm (and its accompanying Common Locale Data Repository) is in the standard and maintained by the consortium, because I definitely wouldn't want to maintain those myself.