3 ms·
> to an American, a user that searches for "cooperation" when your text data happens to include writings by people who write like in The New Yorker, would proba
by josephcsible 3y ago
> to an American, a user that searches for "cooperation" when your text data happens to include writings by people who write like in The New Yorker, would probably to expect to find "coöperation".
Unicode shouldn't be responsible for making such searches work, just like it's not responsible for making searches for "analyze" match text that says "analyse".
- renhanxue 3y agoMy point was simply that the fact that there are multiple representations of characters that look the same is just a tiny part of the complexity involved in making text behave like users want. It's not that uncommon for people to think that "oh I'll just normalize the string and that'll solve my problems", but normalization is just a small part of quote-unquote "proper" Unicode handling. The "proper" way of sorting and comparing Unicode strings is part of the standard; it's called the Unicode Collation Algorithm (https://unicode.org/reports/tr10/ https://unicode.org/reports/tr10/). It is unwieldy to say the least, but it is tuneable (see the "Tailoring" part) and can be used to implement o/ö equivalence if desired. I think it's great that this algorithm (and its accompanying Common Locale Data Repository) is in the standard and maintained by the consortium, because I definitely wouldn't want to maintain those myself.