3 ms·
Is it recognized as equivalent by all text-search programs? The magic words you want to look for are [Unicode canonicalization], which aspires to make that (an
by patio11 13y ago
Is it recognized as equivalent by all text-search programs?
The magic words you want to look for are [Unicode canonicalization], which aspires to make that (and other string-comparison needs) actually work. Implementation quality across the universe of programs is... mixed.
- Someone 13y agoCanonicalization is more restricted and different. More restricted in that it only treats truly identical strings that have multiple representations the same. Normalization won't turn "foo" and "FOO" into the same string, but it will turn "fòó" and "fo<with grave accent>o<with acute accent>" into the same string. Different in the sense that it creates a new string, rather than comparing two strings. Just like you neither need nor want to do a tolower(s) when comparing case-insensitively, you don't need nor want to normalize unicode to do a normalization invariant comparison. The unicode standard uses "equivalence" to treat "<fl ligature>" and "fl" as equivalent (see http://en.wikipedia.org/wiki/Unicode_equivalence; http://en.wikipedia.org/wiki/Unicode_equivalence; Unicode technical report on normalization at http://www.unicode.org/reports/tr15/tr15-18.html http://www.unicode.org/reports/tr15/tr15-18.html)