2 ms·
Very interesting post -- dealing with unicode gets tricky. Because dealing with a full character repertoire is tricky, not because unicode does it poorly (unico
by jrochkind1 13y ago
Very interesting post -- dealing with unicode gets tricky. Because dealing with a full character repertoire is tricky, not because unicode does it poorly (unicode actually does it well).
I wonder if Unicode UTR#30 would be an alternative (more reliable?) method of idempotent normalization? http://www.unicode.org/reports/tr30/tr30-4.html http://www.unicode.org/reports/tr30/tr30-4.html
Or maybe the twisted algorithm really is UTR#30, just not labelled that?
As far as I can tell, UTR#30 did not make it to formally being part of the unicode spec, for reasons I'm not entirely clear on -- it is nonetheless quite useful, and this case is an example. Solr for instance still uses it. (http://wiki.apache.org/solr/AnalyzersTokenizersTokenFilters#solr.ICUFoldingFilterFactory http://wiki.apache.org/solr/AnalyzersTokenizersTokenFilters#...).
It might be a pain to find code implementing UTR#30 in your language of choice though (I am not sure if it's part of current ICU libraries or not).
It's also worth pointing out, that in addition to this kind of 'folding' of different-but-look-the-same graphemes, in this sort of use case you ABSOLUTELY need to do byte normalization as per UAX#15 http://unicode.org/reports/tr15/ http://unicode.org/reports/tr15/ . Probably NFKC for this sort of use case.
- jrochkind1 13y agoHuh, and another alternative would be using one of the unicode collation algorithms for normalization -- which, unlike UAX#15, did make it beyond draft stage to be an official part of the unicode spec. http://www.unicode.org/reports/tr30/tr30-4.html#_Toc23 http://www.unicode.org/reports/tr30/tr30-4.html#_Toc23