3 ms·
If you actually want to be UCA-compliant, though, it won't really help you to use codepoints. Different Unicode normalization, contractions and ignorables reall
by Sesse__ 5y ago
If you actually want to be UCA-compliant, though, it won't really help you to use codepoints. Different Unicode normalization, contractions and ignorables really mess up things. (In general, you can't search by case-folding if you want all the edge cases right; you have to apply the full Unicode collation algorithm.) I wonder if it's possible to store UCA weights instead of code points in the trigrams, but I haven't thought deeply about it, and it would surely affect indexing speed.
In plocate, where I have this exact same issue, I solve it half-heartedly by using 3-byte trigrams and then constructing a whole bunch of alternatives, even if they differ in width. (I don't support accent-insensitive search and I don't support contractions, so locate ß won't find SS.) See https://git.sesse.net/?p=plocate;a=blob;f=parse_trigrams.cpp;h=678f520fd03cc7c11c63b3cae7f77e712196c56f;hb=HEAD#l197 https://git.sesse.net/?p=plocate;a=blob;f=parse_trigrams.cpp... — but it doesn't support the Kelvin case if it's in the filename and you give K on the command line.