5 ms·
In fairness, when I'm dealing with unicode data which is actually unicode data, I don't want to set `LC_ALL=C` and fuck up my data set.
by danudey 8y ago
In fairness, when I'm dealing with unicode data which is actually unicode data, I don't want to set `LC_ALL=C` and fuck up my data set.
- scottlamb 8y agoIt's certainly worth considering, but the answer depends on what you're doing with it. If your goal is to binary search over it for an exact string match, for example, the important thing is that your sorting and searching code are consistent. Might as well have both use the simpler/faster LC_ALL=C semantics.
- treffer 8y agoActually sorting should be fine. The order of UTF-8 will always be the same as the codepoints encoded under it if you sort it as binary. I really appreciate this property of UTF-8, and I can highly recommend to do a pen&paper exercise to see why it preserves order :-) It wouldn't work if you want any type of normalisation and especially solving composition/decomposition.
- burntsushi 8y agoLocale specific sorting is not the same as sorting by codepoint. The Unicode collation algorithm is fairly involved: https://www.unicode.org/reports/tr10/ https://www.unicode.org/reports/tr10/