3 ms·
The same LC_ALL=C trick also speeds up GNU tools, sort in particular: $ shuf /var/log/syslog > shuf-syslog $ time sort shuf-syslog > /dev/null rea
by scottlamb 8y ago
The same LC_ALL=C trick also speeds up GNU tools, sort in particular:
$ shuf /var/log/syslog > shuf-syslog
$ time sort shuf-syslog > /dev/null
real 0m0.536s
user 0m0.870s
sys 0m0.031s
$ time LC_ALL=C sort shuf-syslog > /dev/null
real 0m0.094s
user 0m0.109s
sys 0m0.024s
It's more dramatic with a bigger file, of course.
I once got annoyed with sort's speed, threw together a parallel external sort program that dramatically outperformed it, then realized the difference was not as dramatic with LC_ALL=C...oh well, it was a fun afternoon program anyway.
- danudey 8y agoIn fairness, when I'm dealing with unicode data which is actually unicode data, I don't want to set `LC_ALL=C` and fuck up my data set.
- scottlamb 8y agoIt's certainly worth considering, but the answer depends on what you're doing with it. If your goal is to binary search over it for an exact string match, for example, the important thing is that your sorting and searching code are consistent. Might as well have both use the simpler/faster LC_ALL=C semantics.
- treffer 8y agoActually sorting should be fine. The order of UTF-8 will always be the same as the codepoints encoded under it if you sort it as binary. I really appreciate this property of UTF-8, and I can highly recommend to do a pen&paper exercise to see why it preserves order :-) It wouldn't work if you want any type of normalisation and especially solving composition/decomposition.
- burntsushi 8y agoLocale specific sorting is not the same as sorting by codepoint. The Unicode collation algorithm is fairly involved: https://www.unicode.org/reports/tr10/ https://www.unicode.org/reports/tr10/
- james_s_tayler 8y agoIf you are just using sort because you need to use uniq then you can replace the whole thing with an awk command that gets unique lines without sorting. I think it's faster.