3 ms·
For me sort|uniq is nine times slower than LANG=C sort -u but `sort -u` on its own is only marginally faster than sort|uniq. I can't remember exactly what
by repsilat 7y ago
For me sort|uniq is nine times slower than
LANG=C sort -u
but `sort -u` on its own is only marginally faster than sort|uniq.
I can't remember exactly what LANG=C does, but I think the it makes sort not need to do some fancy unicode stuff? If the person writing the article just needs to uniqify IP addresses they should use it.
- majke 7y agoOh boy. This is drastic. Thanks for great advice: marek:~$ time (cat logs-popcount-org.txt | sort -u | wc -l) 39057531 real 2m37.387s user 2m35.626s sys 0m2.937s marek:~$ time (cat logs-popcount-org.txt | LANG=C sort -u -S6G | wc -l) 39057531 real 0m12.908s user 0m42.826s sys 0m3.586s
- perl4ever 7y agoI think I used to use "env LC_ALL=C". I like to set "all" the things and I'm also fond of the useless use of env... This does run faster, but it's also important if you want a predictable order and to distinguish all strings like you would get if you implemented your own text sort naively. From Googling just now: "sort -u doesn't report unique lines, but one of each group of lines that have equal sorting order. So if you do want unique lines, you need a locale where characters are byte and all characters have different sorting order (which the C locale guarantees)."