3 ms·

>time (cat logs.txt | sort | uniq > /dev/null) If I skip two useless pipes, and use "sort -u logs.txt > /dev/null" instead, I'm already twice as fast as origin

by Hitton 7y ago

>time (cat logs.txt | sort | uniq > /dev/null) If I skip two useless pipes, and use "sort -u logs.txt > /dev/null" instead, I'm already twice as fast as original (it seems that piping to sort effectively prevents parallelization).

majke 7y ago
`sort -u` is slower for me. Doing cat from left to right helps readability - it's important. Performance-wise cat gives you 64KiB blocks of data, while direct pipe can give more. My programs (mmuniq-*) use 512KiB input buffer, so indeed with redirection you can reduce the number of read/write syscalls 8x, but it doesn't change much of the timing frankly. Parallelization is an interesting aspect, which we didn't discuss really.
- adrianmonk 7y ago
  > Doing cat from left to right helps readability - it's important. People may balk because it's unfamiliar, but this is syntactically legal: < logs.txt sort | uniq > /dev/null That is, the redirection customarily goes at the end, but it doesn't have to. EDIT: Also, in this specific case, the "sort" command can take a file argument, so you can also do this: sort logs.txt | uniq > /dev/null
repsilat 7y ago
For me sort|uniq is nine times slower than LANG=C sort -u but `sort -u` on its own is only marginally faster than sort|uniq. I can't remember exactly what LANG=C does, but I think the it makes sort not need to do some fancy unicode stuff? If the person writing the article just needs to uniqify IP addresses they should use it.
- majke 7y ago
  Oh boy. This is drastic. Thanks for great advice: marek:~$ time (cat logs-popcount-org.txt | sort -u | wc -l) 39057531 real 2m37.387s user 2m35.626s sys 0m2.937s marek:~$ time (cat logs-popcount-org.txt | LANG=C sort -u -S6G | wc -l) 39057531 real 0m12.908s user 0m42.826s sys 0m3.586s
- perl4ever 7y ago
  I think I used to use "env LC_ALL=C". I like to set "all" the things and I'm also fond of the useless use of env... This does run faster, but it's also important if you want a predictable order and to distinguish all strings like you would get if you implemented your own text sort naively. From Googling just now: "sort -u doesn't report unique lines, but one of each group of lines that have equal sorting order. So if you do want unique lines, you need a locale where characters are byte and all characters have different sorting order (which the C locale guarantees)."

by Hitton 7y ago

majke 7y ago

`sort -u` is slower for me. Doing cat from left to right helps readability - it's important. Performance-wise cat gives you 64KiB blocks of data, while direct pipe can give more. My programs (mmuniq-*) use 512KiB input buffer, so indeed with redirection you can reduce the number of read/write syscalls 8x, but it doesn't change much of the timing frankly. Parallelization is an interesting aspect, which we didn't discuss really.

adrianmonk 7y ago
> Doing cat from left to right helps readability - it's important. People may balk because it's unfamiliar, but this is syntactically legal: < logs.txt sort | uniq > /dev/null That is, the redirection customarily goes at the end, but it doesn't have to. EDIT: Also, in this specific case, the "sort" command can take a file argument, so you can also do this: sort logs.txt | uniq > /dev/null

repsilat 7y ago

For me sort|uniq is nine times slower than LANG=C sort -u but `sort -u` on its own is only marginally faster than sort|uniq. I can't remember exactly what LANG=C does, but I think the it makes sort not need to do some fancy unicode stuff? If the person writing the article just needs to uniqify IP addresses they should use it.

majke 7y ago
Oh boy. This is drastic. Thanks for great advice: marek:~$ time (cat logs-popcount-org.txt | sort -u | wc -l) 39057531 real 2m37.387s user 2m35.626s sys 0m2.937s marek:~$ time (cat logs-popcount-org.txt | LANG=C sort -u -S6G | wc -l) 39057531 real 0m12.908s user 0m42.826s sys 0m3.586s
perl4ever 7y ago
I think I used to use "env LC_ALL=C". I like to set "all" the things and I'm also fond of the useless use of env... This does run faster, but it's also important if you want a predictable order and to distinguish all strings like you would get if you implemented your own text sort naively. From Googling just now: "sort -u doesn't report unique lines, but one of each group of lines that have equal sorting order. So if you do want unique lines, you need a locale where characters are byte and all characters have different sorting order (which the C locale guarantees)."