4 ms·
> perl -MList::Util=uniq -e 'print uniq <>' This one is interesting, because I've come across this issue many times where "uniq" will not work the way you perh
by hnarn 6y ago
> perl -MList::Util=uniq -e 'print uniq <>'
This one is interesting, because I've come across this issue many times where "uniq" will not work the way you perhaps expect it to (if you, like me, don't read "man" first), and will give back unique values from lines later in the file multiple times. Normally I've found that the way to work around this is to sort the output first before passing it to uniq, so I was curious if the perl way was faster or not.
I wrote 10M random numbers between 1-3 to a text file and tried this benchmark:
> time cat hn.txt | sort -n | uniq
real 0m3,929s
> time cat hn.txt | perl -MList::Util=uniq -e 'print uniq <>'
real 0m2,205s
Almost twice as fast. Lesson learned I guess, don't knock on Perl for CLI wrangling :)
I can share one of my frequently used ones as well: if you have a log file with an initial timestamp that is UNIX epoch, you can pipe the text to this perl command to convert it to localtime:
> cat logfile | perl -pe 's/(\d+)/localtime($1)/e'
(I'm sure you could do this with something other than Perl as well, but it does the job well)
- asicsp 6y agoIf you remove `cat`, it gives me nearly identical run time on my machine. I used `shuf -r -n10000000 -i 1-3 > t1` as input. Also, output won't be same, as the perl solution retains input order. `sort -nu` is faster than `sort+uniq` combo. perl -ne 'print if !$h{$_}++' runs faster (but this might depend on input, not sure if it is always faster than uniq) than the `sort -u` or `sort+uniq` combo what's surprising to me is that this was faster than awk '!a[$0]++'
- tyingq 6y agoTry using mawk instead of GNU awk. It's much faster than either Perl or gawk in this scenario for me. Roughly 2x. Also, "export LC_ALL=C" typically speeds up gawk quite a lot.
- asicsp 6y agoI did try LC_ALL=C with gawk, but that didn't give any improvement for this case. mawk indeed was much faster
- tzs 6y ago(You sound like someone who would probably know this, but a lot of people don't). Note that in general "sort -nu" is not equivalent to "sort -n | uniq", although in this particular case they are. If you have more than one field on the line, they can differ. For example, if the input is: 3 first 2 second 3 second 1 second 2 first 1 first then this is the "sort -nu" output: 1 second 2 second 3 first whereas "sort -n | uniq" gives: 1 first 1 second 2 first 2 second 3 first 3 second I'm a bit confused as to how "sort -nu" chooses which record to print when there are more than one with the same key. I would have expected the chosen record to be the one that would have come first in "sort -n" output, so that "sort -n | sort -nu" would be equivalent to "sort -nu", but that is not the case. The pipe gives: 1 first 2 first 3 first It is as if -nu automatically adds -s.
- asicsp 6y agomanual for -u option says "output only the first of an equal run" which is effectively same as what -s option does
- tyingq 6y agoMight be interesting to see how it fares vs "sort -u".