3 ms·
If you remove `cat`, it gives me nearly identical run time on my machine. I used `shuf -r -n10000000 -i 1-3 > t1` as input. Also, output won't be same, as the p
by asicsp 6y ago
If you remove `cat`, it gives me nearly identical run time on my machine. I used `shuf -r -n10000000 -i 1-3 > t1` as input. Also, output won't be same, as the perl solution retains input order. `sort -nu` is faster than `sort+uniq` combo.
perl -ne 'print if !$h{$_}++'
runs faster (but this might depend on input, not sure if it is always faster than uniq) than the `sort -u` or `sort+uniq` combo
what's surprising to me is that this was faster than
awk '!a[$0]++'
- tyingq 6y agoTry using mawk instead of GNU awk. It's much faster than either Perl or gawk in this scenario for me. Roughly 2x. Also, "export LC_ALL=C" typically speeds up gawk quite a lot.
- asicsp 6y agoI did try LC_ALL=C with gawk, but that didn't give any improvement for this case. mawk indeed was much faster
- tzs 6y ago(You sound like someone who would probably know this, but a lot of people don't). Note that in general "sort -nu" is not equivalent to "sort -n | uniq", although in this particular case they are. If you have more than one field on the line, they can differ. For example, if the input is: 3 first 2 second 3 second 1 second 2 first 1 first then this is the "sort -nu" output: 1 second 2 second 3 first whereas "sort -n | uniq" gives: 1 first 1 second 2 first 2 second 3 first 3 second I'm a bit confused as to how "sort -nu" chooses which record to print when there are more than one with the same key. I would have expected the chosen record to be the one that would have come first in "sort -n" output, so that "sort -n | sort -nu" would be equivalent to "sort -nu", but that is not the case. The pipe gives: 1 first 2 first 3 first It is as if -nu automatically adds -s.
- asicsp 6y agomanual for -u option says "output only the first of an equal run" which is effectively same as what -s option does