5 ms·
Out of curiosity (and with the danger of hijacking this thread as a stackoverflow discussion), I changed the awk code so that it could be used with GNU parallel
by e12e 9y ago
Out of curiosity (and with the danger of hijacking this thread as a stackoverflow discussion), I changed the awk code so that it could be used with GNU parallel - the "reduce"-step is essentially the same program as before:
(cat ngrams.tsv \
|parallel --pipe awk -f map.awk \
|awk -f reduce.awk )
max_key: 2006 sum: 22569013
This now runs in 17 to 18 seconds... :-/
$ cat map.awk
{ a[$2] += $3 }
END {
for (i in a) {
print "ignore", i, a[i]
}
}
$ cat reduce.awk
{ a[$2] += $3 }
END {
for (i in a) {
if (a[i] > max) {
maxk = i
max=a[i]
}
}
print "max_key:", maxk, "sum:", max
}
[ed: However, there are faster awks than gawk:
$ time mawk -f sum.awk ngrams.tsv
max_key: 2006 sum: 22569013
real 0m2.826s
user 0m2.391s
sys 0m0.422s
mawk is (a little) faster than pypy on my machine.
]
- smitherfield 9y ago> GNU parallel Which is a 15000-line Perl script, so to be expected.
- ole_tange 9y ago--pipe is well know for being slow. Try --pipe-part instead: parallel -a ngrams.tsv --pipe-part --block -1 awk -f map.awk | awk -f reduce.awk
- e12e 9y agoThanks for the tip, always nice to see the author of tools commenting on hn :-) The (old) version of parallel packaged with Ubuntu 16.04 (linux subsystem for windows) - doesn't have --pipe-part -- but running from upstream, the speed is more reasonable: $ time (./parallel-20170522/src/parallel -a ngrams.tsv \ --pipe-part --block -1 -j4 mawk -f map.awk \ | mawk -f reduce.awk ) max_key: 2006 sum: 22569013 real 0m2.265s user 0m4.672s sys 0m1.672s (Tried a few variants with/without -jN -- and this seems typical for the fast end of the spectrum). $ time (cat ngrams.tsv \ | mawk -f map.awk \ | mawk -f reduce.awk ) max_key: 2006 sum: 22569013 real 0m3.472s user 0m2.891s sys 0m2.406s [ed: btw, did a double-take when I saw your Gnu Privacy Guard id: 0x88888888 :-) ]
- ole_tange 9y ago--line-buffer may or may not give additional speed up.