4 ms·
cat 100GB_data_file_with_DUPLICATE_lines | parallel --pipe awk \'\!a[\$0]++\' > data_file_with_UNIQUE_lines is BETTER since it uses less computer resources tha
by known 10y ago
cat 100GB_data_file_with_DUPLICATE_lines | parallel --pipe awk \'\!a[\$0]++\' > data_file_with_UNIQUE_lines
is BETTER since it uses less computer resources than
gawk '!a[$0]++' 100GB_data_file_with_DUPLICATE_lines > data_file_with_UNIQUE_lines
- snvzz 10y agoThat's a classic misuse of cat. Just use '<'.
- kamijo-k 10y agoYeah, but to me, this "misuse" actually makes the input and output clearer than using "<" and ">" together. After all, it places input on the left side of the command, and output on the right side.
- ole_tange 10y agoExcept for GNU Parallel --pipe. When you use '<' what you typically mean is '--pipepart -a file'.
- decisiveness 10y agoThe parallel example will definitely not work as you expect, and will likely result in most duplicate lines still being present. When using --pipe this way, if you don't declare --block to be the size of the file (in which case there's no benefit to using parallel), each parallel execution will be run on a separate 1MB (default --block size) chunk of the file before outputting results all separately, then together in a single group (stdout), to the output file. If you're looking to spread work across CPUs and correctly get the desired output, I'd do something like: parallel -a input.txt --pipepart mawk \'\!a[\$0]++\' | mawk '!a[$0]++' > output.txt I used mawk because it is typically much more performant on large files.