3 ms·
The sort | uniq method literally needs to sort the file and pipe it to uniq, a far more memory-intensive operation than the single-pass awk check. You can writ
by niggler 14y ago
The sort | uniq method literally needs to sort the file and pipe it to uniq, a far more memory-intensive operation than the single-pass awk check. You can write your own hash function in AWK if you think you may overstep memory, but of course you risk hash collisions. It's a tradeoff.
I tried it on a 1U server with 24GB ram a few years ago and found that the sort was thrashing at the 10GB file size while AWK handled it easily
- carlesfe 14y agoYes, awk does some real black bagic. It's awesome how it can parse really, really big files.
- stevvooe 14y agoSort will actually externally sort blocks into temp files and merge them. Adjusting this block size can help with thrashing. Awk may still be better for uniquification.
- enigmo 14y agoWhat about 'sort -u' ?
- stevvooe 14y agoI'm not sure. I haven't closely studied the difference between each algorithm. My guess would be that sort -u would perform better as the data set gets larger with a good block size setting because it does do an external sort. Cardinality would also affect the performance. If the unique set handily fits in memory, an external sort on a large data set wouldn't be very efficient.
- gav 14y agoOr you can use `sort -u` and not have to pipe to `uniq`.