6 ms·
sort -u <filename> would be faster than all that piping
by naz 16y ago
sort -u <filename>
would be faster than all that piping
- secretasiandan 16y agoI've never used it but there's a -S flag for sort that allows you to specify buffer size. So maybe look into increasing that as large as you can.
- sqrt17 16y ago-S is quite useful. For bigger sorting tasks, I usually use -S50% to speed up the speed of sorting. (sort's default memory size must be about 1MB). It may also help to use compression with the temporary files (something like "--compress-program lzop"), but I've never tried that.
- jemfinch 16y agoCompared to the IO costs, the computational difference must be inconsequential.
- jey 16y agoRight, but sort is apparently clever enough to do a disk-backed mergesort on a file (according to timr's comment), but it doesn't get a chance to recognize that you're sorting a file if you just pipe in the data.
- ori_b 16y agoI believe it uses scratch space in /tmp to do it if you pipe it in. (The manpage says that it allows you to specify an alternative temporary directory than $TMPDIR, however)
- jemfinch 16y agoWhy did you get modded up? You're wrong. Sort will write temporary files to /tmp if you exceed its in-memory sort threshold. Did you really think sort(1) just bails if you pipe it more data than can be contained in memory?