4 ms·
Personally, I would run: split -d -C 1G FILE split.FILE # separate into 1G files on line boundaries # then, on separate cores sort -u split.FILE.00 -o so
by pbh 16y ago
Personally, I would run:
split -d -C 1G FILE split.FILE # separate into 1G files on line boundaries
# then, on separate cores
sort -u split.FILE.00 -o sort.FILE.00
sort -u split.FILE.01 -o sort.FILE.01
...
sort -u split.FILE.N -o sort.FILE.N
# then, on one core
sort -m -u sort.FILE.*
You should be aware that you may not get what you expect from sort unless you set LC_ALL="C". You should also pass "-S" to sort to set the main memory size, probably to a value like "1G". Of course, at some point when you're doing a lot of distributed processing you'll just want to use Hadoop, as other posters have noted.
- pjscott 16y agoIt looks like GNU sort already does this internally, but with smarter temp file management. Increasing the memory size looks like a good idea, and if you have multiple hard drives, it may help to specify a temporary directory on a different hard drive with the -T option, since this problem looks IO-bound.
- pbh 16y agoI believe GNU sort is a single-threaded external multi-pass/multi-way mergesort. http://en.wikipedia.org/wiki/External_sorting http://en.wikipedia.org/wiki/External_sorting The reason to split is not to replicate mergesort, but rather to get around single threading in GNU sort. Of course, there are better options if this is a regular task, but GNU sort just happens to be really common and easy if it's a one-off. I do wish I had a better split (since split seems to read the whole file) and a better sort (that was parallel and reasonably common) though.