3 ms·
It looks like GNU sort already does this internally, but with smarter temp file management. Increasing the memory size looks like a good idea, and if you have m
by pjscott 16y ago
It looks like GNU sort already does this internally, but with smarter temp file management. Increasing the memory size looks like a good idea, and if you have multiple hard drives, it may help to specify a temporary directory on a different hard drive with the -T option, since this problem looks IO-bound.
- pbh 16y agoI believe GNU sort is a single-threaded external multi-pass/multi-way mergesort. http://en.wikipedia.org/wiki/External_sorting http://en.wikipedia.org/wiki/External_sorting The reason to split is not to replicate mergesort, but rather to get around single threading in GNU sort. Of course, there are better options if this is a regular task, but GNU sort just happens to be really common and easy if it's a one-off. I do wish I had a better split (since split seems to read the whole file) and a better sort (that was parallel and reasonably common) though.