4 ms·
Sort a file in "parallel" (note quotes!) in a multicore machine (note I'm showing 2 cores but this could be generalized to more): mkfifo split_aa mkfif
by smlacy 13y ago
Sort a file in "parallel" (note quotes!) in a multicore machine (note I'm showing 2 cores but this could be generalized to more):
mkfifo split_aa
mkfifo split_ab
mkfifo sorted_aa
mkfifo sorted_ab
split -n2 input_file split_ &
sort split_aa >> sorted_aa &
sort split_ab >> sorted_ab &
sort -m sorted_aa sorted_bb > final_sorted
It's left as an exercise for the reader to determine if this actually saves any time, and if this technique could be used to implement a truly parallel sort.
- timClicks 13y agoWhat about? sort --parallel=2 input_file
- smlacy 13y agoWell, yeah, that's the easy way. :) This is mostly to illustrate a technique of "parallel processing using fifos and split input" It's almost like a miniature mapreduce running locally. See also "bashreduce" (https://github.com/erikfrey/bashreduce https://github.com/erikfrey/bashreduce) which I didn't write.