2 ms·
If you're just looking to parallelise an operation over some file then GNU Parallel is an fantastic tool as well. On several occasions recently I've combined th
by jon-wood 12y ago
If you're just looking to parallelise an operation over some file then GNU Parallel is an fantastic tool as well. On several occasions recently I've combined the split command and Parallel to break a large CSV file up into smaller chunks, and then run a Ruby process on each of those chunks.
Parallel is apparently also able to distribute a command over several hosts using SSH, although I've not tried that one.
- icebraining 12y agoDistributed over SSH is awesome. Parallel can push the input files, run the command and pull the output (and then cleanup) all by itself.
- JensRantil 12y agoParallel is awesome. The only problem I have with it is that it doesn't support data localization. If you have 500 GB of data you don't want to copy it to a machine to run a command on it. Better to have the data stored on your cluster directly, which is what FileMap does.