3 ms·
The "parallel concatenation" sounds like a really bad idea, partly because you have no guarantee that the files won't get interleaved, partly because it's solvi
by mpweiher 11y ago
The "parallel concatenation" sounds like a really bad idea, partly because you have no guarantee that the files won't get interleaved, partly because it's solving the wrong problem.
There is virtually no benefit to the actual process of concatenation from the parallelisation, as the process is completely I/O bound (and again, concatenating things in parallel will only work by accident). The problem the parallelisation is solving is the overhead of starting cat for each individual file.
This overhead can be eliminated quite easily by the standard Unix find/xargs combination, which by default will do 5000 files at a time. At that point, you will almost certainly be maxing out your I/O devices, no parallel processing needed, and your files will all be in a predictable order with no interleaving.