4 ms·
Don't do this: ls | parallel -m -j $f "cat {} >> ../transactions_cat/transactions.csv" It only works if all files enumerated by `ls` are smaller than `PIP
by sigil 11y ago
Don't do this:
ls | parallel -m -j $f "cat {} >> ../transactions_cat/transactions.csv"
It only works if all files enumerated by `ls` are smaller than `PIPE_BUF`, which is 4096 bytes on Linux (but was historically only 512 bytes). Writes larger than `PIPE_BUF` are not guaranteed to be atomic. Meaning, if you have input files larger than 512 or 4096 bytes, your output file will have chunks of file A interleaved with chunks of file B.
If I knew for sure all input files were smaller than PIPE_BUF, I'd do this instead:
find . -type f -print0 | xargs -0 -P8 cat > ../transactions.csv
That saves you the overhead of firing up a new shell just to reopen a file for append and exec `cat`.
- coderzach 11y agoI don't think that's the case, the default behavior of gnu parallel is to buffer the entire output before outputting. The output will only be interleaved if the `--ungroup` option is used. Parallel also has --line-buffer, which will interleave lines, rather than arbitrary bytes.
- sigil 11y agoAre you suggesting that the command argument to `parallel` is shell that's parsed and reinterpreted by `parallel`? Buffering the output would make more sense, but the OP's command isn't outputting to stdout, but rather appending directly to a file.
- yeahrightfam 11y agoYep it's flawed. That made read the rest of the post with a "meh" attitude instead of my initial "this could be cool" mindset.