8 ms·
Xargs is a nearly drop in replacement and probably already installed by default in most distros. You may need the -n 1 (one file per) and -P to parallelize.
by wiredfool 3y ago
Xargs is a nearly drop in replacement and probably already installed by default in most distros. You may need the -n 1 (one file per) and -P to parallelize.
xargs -n 1 -P 8
- c-hendricks 3y agofind + xargs has become my go-to "process files in parallel". Tho now I'm wondering if I should be using `-n` instead of `-L` #!/usr/bin/env bash set -e main() { if [ "$1" = "handle-file" ]; then shift handle-file "$@" else find . \ -type f \ -not -path '*/optimized/*' \ -print0 \ | xargs \ -0 \ -L 1 \ -P 8 \ -I {} \ bash -c "cd \"$PWD\" && \"$0\" handle-file \"{}\"" fi } handle-file() { echo "handle-file $1 ..." } main "$@"
- wiredfool 3y agoHuh, I hadn't seen the -L before. Looks pretty similar.
- indymike 3y agoActually, parallel is a drop in for xargs as xargs has been around longer. Parallel has a few big improvements: * Grouped output (prevents one process from writing output in the middle of another's output) * In-order output (task a output first, task b output second even though they ran in parallel) * Better handling of special characters * Remote execution More here: https://www.gnu.org/software/parallel/parallel_alternatives.html#DIFFERENCES-BETWEEN-xargs-AND-GNU-Parallel https://www.gnu.org/software/parallel/parallel_alternatives....