8 ms·
Parallel processing with Unix tools
- enriquto 9y agodo not forget about "make", that will traverse a user-supplied tree of filenames in parallel
- pletnes 9y agoOr any job in general. Also, make 4.x comes with the handy -O flag which makes the output seem "serial" instead of intermixed between processes.
- dahart 9y agoAnd make -j will execute the rules & build targets in parallel. Once I built a general batch parallel system using make that could pause & resume parallel jobs by using each job's log file as the make target. Then I discovered gnu parallel and scrapped my project.
- gnufx 9y agoDependencies in batch resource managers for distributed parallel jobs (or simpler ones) are standard practice. https://arc.liv.ac.uk/SGE/htmlman/htmlman1/qmake.html https://arc.liv.ac.uk/SGE/htmlman/htmlman1/qmake.html is an old adaptation of GNU Make which works within a job.
- SamJson 9y agoFun fact from https://www.gnu.org/software/parallel/history.html https://www.gnu.org/software/parallel/history.html > [GNU Parallel] was originally a wrapper that generated a makefile and used make -j to do the parallelization.
- dahart 9y agoHa! Crazy, I didn't know that, thanks for the pointer. I recall having some issues with scaling to very large jobs, when I got lists of targets too long for make to deal with. (I think... I could be mis-remembering the details, but something in my pipeline would fail with really big batches.) Maybe parallel split away from make when it ran into similar issues, I wonder.
- anitil 9y agoGah story of my life - hours/days of grueling work trying to solve a problem, then someone say 'why don't you just use $tool_that_does_exactly_what_you_want_but_better?'. Code goes in the bin.
- pixelbeat 9y agoVery good point. I added a whole section to the article, implementing the counting lines example with `make -j`, which performs just as well as `xargs -P`
- mtreis86 9y agoQt multithreading http://doc.qt.io/qt-5/threads-technologies.html http://doc.qt.io/qt-5/threads-technologies.html Aria for downloads https://aria2.github.io/ https://aria2.github.io/ Pigz and pbzip2 for compression https://zlib.net/pigz/ https://zlib.net/pigz/ http://compression.ca/pbzip2/ http://compression.ca/pbzip2/
- gjvc 9y agoI have known situations where the compressed output of pbzip2 was not readable by some .NET library a customer was using (I'm afraid I can't remember which one). Fortunately an alternative multithreaded bzip2 implementation, http://lbzip2.org/ http://lbzip2.org/, did not suffer from this problem so just in case...
- vthriller 9y agoOn multiple occasions I found lbzip2[0] to be faster than pbzip2 [0] http://lbzip2.org http://lbzip2.org Edit: here's a quick sample (file is read from tmpfs on a 4-core i7-5500U): $ time lbzip2 < linux-4.13.3.tar > /dev/null real 0m17.410s user 1m7.799s sys 0m0.283s $ time pbzip2 < linux-4.13.3.tar > /dev/null real 0m30.556s user 1m57.557s sys 0m2.169s
- vthriller 9y agoAlso, with modern enough xz-utils you can just use `xz -T` for parallel compression.
- dahart 9y agoIf you're zipping multiple files, is it better to pbzip2 each file, or to parallel bzip2 them? You wouldn't want to parallel pbzip2, would you?
- vthriller 9y agoBy compressing multiple (large) files sequentially you'd be able to gradually free some disk space much sooner, and less free space will be needed to write compressed data to.
- alvil 9y agoI like it using newLISP :) ;calculate primes in a range (define (primes from to) (local (plist) (for (i from to) (if (= 1 (length (factor i))) (push i plist -1))) plist)) (set 'start (time-of-day)) ; start child processes (spawn 'p1 (primes 1 1000000)) (spawn 'p2 (primes 1000001 2000000)) (spawn 'p3 (primes 2000001 3000000)) (spawn 'p4 (primes 3000001 4000000)) ; wait for a maximum of 60 seconds for all tasks to finish ; returns true if all finished in time (sync 60000) ; p1, p2, p3 and p4 now each contain a lists of primes (println "time spawn: " (- (time-of-day) start)) (println "time simple: " (time (primes 1 4000000))) (exit)
- unixhero 9y agoOne day I have to learn how to use xargs
- limaoscarjuliet 9y agoYou and me, both of us. I once combined rpm -e with xargs rpm -ql to the tune of getting all packages uninstalled from the system in one swift command. It was a test system so all good. But I'm still not sure what I was thinking.
- pwg 9y agoThe simplest use case for xargs is that you have a list of filenames and you want to feed that list to a command that only accepts filenames as CLI parameters: I.e., 'find' generates a list of filenames. But 'stat' (for example) only accepts file names as parameters on the command line. You combine these two using 'xargs': find -type f -print0 | xargs -0 stat --format="%y\t%n" And you get an output of human readable modification times, a tab, the filename and a newline. Note, the "-print0" to find instructs find to output null terminated filenames, and the -0 to xargs instructs xargs to expect to receive null terminated filenames. These switches (they may be specific to GNU coreutils) allow for handling filenames with any possible character in them correctly (i.e., spaces cause no troubles). For this use case, remembering how/when to use xargs is easy. The more esoteric usages are the ones where "man xargs" is often handy.
- cat199 9y agoIt's pretty much 'apply' for shell arguments https://en.wikipedia.org/wiki/Apply https://en.wikipedia.org/wiki/Apply use it many places instead of a for loop: for x in ; do <cmd> $x done is: echo |xargs cmd Assuming 'cmd' can take multiple arguments. If not, use 'xargs -L 1 cmd' to run cmd 1x per arg. GNU xargs has -1, but -L 1 is portable to xargs in the BSDs and others.. plus it's good to know -L in case you ever need to -L 2, etc. using echo as a driver is not the best example, but anyhow.
- cat199 9y agoalso of note is: -t : print the commands being run -I %: interpolate file name in command string. e.g. ls .mp4 |xargs -t -L 1 -I % scp % me@somebox:/movies to scp individual files (replaced in %) to a destination, one at a time, printing each command as it is run. Again, not the best example since you can scp .mp4 me@sombeox:/movies or rsync, etc. but you get the idea.
- wyoh 9y agoRust Parallel is nice too: https://github.com/mmstick/parallel https://github.com/mmstick/parallel
- agumonkey 9y agoPerformance of the actual program might be less relevant but it's nice to see it using 1% for itself compared to GNU parallel.
- SamJson 9y agoThe performance is apparently close to xargs (if speed is key, why not just use xargs?). Rust parallel does, however, have some issues that would rule it out for me: https://www.gnu.org/software/parallel/parallel_alternatives.html#DIFFERENCES-BETWEEN-Rust-parallel-AND-GNU-Parallel https://www.gnu.org/software/parallel/parallel_alternatives....
- tejtm 9y agoand the humble pipe, another process may start the next stage before the previous one(s) have finished.
- limaoscarjuliet 9y agoConsumer task will wait on pipe read and the producer will wait on pipe open, so it is synchronized properly.
- grigjd3 9y agoNeed to be aware of the buffer size on the pipe. It can be one of those issues that never comes up until you just cross the threshold, then everything fails ungracefully.
- termie 9y agoBash’s built-in wait is also handy when you want quick and simple parallelism. http://tldp.org/LDP/abs/html/x9644.html http://tldp.org/LDP/abs/html/x9644.html
- philsnow 9y agoI do this pattern a lot for x in $(seq 1 10); do long_process file${x} & done; wait It doesn't take care of not saturating my cpu or anything; when I need to care about that then I try to remember how to use parallel.
- enriquto 9y agoit's actually easier using parallel : for x in ...; do echo long_process file$x ; done | parallel -j 8 EDIT: and in your case, it is even easier with xargs, e.g.: ls files* | xargs -n 1 -P 8 long_process
- sigjuice 9y agoWouldn't this fail if your file names have white space?
- Anthony-G 9y agoParsing `ls` is never a good idea[1]. A more robust way to use globbing in a POSIX shell would be something like this: printf "%s\0" files* | xargs -0 -n 1 -P 8 long_process 1. http://mywiki.wooledge.org/ParsingLs http://mywiki.wooledge.org/ParsingLs
- enriquto 9y agoI disagree with the article that you linked. Filenames are variable names that you get to choose. Half of the game is won by chosing them wisely, so that they are convenient to use.
- enriquto 9y ago
- gigatexal 9y agoCommand line goals.
- Filligree 9y agoFor GNU Parallel, if you pass it -k it will buffer all of its subcommands' output and print it as if it had all been run serially, removing the need for those subcommands to output anything atomically -- a requirement that usually can't be guaranteed. It's a capable tool, if somewhat... complicated.
- protomikron 9y agoThanks, I did not know that. This is useful if you are solving embarrassingly parallel problems that generate lots of log data (e.g. timings for benchmark) and write them to stdout/stderr. AFAIK there is some guaranteed atomicity when writing PIPE_BUF<512 to a pipe, however then your sub programs shouldn't write more than 512 bytes of log-data at any point, I would guess.
- dahart 9y ago> parallel --will-cite I love gnu parallel, but that citation thing is a bit of a drag on parallel. You only have to do this once, but I see why you'd write a blog post to include the flag and not have to discuss the issue. I wish Ole would relax on this. It's not really that appropriate to demand that anyone using parallel for academics cite a magazine article. It's only appropriate for people doing research in parallel algorithms, and in any case it would make citing easier if there was a journal article to reference. There's nothing wrong with asking nicely, and I try to spread the word about gnu parallel. But the heavy handed demand is annoying, even if I only deal with it like once a year.
- bamboozled 9y agoStrongly agree with this!
- gioele 9y agoFor 99% of the use cases moreutils' parallel (AKA Tollef parallel) is an equal, or even better, alternative. "Interesting" thread from 2012 about replacing moreutils' parallel with GNU parallel in Debian: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=597050 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=597050 (TLDR: skip to message #75).
- dahart 9y agoOh, I've seen the --tollef option in the help doc, but I still didn't realize there was a complete alternative to gnu parallel. Definitely will have to try. Mr. Hess makes a lot of good points, most I think I agree with, so I can sympathize with his frustration. The point about transferring files seems interesting. I haven't used gnu parallel in remote/ssh mode aside from simple tests, but that seems wicked useful for some situations. Is there a real and simple alternative to running gnu parallel on multiple remote hosts? Is that something the moreutils version can also do?
- deno 9y agoThere’s also Rust Parallel[1]. It’s really fast, so it makes sense to use even for very quick things (but you might be running a lot so it adds up). Overall it has become my preferred Parallel implementation. [1] https://github.com/mmstick/parallel https://github.com/mmstick/parallel
- wookayin 9y agoThis tool is also great: https://github.com/greymd/tmux-xpanes https://github.com/greymd/tmux-xpanes
- npx 9y agoI realize it's somewhat off topic, but I feel like Joyent Manta deserves an honorable mention. It's an S3 style object store, but you can spin up containers on top of objects and do massively parallel computations with Unix tools. https://apidocs.joyent.com/manta/ https://apidocs.joyent.com/manta/
- zackmorris 9y agoMy biggest gripe with UNIX is that a command like: (sleep 5 && echo hello) > test.txt Causes a race condition where test.txt is created empty, then 5 seconds later written with "hello". Try it without parentheses to see it wait. UNIX doesn't use our intuitive notion of order of operations on its piping, it just runs everything simultaneously. This allowed for tremendous efficiency and concurrency but it's hard to fathom how much this has cost us in bugs and lost development time. I feel like this was a lost opportunity because it prevented the Actor model (as seen in Erlang and Go) from really taking off decades ago. Perhaps this bug/feature was one of the motivations for commands like "parallel". Does anyone have a general workaround for this problem? Some command that we could insert in the chain to force a wait, without having to install any external tools? Thanx! Edit: I'm having a hard time explaining how insidious this race condition is for people who haven't encountered it yet. The gist of it is that file descriptors aren't opened when a pipe sends its first byte, they are opened when the shell command is interpreted. I'm also having a hard time finding examples, here is one I think, though there are many, many others: https://unix.stackexchange.com/questions/174788/am-i-hitting-a-race-condition-in-bash https://unix.stackexchange.com/questions/174788/am-i-hitting...
- adamkruszewski 9y agoI think it does, try it as: sleep 5 && (echo hello > test.txt) and you'll see it creates the file only after sleep ends.
- chubot 9y agoI don't understand what you're trying to say. I would expect these two commands to behave differently: (sleep 5 && echo hello) > test.txt sleep 5 && echo hello > test.txt "our intuitive notion of order of operations on its piping" -- I don't know what your intuitive notion is, and there is no piping in your example. Piping would be: sleep 5 | echo hello (this is an odd example, but it works) I also don't see any race condition in your example. A race would be two processes accessing the same data. They are redirects, and the way redirects like > and < work is to: 1. open a file 2. change the global process state 3. run arbitary commands, either builtin (echo) or external (ls). External commands require a new process and inherit their process state (descriptor table, where stdout and stdin are conected to) from the parent shell. 4. restore the global process state 5. close the file If you understand that, it might be clear why the file test.txt is opened before sleep. There's really no other way it can work, given the fact that you can execute arbitrary commands inside the redirect. I address some related things about redirects here: "Avoid Directly Manipulating File Descriptors in Shell" http://www.oilshell.org/blog/2017/08/12.html http://www.oilshell.org/blog/2017/08/12.html ----- The more unintuitive thing about shell is that what you wrote is technically better as: { sleep 5 && echo hello; } > test.txt () doesn't mean "grouping" like in other languages, it means subshell. {} means grouping, and it has an odd way of parsing, where ; is required before }, but it isn't required before ). This is because () are operators and {} aren't. I wrote about this here, "the subshell construct () is confused with the grouping construct { }": http://www.oilshell.org/blog/2017/02/06.html http://www.oilshell.org/blog/2017/02/06.html ----- EDIT: I will say that I was very confused by the syntax of > and < when I started learning shell. It does seem like ">" means "this goes into that". In fact I think I internalized that from MS-DOS. But that doesn't explain what these mean, and why they are equivalent: tac >out.txt <in.txt tac <in.txt >out.txt They are best read as prefix operators that modify the state of the process invocation. What's even more confusing is that you can do: tac 1>out.txt 0<in.txt tac 0<in.txt 1>out.txt Then they look like binary operators again. But the descriptor is "stuck on", it's actually part of the operator lexically. Valid: tac 1>out.txt 0< in.txt tac 1> out.txt 0< in.txt Invalid: tac 1 >out.txt 0 <in.txt tac 1 > out.txt 0 < in.txt In other words, there can't be any space between the LHS file descriptor and the > or <. The syntax of shell is horrible, and that's something I would like to fix with Oil shell. But I don't see the problem you are pointing out.
- jwilk 9y agoNote that, somewhat counter-intuitively, what "wc -l" counts is the number of newline characters: $ printf 'foo\nbar' | wc -l 1 There are arguably two lines in the input, but the result is 1. Without this (mis)feature, "wc -l" would be more difficult to parallelize.
- LambdaComplex 9y agoThe POSIX definition of a "line" is "A sequence of zero or more non- <newline> characters plus a terminating <newline> character"[0]. I don't think it's counterintuitive at all for POSIX utilities to respect that definition. 0. http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap03.html#tag_03_206 http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_...
- frou_dh 9y ago"terminator, not separator" is an easy way to remember it. I would call bar in GP's comment a fragment.
- jandrese 9y agoThe only difference is you have to check the last character in the file to determine if it is a newline. Maybe the last 4 characters if it is Unicode.
- catern 9y agoSee also http://catern.com/posts/pipes.html http://catern.com/posts/pipes.html
- known 9y agoOleTange, author of parallel chips in https://www.reddit.com/r/programming/comments/5x39jh/counting_lines_60_faster_than_wc_with_clojure/ https://www.reddit.com/r/programming/comments/5x39jh/countin...
- YSFEJ4SWJUVU6 9y agoThe mentioned tool 'turbo-linecount' is, well, odd. It targets what I'd think is a very niche application of counting lines very fast (a domain usually limited by I/O speed), using a rather complex design... and then manages to throw much of the gained advantage away by using what is perhaps one of the slowest ways of counting newlines in a buffer.