8 ms·
GNU Parallel Tutorial
- zevv 10y agoOr, good enough for 95% of the use cases, simply use 'xargs' which is part of findutils and installed by default on all linux distributions. See the '-P/--max-procs' option in the man page for more details.
- deleted 10y ago[deleted]
- bandrami 10y agoRight, but it's those remaining 5% of situations where you want parallel execution on multiple remote hosts that make this cool (xargs itself can be replaced 95% of the time with backticks, for that matter). Combining parallel, pies[1], and shepherd[2] makes for some very cool possibilities, which you can check out with the GNU System Distribution[3], which I've recently migrated all my personal systems to. [1] http://www.gnu.org.ua/software/pies/ http://www.gnu.org.ua/software/pies/ [2] https://www.gnu.org/software/shepherd/ https://www.gnu.org/software/shepherd/ [3] https://www.gnu.org/software/guix/ https://www.gnu.org/software/guix/
- adrianN 10y agoWith xargs I always have the problem that lines get mangled when you have concurrent output, so you have to work around that. GNU parallel prevents this by some magic.
- sigil 10y agoThe "magic" is simply output buffering inside GNU parallel...something you should be aware of if the concurrent programs are producing large amounts of output.
- xorcist 10y agoIf you find yourself depending on the order of output lines, your are probably better off using something like Python Fabric instead.
- adrianN 10y agoReproducing that behavior with just bash and xargs is too difficult for me.
- ole_tange 10y ago"inside" = temporary files - not RAM. See comparison on https://www.gnu.org/software/parallel/parallel_design.html#Buffering-on-disk https://www.gnu.org/software/parallel/parallel_design.html#B...
- deleted 10y ago[deleted]
- psi-squared 10y agoIf I've read this correctly, the 'sem' mode lets you submit several lots of jobs with an overall limit on the total number of tasks running at a time (rather than one limit per lot of jobs). That on its own is super useful for what I'm working on right now. But what would make it even more useful, is: can you get GNU make to use 'sem' instead of its own jobserver? That way I could run almost everything I need to under one overall task limit, and that would be really nice to have. (For this reason, I'm a fan of the idea that every program with its own 'parallel execution' mode should be able to interact with a common jobserver. The 'make' jobserver is, as far as I know, the simplest, and should be pretty easy to support: http://make.mad-scientist.net/papers/jobserver-implementation/ http://make.mad-scientist.net/papers/jobserver-implementatio... )
- dahart 10y agoI don't think you can use a different job server for make, all you can do is have a make rule that launches parallel or something else, but then you lose dependency tracking on the sub-tasks. Are you running parallel make tasks where each task is also doing something multi-threaded or parallel? Like using make -j 8 won't work for you? Make does have the -l load average task limiter when but I've never gotten it to work reliably, it always starts way too many jobs at first and chokes for a while before calming down. Often that won't work for me, but maybe it will help you?
- psi-squared 10y agoMy current workflow has a lot of "Run make -j<lots> to build, followed by parallel -j<lots> to run all the tests", but sometimes I want to compare/test multiple different versions of the code (in a way which, sadly, doesn't work well with incremental builds). In that case it'd be nice to be able to just spin everything up and not have to worry about overloading/under-loading the machine I'm working on. I know what you mean about the load average limiter - parallel behaves like that too. I think the --delay option to parallel is supposed to solve that (I haven't tried it - will try tomorrow), but I don't know if make has anything similar. Finally, on further reading, it definitely seems technically possible, even if it hasn't been done so far. The make documentation has a section on the jobserver protocol, which looks complete enough to write both the client and server parts: https://www.gnu.org/software/make/manual/html_node/Job-Slots.html https://www.gnu.org/software/make/manual/html_node/Job-Slots... So if nothing exists so far, it's something I might look into writing myself.
- oftenwrong 10y agoI am a long-time user of gnu parallel, but I feel it has a bit of "featuritis". Just look at all of those options in the man page!
- pi-rat 10y agoLove this tool, often end up using it with imagemagick. # Resize all jpgs to 800x600 using 8 jobs/cores. parallel -j 8 convert {} -resize 800x600 {.}_small.jpg ::: *.jpg # Or get the help of a few servers (via SSH) to do the same job. parallel -S serverA,serverB -j 8 --transfer --return {.}_small.jpg convert {} -resize 800x600 {.}_small.jpg ::: *.jpg
- dahart 10y agoI do the same thing and resize all the pictures I ever import into both small & thumbnail versions. Fun tip - if your resize script prints only the output file name to stdout, you can pipe the result into another parallel command, e.g., parallel <resize to 800x600> ::: *.jpg | parallel <resize to thumbnail> This way generating thumbnails runs on the small image instead of the full size image, saving more time. Also, if you're on a mac, using sips is quite a bit faster than imagemagick, and sips comes built-in. I've never used servers with parallel for image resizing, what are the benefits? I'd have guessed it would take longer and saturate the network, versus doing it locally. Is it useful for really long running jobs when you don't want to load your local cpu? Is it actually faster sometimes, or are there other more important reasons? I could see it being useful if I didn't have a local imagemagick install, but had access to servers with it there. Maybe it'd be useful in cases where I'm running docker environments for the job processing? What other use cases and scenarios have you run into?
- pi-rat 10y agosips is a good tip, thanks! I often sort and process images with a macbook on my lap curled up on the sofa. Resizing locally makes the mac blazing hot with fans whining - not comfortable and kills the battery. Transfer speed is not too bad over 802.11AC. So I use it mainly as a method of moving cpu intensive work away from my lap. Also, I sometimes use parallel -S server,: ....... The semicolon adds the local machine to the list, it will saturate both the laptop and whatever it manages from the other computer. I have to admit I've never tested this scientifically, but it seems to be faster even with the overhead (gain of remote imagemagick seems to be more than cpu overhead of SSH file transfer).
- haddr 10y agoWow! Parallel is simply unmatched!
- janoc 10y agoPlease, don't: "Install the newest version using your package manager or with this command: (wget -O - pi.dk/3 || curl pi.dk/3/ || fetch -o - http://pi.dk/3) | bash " facepalm
- dahart 10y agoI think this discussion has already gone the rounds on HN multiple times, and it usually gets pointed out that running any installer at all has more or less the same risks as pipe something from the internet straight to the shell -- while this looks scary, it's actually no less safe than how we install any other software. I agree it doesn't look ideal and may not be best practice, but what do you feel is a better realistic alternative? What is the main issue for you? Is it the lack of a hash or checksum to verify what you downloaded, and make sure you didn't get a malicious site or a compromised package?
- heinrich5991 10y agoThis is HTTP though, not HTTPS. You're not just trusting the authors, but also trusting the network.
- dahart 10y agoYes, absolutely! So package validation is the issue for you. But how certain are you about everything else you install using HTTPS? (I'm honestly asking, not being rhetorical.) And what is stopping anything else you install from downloading something over HTTP, and executing it?
- deleted 10y ago[deleted]
- gravypod 10y agoWho is pi.dk? This isn't using https? If you have a package manager--that has developer signatures for every release and has been checked by multiple others in the community--when even recommend using this? This is far less secure then running apt-get, yum, or even pacman.
- CJefferson 10y agoFeel I should mention this -- the confusing and non-clarified (I mailed and asked for clarification, I was told to stop using the software) licence which says if you use this as an academic, you must cite it or pay. I don't use it for that reason -- this request isn't scalable. I either use xargs, or this reimplementation which had all the stuff I need: https://github.com/mmstick/parallel https://github.com/mmstick/parallel
- avar 10y agoIt's licensed under the GPL, where you're not allowed to add any additional terms. The citation notice it spews out is annoying, but can be disabled, and you definitely don't have to include it anywhere. The author just asks for citations because he's in academia, and citation backrubs is how anyone can justify getting time to do anything (including maintaining some software) in academia.
- filmor 10y ago2 questions: - Isn't it common courtesy to cite important tools being used for data processing? Why is this a problem? - Wikipedia says the tool is under GPLv3, so from a bare licence perspective you should be safe to use it for whatever you want, shouldn't it?
- dahart 10y ago> Isn't it common courtesy to cite important tools being used for data processing? No, no it's not. It is common to cite previous work, the research you build on top of. It is not common at all to cite tools you use, unless those tools are directly related to your research. If by "important" you mean gnu parallel counts as previous work, then citation is more of a requirement than a courtesy, otherwise citing all the tools you used can be very discourteous to your readers. It would be extra courteous to give parallel some PR if you love it and it really helped your research get done. > Why is this a problem? What if all gnu tools asked the same thing parallel does? What if using emacs and find and tar all asked you to cite them when using them, with obnoxious and demanding language? I use more than 100 free tools to get any paper implemented and written, my papers would be rejected if I tried to cite them all, the vast majority are not relevant to the research. > you should be safe to use it for whatever you want Yes, correct! GPL does not allow requiring citations.
- rdtsc 10y agoGNU parallel is pretty cool. Another little thing I realized a while back is that `make` (yes the crusty old make + -j flag) can be used to parallelize jobs. We do it for compiling usually, but it can be used for other jobs as well.
- ole_tange 10y agoFun fact: GNU Parallel was originally a wrapper script that used `make` for the parallelization: https://www.gnu.org/software/parallel/history.html https://www.gnu.org/software/parallel/history.html
- rdtsc 10y agoHa! Really cool. That was a fun fact! Make was right there under my nose I just never imagined using it for anything but compiling and building things. In that case I was forced by circumstances (was developing on a constrained ancient version of RHEL), couldn't use GNU Parallel and someone suggested `make`. The use case of obvious once a co-worker mentioned it. But it was definitely It was one of the memorable "thinking outside the box" example as they say.
- KeshandKooley 10y agoGo watch my YouTube video everyone! And remember to Subscribe ️️️ https://m.youtube.com/watch?v=QRQPSp-vTd4 https://m.youtube.com/watch?v=QRQPSp-vTd4