9 ms·
GNU Parallel
- wingless 10y agoAm I the only person who finds GNU parallel way too complicated? I tried to perform a very easy parallel task with it and spent hours reading the documentation and various tutorials. If a person with Unix command-line skills can't easily pick it up, what's the point of having it?
- tyingq 10y agoThere do seem to be some very complex use cases. On the other hand, their example of parallel gzip of files seems straightforward: find . -name '*.html' | parallel gzip --best Generally, using it in places where you would normally use xargs seems uncomplicated.
- amelius 10y agoProbably a stupid question, but how do the remote machines translate the pathnames to local pathnames? And what happens if they fail?
- mynewtb 10y agoWhat was it that you tried to achieve? I find it very easy for trivial parallelisation over files but am quickly lost when it becomes more complex.
- ole_tange 10y agoDid you start by reading the "Reader's guide" which is presented before the first option is introduced in the man page?
- base698 10y agoAs always gets brought up when GNU parallel is mentioned: xargs does most of the use cases you'd need for parallel. xargs -n1 -P4 Would be at most one arg from the arg list run with 4 jobs. http://stackoverflow.com/questions/28357997/running-programs-in-parallel-using-xargs http://stackoverflow.com/questions/28357997/running-programs...
- lyle_nel 10y agoI have a small cluster of machines that I run experiments on. GNU parallel makes the dispatch of jobs on remote machines very easy. In addition, I often use it to search for sequences by running grep in parallel. For example $ parallel 'grep {1} -f haystack.txt' :::: many_needles.txt Where {1} is a single line in many_needles.txt
- justinsaccount 10y agoUnless those patterns are regexes, you should just be using $ fgrep -f many_needles.txt haystack.txt
- nothrabannosir 10y agoHow much faster is a plain text search really than an regexp without special characters? You'd think this would be quite easy to optimise for a regexp engine. I admit I try to use -f all the time but your post suddenly made me realise I'd never actually measured the effect. :/ Edit: yes sorry I meant -F
- burntsushi 10y agoThere are a few things to clear up here. Namely, fgrep (equivalent to grep -F) is orthogonal to -f. fgrep/`grep -F` is for "fixed string" search, where as -f is for reading multiple patterns from a file. So `fgrep -f file` means "do a simple literal (non-regexp) search for each string in `file`." There are two possible reasons why one might want to explicitly use fgrep/`grep -F`: 1) to avoid needing to escape a string that otherwise contains special regex characters and 2) because it permits the search tool to avoid the regexp engine entirely. (1) is always a valid reason and is actually quite useful because escaping regexes can be a bother. But whether (2) is valid or not depends on whether your search tool is smart enough to recognize a simple literal search and automatically avoid the regex engine. Another layer to this is, of course, whether the regex engine itself is smart enough to handle this case for you automatically. Whether these optimizations are actually applied or not is difficult for a casual user to know. I don't actually know of any tool that doesn't optimize the simplest case (when no special regex features are required and it's just a simple literal search), so it seems to me that one should never use fgrep/`grep -F` for performance reasons alone. However, if you use the `-f` flag, then you've asked the tool to do multiple string search. Perhaps in this case, the search tool doesn't try as hard to do simple literal optimizations. Indeed, I can actually witness evidence in favor of this guess. The first command takes 15s and the second command takes 10s: LC_ALL=C grep -c -f queries /tmp/OpenSubtitles2016.raw.en LC_ALL=C grep -c -F -f queries /tmp/OpenSubtitles2016.raw.en The contents of `queries`: $ cat queries Sherlock Holmes John Watson Professor Moriarty Irene Adler grep in this case is GNU grep 2.26. The size of /tmp/OpenSubtitles2016.raw.en is 9.3GB. The only difference between the commands is the presence of the -F switch in the second command. My /tmp is a ramdisk, so the file was already in memory and therefore isn't benchmarking the speed of my disk. The corpus can be downloaded here (warning, multiple GB): http://opus.lingfil.uu.se/OpenSubtitles2016/mono/OpenSubtitles2016.raw.en.gz http://opus.lingfil.uu.se/OpenSubtitles2016/mono/OpenSubtitl... Interestingly, performing a similar test using BSD grep shows no differences in the execution time, which suggests BSD grep isn't doing anything smart even when it knows it has only literals (and I say this because BSD grep is outrageously slow). As a small plug, ripgrep is four times faster than GNU grep on this test and has no difference whether you pass -F or not. (This is only scratching the surface of literal optimizations that a search tool can do. For example, a good search tool will search for `foo` when matching the regex `\w+foo\d+` before ever entering the regex engine itself.)
- noja 10y agoGNU Parallel - an amazing tool with the most user unfriendly brick-wall-in-your-face documentation imaginable. Shame really - it's great.
- flatfilefan 10y agoTo be honest the same can be told about many GNU tools. At least myself I still experience this moment of being totally lost infront of the man screen from time to time. Parallel has a nice tutorial https://www.gnu.org/software/parallel/parallel_tutorial.html https://www.gnu.org/software/parallel/parallel_tutorial.html Have you seen it?
- deleted 10y ago[deleted]
- Mahn 10y agoI always get the feeling that most man pages are written for people who already know how to use the commands, rather than people new to them. For example, if I already understand how tar works and have a general idea of how to use it, man tar is great to drill down and find specific options and switches that I need or don't remember, but if I have no idea what tar files are or why would I want them, the man page really doesn't help much in explaining things. I guess you could argue this is supposed to be so and man pages are doing their job, since they are documentation and not tutorials, but still.
- lowry 10y agoGNU Parallel sucks. Use xargs when possible and paexec when needing fancy features. BTW, paexec even supports piping next to process invocation.
- ole_tange 10y agoDo you have an example of "piping next to process invocation"? It sounds interesting.
- known 10y agocat 100GB_data_file_with_DUPLICATE_lines | parallel --pipe awk \'\!a[\$0]++\' > data_file_with_UNIQUE_lines is BETTER since it uses less computer resources than gawk '!a[$0]++' 100GB_data_file_with_DUPLICATE_lines > data_file_with_UNIQUE_lines
- snvzz 10y agoThat's a classic misuse of cat. Just use '<'.
- kamijo-k 10y agoYeah, but to me, this "misuse" actually makes the input and output clearer than using "<" and ">" together. After all, it places input on the left side of the command, and output on the right side.
- ole_tange 10y agoExcept for GNU Parallel --pipe. When you use '<' what you typically mean is '--pipepart -a file'.
- decisiveness 10y agoThe parallel example will definitely not work as you expect, and will likely result in most duplicate lines still being present. When using --pipe this way, if you don't declare --block to be the size of the file (in which case there's no benefit to using parallel), each parallel execution will be run on a separate 1MB (default --block size) chunk of the file before outputting results all separately, then together in a single group (stdout), to the output file. If you're looking to spread work across CPUs and correctly get the desired output, I'd do something like: parallel -a input.txt --pipepart mawk \'\!a[\$0]++\' | mawk '!a[$0]++' > output.txt I used mawk because it is typically much more performant on large files.
- GolDDranks 10y agoThere is also an in-development GNU Parallel clone/alternative written in Rust. https://github.com/mmstick/parallel https://github.com/mmstick/parallel
- stymaar 10y agoBeing written in Rust is not its main feature (and I think you're being down-voted because people don't like fanboys). This project is cool because it has a really low overhead, which is really cool if you want to parallelize tasks that are not really CPU intensive (but mostly useless if the CPU usage of each task is high).
- mmstick 10y agoDeveloper here. Noticed significant traffic coming from this domain. Not sure if this specific comment is the cause of that though. Kind of difficult to search this site. Anyway, I did and am continuing to develop this with Rust as one of it's biggest points. It's a cool project to showcase Rust to both users and other software developers. I can use it to teach others Rust by example with well documented code, and possibly gain some buzz for this great language. That the application has low overhead is mainly a side effect of choosing to develop this application in a low level language like Rust, but the speed is indeed my biggest selling point in general, for now at least. I'm continuing to commit further development into mitigating costs as much as possible. I can also point out at there is a benefit to using this over the GNU Perl application for large scale computational tasks. Tasks where you are processing hundreds of thousands to hundreds of millions of inputs. This will keep memory consumption low and reduce the amount of CPU cycles required to process those inputs. You could save anywhere from a few minutes to an hour.
- ole_tange 10y agoRust-parallel _is_ fast, and there is clearly a niche here, that GNU Parallel is unlikely to fill: By design GNU Parallel will never require a compiler; this is so you can use GNU Parallel on old systems with no compilers (Think an old, dusty AIX-box that people have forgotten the root password to). This design decision limits how fast GNU Parallel can be compared to compiled alternatives. But the main problem with rust-parallel is that it is not compatible with GNU Parallel (and according to the author, it probably never will be 100% compatible). If you use rust-parallel to walk through GNU Parallel's tutorial (man parallel_tutorial) you will see it fails very quickly. (Full disclosure: I am the author of GNU Parallel. I fully support building other parallelizing tools, but to avoid user confusion, I would recommend calling them something other than 'parallel' if they are not actually compatible with GNU Parallel. History has shown that using the same name will lead to a lot of unnecessary grief: e.g. GNU Parallel vs. Parallel from moreutils).
- flatfilefan 10y agoGNU Parallel rules! I made $100K with just a few lines of code involving Parallel.
- tbrock 10y agoTell the story!
- flatfilefan 10y agothery simple really, but so is UNIX :-) parallel -j 1000 -a urls.txt {wget --recursive {}|lynx --dump|(other unix shell power tools) > url.txt} gets you a powerfull web scraper that maxes out your internet connection. Throw some machine learning on top of it and you can do a lot with that. Can't tell you what exactly I did with the ML that being trade secret about to get a patent, but the rest doesn't involve the parallel anyway.
- witty_username 10y agoOK, forget technical details; how's the money made?
- flatfilefan 10y agobetter sales targeting
- deleted 10y ago[deleted]
- limaoscarjuliet 10y agoI use it daily for parallel build process and I love it. Easy to set up, easy to deal with. Documentation was tough though.
- DigitalJack 10y agoThat's interesting. Make is parallelizable too, although I occasionally run into projects that won't build correctly if you use the feature.
- sheraz 10y agoDoes anyone else remember Slashdot and the endless threads about Beowulf clusters? That was back when "parallel computing" was overly complicated and rather opaque. And most of us had no idea how to take advantage of multiple machines.
- ole_tange 10y agoSure do. Makes you feel that the future is now!
- cmax 10y agoparallel is nice, in the past I would run a small script like https://gist.github.com/CMAD/3077918 https://gist.github.com/CMAD/3077918 because I had to migrate email accounts from a list and it was all very old servers with the package manager broken, good old times, now I will have to run some super convoluted orchestration
- realworldview 10y agoprll https://github.com/exzombie/prll https://github.com/exzombie/prll is a very pleasant and approachable alternative that I've used in preference to GNU Parallel.
- ole_tange 10y agoJust be aware that Stderr will contain stuff you did not ask for: https://www.gnu.org/software/parallel/man.html#DIFFERENCES-BETWEEN-prll-AND-GNU-Parallel https://www.gnu.org/software/parallel/man.html#DIFFERENCES-B... The syntax difference is small for simple tasks.
- blockoperation 10y agoI use parallel for pretty much all batching these days. It's useful even when you don't need parallelisation – here's a simple transcoding example: parallel -j1 'ffmpeg -i {2} -c:v libx264 -tune film -preset veryslow -crf 18 -vf scale={1} -c:a libfdk_aac -vbr 5 conv/{2.}-{1}.mp4' ::: hd480 hd720 hd1080 ::: *.mp4 Obviously in the real world you would want to take some extra steps to avoid making upscaled versions, but this is just a rough example.