5 ms·

As always gets brought up when GNU parallel is mentioned: xargs does most of the use cases you'd need for parallel. xargs -n1 -P4 Would be at most one arg fro

by base698 10y ago

As always gets brought up when GNU parallel is mentioned: xargs does most of the use cases you'd need for parallel. xargs -n1 -P4 Would be at most one arg from the arg list run with 4 jobs. http://stackoverflow.com/questions/28357997/running-programs-in-parallel-using-xargs http://stackoverflow.com/questions/28357997/running-programs...

lyle_nel 10y ago
I have a small cluster of machines that I run experiments on. GNU parallel makes the dispatch of jobs on remote machines very easy. In addition, I often use it to search for sequences by running grep in parallel. For example $ parallel 'grep {1} -f haystack.txt' :::: many_needles.txt Where {1} is a single line in many_needles.txt
- justinsaccount 10y ago
  Unless those patterns are regexes, you should just be using $ fgrep -f many_needles.txt haystack.txt
  - nothrabannosir 10y ago
    How much faster is a plain text search really than an regexp without special characters? You'd think this would be quite easy to optimise for a regexp engine. I admit I try to use -f all the time but your post suddenly made me realise I'd never actually measured the effect. :/ Edit: yes sorry I meant -F
    - burntsushi 10y ago
      There are a few things to clear up here. Namely, fgrep (equivalent to grep -F) is orthogonal to -f. fgrep/`grep -F` is for "fixed string" search, where as -f is for reading multiple patterns from a file. So `fgrep -f file` means "do a simple literal (non-regexp) search for each string in `file`." There are two possible reasons why one might want to explicitly use fgrep/`grep -F`: 1) to avoid needing to escape a string that otherwise contains special regex characters and 2) because it permits the search tool to avoid the regexp engine entirely. (1) is always a valid reason and is actually quite useful because escaping regexes can be a bother. But whether (2) is valid or not depends on whether your search tool is smart enough to recognize a simple literal search and automatically avoid the regex engine. Another layer to this is, of course, whether the regex engine itself is smart enough to handle this case for you automatically. Whether these optimizations are actually applied or not is difficult for a casual user to know. I don't actually know of any tool that doesn't optimize the simplest case (when no special regex features are required and it's just a simple literal search), so it seems to me that one should never use fgrep/`grep -F` for performance reasons alone. However, if you use the `-f` flag, then you've asked the tool to do multiple string search. Perhaps in this case, the search tool doesn't try as hard to do simple literal optimizations. Indeed, I can actually witness evidence in favor of this guess. The first command takes 15s and the second command takes 10s: LC_ALL=C grep -c -f queries /tmp/OpenSubtitles2016.raw.en LC_ALL=C grep -c -F -f queries /tmp/OpenSubtitles2016.raw.en The contents of `queries`: $ cat queries Sherlock Holmes John Watson Professor Moriarty Irene Adler grep in this case is GNU grep 2.26. The size of /tmp/OpenSubtitles2016.raw.en is 9.3GB. The only difference between the commands is the presence of the -F switch in the second command. My /tmp is a ramdisk, so the file was already in memory and therefore isn't benchmarking the speed of my disk. The corpus can be downloaded here (warning, multiple GB): http://opus.lingfil.uu.se/OpenSubtitles2016/mono/OpenSubtitles2016.raw.en.gz http://opus.lingfil.uu.se/OpenSubtitles2016/mono/OpenSubtitl... Interestingly, performing a similar test using BSD grep shows no differences in the execution time, which suggests BSD grep isn't doing anything smart even when it knows it has only literals (and I say this because BSD grep is outrageously slow). As a small plug, ripgrep is four times faster than GNU grep on this test and has no difference whether you pass -F or not. (This is only scratching the surface of literal optimizations that a search tool can do. For example, a good search tool will search for `foo` when matching the regex `\w+foo\d+` before ever entering the regex engine itself.)
      - recuter 10y ago
        This is a great comment and ripgrep deserves a more prominent plug, clickable: https://github.com/BurntSushi/ripgrep https://github.com/BurntSushi/ripgrep
      - tedunangst 10y ago
        BSD grep decides on a pattern by pattern basis which match engine to use. -F is unlikely to affect performance.
        burntsushi 10y ago
        Oh dear, you appear to be correct. Adding additional queries to `queries` (while being careful not to increase total match count by much) appears to increase search time linearly. From that, it looks like BSD grep is just running an additional pass over each line for each query.
- bane 10y ago
  If you find yourself searching lots of haystacks, and your needles are just text and not a regex, a better approach is to stuff all the needles into some kind of index, then chop up the haystack into overlapping tiles (of variable width from the smallest needle to the largest), then search each tile against the index of needles. This effectively searches all the needles at once and turns the operation from O(n) where n is the number of needles to O(m) where m is the number of tiles in haystack.txt. It may seem to be a trivial difference, but then you can search multiple haystacks at once fairly easily, and this approach scales to hundreds of millions of needles at once. The code for it isn't very difficult either, heck you can just use an in memory SQLite dB to get a searchable, temporary, index and rely on using some of the most tested software in history.
  - agentgt 10y ago
    This also works for sorting as well and is typically called Radix Sort or Bucket Sort. Basically using unique attributes of the data you then divide and conquer on those attributes (e.g. for Radix you make buckets based on the digits).
- lgessler 10y ago
  (Sorry, this is mildly off-topic.) Not sure if this fits your usecase, but you should check out codesearch if you haven't already: https://github.com/google/codesearch https://github.com/google/codesearch (Russ Cox's excellent writeup is here: https://swtch.com/~rsc/regexp/regexp4.html https://swtch.com/~rsc/regexp/regexp4.html)
dagw 10y ago
I guess this is very much a case of YMMV. Virtually all my parallel use cases involve copying files to and from remote machines and starting processes there.
jmiserez 10y ago
I've also always gotten everything to run with just xargs and minimal scripting. For my taste GNU parallel gets recommended a bit too quickly e.g. on Stackoverflow when the standard tools would do just fine. Your linked SO question is a prime example of that, xargs specific question yet there's a response that dismisses it entirely and suggests parallel when xargs can easily do the task at hand.
Sir_Cmpwn 10y ago
xargs -P is not POSIX. Please don't use it if you ship your scripts to end users.
- oblio 10y ago
  Well, GNU Parallel is not POSIX either, so it boils down to which tool you want to ship instead.
  - Sir_Cmpwn 10y ago
    Well, it's a different matter to depend on one standalone tool then to depend on a specific coreutils.
    - duskwuff 10y ago
      I'd bet on GNU xargs being more likely to be installed than GNU Parallel. One of the two is part of a standard package that all Linux systems, and many non-Linux systems, have installed. The other is a special-purpose tool.
      - Sir_Cmpwn 10y ago
        >I'd bet on GNU xargs being more likely to be installed than GNU Parallel. Your bet is irrelevant in the face of widely implemented standards. POSIX will outlive GNU coreutils. I'd bet your system also probably has a package manager that makes it easy to mark parallel as a dependency of your software. >many non-Linux systems Linux is pretty much the only system that typically ships GNU coreutils. The only one that comes to mind is Hurd.
  - cstrahan 10y ago
    For use within scripts, I think it's a much easier matter to determine that parallel is on $PATH than to try to do feature detection on xargs. Granted, it's always possible that the parallel on $PATH might not be the one you're thinking of, but... Whatevs.
RX14 10y ago
I've taken to using gnu parallel over xargs at all opportunities because I find it much easier to use. I find the parallel commands much shorter and easier to understand. Added to the fact it's more featureful and I find it pretty much everywhere already, why not?
jadbox 10y ago
This reminded me to fill a bug for BusyBox/Alpine's version of xargs which does not support parallel operations. https://bugs.busybox.net/show_bug.cgi?id=9511 https://bugs.busybox.net/show_bug.cgi?id=9511
krick 10y ago
xargs is fucked up when it comes to handling spaces in filenames and such. In fact, this is pretty much the only reason I use GNU parallel — I rarely need to actually run stuff in parallel (but it usually doesn't hurt either), but I need to do something pretty complicated (list | grep | sort | uniq | feed it to feh / whatever) over multiple arguments, without having to worry if these arguments contain spaces, quotes, unicode symbols, etc.
- martinflack 10y ago
  Protip: for "strange" characters, see the -0 argument to xargs. If the file list comes from find, see the -print0 argument there. find -print0 | xargs -0 -n1 echo echo -n "file '#1'\0file '#2'\0" | xargs -0 -n1 -I{} echo "the file is \"{}\", ok?"
  - krick 10y ago
    Oh yeah, great I idea. Now insert sorting somewhere between the pipes and use at least one filename with '"' character. Good luck with your xargs. parallel handles all this automatically, you don't even have to know there might be any problems with escaping here.
    - martinflack 10y ago
      Thanks - I will take a closer look at parallel.

by base698 10y ago

lyle_nel 10y ago

I have a small cluster of machines that I run experiments on. GNU parallel makes the dispatch of jobs on remote machines very easy. In addition, I often use it to search for sequences by running grep in parallel. For example $ parallel 'grep {1} -f haystack.txt' :::: many_needles.txt Where {1} is a single line in many_needles.txt

justinsaccount 10y ago
Unless those patterns are regexes, you should just be using $ fgrep -f many_needles.txt haystack.txt
- nothrabannosir 10y ago
  How much faster is a plain text search really than an regexp without special characters? You'd think this would be quite easy to optimise for a regexp engine. I admit I try to use -f all the time but your post suddenly made me realise I'd never actually measured the effect. :/ Edit: yes sorry I meant -F
  - burntsushi 10y ago
    There are a few things to clear up here. Namely, fgrep (equivalent to grep -F) is orthogonal to -f. fgrep/`grep -F` is for "fixed string" search, where as -f is for reading multiple patterns from a file. So `fgrep -f file` means "do a simple literal (non-regexp) search for each string in `file`." There are two possible reasons why one might want to explicitly use fgrep/`grep -F`: 1) to avoid needing to escape a string that otherwise contains special regex characters and 2) because it permits the search tool to avoid the regexp engine entirely. (1) is always a valid reason and is actually quite useful because escaping regexes can be a bother. But whether (2) is valid or not depends on whether your search tool is smart enough to recognize a simple literal search and automatically avoid the regex engine. Another layer to this is, of course, whether the regex engine itself is smart enough to handle this case for you automatically. Whether these optimizations are actually applied or not is difficult for a casual user to know. I don't actually know of any tool that doesn't optimize the simplest case (when no special regex features are required and it's just a simple literal search), so it seems to me that one should never use fgrep/`grep -F` for performance reasons alone. However, if you use the `-f` flag, then you've asked the tool to do multiple string search. Perhaps in this case, the search tool doesn't try as hard to do simple literal optimizations. Indeed, I can actually witness evidence in favor of this guess. The first command takes 15s and the second command takes 10s: LC_ALL=C grep -c -f queries /tmp/OpenSubtitles2016.raw.en LC_ALL=C grep -c -F -f queries /tmp/OpenSubtitles2016.raw.en The contents of `queries`: $ cat queries Sherlock Holmes John Watson Professor Moriarty Irene Adler grep in this case is GNU grep 2.26. The size of /tmp/OpenSubtitles2016.raw.en is 9.3GB. The only difference between the commands is the presence of the -F switch in the second command. My /tmp is a ramdisk, so the file was already in memory and therefore isn't benchmarking the speed of my disk. The corpus can be downloaded here (warning, multiple GB): http://opus.lingfil.uu.se/OpenSubtitles2016/mono/OpenSubtitles2016.raw.en.gz http://opus.lingfil.uu.se/OpenSubtitles2016/mono/OpenSubtitl... Interestingly, performing a similar test using BSD grep shows no differences in the execution time, which suggests BSD grep isn't doing anything smart even when it knows it has only literals (and I say this because BSD grep is outrageously slow). As a small plug, ripgrep is four times faster than GNU grep on this test and has no difference whether you pass -F or not. (This is only scratching the surface of literal optimizations that a search tool can do. For example, a good search tool will search for `foo` when matching the regex `\w+foo\d+` before ever entering the regex engine itself.)
    - recuter 10y ago
      This is a great comment and ripgrep deserves a more prominent plug, clickable: https://github.com/BurntSushi/ripgrep https://github.com/BurntSushi/ripgrep
    - tedunangst 10y ago
      BSD grep decides on a pattern by pattern basis which match engine to use. -F is unlikely to affect performance.
      - burntsushi 10y ago
        Oh dear, you appear to be correct. Adding additional queries to `queries` (while being careful not to increase total match count by much) appears to increase search time linearly. From that, it looks like BSD grep is just running an additional pass over each line for each query.
bane 10y ago
If you find yourself searching lots of haystacks, and your needles are just text and not a regex, a better approach is to stuff all the needles into some kind of index, then chop up the haystack into overlapping tiles (of variable width from the smallest needle to the largest), then search each tile against the index of needles. This effectively searches all the needles at once and turns the operation from O(n) where n is the number of needles to O(m) where m is the number of tiles in haystack.txt. It may seem to be a trivial difference, but then you can search multiple haystacks at once fairly easily, and this approach scales to hundreds of millions of needles at once. The code for it isn't very difficult either, heck you can just use an in memory SQLite dB to get a searchable, temporary, index and rely on using some of the most tested software in history.
- agentgt 10y ago
  This also works for sorting as well and is typically called Radix Sort or Bucket Sort. Basically using unique attributes of the data you then divide and conquer on those attributes (e.g. for Radix you make buckets based on the digits).
lgessler 10y ago
(Sorry, this is mildly off-topic.) Not sure if this fits your usecase, but you should check out codesearch if you haven't already: https://github.com/google/codesearch https://github.com/google/codesearch (Russ Cox's excellent writeup is here: https://swtch.com/~rsc/regexp/regexp4.html https://swtch.com/~rsc/regexp/regexp4.html)

dagw 10y ago

I guess this is very much a case of YMMV. Virtually all my parallel use cases involve copying files to and from remote machines and starting processes there.

jmiserez 10y ago

I've also always gotten everything to run with just xargs and minimal scripting. For my taste GNU parallel gets recommended a bit too quickly e.g. on Stackoverflow when the standard tools would do just fine. Your linked SO question is a prime example of that, xargs specific question yet there's a response that dismisses it entirely and suggests parallel when xargs can easily do the task at hand.

Sir_Cmpwn 10y ago

xargs -P is not POSIX. Please don't use it if you ship your scripts to end users.

oblio 10y ago
Well, GNU Parallel is not POSIX either, so it boils down to which tool you want to ship instead.
- Sir_Cmpwn 10y ago
  Well, it's a different matter to depend on one standalone tool then to depend on a specific coreutils.
  - duskwuff 10y ago
    I'd bet on GNU xargs being more likely to be installed than GNU Parallel. One of the two is part of a standard package that all Linux systems, and many non-Linux systems, have installed. The other is a special-purpose tool.
    - Sir_Cmpwn 10y ago
      >I'd bet on GNU xargs being more likely to be installed than GNU Parallel. Your bet is irrelevant in the face of widely implemented standards. POSIX will outlive GNU coreutils. I'd bet your system also probably has a package manager that makes it easy to mark parallel as a dependency of your software. >many non-Linux systems Linux is pretty much the only system that typically ships GNU coreutils. The only one that comes to mind is Hurd.
- cstrahan 10y ago
  For use within scripts, I think it's a much easier matter to determine that parallel is on $PATH than to try to do feature detection on xargs. Granted, it's always possible that the parallel on $PATH might not be the one you're thinking of, but... Whatevs.

RX14 10y ago

I've taken to using gnu parallel over xargs at all opportunities because I find it much easier to use. I find the parallel commands much shorter and easier to understand. Added to the fact it's more featureful and I find it pretty much everywhere already, why not?

jadbox 10y ago

This reminded me to fill a bug for BusyBox/Alpine's version of xargs which does not support parallel operations. https://bugs.busybox.net/show_bug.cgi?id=9511 https://bugs.busybox.net/show_bug.cgi?id=9511

krick 10y ago

xargs is fucked up when it comes to handling spaces in filenames and such. In fact, this is pretty much the only reason I use GNU parallel — I rarely need to actually run stuff in parallel (but it usually doesn't hurt either), but I need to do something pretty complicated (list | grep | sort | uniq | feed it to feh / whatever) over multiple arguments, without having to worry if these arguments contain spaces, quotes, unicode symbols, etc.

martinflack 10y ago
Protip: for "strange" characters, see the -0 argument to xargs. If the file list comes from find, see the -print0 argument there. find -print0 | xargs -0 -n1 echo echo -n "file '#1'\0file '#2'\0" | xargs -0 -n1 -I{} echo "the file is \"{}\", ok?"
- krick 10y ago
  Oh yeah, great I idea. Now insert sorting somewhere between the pipes and use at least one filename with '"' character. Good luck with your xargs. parallel handles all this automatically, you don't even have to know there might be any problems with escaping here.
  - martinflack 10y ago
    Thanks - I will take a closer look at parallel.