5 ms·

This can be a good entry into the "why" for many people that use xargs now - https://www.gnu.org/software/parallel/parallel_alternatives.html#differences-betwe

This can be a good entry into the "why" for many people that use xargs now - https://www.gnu.org/software/parallel/parallel_alternatives.html#differences-between-xargs-and-gnu-parallel https://www.gnu.org/software/parallel/parallel_alternatives.... It has a lot of features that can feel excessive at first glance but if you have felt some pain in building jobs, most of it is pretty sensible and much better than rollyourown.

cb321 5y ago
The comparison is not very fair to modern day xargs. `nproc` is a relatively standard utility (coreutils). So, xargs -P$(nproc) gets you core (or core-proportional) parallelism. Grouping output/Making a safe parallel grep is also easy-ish with `--process-slot-var=slot` and sending to `tmpOut.$slot`. Jobs on remote computers can be done similarly with any kind of `arrayVar[$slot]` setup where `arrayVar` has a bunch of `ssh` targets, possibly duplicates if you want to run >1 job per host. (In pure POSIX sh you could use eval and $1, $2 positional args with shell arithmetic..) Anyway, those three are just off the top of my head, unfairness-wise. Last I looked at the source for GNU parallel it looked like mountains upon mountains of Perl I would rather not depend upon, personally, but to each his own.
- ZoomZoomZoom 5y ago
  > Anyway, those three are just off the top of my head, unfairness-wise. Last I looked at the source for GNU parallel it looked like mountains upon mountains of Perl I would rather not depend upon, personally, but to each his own. Well, there was a Rust version with zero Perl, now unfortunately archived. It wasn't 100% on a par with the original and wasn't really finished. On the other hand, built easily for Windows and helped me on a few occasions. https://github.com/mmstick/parallel https://github.com/mmstick/parallel
  - cb321 5y ago
    Well, some archived project is not so great either. The core functionality is not even a 20 line bash script since bash grew wait -n, though: #!/bin/bash if [ "${1-0}" -lt 1 ]; then # No arg / arg not a number >= 1 echo "Usage: $0 <N>"; echo "reads cmds from stdin, running up to N at once." exit 1 fi TMP=`mktemp -t stripen.XXXXXX` trap 'rm -f $TMP; exit 0' HUP INT TERM EXIT STRIPE_SEQ=1 while read cmd; do jobs > $TMP # jobs | wc -l does not work if [ $(wc -l < $TMP) -ge $1 ]; then wait -n # Wait for 1/more jobs to finish fi # Could accumulate total $? above, but would need to replace final wait. STRIPE_SEQ=$((STRIPE_SEQ + 1)) ( eval "$cmd" ) < /dev/null & # Run job in a subshell in bg done rm -f $TMP wait # Wait for all to finish and if bash ever grows some magic environment variable $NUM_BG_JOBS or you don't want auto-help or sequence numbers or etc. it can be even simpler.
    - arrakeen 5y ago
      sure a "parallel xargs" can ostensibly be implemented in POSIX sh but that's merely the tip of the iceberg with what parallel can do. why not just skim the documentation and give it a try?
      - cb321 5y ago
        I have skimmed. I did give it a try. Not for me.
- forgotmypw17 5y ago
  Perl has retained stable backwards compatibility and no breaking changes for 20+ years. What's wrong with Perl?
  - smegsicle 5y ago
    "perl has common-lisp levels of stability"
  - rurban 5y ago
    Lots of unstable breakages over the last year's. automake got broken by an unnecessary deprecation, signatures got broken, encodings, and dozens more. but still miles better than other such languages, and esp. if it would have been written in C, just as the incompatible moreutils counterpart.
    - MainJane 5y ago
      I was curious how much breakage GNU Parallel has suffered. So I fetched all versions (in parallel) and ran: parallel -k --tag --argsep -- {} echo ::: 1 -- parallel-* Every version since 20120622 work (except for 20121022). That is code which is almost 10 years old.
      - rurban 5y ago
        you need to try with all the perl versions, not the parallel versions.
        forgotmypw17 5y ago
        In my anecdotal, n=1 experience, nothing Perl-based I've ever used has EVER broken over 20+ years, not even ONCE. Compare this with PHP, whose breaking changes between releases has taken down my sites on multiple occasions. Compare this with Python, whose breaking changes prevent me from running the overwhelming majority of Python things I've tried to use.
    - arunix 5y ago
- arrakeen 5y ago
  > Last I looked at the source for GNU parallel it looked like mountains upon mountains of Perl I would rather not depend upon, personally, but to each his own. i used parallel for years under the assumption that it was written in C and only recently learned it was written in perl when i decided to dive deeply into its documentation. if you're using a package manager to install parallel and it runs fast enough for your needs (it does) then who cares what language it was implemented in?
  - cb321 5y ago
    Some users like to add their own features or have problems to debug. They surely care. Others may want to move coordination to a remote host and care about some single-file transfer of the exact version of the tool. They also care. It's ok that you don't, personally, of course.
- smegsicle 5y ago
  > mountains upon mountains of Perl I would rather not depend upon how bad was the perl?
  - cb321 5y ago
    Subjective. Judge for yourself: https://git.savannah.gnu.org/git/parallel.git https://git.savannah.gnu.org/git/parallel.git
- MainJane 5y ago
  > The comparison is not very fair to modern day xargs. I am curious how you come to that conclusion. > `nproc` is a relatively standard utility (coreutils). So, xargs -P$(nproc) gets you core (or core-proportional) parallelism. I follow you on this point. A bit harder on remote systems, but definitely doable. > Grouping output/Making a safe parallel grep is also easy-ish with `--process-slot-var=slot` and sending to `tmpOut.$slot`. I tried spending 5 minutes on coding this, but the details seem to be very hard to get right: composed commands, grouping stderr, combined with not leaving tmp files behind if killed and allowing for the total output to be bigger than the free space on /tmp. I could not do it. Could you consider spending 5 minutes on showing in code how you would do it? > Jobs on remote computers can be done similarly with any kind of `arrayVar[$slot]` setup where `arrayVar` has a bunch of `ssh` targets, possibly duplicates if you want to run >1 job per host. (In pure POSIX sh you could use eval and $1, $2 positional args with shell arithmetic..) This one seemed even harder to me: It was completely unclear how you would make sure that a given number of jobs were constantly running. And how would you need to quote data, so an eval would not cause "foo space space bar" turn into "foo space bar". And how you would kill remote jobs, if the local script was killed. If you believe this is simple, could you spend 5 minutes on showing the rest of us how you would do it in actual working code? Because it seems the devil is really in the detail. > Last I looked at the source for GNU parallel it looked like mountains upon mountains of Perl I would rather not depend upon, personally, but to each his own. Personally, I would take production tested code over home-made untested code any day - no matter the language in which it was written.
  - cb321 5y ago
    > allowing for the total output to be bigger than the free space on /tmp. I could not do it. This is an unreasonable standard when you do not know in advance how big the output is. What do you imagine GNU parallel does? Use `df` on every host it knows about to fill every disk partition it can? That sounds like a pretty system-hostile behavior to me. Meanwhile, putting your temp files somewhere bigger is obv. as easy as $TMPDIR or such. Best wishes/luck. I only have 5 minutes to explain why nothing can do the impossible like read a user's mind about disk free space management or the value of partial results. All software makes some assumptions... :-)

cb321 5y ago

The comparison is not very fair to modern day xargs. `nproc` is a relatively standard utility (coreutils). So, xargs -P$(nproc) gets you core (or core-proportional) parallelism. Grouping output/Making a safe parallel grep is also easy-ish with `--process-slot-var=slot` and sending to `tmpOut.$slot`. Jobs on remote computers can be done similarly with any kind of `arrayVar[$slot]` setup where `arrayVar` has a bunch of `ssh` targets, possibly duplicates if you want to run >1 job per host. (In pure POSIX sh you could use eval and $1, $2 positional args with shell arithmetic..) Anyway, those three are just off the top of my head, unfairness-wise. Last I looked at the source for GNU parallel it looked like mountains upon mountains of Perl I would rather not depend upon, personally, but to each his own.

ZoomZoomZoom 5y ago
> Anyway, those three are just off the top of my head, unfairness-wise. Last I looked at the source for GNU parallel it looked like mountains upon mountains of Perl I would rather not depend upon, personally, but to each his own. Well, there was a Rust version with zero Perl, now unfortunately archived. It wasn't 100% on a par with the original and wasn't really finished. On the other hand, built easily for Windows and helped me on a few occasions. https://github.com/mmstick/parallel https://github.com/mmstick/parallel
- cb321 5y ago
  Well, some archived project is not so great either. The core functionality is not even a 20 line bash script since bash grew wait -n, though: #!/bin/bash if [ "${1-0}" -lt 1 ]; then # No arg / arg not a number >= 1 echo "Usage: $0 <N>"; echo "reads cmds from stdin, running up to N at once." exit 1 fi TMP=`mktemp -t stripen.XXXXXX` trap 'rm -f $TMP; exit 0' HUP INT TERM EXIT STRIPE_SEQ=1 while read cmd; do jobs > $TMP # jobs | wc -l does not work if [ $(wc -l < $TMP) -ge $1 ]; then wait -n # Wait for 1/more jobs to finish fi # Could accumulate total $? above, but would need to replace final wait. STRIPE_SEQ=$((STRIPE_SEQ + 1)) ( eval "$cmd" ) < /dev/null & # Run job in a subshell in bg done rm -f $TMP wait # Wait for all to finish and if bash ever grows some magic environment variable $NUM_BG_JOBS or you don't want auto-help or sequence numbers or etc. it can be even simpler.
  - arrakeen 5y ago
    sure a "parallel xargs" can ostensibly be implemented in POSIX sh but that's merely the tip of the iceberg with what parallel can do. why not just skim the documentation and give it a try?
    - cb321 5y ago
      I have skimmed. I did give it a try. Not for me.
forgotmypw17 5y ago
Perl has retained stable backwards compatibility and no breaking changes for 20+ years. What's wrong with Perl?
- smegsicle 5y ago
  "perl has common-lisp levels of stability"
- rurban 5y ago
  Lots of unstable breakages over the last year's. automake got broken by an unnecessary deprecation, signatures got broken, encodings, and dozens more. but still miles better than other such languages, and esp. if it would have been written in C, just as the incompatible moreutils counterpart.
  - MainJane 5y ago
    I was curious how much breakage GNU Parallel has suffered. So I fetched all versions (in parallel) and ran: parallel -k --tag --argsep -- {} echo ::: 1 -- parallel-* Every version since 20120622 work (except for 20121022). That is code which is almost 10 years old.
    - rurban 5y ago
      you need to try with all the perl versions, not the parallel versions.
      - forgotmypw17 5y ago
        In my anecdotal, n=1 experience, nothing Perl-based I've ever used has EVER broken over 20+ years, not even ONCE. Compare this with PHP, whose breaking changes between releases has taken down my sites on multiple occasions. Compare this with Python, whose breaking changes prevent me from running the overwhelming majority of Python things I've tried to use.
  - arunix 5y ago
arrakeen 5y ago
> Last I looked at the source for GNU parallel it looked like mountains upon mountains of Perl I would rather not depend upon, personally, but to each his own. i used parallel for years under the assumption that it was written in C and only recently learned it was written in perl when i decided to dive deeply into its documentation. if you're using a package manager to install parallel and it runs fast enough for your needs (it does) then who cares what language it was implemented in?
- cb321 5y ago
  Some users like to add their own features or have problems to debug. They surely care. Others may want to move coordination to a remote host and care about some single-file transfer of the exact version of the tool. They also care. It's ok that you don't, personally, of course.
smegsicle 5y ago
> mountains upon mountains of Perl I would rather not depend upon how bad was the perl?
- cb321 5y ago
  Subjective. Judge for yourself: https://git.savannah.gnu.org/git/parallel.git https://git.savannah.gnu.org/git/parallel.git
MainJane 5y ago
> The comparison is not very fair to modern day xargs. I am curious how you come to that conclusion. > `nproc` is a relatively standard utility (coreutils). So, xargs -P$(nproc) gets you core (or core-proportional) parallelism. I follow you on this point. A bit harder on remote systems, but definitely doable. > Grouping output/Making a safe parallel grep is also easy-ish with `--process-slot-var=slot` and sending to `tmpOut.$slot`. I tried spending 5 minutes on coding this, but the details seem to be very hard to get right: composed commands, grouping stderr, combined with not leaving tmp files behind if killed and allowing for the total output to be bigger than the free space on /tmp. I could not do it. Could you consider spending 5 minutes on showing in code how you would do it? > Jobs on remote computers can be done similarly with any kind of `arrayVar[$slot]` setup where `arrayVar` has a bunch of `ssh` targets, possibly duplicates if you want to run >1 job per host. (In pure POSIX sh you could use eval and $1, $2 positional args with shell arithmetic..) This one seemed even harder to me: It was completely unclear how you would make sure that a given number of jobs were constantly running. And how would you need to quote data, so an eval would not cause "foo space space bar" turn into "foo space bar". And how you would kill remote jobs, if the local script was killed. If you believe this is simple, could you spend 5 minutes on showing the rest of us how you would do it in actual working code? Because it seems the devil is really in the detail. > Last I looked at the source for GNU parallel it looked like mountains upon mountains of Perl I would rather not depend upon, personally, but to each his own. Personally, I would take production tested code over home-made untested code any day - no matter the language in which it was written.
- cb321 5y ago
  > allowing for the total output to be bigger than the free space on /tmp. I could not do it. This is an unreasonable standard when you do not know in advance how big the output is. What do you imagine GNU parallel does? Use `df` on every host it knows about to fill every disk partition it can? That sounds like a pretty system-hostile behavior to me. Meanwhile, putting your temp files somewhere bigger is obv. as easy as $TMPDIR or such. Best wishes/luck. I only have 5 minutes to explain why nothing can do the impossible like read a user's mind about disk free space management or the value of partial results. All software makes some assumptions... :-)