5 ms·

Maybe not any better syntax, but xargs is usually installed by default in most distros, and I've never found parallel to be advantageous over it. I don't think

by kylek 7y ago

Maybe not any better syntax, but xargs is usually installed by default in most distros, and I've never found parallel to be advantageous over it. I don't think this particular wheel needs to be invented again.

tbrock 7y ago
xargs is great but doesn’t run the commands in parallel or chunk input does it?
- saxonww 7y ago
  It can, check the -n and -P arguments.
- kylek 7y ago
  -P sets parallelism, -n number of arguments for each execution. Those are my main weapons. (-i is usually also necessary, and can allow for some pretty creative situations)
  - hyperpallium 7y ago
    I don't swear on HN, but Holy Shit, I just added -P to xargs in a testsuite bash script, and it's 7 times faster. I only have quad core, but -P values higher than 4 give better performance. Maybe because the tests take different amounts of time, and it avoids them backing up. To get x7, I had -P equal to the number of tests.
    - waterhouse 7y ago
      If the tests are ever waiting for I/O or something, or otherwise using less than one full CPU core, then I think that could explain why running more than the number of cores would be an improvement. If you run "top", or use the "time" utility, what fraction of a CPU core does each test use?
      - hyperpallium 7y ago
        Good thinking, but they don't do any file io, and I'm not sure how it could use less than a full CPU (wouldn't it just finish sooner? Though waiting for io could do that). OTOH it's java (on android, so dex format) and does some classloading of user classes... so maybe that is io? Here's time results: real 0m1.818s user 0m2.020s sys 0m0.900s top. (It completes in 1 sec, so put in a while loop - BTW top isn't behaving normally, "1" not giving multiple CPUs and not fitting on screen). Jus the top line: User 51%, System 23%, IOW 0%, IRQ 0%
- deleted 7y ago
  [deleted]
- gkfasdfasdf 7y ago
  xargs can do both and it rocks. https://www.google.com/search?q=xargs+parallel https://www.google.com/search?q=xargs+parallel
- icebraining 7y ago
  It does. What xargs can't do is run stuff in parallel across many machines.
  - emmelaich 7y ago
    Yeah, you'd have to combine it with something like https://github.com/erikfrey/bashreduce https://github.com/erikfrey/bashreduce
    - fmajid 7y ago
      Or pssh
ravoori 7y ago
xargs does nothing to prevent interleaving of output from simultaneously executed commands.
- waterhouse 7y ago
  Yes. I use parallel if I need coherent output; otherwise I use xargs (which is significantly faster).
- kylek 7y ago
  I suspect that's intended behavior. But if I'm understanding correctly, I think you could account for it with something like- ... | xargs -P10 -i sh -c 'command {} 1>{}.out 2>{}.err' or maybe with tee: ... | xargs -P10 -i sh -c 'command {} | tee -a {}.out'
  - mechanical_jane 7y ago
    It would require your input not to have any special shell chars - e.g. no space. It is definitely not something I would dare to run in production.
fmajid 7y ago
I dislike parallel’s confusing options, but xargs does not report on progress.
lelf 7y ago
> xargs is usually installed by default xargs is part of POSIX, so it should be everywhere. (-P/-n are not, but both GNU and BSD versions do support it.)

by kylek 7y ago

tbrock 7y ago

xargs is great but doesn’t run the commands in parallel or chunk input does it?

saxonww 7y ago
It can, check the -n and -P arguments.
kylek 7y ago
-P sets parallelism, -n number of arguments for each execution. Those are my main weapons. (-i is usually also necessary, and can allow for some pretty creative situations)
- hyperpallium 7y ago
  I don't swear on HN, but Holy Shit, I just added -P to xargs in a testsuite bash script, and it's 7 times faster. I only have quad core, but -P values higher than 4 give better performance. Maybe because the tests take different amounts of time, and it avoids them backing up. To get x7, I had -P equal to the number of tests.
  - waterhouse 7y ago
    If the tests are ever waiting for I/O or something, or otherwise using less than one full CPU core, then I think that could explain why running more than the number of cores would be an improvement. If you run "top", or use the "time" utility, what fraction of a CPU core does each test use?
    - hyperpallium 7y ago
      Good thinking, but they don't do any file io, and I'm not sure how it could use less than a full CPU (wouldn't it just finish sooner? Though waiting for io could do that). OTOH it's java (on android, so dex format) and does some classloading of user classes... so maybe that is io? Here's time results: real 0m1.818s user 0m2.020s sys 0m0.900s top. (It completes in 1 sec, so put in a while loop - BTW top isn't behaving normally, "1" not giving multiple CPUs and not fitting on screen). Jus the top line: User 51%, System 23%, IOW 0%, IRQ 0%
deleted 7y ago
[deleted]
gkfasdfasdf 7y ago
xargs can do both and it rocks. https://www.google.com/search?q=xargs+parallel https://www.google.com/search?q=xargs+parallel
icebraining 7y ago
It does. What xargs can't do is run stuff in parallel across many machines.
- emmelaich 7y ago
  Yeah, you'd have to combine it with something like https://github.com/erikfrey/bashreduce https://github.com/erikfrey/bashreduce
  - fmajid 7y ago
    Or pssh

ravoori 7y ago

xargs does nothing to prevent interleaving of output from simultaneously executed commands.

waterhouse 7y ago
Yes. I use parallel if I need coherent output; otherwise I use xargs (which is significantly faster).
kylek 7y ago
I suspect that's intended behavior. But if I'm understanding correctly, I think you could account for it with something like- ... | xargs -P10 -i sh -c 'command {} 1>{}.out 2>{}.err' or maybe with tee: ... | xargs -P10 -i sh -c 'command {} | tee -a {}.out'
- mechanical_jane 7y ago
  It would require your input not to have any special shell chars - e.g. no space. It is definitely not something I would dare to run in production.

fmajid 7y ago

I dislike parallel’s confusing options, but xargs does not report on progress.

lelf 7y ago

> xargs is usually installed by default xargs is part of POSIX, so it should be everywhere. (-P/-n are not, but both GNU and BSD versions do support it.)