3 ms·

On the topic of xargs replacements, I love gnu parallel. The --dry-run flag of parallel made me confident to do more batch processing than I ever did with xarg

On the topic of xargs replacements, I love gnu parallel. The --dry-run flag of parallel made me confident to do more batch processing than I ever did with xargs. Parallel has an option for almost everything, it's almost too much. But I have shopped around for alternatives. The creator Ole Tange maintains a painstakingly long article of the alternatives and their differences. [0] The gnu parallel book and reading materials [1] are excellent too. [0] https://www.gnu.org/software/parallel/parallel_alternatives.html https://www.gnu.org/software/parallel/parallel_alternatives.... [1] https://www.gnu.org/software/parallel/#Tutorial https://www.gnu.org/software/parallel/#Tutorial

somat 3mo ago
I have used echo as a sort of poor mans equivalent for a safe check of a pipeline, removing the echo when I felt the rest of the pipeline was working correctly. shell stuff | xargs -n 1 -I % echo real command and % args I have also been known to write scripts where instead of executing the critical parts it prints them. Then a dry run is script and the real run is script | sh
- bryancoxwell 2mo ago
  Love that idea, thanks for sharing
- krackers 2mo ago
  Shouldn't you use echo instead of cat?
  - somat 2mo ago
    yes echo, total brain fart there. corrected.
quotemstr 3mo ago
I love GNU Parallel too. I love how it's free software and how I can patch out the obnoxious citation notice.
dataflow 3mo ago
Oh man, parallel is awful. I have to rant about this because I literally tried it again today morning. Every single damn time I try parallel and decide to give it another chance, something ends up not working or causing a problem. I can never get it to just do what I want and get out of the way. Today I foolishly thought maybe I was the one who was holding it wrong every single time in the past, so I copy pasted another command that was supposed to work, and thought surely this would be straightforward. Boy was I wrong. I got some manifesto about academic citations and plagiarism, which confused the hell out of me. After I wasted time trying to figure out how to turn off that nonsense, the app just hung there trying to figure out how long its command line can be? Literally doing nothing? What the hell? I killed it but then my terminal didn't close because every time I did this apparently some perl command was spawned in the background blocked on nothing. Why the hell was perl even relevant? Nothing I wrote used Perl. Just run the darn commands I asked in parallel, is that so hard?
- nvme0n1p1 3mo ago
  > the app just hung there trying to figure out how long its command line can be? That's common on linux. Many tools read from stdin if a file path isn't given: cat, xargs, base64, cksum, etc. The citation thing is a little silly, I'll give you that one.
  - deleted 3mo ago
    [deleted]
  - dataflow 2mo ago
    > That's common on linux. Many tools read from stdin if a file path isn't given: cat, xargs, base64, cksum, etc. No. I did pipe to stdin. It's not my first time using Linux... Here's a command line I ran right now, and the output I see: $ echo "http://www.example.com" | parallel -k -j 8 curl -s "{}" Academic tradition requires you to cite works you base your article on. [...more nonsense...] To silence this citation notice: run 'parallel --citation' once. parallel: Warning: Finding the maximal command line length. This may take up to 1 minute. So I wait a few seconds... until I get fed up and look at my process list, and I see perl is just... seemingly sitting there, doing seemingly absolutely nothing. I'm not going to waste a whole minute of my life waiting for this; I see no reason competently written software should take that long just to accomplish such a simple task where nothing is remotely close to reaching any limits. So I Ctrl+C. And then the parent perl process gets killed, but the child apparently keeps running. I press Ctrl+D to exit the terminal, and then: $ # (Ctrl+D pressed) logout ...it just sits there waiting. Ctrl+C and Ctrl+\ do nothing. I have to kill the lingering perl process manually. xargs Just Works without any of this nonsense, yet somehow I'm the one holding GNU parallel wrong?
jeffbee 3mo ago
Parallel is ok but rush is nicer and much faster. https://github.com/shenwei356/rush https://github.com/shenwei356/rush
- foobarqux 3mo ago
  The rush project page on Github itself says it is only "slightly" faster.
  - jeffbee 2mo ago
    Modesty, I suppose. The larger your machine is, the more obviously faster it is.
metadat 3mo ago
Every time I’ve used Gnu Parallel on a new system it required accepting a Eula, which is annoying. I stick to xargs -P99 these days and am happier. Utility software which has non-essential different first-run behavior is hostile to users.
sidewndr46 2mo ago
isn't parallel that tool that dumps some kind of begging message into your output each time you invoke it?
- shawn_w 2mo ago
  No? You can tell it to stop printing the citation notice.
  - saghm 2mo ago
    That sounds a lot more like "yes, but..." than "no"
- Ferret7446 2mo ago
  Do you really equate a one time request to cite something if you use it in an academic paper with begging? I know they say that users of open source software are entitled, but this seems rather extreme.
deleted 2mo ago
[deleted]

somat 3mo ago

I have used echo as a sort of poor mans equivalent for a safe check of a pipeline, removing the echo when I felt the rest of the pipeline was working correctly. shell stuff | xargs -n 1 -I % echo real command and % args I have also been known to write scripts where instead of executing the critical parts it prints them. Then a dry run is script and the real run is script | sh

bryancoxwell 2mo ago
Love that idea, thanks for sharing
krackers 2mo ago
Shouldn't you use echo instead of cat?
- somat 2mo ago
  yes echo, total brain fart there. corrected.

quotemstr 3mo ago

I love GNU Parallel too. I love how it's free software and how I can patch out the obnoxious citation notice.

dataflow 3mo ago

Oh man, parallel is awful. I have to rant about this because I literally tried it again today morning. Every single damn time I try parallel and decide to give it another chance, something ends up not working or causing a problem. I can never get it to just do what I want and get out of the way. Today I foolishly thought maybe I was the one who was holding it wrong every single time in the past, so I copy pasted another command that was supposed to work, and thought surely this would be straightforward. Boy was I wrong. I got some manifesto about academic citations and plagiarism, which confused the hell out of me. After I wasted time trying to figure out how to turn off that nonsense, the app just hung there trying to figure out how long its command line can be? Literally doing nothing? What the hell? I killed it but then my terminal didn't close because every time I did this apparently some perl command was spawned in the background blocked on nothing. Why the hell was perl even relevant? Nothing I wrote used Perl. Just run the darn commands I asked in parallel, is that so hard?

nvme0n1p1 3mo ago
> the app just hung there trying to figure out how long its command line can be? That's common on linux. Many tools read from stdin if a file path isn't given: cat, xargs, base64, cksum, etc. The citation thing is a little silly, I'll give you that one.
- deleted 3mo ago
  [deleted]
- dataflow 2mo ago
  > That's common on linux. Many tools read from stdin if a file path isn't given: cat, xargs, base64, cksum, etc. No. I did pipe to stdin. It's not my first time using Linux... Here's a command line I ran right now, and the output I see: $ echo "http://www.example.com" | parallel -k -j 8 curl -s "{}" Academic tradition requires you to cite works you base your article on. [...more nonsense...] To silence this citation notice: run 'parallel --citation' once. parallel: Warning: Finding the maximal command line length. This may take up to 1 minute. So I wait a few seconds... until I get fed up and look at my process list, and I see perl is just... seemingly sitting there, doing seemingly absolutely nothing. I'm not going to waste a whole minute of my life waiting for this; I see no reason competently written software should take that long just to accomplish such a simple task where nothing is remotely close to reaching any limits. So I Ctrl+C. And then the parent perl process gets killed, but the child apparently keeps running. I press Ctrl+D to exit the terminal, and then: $ # (Ctrl+D pressed) logout ...it just sits there waiting. Ctrl+C and Ctrl+\ do nothing. I have to kill the lingering perl process manually. xargs Just Works without any of this nonsense, yet somehow I'm the one holding GNU parallel wrong?

jeffbee 3mo ago

Parallel is ok but rush is nicer and much faster. https://github.com/shenwei356/rush https://github.com/shenwei356/rush

foobarqux 3mo ago
The rush project page on Github itself says it is only "slightly" faster.
- jeffbee 2mo ago
  Modesty, I suppose. The larger your machine is, the more obviously faster it is.

metadat 3mo ago

Every time I’ve used Gnu Parallel on a new system it required accepting a Eula, which is annoying. I stick to xargs -P99 these days and am happier. Utility software which has non-essential different first-run behavior is hostile to users.

sidewndr46 2mo ago

isn't parallel that tool that dumps some kind of begging message into your output each time you invoke it?

shawn_w 2mo ago
No? You can tell it to stop printing the citation notice.
- saghm 2mo ago
  That sounds a lot more like "yes, but..." than "no"
Ferret7446 2mo ago
Do you really equate a one time request to cite something if you use it in an academic paper with begging? I know they say that users of open source software are entitled, but this seems rather extreme.

deleted 2mo ago

[deleted]