21 ms·
Find|xargs like a boss (Real Example)
- noodly 15y agoComments are closed on that page, so instead I will comment here. There's no need to use xargs, and pipe to sh in this scenario. The same functionality can be achieved by using only find: PROCS=8;cuffs=`which cufflinks`;find `pwd` -name 'accepted_hits.bam' -type f -exec qsub -l ncpus=$PROCS -- $cuffs -p $PROCS -o {}-cufflinks -g genes.gtf {} \;
- barrkel 15y agoCreating a text stream which you then pipe to a shell is a good way of debugging complex operations, especially potentially destructive ones that may need inspection to verify. It's a technique I use a lot, to the point that some of my more destructive scripts don't do anything - they just output shell commands, and it's up to me to pipe them to the shell.
- noodly 15y agoYou can achieve this also by using find and its -ok option, instead of -exec, that way find will ask for confirmation before doing anything.
- JOnAgain 15y agoThanks for the -ok option. was not aware -- but yeah, I've been a heavy user of -exec option for years, and it always blows my mind when people get crazy complicated xargs going.
- barrkel 15y agoThat's not the same thing at all; I could emulate that in a script with 'read' and 'if', but that's not what I do.
- icebraining 15y agoxargs is very useful, although nowadays I usually use GNU parallel instead (it's essentially the same but with a few advantages: https://www.gnu.org/software/parallel/man.html#differences_between_xargs_and_gnu_parallel https://www.gnu.org/software/parallel/man.html#differences_b...)
- makmanalp 15y agoxargs -P is sort of similar
- wging 15y agoThe link in the post to which you replied covers that; it's the section of the man page which deals specifically with differences between xargs and parallel. Having used neither of these tools before, I think two relevant lines from that link would be: >xargs can run a given number of jobs in parallel, but has no support for running number-of-cpu-cores jobs in parallel. >xargs has no support for keeping the order of the output, therefore if running jobs in parallel using xargs the output of the second job cannot be postponed till the first job is done.
- brown9-2 15y agoCame here to say this. GNU Parallel is great.
- saurik 15y agoI would have expected an article on how to use these tools "like a boss" to point out that if there is whitespace in the filenames you will run into problems unless you use "find ... -print0 | xargs -0 ...".
- masklinn 15y agoWhat I'd have expected in an article on how to use this tool is talk about `-exec` and `-exec +` before shooting straight for xargs.
- barrkel 15y agoI very rarely use find -exec, because I usually want to massage the list of files further. So I have a simple utility, print0, which reads lines and outputs null-terminated lines. It turns any list separated with newlines into input suited to xargs -0. File names with embedded newlines are vanishingly rare, compared to those with spaces, so it has always worked out well for me.
- zaptheimpaler 15y agoThis was one I used to backup all websites in a folder: find . -mindepth 1 -maxdepth 1 -type d | awk 'BEGIN {FS="./"}; {print $2}' | xargs -d '\n' tar czf backup1.tar Note that you'll have to change $2 to the correct column number for the name field depending on your output.
- Domenic_S 15y agoWhy not just cp the parent directory?
- sgt 15y agoFor the record, these examples assume GNU xargs, so they won't necessarily work with the BSD xargs that comes with OS X.
- zaphar 15y agoThe author would benefit from replacing his backticks with $(pwd) instead. You can nest this form arbitrarily deep as well as having it interpolate in strings or heredocs.
- gcr 15y ago$(foo) is a bashism; unsupported by different (older) versions of bash and especially sh
- orra 15y agoI don't know how well $(foo) is supported outwith bash in practice, but it is POSIX: http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_06_03 http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3...