6 ms·
It’s a shame that this tool is so incredibly useful but simultaneously so unergonomic to use that it requires a 12 page manual. Every time I reach for it the m
by tbrock 7y ago
It’s a shame that this tool is so incredibly useful but simultaneously so unergonomic to use that it requires a 12 page manual.
Every time I reach for it the magical incantations that finally get it working properly are so completely unmemorable that it’s begging to have a modern replacement (like httpie is to curl).
If someone wrote a version with a better interface it would be instantly adopted in preference to this. Might be a good weekend project.
- feanaro 7y agoI agree with the sentiment, but please, Rust, not Go. In fact, I remember seeing such a project, let me search... Ah, here it is: https://gitlab.redox-os.org/redox-os/parallel https://gitlab.redox-os.org/redox-os/parallel
- tbrock 7y agoOk, whichever language.
- krapht 7y agoWhat? Why? Seriously getting fed up with the Rust lovers in Hacker News. It makes me hate Rust even though I actually really like it as a modern replacement for C++. Memory safety is only one aspect of proving program correctness. It's also less of an issue for garbage collected languages, and tracing GCs are fairly performant. You don't see Coq users crapping on other languages for not proving their programs correct according to a formally defined specification.
- 13415 7y agoThe Rust community is the main reason why I don't use Rust.
- sigmonsays 7y agolets not start a language war here
- hi41 7y agoCould you please elaborate on why Rust would be better than Go? Doesn’t Go offer good support for parallelism, which would be great use for implementing GNU Parallel.
- PeCaN 7y agoOne shouldn't have to elaborate on why Rust is better than anything. It's simply the best language for all tasks in 2019. It's irresponsible not to rewrite everything in Rust at this point.
- mechanical_jane 7y agoSome of the alternatives to GNU Parallel are written in Go: https://www.gnu.org/software/parallel/parallel_alternatives.html https://www.gnu.org/software/parallel/parallel_alternatives.... Speed wise they are comparable to GNU Parallel. So the argument for Rust could be speed. The argument against Rust could be backwards compatibility: How would you do the {= perl expr =} thing? Will it run on the current targets (CentOS 3.9 according to https://www.gnu.org/software/parallel/parallel_design.html#Old-Perl-style https://www.gnu.org/software/parallel/parallel_design.html#O...)? Will it run on systems for which you as a consultant do not have access to a Rust compiler for?
- kylek 7y agoMaybe not any better syntax, but xargs is usually installed by default in most distros, and I've never found parallel to be advantageous over it. I don't think this particular wheel needs to be invented again.
- tbrock 7y agoxargs is great but doesn’t run the commands in parallel or chunk input does it?
- saxonww 7y agoIt can, check the -n and -P arguments.
- kylek 7y ago-P sets parallelism, -n number of arguments for each execution. Those are my main weapons. (-i is usually also necessary, and can allow for some pretty creative situations)
- hyperpallium 7y agoI don't swear on HN, but Holy Shit, I just added -P to xargs in a testsuite bash script, and it's 7 times faster. I only have quad core, but -P values higher than 4 give better performance. Maybe because the tests take different amounts of time, and it avoids them backing up. To get x7, I had -P equal to the number of tests.
- waterhouse 7y agoIf the tests are ever waiting for I/O or something, or otherwise using less than one full CPU core, then I think that could explain why running more than the number of cores would be an improvement. If you run "top", or use the "time" utility, what fraction of a CPU core does each test use?
- 7y ago
- benbenolson 7y agoWhy does the language that it's written in matter? For most users, it's a compiled binary sitting on your system. EDIT: Parent comment edited his post, leaving out his personal preference of language.
- fmajid 7y agoParallel is written in Perl, not a compiled binary. That may be what he was referring to. I also wish Perl (and ruby, and Java) would go away.
- jarfil 7y agoThe biggest problem with parallel is not the implementation but the command line options specification. The shell has no generic way of defining multiple streams feeding multiple placeholders, so parallel ends up using its own syntax which has to be different in order not to interfere with the base shell syntax. If the same functionality was integrated directly into the shell, it could be less complicated and more generic.
- tbrock 7y agoExactly. The implementation is fine, it’s the interface that isn’t great.
- deleted 7y ago[deleted]
- faitswulff 7y ago> ..a 12 page manual. My first thought was, "12 pages isn't so bad." Then I loaded the page and realized you had a typo and meant _112_ page manual. That is enough to scare me off of trying to use parallel for casual tasks.
- LeoPanthera 7y agoI routinely use it for casual tasks and I've never read the manual. Don't confuse complexity with difficulty.
- gnulinux 7y ago> That is enough to scare me off of trying to use parallel for casual tasks. Just because emacs is such a complex software, does that mean no one can start learning it? No. As far as GNU parallel goes simply just understand what `:::` operator does and you can start using it in your workflow today. Software is not complex for the sake of beginners, it is complex so experts have a higher variety of features.
- mechanical_jane 7y agoMaybe the cheat sheet is made for you: https://www.gnu.org/software/parallel/parallel_cheat.pdf https://www.gnu.org/software/parallel/parallel_cheat.pdf
- yesco 7y agoWell I had never heard of httpie before now so thanks for casually dropping that endorsement. Never realized how much I wanted something like this.
- dahart 7y ago> It’s a shame that this tool is so incredibly useful but simultaneously so unergonomic to use that it requires a 12 page manual. It doesn’t require reading the whole manual, nor does the length of the manual have anything to do with parallel’s ergonomics. Do you use bash or ssh? Have you looked at the length of those manuals? Most of parallel’s manual is advanced workflows the casual user doesn’t need, stuff like parallel commands run over a network of machines and collecting the results on the originating host. Maybe you could show some examples of what you mean, because I think parallel is pretty easy to use. Personally I feel like it’s usually easier than Xargs, especially when dealing with tricky quoting. It does have some of it’s own syntax, like the triple-colon :::, and it’s really easy to use. Several pages of manual are about variable substitutions you can use if you want, they’re very similar to bash variable substitutions, and they’re all optional and super convenient. A bunch of parallel’s manual pages are devoted to simple examples, which is nice, and the kind of thing that’s missing from so many man pages. > it’s begging to have a modern replacement I’m not sure what “modern” means. You could certainly propose some changes though, or show some examples of the kind of syntax that isn’t working for you. Httpie and curl are built to do different things.
- Mizza 7y agoIf you're interested, I've written a tool called `loop` which sacrifices some of the abilities of GNU Parallel for a far more intuitive interface (I think so, anyway.) You can check it out here: https://github.com/Miserlou/Loop https://github.com/Miserlou/Loop
- dahart 7y ago> sacrifices some of the abilities of GNU Parallel Like... the ability to run things in parallel? Loop looks nice, don’t get me wrong. I think I’ll try it! It just seems like comparing it to Parallel is misplaced here, and a bad idea in general, since they’re not remotely related. Ole is comparing all these utilities to Parallel to show how general Parallel can be, and suggest you only need his one program and one syntax to emulate hundreds of different utilities: his argument is implicitly that people should use Parallel and not loop.
- Mizza 7y agoYeah, but that syntax is: $ yes | parallel -N0 -j1 echo touch '{= $_=seq()*5 =}'.txt vs $ loop --count-by 5 -- touch $COUNT.txt UNIX is about using the right tool for the right job, not the same tool for every job. Let us remember the koan: A Unix novice came to Master Foo and said: “I am confused. Is it not the Unix way that every program should concentrate on one thing and do it well?” Master Foo nodded. The novice continued: “Isn't it also the Unix way that the wheel should not be reinvented?” Master Foo nodded again. “Why, then, are there several tools with similar capabilities in text processing: sed, awk and Perl? With which one can I best practice the Unix way?” Master Foo asked the novice: “If you have a text file, what tool would you use to produce a copy with a few words in it replaced by strings of your choosing?” The novice frowned and said: “Perl's regexps would be excessive for so simple a task. I do not know awk, and I have been writing sed scripts in the last few weeks. As I have some experience with sed, at the moment I would prefer it. But if the job only needed to be done once rather than repeatedly, a text editor would suffice.” Master Foo nodded and replied: “When you are hungry, eat; when you are thirsty, drink; when you are tired, sleep.” Upon hearing this, the novice was enlightened.
- PeCaN 7y ago> it’s begging to have a modern replacement (like httpie is to curl). This seems like a pretty weird example. All common tasks in curl are very straightforward and I've never even heard of httpie. Let's please not change the interface to basic command line tools every few years because people want something ""modern"".
- brians 7y agoCommon tasks for the web are easy in curl—though it will take me a long time to forget environment variables that could be set to false, true, or two. Httpie makes web apis easier, with better support for api keys and for hashing some headers into others, holding state across several command invocations, that sort of thing.
- RubberSoul 7y agoThere are great videos for understanding how it works [0]. I watched a couple of those and after that was able to do exactly what I needed. For me the trickiest thing is remembering how to deal with redirection. I find the --dry-run option to be super useful when setting up a command. [0]: https://www.youtube.com/playlist?list=PL284C9FF2488BC6D1 https://www.youtube.com/playlist?list=PL284C9FF2488BC6D1
- mechanical_jane 7y agoYeah, quoting redirection is a bitch, but as long as you know GNU Parallel is just a program and not part of the shell, you can usually deduct which parts will be interpreted by the shell before GNU Parallel gets it. I often create bash functions to avoid that shit, and then call the bash functions from GNU Parallel instead.
- msla 7y agoRealistically, the best language for this is C, because everything else is too heavyweight, requires too much of a runtime, or simply isn't available most places.
- mechanical_jane 7y agoDoes that also apply to platforms where you as a consultant for hire does not have access to a C compiler for that given platform? It used to be an issue that non-free UNIXes did not include a C-compiler by default, and then C would really not help. I am not sure if that is the case anymore.
- gioele 7y agoFor those afraid of the complexity of GNU Parallel, there is the tiny moreutils `parallel` (or Tollef's parallel) that covers 90% of the common use cases with a single-page manual. https://manpages.debian.org/testing/moreutils/parallel.1.en.html https://manpages.debian.org/testing/moreutils/parallel.1.en....
- mechanical_jane 7y agoBut looking at the comparison, do you really find GNU Parallel that much harder to understand? parallel sh -c "echo hi; sleep 2; echo bye" -- 1 2 3 parallel echo hi";" sleep 2";" echo bye ::: 1 2 3 parallel -j 3 ufraw -o processed -- *.NEF parallel -j 3 ufraw -o processed ::: *.NEF parallel -j 3 -- ls df "echo hi" parallel -j 3 ::: ls df "echo hi" And if you are looking for the single page overview, why not link to the cheat sheet: https://www.gnu.org/software/parallel/parallel_cheat.pdf https://www.gnu.org/software/parallel/parallel_cheat.pdf
- mechanical_jane 7y agoI think the cheat sheet is made for you: https://www.gnu.org/software/parallel/parallel_cheat.pdf https://www.gnu.org/software/parallel/parallel_cheat.pdf If you have a hard time getting it to work and remember how, then it is likely that the fundamentals of GNU Parallel are not clear to you. Chapter 2 ("Learn GNU Parallel in 15 minutes") of the book should take care of that. > a better interface That is often a tough one. I, for one, like emacs' interface. Others despise it. So when you write 'better interface' I think it will be helpful if you were more specific.
- deleted 7y ago[deleted]