3 ms·
But in the spirit of Unix, shouldn't parallel exec some file transfer program?
by oblio 10y ago
But in the spirit of Unix, shouldn't parallel exec some file transfer program?
- ole_tange 10y agoIt does: Rsync.
- dagw 10y agoThe actual Perl code calls out to ssh and rsync (or can be configured to use something else) when it's time to actually connect and transfer files. It just does it in a way that is nice and reasonably transparent to the end user. It felt like his complaint was that that was 'bloat' since Real Men can achieve almost the same thing by just piping some output through some bash scripts they just hacked together.
- ole_tange 10y agoAnd it is exactly the hacking part that GNU Parallel tries to help with: A lot of the helper functions in GNU Parallel could be done by expert users (--nice, --tmux, --pipepart, env_parallel, --compress, --fifo, --cat, --transfer, --return, --cleanup, --(n)onall). But non-expert users will invariably make mistakes (e.g. get quoting wrong, not getting remote jobs to die if the controlling process is killed, or re-scheduling jobs that were killed by --timeout), and why not just have small wrapper scripts built into GNU Parallel that are well-tested, so the non-expert users can enjoy the same stability as the expert users?
- dagw 10y agoHaving written my fair share of those hacky wrapper scripts before I discovered GUN parallel I certainly am very happy that they offer everything I need in a single easy to use command.