3 ms·

rclone --multi-thread-streams allows transfers in parallel, like robocopy /MT You can also run multiple instances of rsync, the problem seems how to efficientl

by cachius 8mo ago

rclone --multi-thread-streams allows transfers in parallel, like robocopy /MT You can also run multiple instances of rsync, the problem seems how to efficiently divide the set of files.

pama 8mo ago
Sometimes find (with desired maxdepth) piped to gnu-parallel rsync is fine.
SoftTalker 8mo ago
robocopy! Wow, blast from the past. Used to use it all the time when I worked in a Windows shop.
- bob1029 8mo ago
  I am using robocopy right now on a project. The /MIR option is extremely useful for incrementally maintaining copies of large local directories.
cachius 8mo ago
> efficiently divide the set of files. It turns out, fpart does just that! Fpart is a Filesystem partitioner. It helps you sort file trees and pack them into bags (called "partitions"). It is developed in C and available under the BSD license. It comes with an rsync wrapper, fpsync. Now I'd like to see a benchmark of that vs rclone! via https://unix.stackexchange.com/q/189878/#688469 https://unix.stackexchange.com/q/189878/#688469 via https://stackoverflow.com/q/24058544/#comment93435424_25532027 https://stackoverflow.com/q/24058544/#comment93435424_255320... https://www.fpart.org/ https://www.fpart.org/
adolph 8mo ago
My go-to for fast and easy parallelization is xargs -P. find a-bunch-of-files | xargs -P 10 do-something-with-a-file -P max-procs --max-procs=max-procs Run up to max-procs processes at a time; the default is 1. If max-procs is 0, xargs will run as many processes as possible at a time.
- akdev1l 8mo ago
  note that one should use -print0 and -0 for safety
  - adolph 8mo ago
    Thanks! I've been using the -F{} do-something-tofile "{}" approach which is also handy for times in which the input is one pram among others. -0 is much faster. Edit: Looks like when doing file-by-file -F{} is still needed: # find tmp -type f | xargs -0 ls ls: cannot access 'tmp/b file.md'$'\n''tmp/a file.md'$'\n''tmp/c file.md'$'\n': No such file or directory
    - elteto 8mo ago
      You have to do `find ... -print0` so find also uses \0 as the separator.
    - akdev1l 8mo ago
      find -print0 will print the files with null bytes as separators xargs -0 will use a null byte as separator for each argument printf 'a\0b\0c\0' | xargs -tI{} echo “file -> {}"

by cachius 8mo ago

rclone --multi-thread-streams allows transfers in parallel, like robocopy /MT You can also run multiple instances of rsync, the problem seems how to efficiently divide the set of files.

pama 8mo ago

Sometimes find (with desired maxdepth) piped to gnu-parallel rsync is fine.

SoftTalker 8mo ago

robocopy! Wow, blast from the past. Used to use it all the time when I worked in a Windows shop.

bob1029 8mo ago
I am using robocopy right now on a project. The /MIR option is extremely useful for incrementally maintaining copies of large local directories.

cachius 8mo ago

> efficiently divide the set of files. It turns out, fpart does just that! Fpart is a Filesystem partitioner. It helps you sort file trees and pack them into bags (called "partitions"). It is developed in C and available under the BSD license. It comes with an rsync wrapper, fpsync. Now I'd like to see a benchmark of that vs rclone! via https://unix.stackexchange.com/q/189878/#688469 https://unix.stackexchange.com/q/189878/#688469 via https://stackoverflow.com/q/24058544/#comment93435424_25532027 https://stackoverflow.com/q/24058544/#comment93435424_255320... https://www.fpart.org/ https://www.fpart.org/

adolph 8mo ago

My go-to for fast and easy parallelization is xargs -P. find a-bunch-of-files | xargs -P 10 do-something-with-a-file -P max-procs --max-procs=max-procs Run up to max-procs processes at a time; the default is 1. If max-procs is 0, xargs will run as many processes as possible at a time.

akdev1l 8mo ago
note that one should use -print0 and -0 for safety
- adolph 8mo ago
  Thanks! I've been using the -F{} do-something-tofile "{}" approach which is also handy for times in which the input is one pram among others. -0 is much faster. Edit: Looks like when doing file-by-file -F{} is still needed: # find tmp -type f | xargs -0 ls ls: cannot access 'tmp/b file.md'$'\n''tmp/a file.md'$'\n''tmp/c file.md'$'\n': No such file or directory
  - elteto 8mo ago
    You have to do `find ... -print0` so find also uses \0 as the separator.
  - akdev1l 8mo ago
    find -print0 will print the files with null bytes as separators xargs -0 will use a null byte as separator for each argument printf 'a\0b\0c\0' | xargs -tI{} echo “file -> {}"