3 ms·

I prefer rsync because of its delta transfer which doesn't resend files already on the destination, saving bandwidth. This combined with rsync's ability to work

I prefer rsync because of its delta transfer which doesn't resend files already on the destination, saving bandwidth. This combined with rsync's ability to work over ssh lets me sync anywhere rsync runs, including the cloud. It may not be faster than rclone but it is more conserving on bandwidth.

HPsquared 9mo ago
Rclone can "sync" with a range of different ways to check if the existing files are the same. If no hashes are available (e.g. WebDAV) I think you can set it to check by timestamp (with a tolerance) and size. Edit: oh I see, delta transfer only sends the changed parts of files?
- newsoftheday 9mo ago
  It only sends the changed parts of files (the diffs) is my understanding which saves bandwidth.
  - mnw21cam 9mo ago
    However this is only really good for slow connections. If your connection is faster than about 50MB/s then the delta calculation mechanism becomes the bottleneck. On fast connections you should use the -W option for rsync which switches the delta algorithm off.
    - newsoftheday 9mo ago
      Saving on transmitted bandwidth costs is more important to me than speed of transmission for my use cases which I feel are likely to be common use cases.
plagiarist 9mo ago
Does rclone not do that? I thought they were specifically naming themselves similarly because they also did that.
- newsoftheday 9mo ago
  My understanding is that rclone does not do true delta sync sending only the differing parts of files like rsync.
kbr2000 9mo ago
The delta-transfer algorithm [0] is about detecting which chunks of a file differ on source and target [1], and limiting the transfer to those chunks. The savings depend on how and where they differ, and ofcourse there's tradeoffs... You seem to be referring to the selection of candidates of files to transfer (along several possible criteria like modification time, file size or file contents using checksumming) [2] Rsync is great. However for huge filesystems (many files and directories) with relatively less change, you'll need to think about "assisting" it somewhat (by feeding it its candidates obtained in a more efficient way, using --files-from=). For example: in a renderfarm system you would have additions of files, not really updates. Keep a list of frames that have finished rendering (in a cinematic film production this could be eg. 10h/frame), and use it to feed rsync. Otherwise you'll be spending hours for rsync to build its index (both sides) over huge filesystems, instead of transferring relatively few big and new files. In workloads where you have many sync candidates (files) that have a majority of differing chunks, it might be worth rather disabling the delta-transfer algorithm (--whole-file) and saving on the tradeoffs. [0] https://www.andrew.cmu.edu/course/15-749/READINGS/required/cas/tridgell96.pdf https://www.andrew.cmu.edu/course/15-749/READINGS/required/c... [1] https://en.wikipedia.org/wiki/Rsync#Determining_which_parts_of_a_file_have_changed https://en.wikipedia.org/wiki/Rsync#Determining_which_parts_... [2] https://en.wikipedia.org/wiki/Rsync#Determining_which_files_to_send https://en.wikipedia.org/wiki/Rsync#Determining_which_files_...
kalleboo 9mo ago
In my case I get 8 MB/s with rsync, 80 MB/s with rclone. This is with cross-continent transfers so 200ms latency. If it was just a 2x difference I'd be fine with rsync but 10x is very hard to ignore.
- newsoftheday 9mo ago
  For me it's about saving bandwidth which is why I use rsync.

HPsquared 9mo ago

Rclone can "sync" with a range of different ways to check if the existing files are the same. If no hashes are available (e.g. WebDAV) I think you can set it to check by timestamp (with a tolerance) and size. Edit: oh I see, delta transfer only sends the changed parts of files?

newsoftheday 9mo ago
It only sends the changed parts of files (the diffs) is my understanding which saves bandwidth.
- mnw21cam 9mo ago
  However this is only really good for slow connections. If your connection is faster than about 50MB/s then the delta calculation mechanism becomes the bottleneck. On fast connections you should use the -W option for rsync which switches the delta algorithm off.
  - newsoftheday 9mo ago
    Saving on transmitted bandwidth costs is more important to me than speed of transmission for my use cases which I feel are likely to be common use cases.

plagiarist 9mo ago

Does rclone not do that? I thought they were specifically naming themselves similarly because they also did that.

newsoftheday 9mo ago
My understanding is that rclone does not do true delta sync sending only the differing parts of files like rsync.

kbr2000 9mo ago

The delta-transfer algorithm [0] is about detecting which chunks of a file differ on source and target [1], and limiting the transfer to those chunks. The savings depend on how and where they differ, and ofcourse there's tradeoffs... You seem to be referring to the selection of candidates of files to transfer (along several possible criteria like modification time, file size or file contents using checksumming) [2] Rsync is great. However for huge filesystems (many files and directories) with relatively less change, you'll need to think about "assisting" it somewhat (by feeding it its candidates obtained in a more efficient way, using --files-from=). For example: in a renderfarm system you would have additions of files, not really updates. Keep a list of frames that have finished rendering (in a cinematic film production this could be eg. 10h/frame), and use it to feed rsync. Otherwise you'll be spending hours for rsync to build its index (both sides) over huge filesystems, instead of transferring relatively few big and new files. In workloads where you have many sync candidates (files) that have a majority of differing chunks, it might be worth rather disabling the delta-transfer algorithm (--whole-file) and saving on the tradeoffs. [0] https://www.andrew.cmu.edu/course/15-749/READINGS/required/cas/tridgell96.pdf https://www.andrew.cmu.edu/course/15-749/READINGS/required/c... [1] https://en.wikipedia.org/wiki/Rsync#Determining_which_parts_of_a_file_have_changed https://en.wikipedia.org/wiki/Rsync#Determining_which_parts_... [2] https://en.wikipedia.org/wiki/Rsync#Determining_which_files_to_send https://en.wikipedia.org/wiki/Rsync#Determining_which_files_...

kalleboo 9mo ago

In my case I get 8 MB/s with rsync, 80 MB/s with rclone. This is with cross-continent transfers so 200ms latency. If it was just a 2x difference I'd be fine with rsync but 10x is very hard to ignore.

newsoftheday 9mo ago
For me it's about saving bandwidth which is why I use rsync.