3 ms·

Those who do not study ~~history~~ ~~Unix~~ rsync are doomed to repeat it poorly.

• bitL 7y ago
  I used rsync to mirror one encrypted 14TB drive to another one. It took ~2 weeks. I wouldn't call rsync "good enough".
  • manicdee 7y ago
    What was the bottleneck?
    • mynegation 7y ago
      I am not bitl but my guesses, in order of probability would be: * using filesystem that is not natively supported on one or both drives, like fuse ntfs on Linux or ext3 on Mac. * copying over slower network like wifi * lots and lots of small files especially in a single directory * usb 2.0
  • michaelcampbell 7y ago
    Something is very, very wrong then. rsync will use 100% of the bandwidth provided. It won't be any slower than anything else you might try over the same network. If you found that to be the case, it's almost certainly something else.
    • bitL 7y ago
      I guess if you have many overlapping files on both drives and you force rsync to actually look inside and compare them, its efficiency might suffer? It wasn't just a simple copy from one full drive to another empty one, rather two drives with overlapping files that had to be synced. I experimented with rsync to see if I could replace RAID with it, i.e. one primary drive, one backup drive synced on the background instead of mirroring directly (and getting the same low-level bugs spread to both drives or getting both drives to die at around the same time). Both drives were encrypted on the device level (dm-crypt AES128) with standard ext4 on top of mapped drive. I previously ran configs with RAID-1 as well as BTRFS RAID-5 (horribly buggy and dangerous) and wanted to see if rsync makes sense for this purpose (it doesn't). I couldn't use ZFS as that NAS didn't have ECC.