3 ms·
I'm an rsync.net customer, and I use ZFS locally, but I'm not using ZFS replication because I can't control in what order the files in the filesystem get backed
by LeoPanthera 11y ago
I'm an rsync.net customer, and I use ZFS locally, but I'm not using ZFS replication because I can't control in what order the files in the filesystem get backed up. (And I'm also unsure whether it's even possible interrupt a transfer and resume it later, if the underlying data has changed. rsync does this no problem.)
My filesystems are pretty massive, but my internet connection is pretty crappy. I currently am running a script I hacked together which rsyncs my files up to rsync.net in (roughly) file size order - smallest first, largest last. This way, changes to huge files don't spend hours/days backing up, and preventing the backup of smaller files.
I haven't yet found a better way to do this.
- gaadd33 11y agoDoesn't ZFS replication send a snapshot? So the underlying data changing doesn't have any impact on the snapshot's view of the world.
- LeoPanthera 11y agoYes, but after the data has changed, I don't want to waste time sending data in the snapshot that is no longer in the live data. I'd rather just carry on from where it is now. My current system does this just fine.
- masklinn 11y agoSo you add a large amount of data, you snapshot, you delete that data and at that point you send the snapshot but don't want to send the deleted data? Why not just re-snapshot and send the new snapshot instead of the old one? > And I'm also unsure whether it's even possible interrupt a transfer and resume it later, if the underlying data has changed There is no "underlying data has changed" in a snapshot, it's a revision of the filesystem. As for interrupting and resuming later, Delphix apparently has contributed a resumable zfs send/receive to OpenZFS, though I don't know if it's available in yours or rsync.net's distribution: http://blog.delphix.com/matt/2015/03/25/resumable-zfs-sendreceive/ http://blog.delphix.com/matt/2015/03/25/resumable-zfs-sendre... As for the concerns of sending small files first, I don't think ZFS could even use that: I don't believe it's content-addressed, so if you send a set of blocks from snapshot 1, interrupt and send snapshot 2 instead the blocks common to both will still be re-sent, ZFS only knows the block commonality from snapshots and since snapshot 1 hasn't been pushed its contents can't be used for snapshot 2. Unless the resumable replication support uses content addressing but that doesn't seem to be the case.
- LeoPanthera 11y ago> Why not just re-snapshot and send the new snapshot instead of the old one? Because then you have to start form scratch every time. The snapshots are so large that if you wanted to send a complete one, a large amount of stuff will have changed.
- tjfontaine 11y agoOpenZFS has resumable send and receive -- so even if they don't have it today, they could have it in the future. http://blog.delphix.com/matt/2015/03/25/resumable-zfs-sendreceive/ http://blog.delphix.com/matt/2015/03/25/resumable-zfs-sendre... https://www.illumos.org/issues/2605 https://www.illumos.org/issues/2605
- rincebrain 11y agoOZFS has it, but ZoL doesn't (yet), so I'd guess they don't yet. https://github.com/zfsonlinux/zfs/issues/3896 https://github.com/zfsonlinux/zfs/issues/3896 e: Ah, they run ZFS on FreeBSD, but I don't think FBSD has it yet either...
- jlgaddis 11y agoIt's not in FreeBSD yet (1Q or 2Q16, IIRC, for -current) but when it is, I assume rsync.net will definitely support it.