5 ms·

I don't argue that it would work but when you do that you're still sending all the data over some data link. For example if I have 1.5TB of VM and only 10-20Gb

by icefo 6y ago

I don't argue that it would work but when you do that you're still sending all the data over some data link. For example if I have 1.5TB of VM and only 10-20Gb changed today I'm still sending everything to a remote storage. Snapshotting locally and sending the zfs volume seems to be a more efficient solution

beagle3 6y ago
That's not at all true for bup and borg, almost certainly not true for restic (Haven't used it myself and can't tell for sure). bup and borg (and likely restic) keep local hash summaries that are synchronized with the repository if it is remote. They would still have to _read_ the entire 1.5TB VMS if every vm file has changed (there are no facilities on a regular file system that they can use to figure out which parts changed), but they WILL use those local hashes to find out parts actually changed, and only send those changes to the remote storage. The local hashes typically occupy 0.5% of the data size (for bup, by default: 20 byte SHA1 per 8192 bytes on average; You can change the chunking parameters for bup/borg/restic for a tradeoff between cache size and granularity of change detected; for VMs it might make more sense to e.g. have 256KB chunks in which case you'll have 0.01% local storage overhead, and likely still negligible network transfer overhead. > I don't argue that it would work but when you do that you're still sending all the data over some data link. That's only true if you include e.g. the SATA link in that list; To avoid that, you need a filesystem that effectively tracks change ("damage") regions, or VM storage that uses lots of underlying files so you can track it on a regular file system (parallels on the mac used to do that so time machine backups are efficient). no way around it.
- toomuchtodo 6y ago
  > there are no facilities on a regular file system that they can use to figure out which parts changed Doesn't ZFS do block level checksums that could be used for this, and backup tooling could use block pointer metadata for this purpopse?
  - beagle3 6y ago
    Yes, I mentioned that you need a file system that can do that (afaik, zfs, btrfs on Linux, and hammer2 on dragonfly) Or that you’ll need to read it all.
amarshall 6y ago
This should be able to be avoided if the backup target is also ZFS by using `zfs send`.

by icefo 6y ago

beagle3 6y ago

That's not at all true for bup and borg, almost certainly not true for restic (Haven't used it myself and can't tell for sure). bup and borg (and likely restic) keep local hash summaries that are synchronized with the repository if it is remote. They would still have to _read_ the entire 1.5TB VMS if every vm file has changed (there are no facilities on a regular file system that they can use to figure out which parts changed), but they WILL use those local hashes to find out parts actually changed, and only send those changes to the remote storage. The local hashes typically occupy 0.5% of the data size (for bup, by default: 20 byte SHA1 per 8192 bytes on average; You can change the chunking parameters for bup/borg/restic for a tradeoff between cache size and granularity of change detected; for VMs it might make more sense to e.g. have 256KB chunks in which case you'll have 0.01% local storage overhead, and likely still negligible network transfer overhead. > I don't argue that it would work but when you do that you're still sending all the data over some data link. That's only true if you include e.g. the SATA link in that list; To avoid that, you need a filesystem that effectively tracks change ("damage") regions, or VM storage that uses lots of underlying files so you can track it on a regular file system (parallels on the mac used to do that so time machine backups are efficient). no way around it.

toomuchtodo 6y ago
> there are no facilities on a regular file system that they can use to figure out which parts changed Doesn't ZFS do block level checksums that could be used for this, and backup tooling could use block pointer metadata for this purpopse?
- beagle3 6y ago
  Yes, I mentioned that you need a file system that can do that (afaik, zfs, btrfs on Linux, and hammer2 on dragonfly) Or that you’ll need to read it all.

amarshall 6y ago

This should be able to be avoided if the backup target is also ZFS by using `zfs send`.