5 ms·

Use a CoW/snapshotting backing file system for your backup (btrfs/zfs), or one of the modern de-duplicating backup systems (bup/borg/restic), and the space take

Use a CoW/snapshotting backing file system for your backup (btrfs/zfs), or one of the modern de-duplicating backup systems (bup/borg/restic), and the space taken will be proportional to the changed data, while still providing random-access to any snapshot -- Not much more you can expect in terms of space usage. snapshotting filesystems will be much more efficient in terms of time/effort compared to de-duplicating later, but other than that, all of these solutions are mostly equivalent.

icefo 6y ago
I don't argue that it would work but when you do that you're still sending all the data over some data link. For example if I have 1.5TB of VM and only 10-20Gb changed today I'm still sending everything to a remote storage. Snapshotting locally and sending the zfs volume seems to be a more efficient solution
- beagle3 6y ago
  That's not at all true for bup and borg, almost certainly not true for restic (Haven't used it myself and can't tell for sure). bup and borg (and likely restic) keep local hash summaries that are synchronized with the repository if it is remote. They would still have to _read_ the entire 1.5TB VMS if every vm file has changed (there are no facilities on a regular file system that they can use to figure out which parts changed), but they WILL use those local hashes to find out parts actually changed, and only send those changes to the remote storage. The local hashes typically occupy 0.5% of the data size (for bup, by default: 20 byte SHA1 per 8192 bytes on average; You can change the chunking parameters for bup/borg/restic for a tradeoff between cache size and granularity of change detected; for VMs it might make more sense to e.g. have 256KB chunks in which case you'll have 0.01% local storage overhead, and likely still negligible network transfer overhead. > I don't argue that it would work but when you do that you're still sending all the data over some data link. That's only true if you include e.g. the SATA link in that list; To avoid that, you need a filesystem that effectively tracks change ("damage") regions, or VM storage that uses lots of underlying files so you can track it on a regular file system (parallels on the mac used to do that so time machine backups are efficient). no way around it.
  - toomuchtodo 6y ago
    > there are no facilities on a regular file system that they can use to figure out which parts changed Doesn't ZFS do block level checksums that could be used for this, and backup tooling could use block pointer metadata for this purpopse?
    - beagle3 6y ago
      Yes, I mentioned that you need a file system that can do that (afaik, zfs, btrfs on Linux, and hammer2 on dragonfly) Or that you’ll need to read it all.
- amarshall 6y ago
  This should be able to be avoided if the backup target is also ZFS by using `zfs send`.

icefo 6y ago

I don't argue that it would work but when you do that you're still sending all the data over some data link. For example if I have 1.5TB of VM and only 10-20Gb changed today I'm still sending everything to a remote storage. Snapshotting locally and sending the zfs volume seems to be a more efficient solution

beagle3 6y ago
That's not at all true for bup and borg, almost certainly not true for restic (Haven't used it myself and can't tell for sure). bup and borg (and likely restic) keep local hash summaries that are synchronized with the repository if it is remote. They would still have to _read_ the entire 1.5TB VMS if every vm file has changed (there are no facilities on a regular file system that they can use to figure out which parts changed), but they WILL use those local hashes to find out parts actually changed, and only send those changes to the remote storage. The local hashes typically occupy 0.5% of the data size (for bup, by default: 20 byte SHA1 per 8192 bytes on average; You can change the chunking parameters for bup/borg/restic for a tradeoff between cache size and granularity of change detected; for VMs it might make more sense to e.g. have 256KB chunks in which case you'll have 0.01% local storage overhead, and likely still negligible network transfer overhead. > I don't argue that it would work but when you do that you're still sending all the data over some data link. That's only true if you include e.g. the SATA link in that list; To avoid that, you need a filesystem that effectively tracks change ("damage") regions, or VM storage that uses lots of underlying files so you can track it on a regular file system (parallels on the mac used to do that so time machine backups are efficient). no way around it.
- toomuchtodo 6y ago
  > there are no facilities on a regular file system that they can use to figure out which parts changed Doesn't ZFS do block level checksums that could be used for this, and backup tooling could use block pointer metadata for this purpopse?
  - beagle3 6y ago
    Yes, I mentioned that you need a file system that can do that (afaik, zfs, btrfs on Linux, and hammer2 on dragonfly) Or that you’ll need to read it all.
amarshall 6y ago
This should be able to be avoided if the backup target is also ZFS by using `zfs send`.