4 ms·

ZFS does that with snapshots already. (ZFS is not encrypted, but the principle is the same) I believe immutable trees would be the correct algorithm here. The

by Nican 7y ago

ZFS does that with snapshots already. (ZFS is not encrypted, but the principle is the same) I believe immutable trees would be the correct algorithm here. The algorithm relies on a tree structure, where the leaves are the files. Every time a new snapshot is written, every leaf node changed writes a new path to the root node, while leaving the rest of the tree intact. When a snapshot needs to be deleted, you just remove all the old root nodes and have a garbage-collection like-process delete all the old files. EDIT: Look at example here: https://en.wikipedia.org/wiki/Persistent_data_structure#Trees https://en.wikipedia.org/wiki/Persistent_data_structure#Tree... . When xs is deleted, nodes a,c,f can also be deleted. EDIT2: I feel like I can never make posts like these while keeping it short and sweet, and adding enough details. ZFS can be encrypted, but accessing any files is right at its fingertips, not over a slow network connection. And there are caveats about how to encrypt the metadata, but you do not need to decrypt the whole backup to figure out what to do delete.

tynorf 7y ago
AFAIK that is exactly how restic works. You use `restic forget` to remove the roots (snapshots), then `restic prune` garbage collects unreachable blobs. ETA: However, nearly all data in restic is encrypted. This includes the index files. So you still need to have the encryption key to look at snapshots and walk their trees.
e12e 7y ago
> ZFS is not encrypted It certainly can be (as of zfzol 0.8, and oracle zfs has its own encryption scheme). https://github.com/zfsonlinux/zfs/pull/5769 https://github.com/zfsonlinux/zfs/pull/5769
rcthompson 7y ago
If the backup server has access to the tree structure, then the backup isn't fully encrypted, or at least the metadata isn't. Now you need to have a conversation about what metadata is allowed to be stored unencrypted on the server and what metadata isn't. If you want a fully encrypted deduplicated backup system, I don't see how it's possible to prune without the ability to decrypt, because there's no way to know what needs to be pruned.
blattimwind 7y ago
> (ZFS is not encrypted, but the principle is the same) Neither.

by Nican 7y ago

tynorf 7y ago

AFAIK that is exactly how restic works. You use `restic forget` to remove the roots (snapshots), then `restic prune` garbage collects unreachable blobs. ETA: However, nearly all data in restic is encrypted. This includes the index files. So you still need to have the encryption key to look at snapshots and walk their trees.

e12e 7y ago

> ZFS is not encrypted It certainly can be (as of zfzol 0.8, and oracle zfs has its own encryption scheme). https://github.com/zfsonlinux/zfs/pull/5769 https://github.com/zfsonlinux/zfs/pull/5769

rcthompson 7y ago

If the backup server has access to the tree structure, then the backup isn't fully encrypted, or at least the metadata isn't. Now you need to have a conversation about what metadata is allowed to be stored unencrypted on the server and what metadata isn't. If you want a fully encrypted deduplicated backup system, I don't see how it's possible to prune without the ability to decrypt, because there's no way to know what needs to be pruned.

blattimwind 7y ago

> (ZFS is not encrypted, but the principle is the same) Neither.