6 ms·
I started using Restic recently. It's good and I'm going to continue using it. That said, there are a couple of bad problems with it: Firstly, if you want to p
by mike-cardwell 7y ago
I started using Restic recently. It's good and I'm going to continue using it. That said, there are a couple of bad problems with it:
Firstly, if you want to prune old backups, e.g keep the last N1 hourly backups, and the last N2 weekly backups, etc, then it has that ability, however whilst it's doing it, the client has to download and upload a tonne of data in order to repackage the backup files that contain some data that needs removing and other data which doesn't.
Secondly, I've set up an "append only" system, where my various hosts can append to their own backups, but not overwrite or delete them. I wanted the backup server to be unable to read the backups (easy enough, don't supply the encryption keys to the backup server), however at the same time I wanted the backup server to be able to automatically prune old backups. It can not do that without the key. I don't want to give it the keys to the backups as then a compromised backup server means all of my hosts data are suddenly compromised.
- rcthompson 7y agoHow would you expect the backup server to be able to prune old encrypted backups if it can't decrypt them? How should it know what to delete?
- frenchie4111 7y agoThe backup server should have information about when a particular backup was made, it should be able to prune based on backup date without having the ability to read any of the data inside the backup.
- pierrebai 7y agoI assume the backups are incremental, so you cannot just toss old encrypted files without losing backed up files.
- zaarn 7y agoWorse, it's deduplicated, using a similar algorithm to git. You can forget old backups and theoretically that could be done without decrypting but deleting the actual data will require a decrypt.
- rcthompson 7y agoThis doesn't make any sense when the backups are deduplicated (or incremental). A given file/block/whatever unit of data isn't uniquely associated with a single backup.
- pfranz 7y agoI'm no encryption expert, but depending on how the data is partitioned it can be easier to securely prune encrypted data than unencrypted data. For example, securely wiping an encrypted drive is as easy as tossing the key.
- tyri_kai_psomi 7y agoIf you wish to "securely" wipe an encrypted drive, it's best to write over the old data a few times using randomly generated keys, tossing the keys each time you do a write over.
- dividuum 7y agoThat sounds pointless. If you have real crypto, tossing the key (in a way that it cannot be recovered) is absolutely enough.
- rhencke 7y agoIt's not pointless - it's pretty standard practice. For example, although drives in Google Cloud are encrypted at rest, once they are decommissioned the drives are physically destroyed: https://cloud.google.com/security/deletion/#ensuring_safe_and_secure_media_sanitization https://cloud.google.com/security/deletion/#ensuring_safe_an...
- dividuum 7y ago> When a hard drive is retired, authorized individuals verify that the disk is erased by overwriting the drive with zeros and performing a multi-step verification process to ensure the drive contains no data. Thanks. I guess the reason is to avoid future decryption due to either a new crypto attack or to avoid being vulnerable to potential errors in key generation.
- LeoPanthera 7y agoThis is massive overkill. Once is enough. If the drive is already encrypted, just wipe the key.
- amelius 7y agoPerhaps by using homomorphic encryption? https://en.wikipedia.org/wiki/Homomorphic_encryption https://en.wikipedia.org/wiki/Homomorphic_encryption
- Nican 7y agoZFS does that with snapshots already. (ZFS is not encrypted, but the principle is the same) I believe immutable trees would be the correct algorithm here. The algorithm relies on a tree structure, where the leaves are the files. Every time a new snapshot is written, every leaf node changed writes a new path to the root node, while leaving the rest of the tree intact. When a snapshot needs to be deleted, you just remove all the old root nodes and have a garbage-collection like-process delete all the old files. EDIT: Look at example here: https://en.wikipedia.org/wiki/Persistent_data_structure#Trees https://en.wikipedia.org/wiki/Persistent_data_structure#Tree... . When xs is deleted, nodes a,c,f can also be deleted. EDIT2: I feel like I can never make posts like these while keeping it short and sweet, and adding enough details. ZFS can be encrypted, but accessing any files is right at its fingertips, not over a slow network connection. And there are caveats about how to encrypt the metadata, but you do not need to decrypt the whole backup to figure out what to do delete.
- tynorf 7y agoAFAIK that is exactly how restic works. You use `restic forget` to remove the roots (snapshots), then `restic prune` garbage collects unreachable blobs. ETA: However, nearly all data in restic is encrypted. This includes the index files. So you still need to have the encryption key to look at snapshots and walk their trees.
- e12e 7y ago> ZFS is not encrypted It certainly can be (as of zfzol 0.8, and oracle zfs has its own encryption scheme). https://github.com/zfsonlinux/zfs/pull/5769 https://github.com/zfsonlinux/zfs/pull/5769
- rcthompson 7y agoIf the backup server has access to the tree structure, then the backup isn't fully encrypted, or at least the metadata isn't. Now you need to have a conversation about what metadata is allowed to be stored unencrypted on the server and what metadata isn't. If you want a fully encrypted deduplicated backup system, I don't see how it's possible to prune without the ability to decrypt, because there's no way to know what needs to be pruned.
- 7y ago
- mike-cardwell 7y agoIt should be possible to have an index that says "this encrypted blob belongs to this particular snapshot". It introduces a "plain text" component of course, in that the snapshot information (name/date or whatever) is readable. But then, I assume the way it is written, there are some blobs that belong to multiple snapshots, and some bits of blobs that belong to one snapshot but not another etc. So given the implementation, it may not be possible, but it's still a drawback of this particular implementation that people should be aware of.
- nshepperd 7y agoCouldn't the client just tell the server what to delete? Which would require only downloading an (encrypted) index of what data is in use, not downloading and reuploading the data itself.
- cies 7y agoI think that's what it does, but that downloading of encrypted indexes and comparing them takes time. Which is what was complained about.
- eyo2019 7y agoMeet this guy!: https://news.ycombinator.com/item?id=21411467 https://news.ycombinator.com/item?id=21411467
- dividuum 7y ago> Secondly, I've set up an "append only" system, where my various hosts can append to their own backups, but not overwrite or delete them. To my knowledge that's not possible with restic. You can either have append only (using for example restic/rest-server) or pruning. But not both. My solution so far is to use a backend like rsync.net. The backup uses the normal backup + prune routine. And rsync on their server side creates daily undeleteable snapshots of all files and deduplicates the encrypted files in the process without the need to look into them. For me that's seems like a reasonable compromise for now.
- mike-cardwell 7y agoI created my own restic backend using Nginx, which allows two different users to authenticate against a repo. The host being backed up knows the credentials that allow it to append. A different host knows the pruning credentials and performs that task. I posted this in another comment, but I did this as follows (I'm the author) - https://www.grepular.com/Nginx_Restic_Backend https://www.grepular.com/Nginx_Restic_Backend
- Erlich_Bachman 7y agoAnother major problem is that it does not support compression: all files are stored uncompressed on the backup server (even though they are deduplicated) (sic).
- mike-cardwell 7y agoYou know, I actually assumed it had compression. It looks like it will get it eventually though, so I'm not too worried. That said, I can get some benefits by compressing some of my backups myself. For example, I currently backup my postgres dbs with cron jobs like: $ pg_dump dbname | restic backup --stdin --stdin-filename dbname.dump --tag dbname -q With restoring as simple as: $ restic dump latest --tag dbname dbname.dump | psql dbname I should probably stick a gzip/gunzip between those two commands.
- adrianmonk 7y agoIf you stick a cheese up in there, restic will probably lose some of its ability to deduplicate. If it does compression, then it's not an opaque process to it. But if you do it, you are essentially obfuscating the structure of the data.
- mike-cardwell 7y agoGood point. I'll stay away from cheesing up my commands and wait for it to be built in.
- mynegation 7y agoWhat is "cheesing up"?
- adrianmonk 7y agoIt's what Google speech-to-text thinks I meant when I said "gzip" into my phone.
- 7y ago
- Scaevolus 7y agoThere's an open PR to optimize prune-- essentially it lets you decide a percentage of waste per chunk you'll allow before repacking it. https://github.com/restic/restic/issues/2162 https://github.com/restic/restic/issues/2162