4 ms·

Hello, I'm the author of Snebu (the first item mentioned on the list) -- I've been trying to figure out how to get more exposure to it before I let it fly mains

by derekp7 12y ago

Hello, I'm the author of Snebu (the first item mentioned on the list) -- I've been trying to figure out how to get more exposure to it before I let it fly mainstream (i.e., submitting packages to the various distros, etc). On the complaint that it doesn't do encryption -- That is one item that I'd like to seek some advice on. My plan is that if you want encryption, then use a LUKS encrypted filesystem on the target (communications to the target is already encrypted with ssh). The main reason is that I'm not a cryptographer, and even if I use existing libraries there is still a strong chance that I'd miss something and end up using them wrong. Just for example -- to do it right, you would add unique salt to each object you are encrypting (from the client side). That would then render any type of deduplication useless on the backend side, since multiple files with the same contents would have different encrypted contents. That being said, I am adding code that lets you have replicas to other storage devices (tape, cloud, other disk based storage). So you would do your primary backup to a local disk device (that possibly has an encrypted volume), then the secondary stage would be packing files together into an (encrypted) object, for sending to a remote location. I've got a small list going, after I redo the web site I'll put up a planned feature list along with comparisons with other backup utilities. If anyone has ideas to contribute, either drop me an email or open an issue on the Github page. Thanks. Edit: On the encryption side, am I correct in thinking that multiple files with the same contents should encrypt to different streams (via a random salt)? Also, should the file names themselves be encrypted? Finally, metadata, such as mod date, file size, and checksum -- should that all be encrypted too? Thanks.

warmwaffles 12y ago
I like your solution of sending to an already encrypted volume. Let another dedicated crypto program handle that. I'll definitely look at this.
- derekp7 12y ago
  Here's a slight variation -- Normally, the client side runs a "find" command to generate a file manifest, and sends it to the "snebu" command on the server side. Which then generates file list of files that it doesn't already have. The client then responds with a tar command output of those files, sending that back into the "snebu" command on the server (all via ssh). The variation: Run snebu locally, and have the file vault loaded on an iscsi device targeted to the remote backup server. Then you can run LUKS locally, and not have to worry about trusting the remote server. Just make sure you send a copy of the sqlite database file (containing the backup catalog) to the remote volume after a backup. However, this still doesn't address having a remote encrypted volume that multiple clients can send to (in that case, you will need to do the other approach of sending to a local server under your control, then replicating it with a plugin that supports encryption [to be developed yet]). Note, this method should work for any of the other backup utilities that don't support encryption also.
wampus 12y ago
Encryption can be implemented flexibly on systems you control. When it come to encrypting backups, the toughest issue is doing it safely on target systems you don't control (duplicity attacks this problem well). I'd recommend starting there. Encrypted replication is a very interesting idea, and actually helps with the problem of maintaining local & remote backups. I usually stagger them, but being able to replicate to arbitrary targets would be very convenient (I don't like to rsync backups after the fact, because you risk corrupting the target if there is a local failure).
- derekp7 12y ago
  One of the items I struggle with is that I want to keep the current client-side simplicity of Snebu -- on the client, the only thing that is required is bash, find, and tar (although GNU find is required, and older versions of find don't support enough options). However, I think I have a potential solution, which I'll work on this weekend. The solution is: on the client side, include an optional encrypting tar filter, which takes a tar file as input, and encrypts the file contents of each file within the archive, delivering a tar file output with regular headers, but compressed encrypted files. I'll probably have to take some liberties with the tar file format, but as long as the snebu backend is similarly modified (to recognize a pre-compressed and encrypted file segment), then I should be able to get it to work without any major compromises. The only issues with my current plan are: 1) Multiple files with the same contents will still be de-duplicated, which may leak some information (if an attacker already knows file A's contents, and B is marked as being a copy of A, then the attacker will know what file B is). 2) The metadata (file name, size, owner, possibly SHA1) will still be visible. Although I may just use the SHA1 of the encrypted version as the file reference. 3) Sparse files will still show where the "holes" in the files are, unless you tell it not to preserve file sparseness.
wongarsu 12y ago
>if you want encryption, then use a LUKS encrypted filesystem on the target If you control the server the main purpose of encryption is to protect the backups even if somebody gains access to the backup server. An encrypted filesystem isn't a good solution because by design the backup server has the key. This only protects against intruders that are foolish enough to power down the server. As long as the server is running, an encrypted filesystem offers no protection at all.

by derekp7 12y ago

warmwaffles 12y ago

I like your solution of sending to an already encrypted volume. Let another dedicated crypto program handle that. I'll definitely look at this.

derekp7 12y ago
Here's a slight variation -- Normally, the client side runs a "find" command to generate a file manifest, and sends it to the "snebu" command on the server side. Which then generates file list of files that it doesn't already have. The client then responds with a tar command output of those files, sending that back into the "snebu" command on the server (all via ssh). The variation: Run snebu locally, and have the file vault loaded on an iscsi device targeted to the remote backup server. Then you can run LUKS locally, and not have to worry about trusting the remote server. Just make sure you send a copy of the sqlite database file (containing the backup catalog) to the remote volume after a backup. However, this still doesn't address having a remote encrypted volume that multiple clients can send to (in that case, you will need to do the other approach of sending to a local server under your control, then replicating it with a plugin that supports encryption [to be developed yet]). Note, this method should work for any of the other backup utilities that don't support encryption also.

wampus 12y ago

Encryption can be implemented flexibly on systems you control. When it come to encrypting backups, the toughest issue is doing it safely on target systems you don't control (duplicity attacks this problem well). I'd recommend starting there. Encrypted replication is a very interesting idea, and actually helps with the problem of maintaining local & remote backups. I usually stagger them, but being able to replicate to arbitrary targets would be very convenient (I don't like to rsync backups after the fact, because you risk corrupting the target if there is a local failure).

derekp7 12y ago
One of the items I struggle with is that I want to keep the current client-side simplicity of Snebu -- on the client, the only thing that is required is bash, find, and tar (although GNU find is required, and older versions of find don't support enough options). However, I think I have a potential solution, which I'll work on this weekend. The solution is: on the client side, include an optional encrypting tar filter, which takes a tar file as input, and encrypts the file contents of each file within the archive, delivering a tar file output with regular headers, but compressed encrypted files. I'll probably have to take some liberties with the tar file format, but as long as the snebu backend is similarly modified (to recognize a pre-compressed and encrypted file segment), then I should be able to get it to work without any major compromises. The only issues with my current plan are: 1) Multiple files with the same contents will still be de-duplicated, which may leak some information (if an attacker already knows file A's contents, and B is marked as being a copy of A, then the attacker will know what file B is). 2) The metadata (file name, size, owner, possibly SHA1) will still be visible. Although I may just use the SHA1 of the encrypted version as the file reference. 3) Sparse files will still show where the "holes" in the files are, unless you tell it not to preserve file sparseness.

wongarsu 12y ago

>if you want encryption, then use a LUKS encrypted filesystem on the target If you control the server the main purpose of encryption is to protect the backups even if somebody gains access to the backup server. An encrypted filesystem isn't a good solution because by design the backup server has the key. This only protects against intruders that are foolish enough to power down the server. As long as the server is running, an encrypted filesystem offers no protection at all.