8 ms·

borg[1] has become the de facto standard for this use-case.[2] It can run over SSH with the borg binary on the remote server or it can run in an SFTP mode with

by rsync 6y ago

borg[1] has become the de facto standard for this use-case.[2] It can run over SSH with the borg binary on the remote server or it can run in an SFTP mode with nothing installed on the destination. [1] https://www.borgbackup.org/ https://www.borgbackup.org/ [2] https://www.stavros.io/posts/holy-grail-backups/ https://www.stavros.io/posts/holy-grail-backups/

noodlesUK 6y ago
Don’t forget restic - it also works great on s3 compatible services! https://restic.net https://restic.net
- nix23 6y ago
  Yes restic is great! No python no dependencies (like borg), just one GO exe and that's it.
- sjellis 6y ago
  Yes, restic would be my pick: easy to deploy and use, well-maintained. If you want some assurance, CERN are deploying restic: https://indico.cern.ch/event/862873/contributions/3724442/ https://indico.cern.ch/event/862873/contributions/3724442/
- josephernest 6y ago
  I'll have a look too. When using with SFTP, do Restic binaries need to be installed on both local and remote, or just local?
josephernest 6y ago
As I discover it, borg seems to be pretty good for this use case indeed. But still there is one thing (at least) that can be of interest to some people with nFreezer: it is very simple, it does the job in only 249 lines of code. You can then read the FULL source code in a few hours, to see if you trust it or not. See here: https://github.com/josephernest/nfreezer/blob/master/nfreezer.py https://github.com/josephernest/nfreezer/blob/master/nfreeze... If I want to do this with the source code of the tool you mentioned, I would have to spend at least one full week. (this is normal: this program has 100 times more features). The key point is: if you're looking for a solution for which you don't want to trust a remote server, then you probably don't want to trust the backup tool of a random internet person either. And you probably want to read the source code of the program. So having only < 300 lines of code to read in a single .py can be an advantage.
- giomasce 6y ago
  I don't see the key point: the two things (trusting the remote storage and the tool) are rather independent for me. I already trust probably billions lines of code which handle my data (basically, every program installed on my system). I don't trust them because I checked all of them, but because I know that the same lines are used by many other people, some of which have actually read parts of them. There is a global trust repository to which every programmer contributes a little bit and every user benefits a little bit. For this reason, I find more trustworthy a project already used and developed by many people, like borg. I didn't just take that for granted, of course: I read the documentation (which, in the case of borg, is very clean and extensive), I looked how it work, I even looked a part of the source code, and I was satisfied. And all of this has nothing to do with where I am putting that data in the end. On the other hand, missing features in the name of small code size is not a great advantage, if those missing features make my backup less reliable or quick or compact. And maybe I end up backing up less stuff because (say) the simple tool does not handle deduplication or compression as effectively as the complicated one, and so is not viable for me. This would be a net loss. Be careful, I am just saying my perspective. You have, of course, the entire right to decide what are your priorities and which tool is better for you.
- rufugee 6y ago
  Very well said. Sometimes simplicity carries immense value and reassurance.
- rsync 6y ago
  "But still there is one thing (at least) that can be of interest to some people with nFreezer: it is very simple, it does the job in only 249 lines of code. You can then read the FULL source code in a few hours, to see if you trust it or not. See here: https://github.com/josephernest/nfreezer/blob/master/nfreeze https://github.com/josephernest/nfreezer/blob/master/nfreeze..." I really appreciate this and find this very interesting. I would be happy to give you a free dev/test account at rsync.net if that would help you continue this development.
- rakoo 6y ago
  From https://github.com/josephernest/nfreezer/blob/master/nfreezer.py https://github.com/josephernest/nfreezer/blob/master/nfreeze..., this is how you decrypt files: [...] with open(f2, 'wb') as f, src_cm.open(chunkid.hex(), 'rb') as g: decrypt(g, pwd=encryptionpwd, out=f) [...] def decrypt(f=None, s=None, pwd=None, out=None): [...] while True: block = f.read(BLOCKSIZE) if not block: break out.write(cipher.decrypt(block)) try: cipher.verify(tag) except ValueError: print('Incorrect key or file corrupted.') So, basically, you decrypt the whole file and write the result before checking the tag. You're using a block cipher in a streaming fashion, and as has already been said before (see https://www.imperialviolet.org/2015/05/16/aeads.html https://www.imperialviolet.org/2015/05/16/aeads.html, "AEADs with large plaintexts") this is dangerous if you don't do it correctly. Your data may be garbage, but it's too late it's already written on the disk before you know it's bad, it's not deleted and you won't know which file it was. As some HN crypto celebrity said some time ago, if you write "AES" in your code then you're wrong. You MUST use misuse-resistant libraries unless you know exactly what you're doing. TL;DR: your crypto is broken, use NaCl instead of doing it yourself.
tonyhb 6y ago
Also want to plug an old HN favourite - tarsnap. https://www.tarsnap.com/ https://www.tarsnap.com/

by rsync 6y ago

noodlesUK 6y ago

Don’t forget restic - it also works great on s3 compatible services! https://restic.net https://restic.net

nix23 6y ago
Yes restic is great! No python no dependencies (like borg), just one GO exe and that's it.
sjellis 6y ago
Yes, restic would be my pick: easy to deploy and use, well-maintained. If you want some assurance, CERN are deploying restic: https://indico.cern.ch/event/862873/contributions/3724442/ https://indico.cern.ch/event/862873/contributions/3724442/
josephernest 6y ago
I'll have a look too. When using with SFTP, do Restic binaries need to be installed on both local and remote, or just local?

josephernest 6y ago

As I discover it, borg seems to be pretty good for this use case indeed. But still there is one thing (at least) that can be of interest to some people with nFreezer: it is very simple, it does the job in only 249 lines of code. You can then read the FULL source code in a few hours, to see if you trust it or not. See here: https://github.com/josephernest/nfreezer/blob/master/nfreezer.py https://github.com/josephernest/nfreezer/blob/master/nfreeze... If I want to do this with the source code of the tool you mentioned, I would have to spend at least one full week. (this is normal: this program has 100 times more features). The key point is: if you're looking for a solution for which you don't want to trust a remote server, then you probably don't want to trust the backup tool of a random internet person either. And you probably want to read the source code of the program. So having only < 300 lines of code to read in a single .py can be an advantage.

giomasce 6y ago
I don't see the key point: the two things (trusting the remote storage and the tool) are rather independent for me. I already trust probably billions lines of code which handle my data (basically, every program installed on my system). I don't trust them because I checked all of them, but because I know that the same lines are used by many other people, some of which have actually read parts of them. There is a global trust repository to which every programmer contributes a little bit and every user benefits a little bit. For this reason, I find more trustworthy a project already used and developed by many people, like borg. I didn't just take that for granted, of course: I read the documentation (which, in the case of borg, is very clean and extensive), I looked how it work, I even looked a part of the source code, and I was satisfied. And all of this has nothing to do with where I am putting that data in the end. On the other hand, missing features in the name of small code size is not a great advantage, if those missing features make my backup less reliable or quick or compact. And maybe I end up backing up less stuff because (say) the simple tool does not handle deduplication or compression as effectively as the complicated one, and so is not viable for me. This would be a net loss. Be careful, I am just saying my perspective. You have, of course, the entire right to decide what are your priorities and which tool is better for you.
rufugee 6y ago
Very well said. Sometimes simplicity carries immense value and reassurance.
rsync 6y ago
"But still there is one thing (at least) that can be of interest to some people with nFreezer: it is very simple, it does the job in only 249 lines of code. You can then read the FULL source code in a few hours, to see if you trust it or not. See here: https://github.com/josephernest/nfreezer/blob/master/nfreeze https://github.com/josephernest/nfreezer/blob/master/nfreeze..." I really appreciate this and find this very interesting. I would be happy to give you a free dev/test account at rsync.net if that would help you continue this development.
rakoo 6y ago
From https://github.com/josephernest/nfreezer/blob/master/nfreezer.py https://github.com/josephernest/nfreezer/blob/master/nfreeze..., this is how you decrypt files: [...] with open(f2, 'wb') as f, src_cm.open(chunkid.hex(), 'rb') as g: decrypt(g, pwd=encryptionpwd, out=f) [...] def decrypt(f=None, s=None, pwd=None, out=None): [...] while True: block = f.read(BLOCKSIZE) if not block: break out.write(cipher.decrypt(block)) try: cipher.verify(tag) except ValueError: print('Incorrect key or file corrupted.') So, basically, you decrypt the whole file and write the result before checking the tag. You're using a block cipher in a streaming fashion, and as has already been said before (see https://www.imperialviolet.org/2015/05/16/aeads.html https://www.imperialviolet.org/2015/05/16/aeads.html, "AEADs with large plaintexts") this is dangerous if you don't do it correctly. Your data may be garbage, but it's too late it's already written on the disk before you know it's bad, it's not deleted and you won't know which file it was. As some HN crypto celebrity said some time ago, if you write "AES" in your code then you're wrong. You MUST use misuse-resistant libraries unless you know exactly what you're doing. TL;DR: your crypto is broken, use NaCl instead of doing it yourself.

tonyhb 6y ago

Also want to plug an old HN favourite - tarsnap. https://www.tarsnap.com/ https://www.tarsnap.com/