8 ms·

Really looking forward to the day I can ditch borg backup and switch over but really can't until https://github.com/restic/restic/issues/21 https://github.com/r

Really looking forward to the day I can ditch borg backup and switch over but really can't until https://github.com/restic/restic/issues/21 https://github.com/restic/restic/issues/21 is addressed. I have to pay for cloud storage and the lack of compression would easily double my costs based on my testing.

l33tman 5y ago
Borg locally in combination with rclone to AWS (or other) works very well, compression and deblocking.
- lucb1e 5y ago
  What is deblocking? Did you mean deduplication?
  - l33tman 5y ago
    Yes, dedup of blocks :)
andrepd 5y ago
Why are you looking forward to ditching borg? I've started to use it as my backup solution and it works wonders.
- ollien 5y ago
  My main gripe with borg, personally, is that it's push only. I want to be able to back up my VPS without having to have it ssh into my home network.
  - andrepd 5y ago
    Ah true, that's a good point. I can't use it to back up my Android phone, for instance.
    - rubenbe 5y ago
      I do backup my android phone using borg installed via termux.
      - lucb1e 5y ago
        Holy crap that's just `pkg install borgbackup`. I had no idea (my phone is already rooted anyhow, so this will also be able to get data folders). This changes everything. There is also `pkg install restic` btw. Based on the problems with append-only in borg and the lack of those in restic's implementation (I did a short audit on that part of the `restic/rest-server` code, looks solid but don't take my word for it), I might go with the latter but this is a great tip regardless.
        rubenbe 5y ago
        The reason I chose borg over restic is there are at least two commercial providers (useful for an offsite backup). borgbase.com and rsync.net too iirc
      - anotherevan 5y ago
        That's an interesting solution. Do you have any more details or a blog post to share?
        rubenbe 5y ago
        I never wrote a blog post about it, but it is triggered when I plug in my charger and the phone is on Wifi. There are hooks in termux to do so. Thanks for the suggestion to write a blog post about it ;)
        anotherevan 5y ago
        Quid pro quo: I've been using Titanium Backup[1] to make backups of all my apps, however it does not working properly with Android 11 and seems to perhaps be abandoned. So I'm now also using OAndBackupX[2] as well, which seems to be doing the job. I then use FolderSync[3] to SFTP synchronise those two backup folders across to my server regularly when the phone is on the home wifi. (I also two-way sync my photos folder which is really quite handy.) I use to also occasionally do a full sync of my phone contents to my server using FTP[4] although since upgrading, Android 11 has clobbered access to the Android/data folder making that problematic. Using Termux + Borg (or Restic) so push full full backups looks attractive. Never seen Termus before. Thanks. [1] https://play.google.com/store/apps/details?id=com.keramidas.TitaniumBackup https://play.google.com/store/apps/details?id=com.keramidas.... [2] https://github.com/machiav3lli/oandbackupx https://github.com/machiav3lli/oandbackupx [3] https://play.google.com/store/apps/details?id=dk.tacit.android.foldersync.lite https://play.google.com/store/apps/details?id=dk.tacit.andro... [4] https://play.google.com/store/apps/details?id=com.theolivetree.ftpserver https://play.google.com/store/apps/details?id=com.theolivetr...
  - pja 5y ago
    Do I have the horrible hack for you! It is possible to do pull backups with borg, with some gruesome ssh hackery. On the backup client side, you need to have a /root/.authorized_keys line like this (edit borg options to suit): command="BORG_PASSPHRASE=$(cat /root/.borg-passphrase) borg create --rsh 'ssh -o \"ProxyCommand socat - UNIX-CLIENT:/root/.socket/borg-socket\" ' --compression auto,zstd -s -x ssh://borg-backup@localhost/<repo location>::<backup name> <backup dirs>" ssh-ed25519 <keydata> <keyname> Then, on your borg server, you need an authorized_keys file like this: command="borg serve --append-only --restrict-to-repo <repo location>",restrict ssh-ed25519 <keydata> <keyname> Finally, run a small shell script like this from the borg server to trigger a pull backup: #!/bin/bash eval $(ssh-agent) > /dev/null ssh-add -q /home/borg-backup/.ssh/<keyname> ssh -A -R /root/.socket/borg-socket:localhost:22 -i .ssh/<keyname> root@borg-client ssh-agent -k I trigger this using cron every night, but systemd timers will work too. The first neat thing about this setup is that the client never even sees the private key that it uses to authenticate to the borg server - the key stays on the server & authentication is tunnelled between client & server via ssh-agent. You don't even need to be able to make a tcp connection from the client to the server - so long as the borg server can make an outgoing tcp connection to the client then everything just works. The client connects back to the server via a socat connection through a unix socket created by the outgoing ssh connection that tunnels any tcp connection made through it back to the sshd on the server. (You could probably tunnel the repo passphrase through as well, if you really wanted to.) The second neat thing is the use of authorized_keys commands which are tied to an ssh keypair means that you're giving the minimal possible access - each ssh connection can only trigger that specific command & no other. You can issue ssh keys on a per-host basis & revoke them individually if necessary. You have to use socat as a proxy program for the return ssh connection as ssh doesn't know how to connect to a unix socket & this setup requires config StreamLocalBindUnlink yes in the .ssh/config on the client (possibly both client + server?), as otherwise the unix socket doesn't get cleaned up after the connection ends & the whole thing only works once before you have to remove the socket by hand. I'm not sure why this isn't the default for ssh to be honest. This method is outlined in the ssh-agent section of https://borgbackup.readthedocs.io/en/stable/deployment/pull-backup.html https://borgbackup.readthedocs.io/en/stable/deployment/pull-... but the docs don't really call it out as a method of getting pull backups working properly. It's a bit convoluted, but it does work! (If your client can make a direct tcp connection to the server you can skip the whole song + dance with the unix sockets of course.)
  - mindslight 5y ago
    I mirror data on VPSs to my local storage array with rsync/unison, and then backup the whole thing with borg.
  - smw 5y ago
    tailscale
derekp7 5y ago
You might like Snebu then, it has always had compression and deduplication, now does public key encryption. No direct cloud support, although you can keep your repository sync'd to a bucket if you want. (Disclosure -- I'm the author)
GordonS 5y ago
Restic looked interesting - I would never have guessed it didn't support such a fundamental feature as compression?! So thanks for mentioning this and saving me from wasting my time. Note: I'm not trying to be horrible or disrespectful to the Restic devs; it's just that backup without compression is a complete show-stopper, especially if you want to use cloud storage (where storage space, storage operation, and bandwidth all eat money).
- dunham 5y ago
  There can be security issues when using encryption on top of compression. These attacks have been used against HTTPS. I don't know if they apply to this specific case, but I assumed that the restic developers were being extra paranoid.
  - GordonS 5y ago
    Sure, that's intrinsic when combining encryption with compression - but it shouldn't be a reason not to make compression optional!
  - aborsy 5y ago
    Known plaintext attacks aren’t much relevant in backup context.
- OJFord 5y ago
  It does block-wise (restic definition, file chunks, not disk) de-duplication, so it's not as bad as you are perhaps imagining.
  - Fazel94 5y ago
    In my experience, the deduplication on SQL dumps is way inferior to borg, even without the compression.
DevX101 5y ago
Kopia supports compression: https://kopia.io/ https://kopia.io/
deleted 5y ago
[deleted]

l33tman 5y ago

Borg locally in combination with rclone to AWS (or other) works very well, compression and deblocking.

lucb1e 5y ago
What is deblocking? Did you mean deduplication?
- l33tman 5y ago
  Yes, dedup of blocks :)

andrepd 5y ago

Why are you looking forward to ditching borg? I've started to use it as my backup solution and it works wonders.

ollien 5y ago
My main gripe with borg, personally, is that it's push only. I want to be able to back up my VPS without having to have it ssh into my home network.
- andrepd 5y ago
  Ah true, that's a good point. I can't use it to back up my Android phone, for instance.
  - rubenbe 5y ago
    I do backup my android phone using borg installed via termux.
    - lucb1e 5y ago
      Holy crap that's just `pkg install borgbackup`. I had no idea (my phone is already rooted anyhow, so this will also be able to get data folders). This changes everything. There is also `pkg install restic` btw. Based on the problems with append-only in borg and the lack of those in restic's implementation (I did a short audit on that part of the `restic/rest-server` code, looks solid but don't take my word for it), I might go with the latter but this is a great tip regardless.
      - rubenbe 5y ago
        The reason I chose borg over restic is there are at least two commercial providers (useful for an offsite backup). borgbase.com and rsync.net too iirc
    - anotherevan 5y ago
      That's an interesting solution. Do you have any more details or a blog post to share?
      - rubenbe 5y ago
        I never wrote a blog post about it, but it is triggered when I plug in my charger and the phone is on Wifi. There are hooks in termux to do so. Thanks for the suggestion to write a blog post about it ;)
        anotherevan 5y ago
        Quid pro quo: I've been using Titanium Backup[1] to make backups of all my apps, however it does not working properly with Android 11 and seems to perhaps be abandoned. So I'm now also using OAndBackupX[2] as well, which seems to be doing the job. I then use FolderSync[3] to SFTP synchronise those two backup folders across to my server regularly when the phone is on the home wifi. (I also two-way sync my photos folder which is really quite handy.) I use to also occasionally do a full sync of my phone contents to my server using FTP[4] although since upgrading, Android 11 has clobbered access to the Android/data folder making that problematic. Using Termux + Borg (or Restic) so push full full backups looks attractive. Never seen Termus before. Thanks. [1] https://play.google.com/store/apps/details?id=com.keramidas.TitaniumBackup https://play.google.com/store/apps/details?id=com.keramidas.... [2] https://github.com/machiav3lli/oandbackupx https://github.com/machiav3lli/oandbackupx [3] https://play.google.com/store/apps/details?id=dk.tacit.android.foldersync.lite https://play.google.com/store/apps/details?id=dk.tacit.andro... [4] https://play.google.com/store/apps/details?id=com.theolivetree.ftpserver https://play.google.com/store/apps/details?id=com.theolivetr...
- pja 5y ago
  Do I have the horrible hack for you! It is possible to do pull backups with borg, with some gruesome ssh hackery. On the backup client side, you need to have a /root/.authorized_keys line like this (edit borg options to suit): command="BORG_PASSPHRASE=$(cat /root/.borg-passphrase) borg create --rsh 'ssh -o \"ProxyCommand socat - UNIX-CLIENT:/root/.socket/borg-socket\" ' --compression auto,zstd -s -x ssh://borg-backup@localhost/<repo location>::<backup name> <backup dirs>" ssh-ed25519 <keydata> <keyname> Then, on your borg server, you need an authorized_keys file like this: command="borg serve --append-only --restrict-to-repo <repo location>",restrict ssh-ed25519 <keydata> <keyname> Finally, run a small shell script like this from the borg server to trigger a pull backup: #!/bin/bash eval $(ssh-agent) > /dev/null ssh-add -q /home/borg-backup/.ssh/<keyname> ssh -A -R /root/.socket/borg-socket:localhost:22 -i .ssh/<keyname> root@borg-client ssh-agent -k I trigger this using cron every night, but systemd timers will work too. The first neat thing about this setup is that the client never even sees the private key that it uses to authenticate to the borg server - the key stays on the server & authentication is tunnelled between client & server via ssh-agent. You don't even need to be able to make a tcp connection from the client to the server - so long as the borg server can make an outgoing tcp connection to the client then everything just works. The client connects back to the server via a socat connection through a unix socket created by the outgoing ssh connection that tunnels any tcp connection made through it back to the sshd on the server. (You could probably tunnel the repo passphrase through as well, if you really wanted to.) The second neat thing is the use of authorized_keys commands which are tied to an ssh keypair means that you're giving the minimal possible access - each ssh connection can only trigger that specific command & no other. You can issue ssh keys on a per-host basis & revoke them individually if necessary. You have to use socat as a proxy program for the return ssh connection as ssh doesn't know how to connect to a unix socket & this setup requires config StreamLocalBindUnlink yes in the .ssh/config on the client (possibly both client + server?), as otherwise the unix socket doesn't get cleaned up after the connection ends & the whole thing only works once before you have to remove the socket by hand. I'm not sure why this isn't the default for ssh to be honest. This method is outlined in the ssh-agent section of https://borgbackup.readthedocs.io/en/stable/deployment/pull-backup.html https://borgbackup.readthedocs.io/en/stable/deployment/pull-... but the docs don't really call it out as a method of getting pull backups working properly. It's a bit convoluted, but it does work! (If your client can make a direct tcp connection to the server you can skip the whole song + dance with the unix sockets of course.)
- mindslight 5y ago
  I mirror data on VPSs to my local storage array with rsync/unison, and then backup the whole thing with borg.
- smw 5y ago
  tailscale

derekp7 5y ago

You might like Snebu then, it has always had compression and deduplication, now does public key encryption. No direct cloud support, although you can keep your repository sync'd to a bucket if you want. (Disclosure -- I'm the author)

GordonS 5y ago

Restic looked interesting - I would never have guessed it didn't support such a fundamental feature as compression?! So thanks for mentioning this and saving me from wasting my time. Note: I'm not trying to be horrible or disrespectful to the Restic devs; it's just that backup without compression is a complete show-stopper, especially if you want to use cloud storage (where storage space, storage operation, and bandwidth all eat money).

dunham 5y ago
There can be security issues when using encryption on top of compression. These attacks have been used against HTTPS. I don't know if they apply to this specific case, but I assumed that the restic developers were being extra paranoid.
- GordonS 5y ago
  Sure, that's intrinsic when combining encryption with compression - but it shouldn't be a reason not to make compression optional!
- aborsy 5y ago
  Known plaintext attacks aren’t much relevant in backup context.
OJFord 5y ago
It does block-wise (restic definition, file chunks, not disk) de-duplication, so it's not as bad as you are perhaps imagining.
- Fazel94 5y ago
  In my experience, the deduplication on SQL dumps is way inferior to borg, even without the compression.

DevX101 5y ago

Kopia supports compression: https://kopia.io/ https://kopia.io/

deleted 5y ago

[deleted]