5 ms·

We[1] built borg into our environment[2] as soon as it was stable, release software. In the years since, it has (ironically) supplanted use of rsync as the de

by rsync 7y ago

We[1] built borg into our environment[2] as soon as it was stable, release software. In the years since, it has (ironically) supplanted use of rsync as the de facto standard that our users back up to us with. As one of our users have said[2], borg is "the holy grail" of backups as it does everything rsync always did, and produces remotely encrypted backups that the provider has zero insight into. It also does not have the inefficiencies that the older, duplicity software has. If you are willing to go without (borg specific) technical support and do your retention with borg instead of our zfs snapshots, there is a special, discounted rate available.[4] [1] rsync.net [2] https://news.ycombinator.com/item?id=17408624 https://news.ycombinator.com/item?id=17408624 [3] https://www.stavros.io/posts/holy-grail-backups/ https://www.stavros.io/posts/holy-grail-backups/ [4] https://www.rsync.net/products/borg.html https://www.rsync.net/products/borg.html

millettjon 7y ago
I use borg with both rsync.net and borgbase.com. Both are great.
blakesterz 7y ago
inefficiencies that the older, duplicity software has. As much as I love duplicity, it does sort of suck when you get into larger numbers of files and gigabytes of data. It's so darn slow sometimes it's almost unusable. Good to know borg is better, I've been meaning to check that out!
- JackMcMack 7y ago
  It's incredibly slow (backing up 1.5TB took more than a day), but the built-in support for PAR2 makes it worth it (for me). Sadly, it seems like borg has no built-in support for any kind of redundancy [1] [1] https://github.com/borgbackup/borg/issues/225 https://github.com/borgbackup/borg/issues/225
- Erlich_Bachman 7y ago
  Duplicity and Borg use two very distinct technologies under the hood - duplicity does differential partial backup archives (which all require a full backup AND a full list of all previous differential archives) and Borg uses deduplication. So yes it is much more efficient in many ways, not simply because it is newer and somehow is more efficiently coded, it is so because it uses different technology and backup model.
dx87 7y ago
I've been using borg to backup small things, like schoolwork, config files, mail, and code, to rsync.net for ~1 year now, and can vouch that it's pretty simple to setup. If you've got a simple use case like I do, the example config in the borg tutorial is pretty much all you need. I've got it set to automatically run as a cron job, and I haven't had to touch it except to occasjionally do sanity checks that everything is being back up correctly.
theamk 7y ago
Nice to see someone offering public borg hosting! I have two security questions: (1) Are you doing any hardening of the borg remote side? The server is pretty complex and it has lots of code paths which can be exercised by a potentially untrusted client. (2) Is there a way to protect compromised clients (like a cryptolocker)? borg offers "append only" mode, but it does not allow pruning (obviously). And trying to run "prune" using separate trusted connection will commit untrusted changes as well.
pizza234 7y ago
Borg is definitely not the holy grail of backups, unless one has very loose requirements. Desktop backup solutions have typically a set of functionalities to compare against (block-based, safe on untrusted repositories, with realtime backup, compression-efficient, and optionally, with a functional GUI), that show how open source solutions are strangely lacking, each in a way or another. > produces remotely encrypted backups that the provider has zero insight into. If you're implying an untrusted repository, this is not entirely correct. From https://borgbackup.readthedocs.io/en/stable/internals/security.html https://borgbackup.readthedocs.io/en/stable/internals/securi...: > in a multiple-client scenario a repository can trick a client into reusing counter values by ignoring counter reservations and replaying the manifest The last discussion on HN was actually significantly more critical: https://news.ycombinator.com/item?id=18952839 https://news.ycombinator.com/item?id=18952839. Borg also has no efficient compression, as it doesn't use multithreading. There are multiple GitHub issues on this subject; I've just checked, and they're closed.
- kbenson 7y ago
  > Borg also has no efficient compression, as it doesn't use multithreading. There are multiple GitHub issues on this subject; I've just checked, and they're closed. Depending on your use model, this may or may not affect you. Whether you use multiple threads to compress a single stream or not probably matters a lot less if you have 15-30 backups going on at the same time.
cies 7y ago
How do I get rotation schemas to work with borg? Say I want to keep the last 3 daily snapshots, one per week for the last 4 weeks, and one per month for the last 6 months... I understand this is kinda hard with "zero knowledge" encryption, but it is possible and not a wild feature to request from the self-named holy grail of backups soft.
- corobo 7y ago
  The prune option is what you’re after there https://borgbackup.readthedocs.io/en/stable/usage/prune.html https://borgbackup.readthedocs.io/en/stable/usage/prune.html
  - ngrilly 7y ago
    Yes, but which server will run the purge command? It cannot be the same server as the one you are backuping, since one of the goal of the backup is to protect the server if it is compromised.
    - Erlich_Bachman 7y ago
      You can run command from inside your backup script, from the client. It needs the encryption key anyway to add backups, this way it is just one extra command that does all the pruning (quite quickly as well). And you don't have to worry about full/incremental backups, nor really about any "rotation" or managing them manually, it's all deduplicated and Borg just keeps X number of daily/weekly/whatever snapshots (like duplicacy, like restic, like bup, etc. etc.).
      - ngrilly 7y ago
        Yes, I got this. But if your server is compromised, then the attacker can erase your backups, which ruins the point of doing backups in the first place. This is why there is an append-only mode.
        Erlich_Bachman 7y ago
m3nu 7y ago
If your customers back up multiple servers, are they still under the same rsync.net account and could they see each other's data? This is one of the issues I aimed to solve with BorgBase.com[1]: every single repo is its own backup user and can't see other repos. This separation allowed to add some Borg-specific features, like append-only mode, monitoring for stale backups or using a specific Borg version. 1: https://www.borgbase.com https://www.borgbase.com
- rsync 7y ago
  rsync.net has "subaccounts" which allow you to control side-by-side backup directories using standard UNIX permissions and ownerships. That is, if the default configuration is not exactly what you'd like. Subaccounts even get their own .ssh/authorized_keys file. It's all very unix centered and command-line focused. There's no web interface for features like this.
- theamk 7y ago
  I see that you offer "append only mode" as a core feature. How do you deal with pruning of old backups in this case? Regular borg only offers two options: either pruning does not decrease space used at all; or pruning permanently commits changes, negating any security advantages. Both of those seem bad enough to prevent offering "append-only" commerically?
balladeer 7y ago
It says 1.5c per GB month but then starts at 100GB minimum. Any options for users who would want to use less storage than that?
- sliken 7y ago
  Generally it's not worth using a credit card for something less than $1.5 per month. Various cloud services will give you 100GB for free.
novok 7y ago
Why aren't you price competitive with Google Drive, Amazon Drive, Wasabi, etc? Their rate is ~$0.005/GB/month once you start buying +1TB
- Youden 7y ago
  $5/TB is getting pretty common now (Backblaze B2 comes to mind) and if you roll your own you can get ~1.6EUR/TB from providers like Hetzner (https://www.hetzner.com/dedicated-rootserver/matrix-sx https://www.hetzner.com/dedicated-rootserver/matrix-sx).
  - j1elo 7y ago
    How do you arrive to the 1.6€/TB/month? In your link the cheapest option is 76€, with an 82€ setup fee. EDIT - I see it now, had read 4 TB for the cheapest option, but it is actually 40 TB !! Too much for my typical home user needs, though :-)
  - novok 7y ago
    How reliable is Hetzner? Is there major downtime, enough bandwidth (Around 100Mbps minimum during congested times?)
    - Youden 7y ago
      In my experience pretty reliable. I've not seen major downtime and bandwidth will depend entirely on routing to you in particular (FWIW my ISP peers with Hetzner directly and congestion hasn't been an issue, I can typically saturate my 1Gbps connection at any time of day).
jolmg 7y ago
> borg is "the holy grail" of backups as it does everything rsync always did I tried borg for some time for full backups from / and there were 3 things I didn't like about it: 1. Mounting and navigating a snapshot was extremely slow. Like wait a few seconds for any `ls` to finish. 2. It needed the encryption password for many things I think it shouldn't. 3. The resulting backups are very opaque. I've since switched to making my backups by rsync'ing to btrfs subvolumes. I snapshot the previous backup and rsync the differences to the new one. I get to navigate each snapshot with the same snappiness as any other directory in the filesystem. btrfs can also instantly tell me which files differ between 2 snapshots, no matter how huge they are. I don't remember all the options that borg offers on synching, but I doubt it has the same breadth of options as rsync. So for me, rsync + COW filesystem > borg. I am a little concerned with my use of btrfs, but I'm planning to do the same thing using ext4's reflinks for some disk and filesystem redundancy.

by rsync 7y ago

millettjon 7y ago

I use borg with both rsync.net and borgbase.com. Both are great.

blakesterz 7y ago

inefficiencies that the older, duplicity software has. As much as I love duplicity, it does sort of suck when you get into larger numbers of files and gigabytes of data. It's so darn slow sometimes it's almost unusable. Good to know borg is better, I've been meaning to check that out!

JackMcMack 7y ago
It's incredibly slow (backing up 1.5TB took more than a day), but the built-in support for PAR2 makes it worth it (for me). Sadly, it seems like borg has no built-in support for any kind of redundancy [1] [1] https://github.com/borgbackup/borg/issues/225 https://github.com/borgbackup/borg/issues/225
Erlich_Bachman 7y ago
Duplicity and Borg use two very distinct technologies under the hood - duplicity does differential partial backup archives (which all require a full backup AND a full list of all previous differential archives) and Borg uses deduplication. So yes it is much more efficient in many ways, not simply because it is newer and somehow is more efficiently coded, it is so because it uses different technology and backup model.

dx87 7y ago

I've been using borg to backup small things, like schoolwork, config files, mail, and code, to rsync.net for ~1 year now, and can vouch that it's pretty simple to setup. If you've got a simple use case like I do, the example config in the borg tutorial is pretty much all you need. I've got it set to automatically run as a cron job, and I haven't had to touch it except to occasjionally do sanity checks that everything is being back up correctly.

theamk 7y ago

Nice to see someone offering public borg hosting! I have two security questions: (1) Are you doing any hardening of the borg remote side? The server is pretty complex and it has lots of code paths which can be exercised by a potentially untrusted client. (2) Is there a way to protect compromised clients (like a cryptolocker)? borg offers "append only" mode, but it does not allow pruning (obviously). And trying to run "prune" using separate trusted connection will commit untrusted changes as well.

pizza234 7y ago

Borg is definitely not the holy grail of backups, unless one has very loose requirements. Desktop backup solutions have typically a set of functionalities to compare against (block-based, safe on untrusted repositories, with realtime backup, compression-efficient, and optionally, with a functional GUI), that show how open source solutions are strangely lacking, each in a way or another. > produces remotely encrypted backups that the provider has zero insight into. If you're implying an untrusted repository, this is not entirely correct. From https://borgbackup.readthedocs.io/en/stable/internals/security.html https://borgbackup.readthedocs.io/en/stable/internals/securi...: > in a multiple-client scenario a repository can trick a client into reusing counter values by ignoring counter reservations and replaying the manifest The last discussion on HN was actually significantly more critical: https://news.ycombinator.com/item?id=18952839 https://news.ycombinator.com/item?id=18952839. Borg also has no efficient compression, as it doesn't use multithreading. There are multiple GitHub issues on this subject; I've just checked, and they're closed.

kbenson 7y ago
> Borg also has no efficient compression, as it doesn't use multithreading. There are multiple GitHub issues on this subject; I've just checked, and they're closed. Depending on your use model, this may or may not affect you. Whether you use multiple threads to compress a single stream or not probably matters a lot less if you have 15-30 backups going on at the same time.

cies 7y ago

How do I get rotation schemas to work with borg? Say I want to keep the last 3 daily snapshots, one per week for the last 4 weeks, and one per month for the last 6 months... I understand this is kinda hard with "zero knowledge" encryption, but it is possible and not a wild feature to request from the self-named holy grail of backups soft.

corobo 7y ago
The prune option is what you’re after there https://borgbackup.readthedocs.io/en/stable/usage/prune.html https://borgbackup.readthedocs.io/en/stable/usage/prune.html
- ngrilly 7y ago
  Yes, but which server will run the purge command? It cannot be the same server as the one you are backuping, since one of the goal of the backup is to protect the server if it is compromised.
  - Erlich_Bachman 7y ago
    You can run command from inside your backup script, from the client. It needs the encryption key anyway to add backups, this way it is just one extra command that does all the pruning (quite quickly as well). And you don't have to worry about full/incremental backups, nor really about any "rotation" or managing them manually, it's all deduplicated and Borg just keeps X number of daily/weekly/whatever snapshots (like duplicacy, like restic, like bup, etc. etc.).
    - ngrilly 7y ago
      Yes, I got this. But if your server is compromised, then the attacker can erase your backups, which ruins the point of doing backups in the first place. This is why there is an append-only mode.
      - Erlich_Bachman 7y ago

m3nu 7y ago

If your customers back up multiple servers, are they still under the same rsync.net account and could they see each other's data? This is one of the issues I aimed to solve with BorgBase.com[1]: every single repo is its own backup user and can't see other repos. This separation allowed to add some Borg-specific features, like append-only mode, monitoring for stale backups or using a specific Borg version. 1: https://www.borgbase.com https://www.borgbase.com

rsync 7y ago
rsync.net has "subaccounts" which allow you to control side-by-side backup directories using standard UNIX permissions and ownerships. That is, if the default configuration is not exactly what you'd like. Subaccounts even get their own .ssh/authorized_keys file. It's all very unix centered and command-line focused. There's no web interface for features like this.
theamk 7y ago
I see that you offer "append only mode" as a core feature. How do you deal with pruning of old backups in this case? Regular borg only offers two options: either pruning does not decrease space used at all; or pruning permanently commits changes, negating any security advantages. Both of those seem bad enough to prevent offering "append-only" commerically?

balladeer 7y ago

It says 1.5c per GB month but then starts at 100GB minimum. Any options for users who would want to use less storage than that?

sliken 7y ago
Generally it's not worth using a credit card for something less than $1.5 per month. Various cloud services will give you 100GB for free.

novok 7y ago

Why aren't you price competitive with Google Drive, Amazon Drive, Wasabi, etc? Their rate is ~$0.005/GB/month once you start buying +1TB

Youden 7y ago
$5/TB is getting pretty common now (Backblaze B2 comes to mind) and if you roll your own you can get ~1.6EUR/TB from providers like Hetzner (https://www.hetzner.com/dedicated-rootserver/matrix-sx https://www.hetzner.com/dedicated-rootserver/matrix-sx).
- j1elo 7y ago
  How do you arrive to the 1.6€/TB/month? In your link the cheapest option is 76€, with an 82€ setup fee. EDIT - I see it now, had read 4 TB for the cheapest option, but it is actually 40 TB !! Too much for my typical home user needs, though :-)
- novok 7y ago
  How reliable is Hetzner? Is there major downtime, enough bandwidth (Around 100Mbps minimum during congested times?)
  - Youden 7y ago
    In my experience pretty reliable. I've not seen major downtime and bandwidth will depend entirely on routing to you in particular (FWIW my ISP peers with Hetzner directly and congestion hasn't been an issue, I can typically saturate my 1Gbps connection at any time of day).

jolmg 7y ago

> borg is "the holy grail" of backups as it does everything rsync always did I tried borg for some time for full backups from / and there were 3 things I didn't like about it: 1. Mounting and navigating a snapshot was extremely slow. Like wait a few seconds for any `ls` to finish. 2. It needed the encryption password for many things I think it shouldn't. 3. The resulting backups are very opaque. I've since switched to making my backups by rsync'ing to btrfs subvolumes. I snapshot the previous backup and rsync the differences to the new one. I get to navigate each snapshot with the same snappiness as any other directory in the filesystem. btrfs can also instantly tell me which files differ between 2 snapshots, no matter how huge they are. I don't remember all the options that borg offers on synching, but I doubt it has the same breadth of options as rsync. So for me, rsync + COW filesystem > borg. I am a little concerned with my use of btrfs, but I'm planning to do the same thing using ext4's reflinks for some disk and filesystem redundancy.