5 ms·

I get that performance is interesting to graph, but it's very much secondary in importance when compared to the backup solution being bulletproof. I've found en

I get that performance is interesting to graph, but it's very much secondary in importance when compared to the backup solution being bulletproof. I've found encrypted Borg very difficult to get wrong and setup is very simple. I've also successfully recovered two systems with the tool without issue. Not saying that Borg is necessarily the best solution, just that we should evaluate the important metrics.

rectang 6y ago
How does developing a backup application differ from developing other projects? What do you test differently? What do you avoid changing? How do you avoid breaking things? How do you get enough beta testers when the costs of failure are so extreme?
- uselpa 6y ago
  Each project can have more or less the same metrics, but different priorities. In a backup tool for example, integrity > restore speed > backup speed (unless the backup speed is so slow that you cannot backup). In a project that can afford to loose data, speed may be more important than integrity.
regecks 6y ago
>but it's very much secondary in importance when compared to the backup solution being bulletproof To an extent, anyway. I moved to Borgmatic precisely because the restore speed on Tarsnap made it unsuitable for my needs. Unfortunately I didn't uncover this problem until I actually needed to restore, which extended an outage by about half a day. Having a vague sense of the tool's performance is a good thing, I think.
- Chris_Newton 6y ago
  I moved to Borgmatic precisely because the restore speed on Tarsnap made it unsuitable for my needs. Yes, as long-time Tarsnap users, we got a very unpleasant shock a few months ago when our main server literally went pop one morning and we needed to do a full restore of everything to get the replacement up and running. It took many times longer than we were expecting to download all the data, and consequently caused us extra days of unnecessary downtime. To his credit, Colin was responsive when we asked for advice and did suggest something we could try to make things a bit faster (which it did). But really, when you’ve just had a catastrophic failure, messing around hand-holding your backup tool just to get your essential data restored as quickly as possible shouldn’t be needed. Backup tools need to Just Work, quickly and reliably, every time. Given that Tarsnap also seems to work out very expensive these days, it’s now in the category of software where I’m happy to have used it and it did do its job even if not ideally, but unless there are dramatic improvements on both counts in the near future I think we’ll probably be looking for another option.
jusssi 6y ago
> I get that performance is interesting to graph, but it's very much secondary in importance when compared to the backup solution being bulletproof. No. I need a backup that consistently finishes before I close the lid to put my laptop in my backpack at the end of the day. Too slow -> no backup.
- _flux 6y ago
  Unless daily work datasets are huge, you would be hard pressed to find a backup system that is unable to pull that off, though it's unclear from your description when you consider your backup to start. My multi-daily backups (always incremental in Kopia) take 3 minutes, with a working set of 4 million files.
  - jusssi 6y ago
    Many dev environments come in virtual machine images. On a good day they're throwaway Vagrant builds, but sometimes there's a significant manual setup required and you want to include them in a backup. Incidentally, some backup utilities run into unexplainable performance issues trying to delta diff very large files such as VM images. Rsync in particular sometimes stalls completely when trying to sync a 30GB VM image to a remote server.
    - wila 6y ago
      There's backup tools that can make a complete backup of your VM while it is still running. At least for VMware virtual machines [0] You can schedule the backup to start some time before you leave at the end of the day. [0] https://www.vimalin.com https://www.vimalin.com disclaimer: I am the author of vimalin
    - _flux 6y ago
      I think Kopia should deal fine with this, though I don't think I have many such large files that I modify. The reason why I think it's fine is that it doesn't really try to do "delta diffs" at all. Instead is uses a rolling hash to cut the input file into variable length chunks and those chunks are added to the chunk storage, and then actual files are constructed from those chunks. Obviously such copies should be done when the files are stable, so either not running or using a snapshot. (If using LVM, this can be arranged easily with any filesystem, except ones that don't like getting mounted multiple times..) So while I guess size-related bugs could be there, there's no any fundamental property of making the task bigger as the files themselves grow bigger.
smarx007 6y ago
Worth noting that Restic is suitable for Windows. Not sure if Bupstash was tested on macOS. Borg also has FreeBSD support and experimental support for Windows via WSL2 and Cygwin.
- 8fingerlouie 6y ago
  I was facing this problem when selecting a backup strategy for client machines. Servers are one thing where i can rely on them to be always running, but clients go off grid occasionally. As for restic, yes it's available on Windows, but has a serious speed degradation when datasets increase above 2TB, mainly due to chunks being 2-5MB, where Borg uses 400MB chunks. And we're not talking a 10% slowdown. It's to the point where prune operations on repositories >2TB can/will take multiple days to complete. For clients, I evaluated multiple solutions, one was Duplicati (https://www.duplicati.com/ https://www.duplicati.com/), and while it appears to work well most of the time, it sometimes stops backing up for no apparent reason. Backuppc also looked promising, but doesn't (didn't ?) support source encrypted backups. In the end i ended up with Arq (https://www.arqbackup.com/ https://www.arqbackup.com/) for Windows and Mac clients. It works reliably, and checks all the right boxes. For servers i was initially really hooked on restic as it allows backups over HTTPS instead of SSH, but the performance issues were a showstopper.

rectang 6y ago

How does developing a backup application differ from developing other projects? What do you test differently? What do you avoid changing? How do you avoid breaking things? How do you get enough beta testers when the costs of failure are so extreme?

uselpa 6y ago
Each project can have more or less the same metrics, but different priorities. In a backup tool for example, integrity > restore speed > backup speed (unless the backup speed is so slow that you cannot backup). In a project that can afford to loose data, speed may be more important than integrity.

regecks 6y ago

>but it's very much secondary in importance when compared to the backup solution being bulletproof To an extent, anyway. I moved to Borgmatic precisely because the restore speed on Tarsnap made it unsuitable for my needs. Unfortunately I didn't uncover this problem until I actually needed to restore, which extended an outage by about half a day. Having a vague sense of the tool's performance is a good thing, I think.

Chris_Newton 6y ago
I moved to Borgmatic precisely because the restore speed on Tarsnap made it unsuitable for my needs. Yes, as long-time Tarsnap users, we got a very unpleasant shock a few months ago when our main server literally went pop one morning and we needed to do a full restore of everything to get the replacement up and running. It took many times longer than we were expecting to download all the data, and consequently caused us extra days of unnecessary downtime. To his credit, Colin was responsive when we asked for advice and did suggest something we could try to make things a bit faster (which it did). But really, when you’ve just had a catastrophic failure, messing around hand-holding your backup tool just to get your essential data restored as quickly as possible shouldn’t be needed. Backup tools need to Just Work, quickly and reliably, every time. Given that Tarsnap also seems to work out very expensive these days, it’s now in the category of software where I’m happy to have used it and it did do its job even if not ideally, but unless there are dramatic improvements on both counts in the near future I think we’ll probably be looking for another option.

jusssi 6y ago

> I get that performance is interesting to graph, but it's very much secondary in importance when compared to the backup solution being bulletproof. No. I need a backup that consistently finishes before I close the lid to put my laptop in my backpack at the end of the day. Too slow -> no backup.

_flux 6y ago
Unless daily work datasets are huge, you would be hard pressed to find a backup system that is unable to pull that off, though it's unclear from your description when you consider your backup to start. My multi-daily backups (always incremental in Kopia) take 3 minutes, with a working set of 4 million files.
- jusssi 6y ago
  Many dev environments come in virtual machine images. On a good day they're throwaway Vagrant builds, but sometimes there's a significant manual setup required and you want to include them in a backup. Incidentally, some backup utilities run into unexplainable performance issues trying to delta diff very large files such as VM images. Rsync in particular sometimes stalls completely when trying to sync a 30GB VM image to a remote server.
  - wila 6y ago
    There's backup tools that can make a complete backup of your VM while it is still running. At least for VMware virtual machines [0] You can schedule the backup to start some time before you leave at the end of the day. [0] https://www.vimalin.com https://www.vimalin.com disclaimer: I am the author of vimalin
  - _flux 6y ago
    I think Kopia should deal fine with this, though I don't think I have many such large files that I modify. The reason why I think it's fine is that it doesn't really try to do "delta diffs" at all. Instead is uses a rolling hash to cut the input file into variable length chunks and those chunks are added to the chunk storage, and then actual files are constructed from those chunks. Obviously such copies should be done when the files are stable, so either not running or using a snapshot. (If using LVM, this can be arranged easily with any filesystem, except ones that don't like getting mounted multiple times..) So while I guess size-related bugs could be there, there's no any fundamental property of making the task bigger as the files themselves grow bigger.

smarx007 6y ago

Worth noting that Restic is suitable for Windows. Not sure if Bupstash was tested on macOS. Borg also has FreeBSD support and experimental support for Windows via WSL2 and Cygwin.

8fingerlouie 6y ago
I was facing this problem when selecting a backup strategy for client machines. Servers are one thing where i can rely on them to be always running, but clients go off grid occasionally. As for restic, yes it's available on Windows, but has a serious speed degradation when datasets increase above 2TB, mainly due to chunks being 2-5MB, where Borg uses 400MB chunks. And we're not talking a 10% slowdown. It's to the point where prune operations on repositories >2TB can/will take multiple days to complete. For clients, I evaluated multiple solutions, one was Duplicati (https://www.duplicati.com/ https://www.duplicati.com/), and while it appears to work well most of the time, it sometimes stops backing up for no apparent reason. Backuppc also looked promising, but doesn't (didn't ?) support source encrypted backups. In the end i ended up with Arq (https://www.arqbackup.com/ https://www.arqbackup.com/) for Windows and Mac clients. It works reliably, and checks all the right boxes. For servers i was initially really hooked on restic as it allows backups over HTTPS instead of SSH, but the performance issues were a showstopper.