4 ms·

Unless it's for experimenting, I've stopped caring for backup solutions other than borg and zfs as the only way to prove their stability is to have them exist f

Unless it's for experimenting, I've stopped caring for backup solutions other than borg and zfs as the only way to prove their stability is to have them exist for a while without big complaints and new ones all seem to have complaints. Just having no data loss isn't enough which is the absolute base point but huge memory consumption and other operational issues are also showstoppers.

cma 5y ago
Borg being single-threaded is painful in the era of consumer 12 and 16 core CPUs, and even prosumer 64-core.
- sigio 5y ago
  Don't notice this much.... disk is usually the bottleneck, and otherwise it will be the network to the remote backup location. Still backups complete in seconds: Repository: ssh://backup/./backups/mungedhostname.borg Archive name: 20220327-2201 Archive fingerprint: 8b710144579c8d531e7c4a0192304323081b14a71445557608d859494bbe84b6 Time (start): Sun, 2022-03-27 22:01:35 Time (end): Sun, 2022-03-27 22:01:52 Duration: 17.28 seconds Number of files: 45678 Original size Compressed size Deduplicated size This archive: 24.22 GB 9.35 GB 47.78 MB All archives: 363.60 GB 140.91 GB 7.81 GB Unique chunks Total chunks Chunk index: 59857 852257
  - sliken 5y ago
    Sure, for spinning disks, but deduplication, compression, and encryption can easily be the bottleneck if you have some NVMEs.
    - cma 5y ago
      And you can turn the compression way up to where it is very expensive but gives much better space savings.
- silon42 5y ago
  I like that it runs in the background without heating my laptop much.
- mekster 5y ago
  You could spread the backup into several isolated borg repos and run in parallel if CPU is the bottleneck.
- ThomasWaldmann 5y ago
  I agree that a multithreaded borg could utilise resources better, esp. IF you have a lot of changed / new data to back up. But OTOH, for many users this is primarily the case for the first backup, but not for their daily / hourly backups when most files are unchanged - for that, I guess speed is I/O bound and all your CPU cores won't help you with that. So for N-1 backup runs, it just works good enough already for many users. And for that 1 initial backup, some patience helps. :-) Implementing MT is planned since long, but due to the above, other stuff had higher priority.
aborsy 5y ago
Restic in my experience has been rock solid. I actually switched from Borg. Borg’s crypto has known limitations; its Python error messages are long and messy; it complained more frequently. Restic’s repository format is simple and well documented, which is important for long term data recovery (and fixes in case changes occur in the repo). The crypto is from a good source, and well regarded. Multithreaded, fast, nice and clean output. ZFS is a file system, and has serious limitation when used as a backup tool. It needs a ZFS backend, ruling out almost any provider (basically self host your ZFS system, which is costly and error prone). It needs more RAM than Borg and restic. And I personally feel uncomfortable with native encryption in ZFS til some time. Lower level system encryption is probably not what you want in backups. One feature I miss from these tools (other than ZFS): error correction. They could use a Reed Solomon code or similar and add parities in case there is an accidental change in the repository.
- wink 5y ago
  Didn't encounter any problems on Linux... but on Windows I ran into an issue the first time I used it (might me by personal bad luck though). If you're using restic on Windows, maybe take a minute to check https://github.com/restic/restic/pull/3637 https://github.com/restic/restic/pull/3637 - it was my first contribution to restic and it seems to have stalled, but I'm not sure if it's a general "oh, windows" problem no one takes seriously in a project that 90% of the maintainers use on Linux like so very often :P
- mekster 5y ago
  > ZFS is a file system, and has serious limitation when used as a backup tool. But it's definitely the best as it knows everything that happened on the filesystem unlike any other tools that require entire scanning of the repo directories on every run. You can read how zfs performs better than a rsync run. https://arstechnica.com/information-technology/2015/12/rsync-net-zfs-replication-to-the-cloud-is-finally-here-and-its-fast/?amp=1 https://arstechnica.com/information-technology/2015/12/rsync... Also, you can even take database backup as a file system snapshot which is far easier than any other database backup which isn't always simple. > basically self host your ZFS system, which is costly and error prone How is this so? Just run Ubuntu, install zfs userland and it works or just use rsync.net (not affiliated but can't find a better service that accepts zfs send). Don't try on RH based distro as its support is pretty bad on zfs.

cma 5y ago

Borg being single-threaded is painful in the era of consumer 12 and 16 core CPUs, and even prosumer 64-core.

sigio 5y ago
Don't notice this much.... disk is usually the bottleneck, and otherwise it will be the network to the remote backup location. Still backups complete in seconds: Repository: ssh://backup/./backups/mungedhostname.borg Archive name: 20220327-2201 Archive fingerprint: 8b710144579c8d531e7c4a0192304323081b14a71445557608d859494bbe84b6 Time (start): Sun, 2022-03-27 22:01:35 Time (end): Sun, 2022-03-27 22:01:52 Duration: 17.28 seconds Number of files: 45678 Original size Compressed size Deduplicated size This archive: 24.22 GB 9.35 GB 47.78 MB All archives: 363.60 GB 140.91 GB 7.81 GB Unique chunks Total chunks Chunk index: 59857 852257
- sliken 5y ago
  Sure, for spinning disks, but deduplication, compression, and encryption can easily be the bottleneck if you have some NVMEs.
  - cma 5y ago
    And you can turn the compression way up to where it is very expensive but gives much better space savings.
silon42 5y ago
I like that it runs in the background without heating my laptop much.
mekster 5y ago
You could spread the backup into several isolated borg repos and run in parallel if CPU is the bottleneck.
ThomasWaldmann 5y ago
I agree that a multithreaded borg could utilise resources better, esp. IF you have a lot of changed / new data to back up. But OTOH, for many users this is primarily the case for the first backup, but not for their daily / hourly backups when most files are unchanged - for that, I guess speed is I/O bound and all your CPU cores won't help you with that. So for N-1 backup runs, it just works good enough already for many users. And for that 1 initial backup, some patience helps. :-) Implementing MT is planned since long, but due to the above, other stuff had higher priority.

aborsy 5y ago

Restic in my experience has been rock solid. I actually switched from Borg. Borg’s crypto has known limitations; its Python error messages are long and messy; it complained more frequently. Restic’s repository format is simple and well documented, which is important for long term data recovery (and fixes in case changes occur in the repo). The crypto is from a good source, and well regarded. Multithreaded, fast, nice and clean output. ZFS is a file system, and has serious limitation when used as a backup tool. It needs a ZFS backend, ruling out almost any provider (basically self host your ZFS system, which is costly and error prone). It needs more RAM than Borg and restic. And I personally feel uncomfortable with native encryption in ZFS til some time. Lower level system encryption is probably not what you want in backups. One feature I miss from these tools (other than ZFS): error correction. They could use a Reed Solomon code or similar and add parities in case there is an accidental change in the repository.

wink 5y ago
Didn't encounter any problems on Linux... but on Windows I ran into an issue the first time I used it (might me by personal bad luck though). If you're using restic on Windows, maybe take a minute to check https://github.com/restic/restic/pull/3637 https://github.com/restic/restic/pull/3637 - it was my first contribution to restic and it seems to have stalled, but I'm not sure if it's a general "oh, windows" problem no one takes seriously in a project that 90% of the maintainers use on Linux like so very often :P
mekster 5y ago
> ZFS is a file system, and has serious limitation when used as a backup tool. But it's definitely the best as it knows everything that happened on the filesystem unlike any other tools that require entire scanning of the repo directories on every run. You can read how zfs performs better than a rsync run. https://arstechnica.com/information-technology/2015/12/rsync-net-zfs-replication-to-the-cloud-is-finally-here-and-its-fast/?amp=1 https://arstechnica.com/information-technology/2015/12/rsync... Also, you can even take database backup as a file system snapshot which is far easier than any other database backup which isn't always simple. > basically self host your ZFS system, which is costly and error prone How is this so? Just run Ubuntu, install zfs userland and it works or just use rsync.net (not affiliated but can't find a better service that accepts zfs send). Don't try on RH based distro as its support is pretty bad on zfs.