9 ms·

Restic 0.13.0

111 points by soheilpro 5y ago

tekronis 5y ago
Besides Homebrew, restic is also available via MacPorts: https://ports.macports.org/port/restic/ https://ports.macports.org/port/restic/
mekster 5y ago
Unless it's for experimenting, I've stopped caring for backup solutions other than borg and zfs as the only way to prove their stability is to have them exist for a while without big complaints and new ones all seem to have complaints. Just having no data loss isn't enough which is the absolute base point but huge memory consumption and other operational issues are also showstoppers.
- cma 5y ago
  Borg being single-threaded is painful in the era of consumer 12 and 16 core CPUs, and even prosumer 64-core.
  - sigio 5y ago
    Don't notice this much.... disk is usually the bottleneck, and otherwise it will be the network to the remote backup location. Still backups complete in seconds: Repository: ssh://backup/./backups/mungedhostname.borg Archive name: 20220327-2201 Archive fingerprint: 8b710144579c8d531e7c4a0192304323081b14a71445557608d859494bbe84b6 Time (start): Sun, 2022-03-27 22:01:35 Time (end): Sun, 2022-03-27 22:01:52 Duration: 17.28 seconds Number of files: 45678 Original size Compressed size Deduplicated size This archive: 24.22 GB 9.35 GB 47.78 MB All archives: 363.60 GB 140.91 GB 7.81 GB Unique chunks Total chunks Chunk index: 59857 852257
    - sliken 5y ago
      Sure, for spinning disks, but deduplication, compression, and encryption can easily be the bottleneck if you have some NVMEs.
      - cma 5y ago
        And you can turn the compression way up to where it is very expensive but gives much better space savings.
  - silon42 5y ago
    I like that it runs in the background without heating my laptop much.
  - mekster 5y ago
    You could spread the backup into several isolated borg repos and run in parallel if CPU is the bottleneck.
  - ThomasWaldmann 5y ago
    I agree that a multithreaded borg could utilise resources better, esp. IF you have a lot of changed / new data to back up. But OTOH, for many users this is primarily the case for the first backup, but not for their daily / hourly backups when most files are unchanged - for that, I guess speed is I/O bound and all your CPU cores won't help you with that. So for N-1 backup runs, it just works good enough already for many users. And for that 1 initial backup, some patience helps. :-) Implementing MT is planned since long, but due to the above, other stuff had higher priority.
- aborsy 5y ago
  Restic in my experience has been rock solid. I actually switched from Borg. Borg’s crypto has known limitations; its Python error messages are long and messy; it complained more frequently. Restic’s repository format is simple and well documented, which is important for long term data recovery (and fixes in case changes occur in the repo). The crypto is from a good source, and well regarded. Multithreaded, fast, nice and clean output. ZFS is a file system, and has serious limitation when used as a backup tool. It needs a ZFS backend, ruling out almost any provider (basically self host your ZFS system, which is costly and error prone). It needs more RAM than Borg and restic. And I personally feel uncomfortable with native encryption in ZFS til some time. Lower level system encryption is probably not what you want in backups. One feature I miss from these tools (other than ZFS): error correction. They could use a Reed Solomon code or similar and add parities in case there is an accidental change in the repository.
  - wink 5y ago
    Didn't encounter any problems on Linux... but on Windows I ran into an issue the first time I used it (might me by personal bad luck though). If you're using restic on Windows, maybe take a minute to check https://github.com/restic/restic/pull/3637 https://github.com/restic/restic/pull/3637 - it was my first contribution to restic and it seems to have stalled, but I'm not sure if it's a general "oh, windows" problem no one takes seriously in a project that 90% of the maintainers use on Linux like so very often :P
  - mekster 5y ago
    > ZFS is a file system, and has serious limitation when used as a backup tool. But it's definitely the best as it knows everything that happened on the filesystem unlike any other tools that require entire scanning of the repo directories on every run. You can read how zfs performs better than a rsync run. https://arstechnica.com/information-technology/2015/12/rsync-net-zfs-replication-to-the-cloud-is-finally-here-and-its-fast/?amp=1 https://arstechnica.com/information-technology/2015/12/rsync... Also, you can even take database backup as a file system snapshot which is far easier than any other database backup which isn't always simple. > basically self host your ZFS system, which is costly and error prone How is this so? Just run Ubuntu, install zfs userland and it works or just use rsync.net (not affiliated but can't find a better service that accepts zfs send). Don't try on RH based distro as its support is pretty bad on zfs.
soheilpro 5y ago
I'm so glad to finally see the --dry-run option in Restic.
wmf 5y ago
Still no compression?
- spasche 5y ago
  I'm afraid not: https://github.com/restic/restic/issues/21 https://github.com/restic/restic/issues/21
crecker 5y ago
Where can I read about the best solutions for backups? For nerdy people
- CameronNemo 5y ago
  Restic and BorgBackup really seem to be the favored solutions out there. Restic for encryption, Borg for deduplication and compression. Or maybe bacula if you want pull based backups instead of push based. https://restic.readthedocs.io/en/stable/ https://restic.readthedocs.io/en/stable/ https://www.borgbackup.org/ https://www.borgbackup.org/ https://www.bacula.org/documentation/documentation/ https://www.bacula.org/documentation/documentation/
  - photon-torpedo 5y ago
    BorgBackup can be made to operate in pull mode too, see: https://borgbackup.readthedocs.io/en/stable/deployment/pull-backup.html https://borgbackup.readthedocs.io/en/stable/deployment/pull-... At work I've implemented a variant of the ssh-agent method described there. Admittedly, it requires some scripting :)
- O_H_E 5y ago
  When talking about backups for nerds, https://www.rsync.net/ https://www.rsync.net/ always deserves a mention.
  - funcDropShadow 5y ago
    ... as well as the http://taobackup.com/ http://taobackup.com/ .
- lucgommans 5y ago
  There is also https://github.com/restic/others https://github.com/restic/others which has some keywords (e.g. is it encrypted, does it do compression) for most FOSS backup solutions. It can be outdated or incomplete for some entries, though.
- rsync 5y ago
lucgommans 5y ago
This point hides a lot of goodness in something that I didn't even understand on the first read: > - We have added checksums for various backends so data uploaded to a backend can be checked there. All data is already stored in files with as filename the sha256sum of the contents, so clearly it's all already checksummed and can be verified right? Looking into the changelog entry[1], this is about verifying the integrity upon uploading: > The verification works by informing the backend about the expected hash of the uploaded file. The backend then verifies the upload and thereby rules out any data corruption during upload. \n\n [...] besides integrity checking for uploads [this] also means that restic can now be used to store backups in S3 buckets which have Object Lock enabled. Object lock is mentioned in passing somewhere down the changelog, but it's a big feature. S3 docs: > Object Lock can help prevent objects from being deleted or overwritten for a fixed amount of time or indefinitely. i.e. ransomware protection. Good luck wiping backups if your file hoster refuses to overwrite or delete the files. And you know Amazon didn't mess with the files because they're authenticated. Extortion is still a thing, but if people would use this, it more-or-less wipes out the attack vector of ransomware. The only risk is if the attacker is in your systems long enough to outlast your retention period and creates useless backups in the meantime so you're not tipped off. Did anyone say "test your backups"? For self-hosting, restic has a custom back-end called rest-server[2] which supports a so-called "append-only mode" (no overwriting or deleting). I worked on the docs for this[3] together with rawtaz and MichaelEischer to make this more secure, because eventually, of course, your disks are full or you want to stop paying for outdated snapshots on S3, and an attacker could have added dummy backups to fool your automatic removal script into thinking it needs to leave only the dummy backups. Using the right retention options, this attack cannot happen. Others are doing some pretty cool stuff in the backup sphere as well, e.g. bupstash[4] has public key encryption so you don't need to have the decryption keys as a backup client. [1] https://github.com/restic/restic/releases/v0.13.0 https://github.com/restic/restic/releases/v0.13.0 [2] https://github.com/restic/rest-server/ https://github.com/restic/rest-server/ [3] https://restic.readthedocs.io/en/latest/060_forget.html#security-considerations-in-append-only-mode https://restic.readthedocs.io/en/latest/060_forget.html#secu... [4] https://github.com/andrewchambers/bupstash/ https://github.com/andrewchambers/bupstash/
Mo3 5y ago
I have to say, this is an excellent solution and I am seriously contemplating on deploying it for all of our servers. I'm a little unclear on one thing, are alternative S3 providers supported?
- bccdee 5y ago
  Yes! I back up directly into Backblaze B2 using restic.
  - Mo3 5y ago
    Excellent. One more question. Assuming someone gains access to a node - will they be able to access the backups or screw with them in any way?
    - lucgommans 5y ago
      Depends which back-end you use and which configuration. (Assuming "node" here means a backup client and not a backup-hosting server.) If you just put stuff on some standard storage (FTP, B2, etc.) without any permissions set up, then no backup client could stop the attacker from using the credentials you've deployed to the backup client to login to your storage and delete your backup files. This is not really specific to restic. Accessing old files from a backup client is a weakness. Though, personally, I don't store things on a system that this system is not supposed to know, so if someone compromises this system and they see data from the past year... that's nearly the same as just seeing the data that is on this system today. Nevertheless, this could be solved by using public keys, so the backup clients has an encryption but no decryption key (of course it's not as trivial as this sounds, e.g. how could it still do deduplication etc.), but restic does not do this.
      - Mo3 5y ago
        Got it. So I guess the best solution to go forward here would be to deploy Minio and then backup the backup directory itself in regular intervals onto different storage not accessible from anywhere.
paraph1n 5y ago
Any thoughts on how this compares to duplicacy?
- Tijdreiziger 5y ago
  …or Duplicity?
  - sigio 5y ago
    Both restic and borg are lightyears ahead of duplicity.... backups are soo much faster (and smaller), restores are much easier.
  - seized 5y ago
    Duplicacy is much faster than Restic for backups and. That was was the overwhelming consensus when I was researching backup software. I use Duplicacy and Rclone, each for different types of data.
- aborsy 5y ago
  Duplicacy has no mount!
timmg 5y ago
I have a hand-coded backup system for my photo library that writes to S3. It runs every night at 2AM. The one feature I have that's important to me is this: it will figure out what files need to be uploaded and then upload as many as possible for an hour then stop. That means that it runs for at-most an hour a night. The reason I need/wanted this feature is that I might come home from a trip with (eg) 30G worth of photos. My (cable) internet will upload at around 1G an hour. I don't want this thing to saturate my internet for 30 hours straight. Instead, it backs up a small amount every night for 30 days. Am I the only one that wants a feature like this? I've never seen it in any other backup system. (At alternative might be to have configurable bandwidth for uploads.)
- crecker 5y ago
  I wonder if you can obtain that behavior with a bash script, but it's boring to write scripts and I do not know if SIGTERM can exit Restic gracely.
  - lucgommans 5y ago
    If you've ever ctrl+c'd restic, you'll know the message > signal interrupt received, cleaning up
    - crecker 5y ago
      Thanks! Will try tonight to play with Restic
- lucgommans 5y ago
  Makes perfect sense. Restic kind-of supports this because you can just kill the client after an hour and, tomorrow, it'll see which objects are there already. I'm not deep enough into the project to know whether this is like an officially supported use-case, but restic was of course made with the idea that interruptions can happen (your computer can crash) and should be handled safely, and for the deduplication it'll cut files up in a deterministic way and thus (as I understand it) store those chunks in a deterministic place.
- 5y ago
rvieira 5y ago
For someone using Kopia, is there any major advantage in switching to restic?
- omnimus 5y ago
  I am using restic and thinking about switching to Kopia... Mainly because Kopia has compression and seems to have more activity in development. It also has gui. And from what i've seen is faster.

111 points by soheilpro 5y ago

Besides Homebrew, restic is also available via MacPorts: https://ports.macports.org/port/restic/ https://ports.macports.org/port/restic/

mekster 5y ago

Unless it's for experimenting, I've stopped caring for backup solutions other than borg and zfs as the only way to prove their stability is to have them exist for a while without big complaints and new ones all seem to have complaints. Just having no data loss isn't enough which is the absolute base point but huge memory consumption and other operational issues are also showstoppers.

cma 5y ago
Borg being single-threaded is painful in the era of consumer 12 and 16 core CPUs, and even prosumer 64-core.
- sigio 5y ago
  Don't notice this much.... disk is usually the bottleneck, and otherwise it will be the network to the remote backup location. Still backups complete in seconds: Repository: ssh://backup/./backups/mungedhostname.borg Archive name: 20220327-2201 Archive fingerprint: 8b710144579c8d531e7c4a0192304323081b14a71445557608d859494bbe84b6 Time (start): Sun, 2022-03-27 22:01:35 Time (end): Sun, 2022-03-27 22:01:52 Duration: 17.28 seconds Number of files: 45678 Original size Compressed size Deduplicated size This archive: 24.22 GB 9.35 GB 47.78 MB All archives: 363.60 GB 140.91 GB 7.81 GB Unique chunks Total chunks Chunk index: 59857 852257
  - sliken 5y ago
    Sure, for spinning disks, but deduplication, compression, and encryption can easily be the bottleneck if you have some NVMEs.
    - cma 5y ago
      And you can turn the compression way up to where it is very expensive but gives much better space savings.
- silon42 5y ago
  I like that it runs in the background without heating my laptop much.
- mekster 5y ago
  You could spread the backup into several isolated borg repos and run in parallel if CPU is the bottleneck.
- ThomasWaldmann 5y ago
  I agree that a multithreaded borg could utilise resources better, esp. IF you have a lot of changed / new data to back up. But OTOH, for many users this is primarily the case for the first backup, but not for their daily / hourly backups when most files are unchanged - for that, I guess speed is I/O bound and all your CPU cores won't help you with that. So for N-1 backup runs, it just works good enough already for many users. And for that 1 initial backup, some patience helps. :-) Implementing MT is planned since long, but due to the above, other stuff had higher priority.
aborsy 5y ago
Restic in my experience has been rock solid. I actually switched from Borg. Borg’s crypto has known limitations; its Python error messages are long and messy; it complained more frequently. Restic’s repository format is simple and well documented, which is important for long term data recovery (and fixes in case changes occur in the repo). The crypto is from a good source, and well regarded. Multithreaded, fast, nice and clean output. ZFS is a file system, and has serious limitation when used as a backup tool. It needs a ZFS backend, ruling out almost any provider (basically self host your ZFS system, which is costly and error prone). It needs more RAM than Borg and restic. And I personally feel uncomfortable with native encryption in ZFS til some time. Lower level system encryption is probably not what you want in backups. One feature I miss from these tools (other than ZFS): error correction. They could use a Reed Solomon code or similar and add parities in case there is an accidental change in the repository.
- wink 5y ago
  Didn't encounter any problems on Linux... but on Windows I ran into an issue the first time I used it (might me by personal bad luck though). If you're using restic on Windows, maybe take a minute to check https://github.com/restic/restic/pull/3637 https://github.com/restic/restic/pull/3637 - it was my first contribution to restic and it seems to have stalled, but I'm not sure if it's a general "oh, windows" problem no one takes seriously in a project that 90% of the maintainers use on Linux like so very often :P
- mekster 5y ago
  > ZFS is a file system, and has serious limitation when used as a backup tool. But it's definitely the best as it knows everything that happened on the filesystem unlike any other tools that require entire scanning of the repo directories on every run. You can read how zfs performs better than a rsync run. https://arstechnica.com/information-technology/2015/12/rsync-net-zfs-replication-to-the-cloud-is-finally-here-and-its-fast/?amp=1 https://arstechnica.com/information-technology/2015/12/rsync... Also, you can even take database backup as a file system snapshot which is far easier than any other database backup which isn't always simple. > basically self host your ZFS system, which is costly and error prone How is this so? Just run Ubuntu, install zfs userland and it works or just use rsync.net (not affiliated but can't find a better service that accepts zfs send). Don't try on RH based distro as its support is pretty bad on zfs.

soheilpro 5y ago

I'm so glad to finally see the --dry-run option in Restic.

wmf 5y ago

Still no compression?

spasche 5y ago
I'm afraid not: https://github.com/restic/restic/issues/21 https://github.com/restic/restic/issues/21

crecker 5y ago

Where can I read about the best solutions for backups? For nerdy people

CameronNemo 5y ago
Restic and BorgBackup really seem to be the favored solutions out there. Restic for encryption, Borg for deduplication and compression. Or maybe bacula if you want pull based backups instead of push based. https://restic.readthedocs.io/en/stable/ https://restic.readthedocs.io/en/stable/ https://www.borgbackup.org/ https://www.borgbackup.org/ https://www.bacula.org/documentation/documentation/ https://www.bacula.org/documentation/documentation/
- photon-torpedo 5y ago
  BorgBackup can be made to operate in pull mode too, see: https://borgbackup.readthedocs.io/en/stable/deployment/pull-backup.html https://borgbackup.readthedocs.io/en/stable/deployment/pull-... At work I've implemented a variant of the ssh-agent method described there. Admittedly, it requires some scripting :)
O_H_E 5y ago
When talking about backups for nerds, https://www.rsync.net/ https://www.rsync.net/ always deserves a mention.
- funcDropShadow 5y ago
  ... as well as the http://taobackup.com/ http://taobackup.com/ .
lucgommans 5y ago
There is also https://github.com/restic/others https://github.com/restic/others which has some keywords (e.g. is it encrypted, does it do compression) for most FOSS backup solutions. It can be outdated or incomplete for some entries, though.
rsync 5y ago

lucgommans 5y ago

This point hides a lot of goodness in something that I didn't even understand on the first read: > - We have added checksums for various backends so data uploaded to a backend can be checked there. All data is already stored in files with as filename the sha256sum of the contents, so clearly it's all already checksummed and can be verified right? Looking into the changelog entry[1], this is about verifying the integrity upon uploading: > The verification works by informing the backend about the expected hash of the uploaded file. The backend then verifies the upload and thereby rules out any data corruption during upload. \n\n [...] besides integrity checking for uploads [this] also means that restic can now be used to store backups in S3 buckets which have Object Lock enabled. Object lock is mentioned in passing somewhere down the changelog, but it's a big feature. S3 docs: > Object Lock can help prevent objects from being deleted or overwritten for a fixed amount of time or indefinitely. i.e. ransomware protection. Good luck wiping backups if your file hoster refuses to overwrite or delete the files. And you know Amazon didn't mess with the files because they're authenticated. Extortion is still a thing, but if people would use this, it more-or-less wipes out the attack vector of ransomware. The only risk is if the attacker is in your systems long enough to outlast your retention period and creates useless backups in the meantime so you're not tipped off. Did anyone say "test your backups"? For self-hosting, restic has a custom back-end called rest-server[2] which supports a so-called "append-only mode" (no overwriting or deleting). I worked on the docs for this[3] together with rawtaz and MichaelEischer to make this more secure, because eventually, of course, your disks are full or you want to stop paying for outdated snapshots on S3, and an attacker could have added dummy backups to fool your automatic removal script into thinking it needs to leave only the dummy backups. Using the right retention options, this attack cannot happen. Others are doing some pretty cool stuff in the backup sphere as well, e.g. bupstash[4] has public key encryption so you don't need to have the decryption keys as a backup client. [1] https://github.com/restic/restic/releases/v0.13.0 https://github.com/restic/restic/releases/v0.13.0 [2] https://github.com/restic/rest-server/ https://github.com/restic/rest-server/ [3] https://restic.readthedocs.io/en/latest/060_forget.html#security-considerations-in-append-only-mode https://restic.readthedocs.io/en/latest/060_forget.html#secu... [4] https://github.com/andrewchambers/bupstash/ https://github.com/andrewchambers/bupstash/

Mo3 5y ago

I have to say, this is an excellent solution and I am seriously contemplating on deploying it for all of our servers. I'm a little unclear on one thing, are alternative S3 providers supported?

bccdee 5y ago
Yes! I back up directly into Backblaze B2 using restic.
- Mo3 5y ago
  Excellent. One more question. Assuming someone gains access to a node - will they be able to access the backups or screw with them in any way?
  - lucgommans 5y ago
    Depends which back-end you use and which configuration. (Assuming "node" here means a backup client and not a backup-hosting server.) If you just put stuff on some standard storage (FTP, B2, etc.) without any permissions set up, then no backup client could stop the attacker from using the credentials you've deployed to the backup client to login to your storage and delete your backup files. This is not really specific to restic. Accessing old files from a backup client is a weakness. Though, personally, I don't store things on a system that this system is not supposed to know, so if someone compromises this system and they see data from the past year... that's nearly the same as just seeing the data that is on this system today. Nevertheless, this could be solved by using public keys, so the backup clients has an encryption but no decryption key (of course it's not as trivial as this sounds, e.g. how could it still do deduplication etc.), but restic does not do this.
    - Mo3 5y ago
      Got it. So I guess the best solution to go forward here would be to deploy Minio and then backup the backup directory itself in regular intervals onto different storage not accessible from anywhere.

paraph1n 5y ago

Any thoughts on how this compares to duplicacy?

Tijdreiziger 5y ago
…or Duplicity?
- sigio 5y ago
  Both restic and borg are lightyears ahead of duplicity.... backups are soo much faster (and smaller), restores are much easier.
- seized 5y ago
  Duplicacy is much faster than Restic for backups and. That was was the overwhelming consensus when I was researching backup software. I use Duplicacy and Rclone, each for different types of data.
aborsy 5y ago
Duplicacy has no mount!

timmg 5y ago

I have a hand-coded backup system for my photo library that writes to S3. It runs every night at 2AM. The one feature I have that's important to me is this: it will figure out what files need to be uploaded and then upload as many as possible for an hour then stop. That means that it runs for at-most an hour a night. The reason I need/wanted this feature is that I might come home from a trip with (eg) 30G worth of photos. My (cable) internet will upload at around 1G an hour. I don't want this thing to saturate my internet for 30 hours straight. Instead, it backs up a small amount every night for 30 days. Am I the only one that wants a feature like this? I've never seen it in any other backup system. (At alternative might be to have configurable bandwidth for uploads.)

crecker 5y ago
I wonder if you can obtain that behavior with a bash script, but it's boring to write scripts and I do not know if SIGTERM can exit Restic gracely.
- lucgommans 5y ago
  If you've ever ctrl+c'd restic, you'll know the message > signal interrupt received, cleaning up
  - crecker 5y ago
    Thanks! Will try tonight to play with Restic
lucgommans 5y ago
Makes perfect sense. Restic kind-of supports this because you can just kill the client after an hour and, tomorrow, it'll see which objects are there already. I'm not deep enough into the project to know whether this is like an officially supported use-case, but restic was of course made with the idea that interruptions can happen (your computer can crash) and should be handled safely, and for the deduplication it'll cut files up in a deterministic way and thus (as I understand it) store those chunks in a deterministic place.
5y ago

rvieira 5y ago

For someone using Kopia, is there any major advantage in switching to restic?

omnimus 5y ago
I am using restic and thinking about switching to Kopia... Mainly because Kopia has compression and seems to have more activity in development. It also has gui. And from what i've seen is faster.