18 ms·
Restic – Backups Done Right
- cfallin 7y agoI've been using Restic recently for backups online (to Backblaze, which it natively supports) after using Duplicity for a while. I'm really, really impressed so far; the hash-based deduplication means: - I don't have to think about deduplicating my own data, which works well with my packrat tendencies (e.g., multiple copies of music library on snapshots of old laptops); - Full-disk backups of multiple machines will share a lot of storage (all system files from my desktop and laptop, both running the same Linux distro); - Restarting a long upload doesn't depend on some finicky state regarding an interrupted previous backup; it can just rescan the whole disk and skip uploading blobs that are already there, which feels much more robust to me.
- Erlich_Bachman 7y agoFYI those are all functions of any deduplicating backup software, of which there are several out there, Restic being only one of them.
- cfallin 7y agoYes, indeed; Restic just happened to be an easy-to-use deduplicating backup program that supports various cloud storage backends, including the one I happen to use (Backblaze). Not needing to patch together some sort of pipeline of dedup --> upload to remote storage is a nice selling point.
- vasili111 7y agoWhat you think about Restic vs Duplicati? Any personal experience?
- wbkang 7y agoAt least on Windows, the biggest difference is that Restic doesn't support VSS snapshot so you can't use it to backup all files in a consistent manner. This makes it a lot less useful than other solutions on Windows.
- jeffdubin 7y agoI've been using Duplicati 2.0 beta (aka "stable") builds, not canary/testing, for 2+ years, and at least three times, on three different machines each with different backends (two using ZFS, scrubbed regularly with no data issues), I've encountered issues with database corruption which wasn't fixable with repair and rebuild. There are lots of things I really like about Duplicati (OSS, backend flexibility, dedup, encryption, active development, etc.), but I'm going to be moving to another solution for a little while until a super stable release is available. Restic was at the top of my list even before this most recent HN nod.
- seized 7y agoCheck out Duplicacy. It seems very solid, none of the Duplicati like issues.
- deleted 7y ago[deleted]
- noja 7y agoFantastic software. Fast, locally encrypted before being shipped to the cloud (or a usb disk), just works.
- brensmith 7y agoI've been using it for about 6 months now, backing up my home directory 3-4 times per week to an external HD. Would also agree that it's fast, easy to use, and also easy to recover files. I'd add that the documentation is really well written.
- rsync 7y agoRestic can operate over (among other methods) plain old SFTP[1]. Therefore, it does - and always has - work perfectly with an rsync.net account[2]. I personally find it reassuring that even though I might be creating and maintaining backups with sophisticated tools like borg or restic or duplicity or rclone ... at any time, and from any system I can grab those backups with dumb old SFTP. [1] https://www.rsync.net/products/sftp.html https://www.rsync.net/products/sftp.html [2] https://forum.restic.net/t/restic-commands-for-rsync-net/2162 https://forum.restic.net/t/restic-commands-for-rsync-net/216...
- ibic 7y agoHow does it compare to Rclone? https://rclone.org/ https://rclone.org/
- dsego 7y agoAfaik, restic does incremental backups (diffs) and encryption. This is great if you need to go back in time and it's safer if something for example corrupts your files (the backups don't get overwritten with corrupt versions). But it is more difficult to restore, the files are split into chunks for deduplication, so you are not able to actually see your backed up files — you need to run the restore process first. I am using rclone right now, and have it set up to archive old file versions (--backup-dir). It's simpler and covers my needs for now.
- specto 7y agoActually you can use rclone as a backend to send your blocks to wherever rclone supports. quite nice.
- gsmecher 7y agoBorg Backup (for similar reasons as Restic) essentially shifted backup from my "I-don't-but-I-should" list to my "solved problems" list. Is there anything similar for binary distribution? Here's what I mean. I develop a software/firmware stack that is typically delivered to users as a set of large (100M) tarballs and binary images. Even though the vast majority of the payload does not change from release to release, it really has proven necessary to distribute just the final blobs. Technologically, it's almost identical, but distribution implies a different set of access controls and an Internet-friendly user interface. I see the idea floating around in e.g. NextCloud forums but this seems like a relatively compact problem without a obvious candidates to solve it.
- dividuum 7y agoI'm not entirely sure I understand what you're asking for, but it sounds like zsync (http://zsync.moria.org.uk/ http://zsync.moria.org.uk/) is what you're looking for.
- curt15 7y agoPerhaps something like OSTree?
- gsmecher 7y agoThanks, OSTree sounds like a good place to start.
- ptype 7y agoI’ve had success with rsync and the “rsyncable” gzip flag in a similar situation
- wongarsu 7y agoThere are a few products in the autoupdater/patcher space that do something like this ("delta update" is the name of that feature).
- xfer 7y agoThere are some programs that do this: https://github.com/systemd/casync https://github.com/systemd/casync . See the readme for similar tools.
- madjam002 7y agoSolid piece of software, I use this every day! One small issue which I need to figure out is the hosts directory being empty when using “restic mount”: https://github.com/restic/restic/issues/1869 https://github.com/restic/restic/issues/1869
- hardwaresofton 7y agoAlso check out Borg: https://www.borgbackup.org https://www.borgbackup.org And some resources on how they're different: - https://github.com/restic/restic/issues/1875 https://github.com/restic/restic/issues/1875 - https://stickleback.dk/borg-or-restic/ https://stickleback.dk/borg-or-restic/ - https://sysadministrivia.com/episodes/S4E5 https://sysadministrivia.com/episodes/S4E5 The general concensus seems to be that restic is borg with more whistles (backing up to various places), but borg is the more trusted tool with the longer history (just use SSH and be done with it). I personally recently used borg for a migration between computers and it worked great for me.
- noja 7y agorestic is simpler than borg, because it doesn't require a special borg server (it just writes out to files - or block storage).
- Quekid5 7y agoThis is important -- it means that restic can trivially be used with e.g. S3.
- res0nat0r 7y agoI've loved using restic sofar. I've got about 3TB of stuff backed up to s3 using the restic cli. It is simple and just works. If anyone though knows how I can more easily restore a file that would be great. I have to supply the restoration snapshot ID right now, and I'd rather just do 'latest' and have it find the newest version of the file in all of the snapshots and restore it. Is this possible ? Something like: restic restore -i '/path/to/file.txt' -i . latest instead of: restic restore -i '/path/to/file.txt' -i . search-file-snapshot-list-for-big-sha-here
- ailox 7y agoyou could always mount the entire backup, using restic mount backup/ and pick the file(s) you are looking for from the `backup/snapshots/latest/` directory.
- PureParadigm 7y agoThis seems very similar to Borg Backup [1], so I'm interested to hear from others who have used both on how it compares. More generally, I've been looking for a solution that helps distribute backups in a peer-to-peer way. I have a few friends with their own home servers, and we want to replicate backups across each other's servers for geographical redundancy. Currently, I have a script that uses rsync to copy some tar archives over daily, but this doesn't scale well as more peers want to join our backup-sharing group, since it requires them granting me SSH access. What I need is a decentralized network to share and retrieve backups from peers. I tried using dat [2] with a Borg Backup repository inside it, but ran into some nasty issues with dat which would cause it to regularly crash and one time even corrupt the data. Does anyone have any suggestions for such a situation? [1] https://www.borgbackup.org/ https://www.borgbackup.org/ [2] https://github.com/datproject/dat https://github.com/datproject/dat
- kevlar1818 7y agoThis might not be quite what you're looking for, but Syncthing[1] is a popular P2P file sharing solution. You could use Restic to make backups to a shared Syncthing folder. [1]: https://syncthing.net/ https://syncthing.net/
- PureParadigm 7y agoThanks for the suggestion. I've tried Syncthing, but it seems to still require that users are explicitly added (i.e., no public access [1]). I'd prefer a solution where anyone could decide to start helping replicate backups, without me having to add them in some way. [1] https://github.com/syncthing/syncthing/issues/1942 https://github.com/syncthing/syncthing/issues/1942
- kevlar1818 7y agoThen you should look into IPFS :) https://github.com/ipfs/ipfs https://github.com/ipfs/ipfs
- 7y ago
- swedtrue 7y agoJust so I understand right and out of curiosity, what does that offer more than the likes of Nextcloud: https://nextcloud.com/ https://nextcloud.com/ or Duple: https://www.duple.io/en/ https://www.duple.io/en/ Since if I understand correctly both provide backup in addition to other functionalities?
- noja 7y agoNextcloud is a different product. restic is for backup.
- swedtrue 7y agoMay you elaborate? What does it offer more? Since Nextcloud and Duple do offer backup as well.
- noja 7y agoNextcloud is a web app that lets you work with files and offers lots of other plugins too, like e-mail, calendaring, office, contacts. It's not a backup program. See https://nextcloud.com/ https://nextcloud.com/ or https://en.wikipedia.org/wiki/Nextcloud https://en.wikipedia.org/wiki/Nextcloud
- ausjke 7y agoor, restic is your NAS backup while Nextcloud is your open source dropbox, roughly.
- NoGravitas 7y agoYou might, for example, use restic or borg to back up your Nextcloud's installation and storage.
- OJFord 7y agoThat's like asking what a markdown renderer offers that Wordpress doesn't.
- mike-cardwell 7y agoI started using Restic recently. It's good and I'm going to continue using it. That said, there are a couple of bad problems with it: Firstly, if you want to prune old backups, e.g keep the last N1 hourly backups, and the last N2 weekly backups, etc, then it has that ability, however whilst it's doing it, the client has to download and upload a tonne of data in order to repackage the backup files that contain some data that needs removing and other data which doesn't. Secondly, I've set up an "append only" system, where my various hosts can append to their own backups, but not overwrite or delete them. I wanted the backup server to be unable to read the backups (easy enough, don't supply the encryption keys to the backup server), however at the same time I wanted the backup server to be able to automatically prune old backups. It can not do that without the key. I don't want to give it the keys to the backups as then a compromised backup server means all of my hosts data are suddenly compromised.
- rcthompson 7y agoHow would you expect the backup server to be able to prune old encrypted backups if it can't decrypt them? How should it know what to delete?
- frenchie4111 7y agoThe backup server should have information about when a particular backup was made, it should be able to prune based on backup date without having the ability to read any of the data inside the backup.
- pierrebai 7y agoI assume the backups are incremental, so you cannot just toss old encrypted files without losing backed up files.
- zaarn 7y agoWorse, it's deduplicated, using a similar algorithm to git. You can forget old backups and theoretically that could be done without decrypting but deleting the actual data will require a decrypt.
- greggman2 7y agoThis looks great. I've been using Arq for years and no complaints but I might switch. I've also been using Syncthing for syncing machines. It's pretty great. i've got it setup so 2 laptops sync to a server and no issues so far.
- LVB 7y agoI'm a long time Arq user too. I looked at Restic but was a little disappointed it didn't support compression. There is an old, still open issue with a ton of comments: https://github.com/restic/restic/issues/21 https://github.com/restic/restic/issues/21
- Erlich_Bachman 7y agoBe aware that Restic does not support compression at all currently, before you might switch.
- booi 7y agoThe biggest problem I've had with backup software is being able to verify when using aws s3 glacier with and without a data pull. Is this possible and efficient with restic? The only software I've found that does this well is Arq
- seized 7y agoI've been researching this use case and it sounds like Restic doesn't handle it well without adding more layers. https://forum.restic.net/t/restic-and-s3-glacier-deep-archive/1551/2 https://forum.restic.net/t/restic-and-s3-glacier-deep-archiv... I've largly decided on Duplicacy over Restic, Borg or Duplicati.
- mike-cardwell 7y agoI recently set this up using Nginx as my restic backend server. Not as a proxy to a backend server, but the actual backend server. No other applications required: https://www.grepular.com/Nginx_Restic_Backend https://www.grepular.com/Nginx_Restic_Backend
- JoshTriplett 7y agoRestic works very well for me, and I've successfully used it to restore lost files. The one issue I've encountered with it: it uses a lot of memory, proportional to the size of the repository indexes, so if you're backing up a lot of data on a machine without a lot of RAM (such as a virtual server), you may run out of memory. Setting GOGC=20 can help slightly, but ultimately, restic needs fixing to support working on indexes larger than memory.
- Scaevolus 7y agoUnfortunately the primary author has mostly moved onto other things, and hasn't elected any maintainers. It's been three months without any commits, and there's 60+ open PRs. The biggest missing feature of Restic is no support for compression. Ah, well.
- aleph- 7y agoAs far as I'm aware from the IRC channel the primary maintainer is mainly busy moving. I do concede it is annoying having to fold in upstream PRs to my own build of it.
- bsg75 7y agoDiscussion from September, including other committers and lack of PR attention: https://forum.restic.net/t/is-restic-dying/2103/5 https://forum.restic.net/t/is-restic-dying/2103/5
- ac29 7y agoThere's a recent open PR adding compression support: https://github.com/restic/restic/pull/2441 https://github.com/restic/restic/pull/2441
- balladeer 7y agoThat’s exactly how Borg started. After it was evident Attic was going to be abandoned. Though I’d say it’s way too early for Restic to land into such a predicament.
- ofrzeta 7y agoI understand that Restic is using deduplication but what's the advantage over rsnapshot? There's AES encryption but how do you handle the credentials with automated backups?
- ansible 7y agoI've been using rsnapshot for a decade or more. For a modest amount of data (a few TB) the performance is adequate, and it is easy to verify the backups. It is also easy to restore files, just go into the directory corresponding to the date of the backup. I could try to run compression in the filesystem itself to save space, but haven't done that so far.
- jaequery 7y agoI've been looking for a solution for something like this and not sure if Restic can do this, so please chime in. Does anyone know of an open-source solution that acts just like Dropbox/GDrive where it detects for any changes in a specified folder and then once detected it automatically uploads to an S3 folder?
- deleted 7y ago[deleted]
- severine 7y agohttps://s3tools.org/s3cmd-sync https://s3tools.org/s3cmd-sync
- rhizome 7y agoSyncthing and s3-fuse to create a local pipe to s3. https://github.com/s3fs-fuse/s3fs-fuse https://github.com/s3fs-fuse/s3fs-fuse
- ramses0 7y agoS3-Fuse has been really unreliable for me (hard-hangs, and similar) ... is that your experience as well, or has it been reliable?
- rhizome 7y agoI haven't used this combination, I haven't used FUSE in awhile, and I forgot to add a disclaimer. FUSE in general has been flaky for me, too. Syncthing has worked pretty well for me though, so I looked to see if Unix Philosophy™ could be used to smash a couple things together. I think that at the end of the day, they are both deceptively complicated technologies.
- throw7 7y agofrom what i read restic has memory scaling issues for certain areas (large amounts of small files, pruning operations). that said, i do use it personally for private and small stuff.
- ahnick 7y agoRestic is probably my favorite pure open source backup solution, but I've been using Duplicacy for years now and have been very happy with it. On the GitHub page (https://github.com/gilbertchen/duplicacy https://github.com/gilbertchen/duplicacy) for Duplicacy you can find a comparison to Restic (along with other backup solutions), which I found informative.
- deleted 7y ago[deleted]
- psanford 7y agoFilippo Valsorda (Go's current crypto maintainer) took a little bit of time to look at the cryptography used by restic: https://blog.filippo.io/restic-cryptography/ https://blog.filippo.io/restic-cryptography/
- Krasnol 7y ago> Easy: Doing backups should be a frictionless process, otherwise you are tempted to skip it. Restic should be easy to configure and use, so that in the unlikely event of a data loss you can just restore it. Likewise, restoring data should not be complicated. So does it have a GUI Version?
- anticristi 7y agoIs backing up still a thing? I prefer to keep my laptop stateless: * I use Ansible to configure a vanilla Ubuntu 18.04 into my workstation [1] * I keep everything that is "source code-y" in some Git repo * I keep non-source-code-y stuff in Dropbox or Seafile (both have a restore to previous version) I prefer everything else to be lost (e.g. some AWS, K8s credentials). I wipe my laptop between customer projects and it works great. [1] https://github.com/cristiklein/stateless-workstation-config https://github.com/cristiklein/stateless-workstation-config
- NoGravitas 7y agoYou're just outsourcing your backups, trusting github, Dropbox, etc. to have good backups.
- deleted 7y ago[deleted]
- kissgyorgy 7y agoThat's pretty cool! I was thinking about this for a while, and just started converting my dotfiles repo to Ansible which would be a step forward.
- throw0101a 7y agoA backup is a coherent copy of data on independent media. Seems that what you're doing is that.
- cactus2093 7y agoI do this too, and am surprised not to see more recommendations for it. It's simpler than a traditional full system backup, lessens security risks like copying credentials into a backup, and IMO keeps things better organized as well. I know Dropbox or git repos are just syncing the files I care about, in a way that I want them organized and can browse on a web interface, on mobile, sync to another device even if it doesn't have all the same software installed, etc. Whereas with a full system backup, there's not always a clear line between system configuration and data files, so depending what apps you're using the data that you're backing up might only be able to be accessed again on an exact copy of the machine running all the same versions of the same apps, which is often not what I actually want.
- Nican 7y agoGlacier Storage/Cold storage, which usually takes ~6h to retrieve files, on Cloud providers are really cheap (I know Azure has $0.99/TB/month), but I have a hard time finding any free backup solutions to support the very-high latency natively. I remember reading a long time ago that Restic was not going to natively support cold storage solutions. Has anything changed?
- adontz 7y agoYou should not backup directly to Glacier, you should backup to S3 and move from S3 to Glacier by retention policy configured for an S3 bucket.
- Nican 7y agoAnd when I have to do incremental backups or retrieve a single file, do I have to move the whole backup from cold storage, and then back again?
- adontz 7y agoI did not understand "back again" part. Glacier allows you to retrieve individual files.
- prirun 7y agoYou should not do this, because getting your data back out of Glacier will be painful. You cannot use S3 tools to restore data from Glacier. Author of Hashbackup, and I've implemented and then depreciated Glacier support because it is very difficult for anything other than WORM-type backups.
- adontz 7y agoIt's not painful, it's the way this specific product works.
- brunoqc 7y agoI use and like Restic but I wish I could easily have "write only" keys like I do with tarsnap so an intruder can't delete all my backups. I heard there's a way to have an "append only" backup or something like that. Is it possible to still prune old backups from time to time?
- alexellisuk 7y agoI put a tutorial together for this not too long ago, a very good tool and great when paired with Minio/s3 - https://github.com/alexellis/restic-minio-civo-learn-guide https://github.com/alexellis/restic-minio-civo-learn-guide
- lucb1e 7y agoI've been using this and it is the best I have found so far. The only issue I encountered was with trying to back up 2.5TB of new data over a shitty German internet connection, which first took forever (restarting 20 times because they force your IP address to change once a day), and then when trying to prune the backup, ran out of RAM memory and actually corrupted it. I'll continue to use it, just not repeating those specific steps, but it was not immediately apparent that the backup was completely unusable. Always test your backups (regardless of whether it's about restic, borg, examplecorp expert enterprise backup, or something else)!
- Sphax 7y agoI've been using restic to backup my workstations for quite some time now and it works well enough. At work I have a systemd service which runs every 10 minutes and use a nfs repository. Performance is good. It has saved me once already after a botched Ubuntu upgrade. At home I have a identical service but it runs every 6 hour and use a backblaze b2 repository. Performance is not great. However I've been backing up ~20 GB for over a year now and it has cost me less than $2 _in total_ so I'd say it's worth it.
- jszymborski 7y agoI've been using Duplicati, which has been doing a great job at keeping my backup sizes low and has a handy interface. I've been keeping encrypted remote backups for ~300 Gb worth of data, which occupies ~150 Gb. On Backblaze B2, it's costing me ~$10CAD/mo, which is a _lot_ cheaper than Tarsnap. I might try Restic (always good to have a backup backup system), but I'm not sure how ergonomic it'd be on Windows. https://www.duplicati.com/ https://www.duplicati.com/
- mmastrac 7y agoMy duplicati experience has not been great - I found that it couldn't recover well from backup corruption (multi-day recovery times for ~1TB of data when everything was local and USB-connected!). The web interface is what keeps me on it.
- tryptophan 7y agoSame here. Duplicati works great until you actually have to restore. It took me 3 hours to restore a 12GB backup. After that I quit using it. Now I just made my home folder a shared folder on Syncthing, in send only mode, so it constantly backs up to my Nas. Much faster and the backup is directly readable.
- jszymborski 7y agoAgreed, backup is slow (although recent versions are leveraging multi-core CPUs, so that's given a modest speed-up). Syncthing in send-only is not a sufficient back-up strategy for 90% of use-cases, however. Without incremental back-ups, you're still susceptible to ransomware and data-loss should syncthing update after the event. I do, however, like to keep a "verbatim" copy in addition to my incrementals since they are less likely to have a "restoration" problem.
- sogubsys 7y agoLearned about this program when pwning Registry on HackTheBox
- Erlich_Bachman 7y agoFor a very good introductory read on what all this deduplicating backup software is all about and how it is a "new school" to the "old school" of incremental backup softwares (where duplicity (which you might want to either also check out, or migrate from) is the king) and its full/incremental model, and about the advantages and disadvantages of the two, I recommend this surprisingly well written guide from Backblaze: https://www.backblaze.com/blog/backing-linux-backblaze-b2-duplicity-restic/ https://www.backblaze.com/blog/backing-linux-backblaze-b2-du...
- acje 7y agoBut does it come with sharepoint integration? Because if you claim to do that right, then this is surely snake oil.
- _ink_ 7y agoI really would like to use Restic, but it seems that their repos randomly break and it seems that it is still not resolved. https://forum.restic.net/t/recovery-options-for-damaged-repositories/1571 https://forum.restic.net/t/recovery-options-for-damaged-repo... Any news on this?
- shekhar101 7y agoTangentially related but I’ll lay out a problem I have for a small office I run. I want to backup and sync my files cheaply somewhere, say GDrive. I want a Uzi that gives me snapshot/versions of files it backed up. Is there a good open source solution for this? Restic and friends are a little too technical for my non-technical office. Thanks
- hrdwdmrbl 7y agoI would argue that the 2nd most important part of a backup is the restoring. Many products don't focus on that enough because it's not the part that most customers will experience until there is an emergency.
- h1d 7y agoThere's also Duplicacy which I feel is a bit more mature than restic (I felt it more performing and it also has a web interface but the lack of mount capability that restic has is what I miss and retention policy is quite easier to specify for restic too), though it's a paid software for non personal usage. But paid is good that the author has less reason to move away from it. (As mentioned elsewhere, restic hasn't been updated in a while but they recently changed their pricing to be much more agressive which I hated. I wish they give it a softer pricing model than cost per machine.) https://duplicacy.com/ https://duplicacy.com/ Their comparison of cloud storage providers gave me a good insight on what to choose in terms of performance. https://github.com/gilbertchen/cloud-storage-comparison/ https://github.com/gilbertchen/cloud-storage-comparison/ I use both restic and Duplicacy, so my backups are done by multiple implementations toward multiple destinations not to get bitten by one of their bugs to avoid the saying "backups aren't working when you need it the most".
- denkmoon 7y agoI like HashBackup. It's not open source unfortunately, but I like the incremental encrypted backups while also being able to pull out a single file easily.
- dewey 7y agoI recently set up Restic to backup my server to my local Synology and it was actually surprisingly easy. Before that I was using Duplicity which broke after a while and it all felt a bit more tricky to get going. Documented it a bit here, mostly so I can go back and copy / paste commands if I have to: https://blog.notmyhostna.me/backup-cloud-server-to-synology-nas/ https://blog.notmyhostna.me/backup-cloud-server-to-synology-...
- lasftew 7y agoI moved from borg to restic because of the native support for B2 buckets (a lot cheaper than dedicated rsync/ssh type cloud file systems). My use case is backing up daily snapshots of my Onedrive (synced locally via rclone). As my important files are mostly immutable (photo library, pdfs), I don’t have to prune snapshots. Pruning is really slow compared to borg. I never backup workstations or home directories, as I can always generate them again using NixOS and home manager. For source code, nothing beats git repos.
- pabs3 7y agoThere are a few downsides to using restic: No support for compression yet No support for deleting data from snapshots No support for continous backups (restic walks the directory tree for each backup). No support for resilience from disk errors using par2 or similar Backing up millions of small/empty files uses a lot of memory
- cies 7y agoSome more: Not too many storage backends. No GUI tool. Hard to implement lifecycle policies like "keep N last hourly backups and M last weekly backup"; and not optimized for this usecase (needs quite a bit of data transfer to accomplish due to the zero-knowledge server-side encryption).
- transfire 7y agoMetadata?