17 ms·
Rsnapshot: filesystem snapshot utility based on rsync
- da_n 10y agoI spent a good few weeks recently researching and testing different back up systems that I could install to a Raspberry Pi with a big USB hard drive attached, to act as an extra offsite backup for some remote servers. rsnapshot was pretty much the only system I found that ticked every requirement, it has proven itself very robust in the month or so I have had it doing nightly snapshots of around 5 servers. Previously I had used a Ruby back up library which was actually really good but needed to be run from the server rather than the back up endpoint.
- stevekemp 10y agoI wonder what you included in your requirements? For me things I value are encryption on the backup-host, and de-duplication. That lead me to obnam, and attic. rsnapshot is a nice tool, because it only needs to transmit things that have changed, but without encryption it isn't something I'd personally want to run again.
- prohor 10y agoThe de-duplication in this case is achieved by hardlinks of unchanged files between snapshots. This means, deduplication is only on file level, so even an append makes a new copy. I've been looking for an rsync-based tool with encryption, but really couldn't find any. There is that uses rsync way of finding differences & keeps change log, but this is in fact normal full-snapshot & incremental backups: http://duplicity.nongnu.org/ http://duplicity.nongnu.org/
- extra88 10y agoMy understanding of what people looking for de-dup want is for their backup store to contain zero files with identical contents. With rsnapshot, if you rename a file, rename a parent directory of it, or move the file to another directory, you will get another copy of that file in your backup. I've used rsnapshot for about ten years, mainly to make remote backups of data on SMB/AFP file servers, and something I had to actively discourage was staff using the moving of files/folders between higher level folders as a method of project management. One group in particular does video production so a project between yesterday and today might have only a change of a few bytes but because the project's folder was moved to a different parent folder (e.g. Completed/), rsnapshot would make a new complete backup of that project which could be tens of gigs. Filesystem snapshots like ZFS's avoid this problem but I haven't worked with how those are copied remotely.
- prohor 10y agoYou might like BoxBackup, but unfortunately not very active project now: https://www.boxbackup.org/ https://www.boxbackup.org/
- kijin 10y agoThe backup tool itself doesn't need to know about encryption at all. I have full control over the backup host, so I would use either full-disk encryption (luks) or filesystem-level encryption (encfs/ecryptfs) on the backup host. The former works better if you want de-duplication. Regular ssh takes care of encryption in transit, so rsync just pipes unencrypted bits through this encrypted tunnel. You could probably add GPG in there somewhere if you wanted. Just a bunch of Unix tools doing Unixy things in the Unix way.
- da_n 10y agoIndeed as kijin states, the drive itself is full disk encrypted using LUKS, and SSH takes care of encrypting everything during transport. I did not require de-dupe but did only want to transmit only the diffs since last back up. I also wanted a something that would be of minimal resource usage on remote end (for rsnapshot was just running an rsync daemon if I recall correctly). In the case of one Ruby library I looked at, the entire system needed to be installed on each remote host to work correctly, which was unacceptable in my scenario. Primarily, it just feels like a solid tool and seems like it can be adapted to suit a lot of use cases (I have seen a config to allow backup of AWS RDS instances for instance).
- cyphar 10y agoThe issue with using a setup like this is that you run the risk of having inconsistent snapshots, where several related files were snapshotted at different points in time. I'm not sure how this lash-up works with something like a database, where you have to have a consistent copy of a single file as well. If you use Btrfs (or ZFS if you're on FreeBSD or illumos) then you can have proper filesystem snapshots with essentially no overhead. Since both filesystems are transactional, you're guaranteed to get a snapshot of a single transaction (resulting in no inconsistency). Unfortunately due to partial writes in database files, Btrfs has some performance issues when having copy-on-write enabled (which is required if you want snapshots) for a database file.
- fl0wenol 10y agoIf you release your snapshot used for mirroring via rsync right away as part of the cron job, and only take it and maintain it in CoW mode during off-peak hours, then it's usually a good compromise for getting something crash consistent.
- inopinatus 10y agoOr you can just put your database in backup mode. We do not need a perfectly consistent file system backup as the starting point. Any internal inconsistency in the backup will be corrected by log replay. - https://www.postgresql.org/docs/current/static/continuous-archiving.html https://www.postgresql.org/docs/current/static/continuous-ar...
- reidrac 10y agoYou're right, but you can use LVM to solve that problem: make an LVM snapshot, run your backup solution (eg, rsnapshot) on the LVM snapshot, remove the LVM snapshot. You still have the problem of caches and transient data, but all files will share the same point in time. In my experience LVM snapshot performance is good, you can even make it small and let it grow to track the changes while you backup is running. EDIT: actually rsnapshot has LVM snapshot support (since version 1.3.1 from Aug 31, 2008), so there you are.
- cyphar 10y ago
- prohor 10y agoI like also rdiff-backup which makes a mirror of a directory + reverse diffs, in case you need to move in time. http://www.nongnu.org/rdiff-backup/ http://www.nongnu.org/rdiff-backup/ Duplicity is also another tools that uses rsync algorithm to find and stores differences, but makes unfortunately incremental backups. On upside - it can store backup on anything that you can push files (FTP, S3, IMAP, ...) and it is the only rsync-related solution with encryption I could find http://duplicity.nongnu.org/ http://duplicity.nongnu.org/ There is also a very nice full solution - BoxBackup, which unfortunately looks not very active now. It is continuous backup, with block level de-duplication, client-side encryption. Unfortunately uses its own format, where I strongly prefer to have things more standard, so you can manually fix them if something goes wrong. https://www.boxbackup.org/ https://www.boxbackup.org/
- ar-jan 10y agoMaybe you'll be interested in https://github.com/restic/restic https://github.com/restic/restic. It does deduplication (with content defined chunking) and client-side encryption, and is actively developed. It does also use its own pack format, but that's fully documented. I'm currently using it for backing up about 1.5TB and it's working very well.
- Bromskloss 10y ago> I like also rdiff-backup which makes a mirror of a directory + reverse diffs, in case you need to move in time. http://www.nongnu.org/rdiff-backup/ http://www.nongnu.org/rdiff-backup/ My favourite backup tool! It's a tragedy that it seems to have gone stale.
- Zitrax 10y agoThere seem to be some activity now in the github repo and the home page has definitely been updated since last time i visited it.
- modeless 10y agoThere are many rsync-based backup systems like this that assume you back up many times and don't restore often, so backups are efficient (deltas, etc) but restoring is not (just download the whole thing). I'd like to use an rsync-like mechanism for software distribution, where the problem is reversed: the software is updated infrequently, but there are many clients that all need to download updates, so "restoring" should be efficient. Is there any open source solution for that? Preferably where the server can be dumb, like an S3 bucket.
- phireal 10y agoLike delta RPMs [0]? Software updates are distributed as diffs and applied to the existing package, so only the differences in the new packages need to be sent each time. This is computationally ineffficient (calculating and applying the delta RPM) but bandwidth efficient (smaller transfers). It depends on what you mean by efficient. [0] http://www.mpipks-dresden.mpg.de/~mueller/docs/suse10.2/html/opensuse-manual_en/manual/sec.rpm.delta.html http://www.mpipks-dresden.mpg.de/~mueller/docs/suse10.2/html... First reference I found on Google
- rakoo 10y agoYou might want to have a look at zsync (http://zsync.moria.org.uk/ http://zsync.moria.org.uk/), which is exactly a reversed rsync: the seed server calculates a signatures file over a file, the clients download that signatures file (through any means, but usually HTTP) and calculate a diff on their side; once they have instructions they can download only the parts of what they need through HTTP Ranges. Which means that any dumb HTTP storage that respects Ranges will work. The big advantage over rsync, on top of being available over HTTP, is that the server only calculates the signatures file once for the life of the content, instead of calculating it for each and every client. I know some distributions use that, but it's not very well known... yet ?
- modeless 10y agoCool! Zsync looks like exactly what I want, except that it doesn't sync directory trees, only individual files.
- beagle3 10y agoWhile rsnapshot and friends are useful as they provide a real file system (e.g., on a portable drive that can just be connected to another computer at any time, no app required), there are "bup", "borg backup" and (IIRC) "obnam", which all provide much faster snapshotting[0], much better space utilization (a change of 1 byte in a 1GB file takes 1GB for rsnapshot; it takes 4KB-128KB for the alternatives), compressed storage, and more. And if you want a "live" system, they provide a FUSE mountable system that makes every past revision look live. I haven't used obnam, but from the interwebs it seems like bup and borg are much much faster, with bup's forte being in the situation where many clients with similar files are backing up to a remote server over ssh (in this specific scenario, borg is slow and cumbersome), and in that it's just a git repository you can poke at. Otherwise, borg has encryption, deletion of old versions (still experimental in bup), and many other goodies. [0] not really an atomic snapshot, of course - they don't make any more or less guarantees than rsnapshot and friends.
- ComodoHacker 10y agoAdd zpaq to the family. It also supports deduplication, encryption and offline catalog. The latter means you don't need access to old backups to perform deduplication. And it's cross-platform.
- beagle3 10y agoI tried zpaq four years ago and it looked very well built, but exceptionally slow (as in, 5-10 times slower than bup at the time). Have things changed?
- ComodoHacker 10y agoI didn't compare with bup and can't say how much has it improved in four years. But yes, subjectively it's slow, mostly because of compression I guess. Not "exceptionally" slow though. With default settings it's a good tradeoff IMO.
- NoGravitas 10y agoI use obnam, and I've just read the manuals for bup and borg. It seems like obnam is more featureful and has better porcelain than either bup or borg, so that the script you end up writing to automate your backups is much simpler. But it is fairly slow. There are some tuning flags that will make it much faster than the default, but I would not at all be surprised if bup and borg are both much faster.
- amelius 10y agoNote that ext3 and ext4 only support 64k hardlinks, so if you already use them a lot, then snapshotting may make you run out of hardlinks even faster.
- pwg 10y agoYes, but that is not a global limit but a per-file limit. So you would need one single file to have 64k hard links to it before you'd run into this issue.
- discreditable 10y agoI used rsnapshot for a long time and I can't recommend it over attic (https://attic-backup.org/ https://attic-backup.org/) anymore. Rsnapshot works quite well but over time I started having problems. I was using it to snapshot a web server with many files uploaded by users. Over time there were so many millions of files that trying to work with the rsnapshot directories required a lot of patience. There was one time when I needed to copy the files to another storage medium, I let rsync run for DAYS and there was no end in sight. There were also two instances where I had some filesystem (ext4) corruption within my rsnapshot tree and in the end the only way I could get it happy again was to delete some suspect backups from my chain. If you're using it to backup anything more than /etc/, I'd strongly suggest using attic. It has these advantages over rsnapshot: * Block-level dedup instead of file-level. If 1b changes in your 1g file, that's another 1g needed in your backup. * Compression. * Encryption via key file, and/or passphrase. Disadvantages over rsnapshot: * You access your backups via a fuse mount. This isn't awful but I admit it isn't as nice as direct access. * If you are running attic from a local host to a remote one, the remote needs attic installed. Alternatively you can store your attic repo locally and mirror it remotely in some other way. * It can take a long time to work with large backup chains.
- uggedal 10y agoattic was removed from Debian after a data-loss bug went unfixed upstream for some time: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=802619 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=802619 It is fixed in the borg fork.
- sciurus 10y agoIn case you haven't seen it, some attic developers forked and started https://github.com/borgbackup/borg https://github.com/borgbackup/borg
- deleted 10y ago[deleted]
- rsync 10y ago"I used rsnapshot for a long time and I can't recommend it over attic (https://attic-backup.org/ https://attic-backup.org/) anymore." rsync.net here. We agree. Attic (well, borg, but whatever) is quickly becoming the de facto standard for nicely versioned, secure, portable remote backups to an SFTP/SSH capable host. We recommended duplicity for many, many years (and in fact, contributed to its development) but it no longer looks like the smart choice given the problems it has with periodic retransmission of the entire backups, etc. FWIW, we support attic and borg and just rolled out borg v1.x into production: http://www.rsync.net/products/attic.html http://www.rsync.net/products/attic.html
- oever 10y agoThere are few more similar project such as Restic (Go) and rdedup (Rust). https://github.com/restic/others https://github.com/restic/others https://github.com/dpc/rdedup https://github.com/dpc/rdedup
- utternerd 10y agoI've used http://backuppc.sourceforge.net/ http://backuppc.sourceforge.net/ for going on 16 years now, it's great and still maintained, albeit by one guy. It just works. Not sure how it stacks up to this.
- gamedna 10y agoI have used backuppc for years as well, and it uses a similar strategy of rsync/hardlinks to make point in time snapshots available to the users.
- aruggirello 10y agoI use Timeshift on my Ubuntu desktop which is basically the same, but wrapped up in a nice GUI.