3 ms·

I've been doing rsync-based backups of close to a thousand systems for ~20 years, most notably for a long time I backed up the python.org infrastructure, and I

I've been doing rsync-based backups of close to a thousand systems for ~20 years, most notably for a long time I backed up the python.org infrastructure, and I have quite a few thoughts on this. I also have a battle-tested rsync wrapper that I'll point to below. - Backups should be automatic, only requiring attention when it is needed. This script philosophy seems to be "Just do your best, mail a log file, and rely on the user to figure out if something didn't work". Even for home backups, this is just wrong. - As an example of the above: This script notes that it fails if a backup takes more than 24 hours. - The "look for other rsyncs running" part of the code is an odd way of approaching locking, but for a single personal "push" backup I guess it is ok. I've got an rsync wrapper that has been battle tested over a couple decades and hundreds of servers here: https://github.com/linsomniac/tummy-backup/blob/master/sbin/zfsharness https://github.com/linsomniac/tummy-backup/blob/master/sbin/... Features of it are: - As the filename implies, the goal is to rsync to a zfs destination, and it will take a zfs snapshot as part of this. It is easy to customize to another backup destination, I've had people report they have customized it for their own laptop backups, for example to an rsync.net destination. - It goes out of its way to detect when rsync has failed and log that. - It does do "inplace" rsyncs, which dramatically save space if you have large files that get appended to (logfiles, ZODB databases). - This is part of a larger system that manages the rsyncs of multiple systems, both local and remote. Things like alerting are done if a backup of a system has failed consistently for many days. - In the case that there are no failures, there is no e-mail sent, meaning the user only gets actionable e-mails. The hardlink trick only works for fairly small data sets. Issues include: Managing hard links takes a lot of overhead, especially on spinning discs. Large files being appended to use a ton of space (a 4GB file with 1K appended every day uses 128GB to store 14 dailies, 6 weeklies, and 12 monthlies). ZFS is a pretty good destination for rsync, as similar snapshots will use 4GB to store.

sitkack 5y ago
This SO post goes over the semantics of `--inplace` and `--no-whole-file` --inplace update destination files in-place --whole-file, -W copy files whole (w/o delta-xfer algorithm) It is very important to debug rsync scripts with a test corpus and not on live data, preferably on test vm or container. https://superuser.com/questions/576035/does-rsync-inplace-write-to-the-entire-file-or-just-to-the-parts-that-need-to https://superuser.com/questions/576035/does-rsync-inplace-wr... Also, when transferring from different systems, make sure that both rsyncs are of a high enough version, again preferably the same. MacOS ships with a really old version of rsync that doesn't support extended attributes at `/usr/bin/rsync` rsync version 2.6.9 protocol version 29 This is kinda a large footgun.
- sgt 5y ago
  HN's new pet word; footgun
  - sitkack 5y ago
    https://hn.algolia.com/?dateRange=pastYear&page=0&prefix=true&query=footgun&sort=byDate&type=all https://hn.algolia.com/?dateRange=pastYear&page=0&prefix=tru...
mekster 5y ago
> - In the case that there are no failures, there is no e-mail sent, meaning the user only gets actionable e-mails. I've always thought this isn't the right approach. How do you know if the email server is borked or you commented out the script in cron to debug it and forgot to put it back in? Either there's a weekly status report to tell things have been green or you could place cron checks like healthchecks.io (You can self host it.) Also it's much better to use 'zfs send' with large volumes to backup if both ends have zfs as zfs knows which files have changed and it doesn't have to scan for what has changed on each go as any other tools do like rsync. https://arstechnica.com/information-technology/2015/12/rsync-net-zfs-replication-to-the-cloud-is-finally-here-and-its-fast/ https://arstechnica.com/information-technology/2015/12/rsync...
- linsomniac 5y ago
  I get your point, and different people have different things they are comfortable with. Some of that also depends on your environment, in my case I've had this script running nightly backups across hundreds of machines over a dozen years. Sending hundreds of e-mails a day into my inbox isn't going to be workable. But if you have one or two machines, maybe it is. I still don't think so, but again different people. Things I have done to ensure reliability (again, this core script has been running for a dozen or more years): - Nagios monitoring of backups: An active check from a monitoring server that alerts if no recent successful backups. - "paper path" monitoring of e-mail: Send an e-mail to an external mailbox and have an active check in Nagios that reports if it has not seen an e-mail recently. - With hundreds of machines, we were in the management interface enough (not daily, but at least monthly) that we would tend to notice before TOO long if something was out of whack. - Regular backup audits: We would perform quarterly backup audits of the important machines, we had a whole workflow for those, which would also give us confidence that the backups were running as expected and that if something got out of whack it didn't go too long. Many of these depend on your definition of "too long". As far as "zfs send", I totally agree. However, even today I have very few machines other than my backup machines that are running ZFS, so that's not really an option for these backups.

sitkack 5y ago

This SO post goes over the semantics of `--inplace` and `--no-whole-file` --inplace update destination files in-place --whole-file, -W copy files whole (w/o delta-xfer algorithm) It is very important to debug rsync scripts with a test corpus and not on live data, preferably on test vm or container. https://superuser.com/questions/576035/does-rsync-inplace-write-to-the-entire-file-or-just-to-the-parts-that-need-to https://superuser.com/questions/576035/does-rsync-inplace-wr... Also, when transferring from different systems, make sure that both rsyncs are of a high enough version, again preferably the same. MacOS ships with a really old version of rsync that doesn't support extended attributes at `/usr/bin/rsync` rsync version 2.6.9 protocol version 29 This is kinda a large footgun.

sgt 5y ago
HN's new pet word; footgun
- sitkack 5y ago
  https://hn.algolia.com/?dateRange=pastYear&page=0&prefix=true&query=footgun&sort=byDate&type=all https://hn.algolia.com/?dateRange=pastYear&page=0&prefix=tru...

mekster 5y ago

> - In the case that there are no failures, there is no e-mail sent, meaning the user only gets actionable e-mails. I've always thought this isn't the right approach. How do you know if the email server is borked or you commented out the script in cron to debug it and forgot to put it back in? Either there's a weekly status report to tell things have been green or you could place cron checks like healthchecks.io (You can self host it.) Also it's much better to use 'zfs send' with large volumes to backup if both ends have zfs as zfs knows which files have changed and it doesn't have to scan for what has changed on each go as any other tools do like rsync. https://arstechnica.com/information-technology/2015/12/rsync-net-zfs-replication-to-the-cloud-is-finally-here-and-its-fast/ https://arstechnica.com/information-technology/2015/12/rsync...

linsomniac 5y ago
I get your point, and different people have different things they are comfortable with. Some of that also depends on your environment, in my case I've had this script running nightly backups across hundreds of machines over a dozen years. Sending hundreds of e-mails a day into my inbox isn't going to be workable. But if you have one or two machines, maybe it is. I still don't think so, but again different people. Things I have done to ensure reliability (again, this core script has been running for a dozen or more years): - Nagios monitoring of backups: An active check from a monitoring server that alerts if no recent successful backups. - "paper path" monitoring of e-mail: Send an e-mail to an external mailbox and have an active check in Nagios that reports if it has not seen an e-mail recently. - With hundreds of machines, we were in the management interface enough (not daily, but at least monthly) that we would tend to notice before TOO long if something was out of whack. - Regular backup audits: We would perform quarterly backup audits of the important machines, we had a whole workflow for those, which would also give us confidence that the backups were running as expected and that if something got out of whack it didn't go too long. Many of these depend on your definition of "too long". As far as "zfs send", I totally agree. However, even today I have very few machines other than my backup machines that are running ZFS, so that's not really an option for these backups.