32 ms·

An almost perfect rsync over SSH backup script

199 points by bpasero 5y ago

esaym 5y ago
Guess they never heard of rsnapshot?? [0] [0]: https://github.com/rsnapshot/rsnapshot https://github.com/rsnapshot/rsnapshot
idealmedtech 5y ago
I think there's a bug in this; the proc_on calculation doesn't account for multiple lines, it just calculates if the output of grep is greater than 1? grep -c may not be available on all systems, so a more portable way to do this is piping to 'wc -l'
rurban 5y ago
Perfect? Ha. Wrong ps grep usage. MacOS specific. No shell quoting. At least he doesn't advocate rm -rf / by an incorrect usage of --delete.
- zeeZ 5y ago
  A shell on a Mac is the "we have x at home" meme to me. Very frustrating.
- BeefWellington 5y ago
  > Perfect? Ha. The cynic in me believes this was intentional to drive engagement and discussion. It's that old adage of "if you want to learn something just say something wrong about it on the Internet" at work.
  - lamontcg 5y ago
    There's a million blog posts out there with "the perfect <whatever> setup" which is this circle jerking way of saying that its the author's preferred configuration. I don't entirely know what the motivation is, but as a grumpy old GenXer, I wish it would stop.
- uhradone 5y ago
  You missed the word "almost" just before the word "perfect": meaning there is space for improvementes and any coding has, if you dig deeper, a never ending context. So, you see something "not so or not at all perfect" just name it to improve it. THX
throwaway81523 5y ago
Too complicated to even try to read. Rsync is great but I've switched to Borg for backups. Borg isn't perfect but it is a higher level approach to backups, as it were. Hetzner recently dropped the price of their Storage Box backup product to about 2 euro per TB per month, and Borg works nicely with it. Borg encrypts all the backup contents and conceals the metadata on the backup server, and yet you can (with the encryption passphrase) mount the backup archive as a read-only file system through FUSE and access it through normal file navigation. It is impressive.
- slig 5y ago
  Is it possible to use it as a read/write mounted drive on macOS?
  - sandreas 5y ago
    I think restic[1] does provide this. [1] https://restic.net/ https://restic.net/
    - slig 5y ago
      Thank you!
- lvass 5y ago
  >Storage Box backup product The script here really only makes sense for servers you physically own. You wouldn't accept SSH access from a key located on a plain VPS, right? Also this script doesn't seem to encrypt the data at all!! Very dangerous on a VPS.
  - throwaway81523 5y ago
    > You wouldn't accept SSH access from a key located on a plain VPS, right? With Borg, I use ssh -A from my laptop to start backups going. I can think of some other schemes like using multiple user accounts on a single Storage Box (Hetzner supports that and I believe offers an API) so that different VPS can't clobber each other's backups, that old backups become read-only, etc. It might be interesting to add some finer access controls on the server side. Borg supports an append-only mode but right now, that's only a config option rather than a security setting, I believe. I've only recently started using Borg so I'm not really familiar with its intricacies yet. There are some things I would change but it is mostly well thought out, imho.
nixcraft 5y ago
rsnapshot[1] is what I used on FreeBSD with a snapshot. It is like a well-tested version of the author's rsync and ssh blog post. I have a blog post here[2] describing my setup. It saved my bacon multiple times. Also, rsnapshot works in pull mode, so no client is needed on my Linux/macOS desktop or server except ssh and rsync. [1] https://github.com/rsnapshot/rsnapshot https://github.com/rsnapshot/rsnapshot [2] https://www.cyberciti.biz/faq/howto-install-rsnapshot-filesystem-snapshot-backup-utility-in-freebsd/ https://www.cyberciti.biz/faq/howto-install-rsnapshot-filesy...
- ansible 5y ago
  Yep. We use rsnapshot with btrfs snapshots on Debian / Ubuntu. You can set up multiple generations easily. Efficient storage using hard links. Very reliable.
  - matja 5y ago
    AFAIK the crucial part of this, is rsync's `--link-dest=DIR` parameter, which hard-links an unmodified file to the respective place in "DIR", rather than making a copy.
    - ansible 5y ago
      The docs for my version of rsnapshot say not to use '--link-dest' if GNU cp is available, because the script will use 'cp -l' instead when copying "daily-1" to "daily-0" (for example). This is before it will overwrite "daily-0" with new files via rsync.
  - xrisk 5y ago
    Could you elaborate on where the btrfs snapshots come into play? I use rsnapshot on a plain ext4 fs and it’s pretty slow (lots of small files).
    - ansible 5y ago
      The snapshots don't help with the speed. Instead, they make sure the files aren't changing / being deleted between when the rsync starts and finishes.
bluedino 5y ago
What seems like a long time ago, I was working at a place that used Linux, but not really any Linux experts. They had backup scripts that assumed (and did not check) that you would connect an external drive. If you didn't, the script would gladly copy files to /mnt/usbdrive, and if the server was full enough, backing up itself to itself would fill the drive and bad things would then happen. There was no email notification if the backup completed or failed, or even started to run. There was no notification of how long it took, how much data was backed up, etc. There were plenty of incidents of data loss.
DarylZero 5y ago
I love the irony of a document about a "perfect" script with a grammatical error in the very first word of the title.
koolba 5y ago
The first two lines of the script are already wrong; #!/bin/bash Should be: #!/usr/bin/env bash set -euo pipefail That’s table stakes for any bash script. With the first piece, exit on error, being critically important.
- switch007 5y ago
  Not using “env” isn’t “wrong”. It depends how many platforms they want to supply Bet yes pipefail and nounset should definitely be set.
  - switch007 5y ago
    Support*
- c0l0 5y ago
  I wish this misguided notion regarding "inofficial strict mode" would go away, as `set -e` is suboptimal (and can be very tedious to deal with on top of that): https://mywiki.wooledge.org/BashFAQ/105 https://mywiki.wooledge.org/BashFAQ/105 It makes sense to opt into errexit for select blocks/sections in scripts and under certain circumstances, but having it default-on is a recipe for quite a bit of head-scratching in the future.
  - koolba 5y ago
    Using -e isn't an excuse to not understand how it works and when it doesn't. It's so that the default in most situation is to exit on failure as that's likely what you want to do. That leads to more terse scripts which hopefully are easier to write correctly and understand. I'd argue that opting in to exit-on-failure for stand alone commands is the right default.
    - eurasiantiger 5y ago
      No, absolutely wrong. The ”right default” is to catch return codes and provide actionable, clear and consistent error messages through an error-exit function.
BeefWellington 5y ago
I wish more of these backup solutions (even simple scripts like this) would include zero trust features. I'd like to have a backup system that I didn't just kludge together myself over a weekend that accomplishes both delta change detection and rotation as well as allows me to keep my data encrypted with a key only I control.
kzrdude 5y ago
I use duplicity. Far from perfect, but I hope it's better than rsync and similar.
hkt 5y ago
Honestly, I just use duplicity. It is easy to test backups and has a nice GUI if you're that way inclined. I'm too old for bash scripts when the stakes are a lifetime of memories or critical data at work.
darrenf 5y ago
Can't help but throw a couple cents/pennies around whenever an "almost perfect" or "advanced" script makes my knee jerk so wildly. So here goes. For this part: pro_on=$(ps aux | grep -c rsync) # if someone is using rsync # grep is also producing one entry so -gt 1 `grep` can be excluded from the output by putting square brackets around one character of the search pattern, like so: pro_on=$(ps aux | grep -c rsyn[c]) Also, this: minutes=$(($minutes - 1)) Can be more easily written as: ((minutes--))
- joosters 5y ago
  > `grep` can be excluded from the output by putting square brackets around one character of the search pattern That's a great trick, thanks! I'm so used to adding '| grep -v grep' at the end of a 'ps | grep' command, your way is much nicer.
  - Symbiote 5y ago
    On Linux, pgrep is probably what you need most of the time, or pkill.
    - varenc 5y ago
      I like the `psgrep` tool which behaves much more like greping ps that pgrep does. And `psgrep` is actually just a bash script itself.
- uhradone 5y ago
  THX the "almost" is getting closer to "almost"!
thg 5y ago
My go-to backup solution, that can also manage rsync over ssh (among plenty other things), is synbak[0]. Short wrapper script to automatically mount the backup medium and it's really simple to (automatically) run a backup job. I use encrypted (LUKS) USB drives for that at home. Highly recommend it. For encrypted backups to e.g. a NAS, Duplicity[1] is my go-to choice (full backup every month or so, with incremental backup every day inbetween). [0]: https://github.com/ugoviti/synbak https://github.com/ugoviti/synbak [1]: https://duplicity.gitlab.io/duplicity-web/ https://duplicity.gitlab.io/duplicity-web/
branon 5y ago
Haven't read the page yet but the header image struck me as interesting. "Molex to SATA, lose all your data" was drilled into my head at an early job and appears to be what's depicted there. If your disks are set up in a similar way, you might have a very immediate need for a good backup script :)
- tjoff 5y ago
  That's a weird mantra, molex to sata is perfectly fine.
  - scottlamb 5y ago
    I'd never heard of it either but found this interesting youtube video: <https://www.youtube.com/watch?v=fAyy_WOSdVc https://www.youtube.com/watch?v=fAyy_WOSdVc>. tl;dw: many of these adapters are so poorly made they might catch on fire. They put the (insulated) wires in and then pour in something kind of like hot-melt glue around them (to hold them in place, and/or for extra insulation, not sure). Sometimes the wires are well separated within that medium, but sometimes they're touching, and possibly the wire's own insulation is melted by the glue. There also seems to be some contaminants in the glue, and possibly corrosion from the way it was soldered. At the end, he shows a better kind where crimped wires are put into hard plastic channels, which reliably keeps them in place without the same risk of compromising their insulation.
- kobalsky 5y ago
  could you explain why it's a bad idea to use molex to sata? did your guys use shitty injection molded connectors that melted or is there another reason?
  - branon 5y ago
    I didn't really know why until I read scottlamb's comment, and yours. I think this is the only plausible explanation - someone had a bad experience with a really crappy connector and it stuck.
- uhradone 5y ago
adrian_b 5y ago
Beware that this script only uses rsync with the "--archive" flag. This may be enough for some users, but "--archive" does not copy all file metadata, so it may cause surprises. rsync must be invoked with "--archive --xattrs --acls" to guarantee complete file copies. Unfortunately all command-line utilities for file copying that are available for UNIX-like operating systems use default options that do not copy most file metadata and they usually need many non-default flags to make complete file copies. Nevertheless, rsync is the best of all alternatives, because when invoked correctly it accurately copies all file metadata even between different file systems and different operating systems, in cases when other copying utilities may lose some file metadata without giving any errors or warnings for the users.
- chronomex 5y ago
  Rsync also doesn't track hardlinks by default! I use "-avPzHAXS" for my backups.
  - Jenda_ 5y ago
    Also have a look at --numeric-ids, otherwise it translates UIDs back and forth according to real user names - which will end up in a terrible mess, especially when you are restoring the backup from a live "CD" (which has different UID ←→ username mapping than the target system).
    - amelius 5y ago
      I use getfacl to save permissions correctly, in case the backup server uses different user/group mappings. And setfacl after restoring. (I wish this was handled by rsync.)
  - deleted 5y ago
    [deleted]
- throwawayboise 5y ago
  This also presumes that the target's rsync implementation and filesystem supports the same metadata.
sirius87 5y ago
For rsync tasks that require scheduling and need to ensure only a single instance executes, I personally prefer creating a systemd service (type=oneshot) and have that run under a systemd timer (set OnBootSec, OnUnitActiveSec) https://wiki.archlinux.org/title/Systemd/Timers https://wiki.archlinux.org/title/Systemd/Timers EDIT: I'd love to hear if anyone knows of any downsides, edge cases for running rsync under systemd timers.
lizknope 5y ago
I didn't read the whole article but the reason I like rsync is that I get files on a normal filesystem. I don't need any special backup software to read or access the files.
- omniamutantur 5y ago
  I totally agree. Do not underestimate the obstacle a family member will face when he needs to open a borg or restic repository to access a backup of your family photos. I use restic for most of my backups but hand-written rsync on an external drive with ntfs for the stuff that would be important to my family.
zoolook 5y ago
[dead]
linsomniac 5y ago
I've been doing rsync-based backups of close to a thousand systems for ~20 years, most notably for a long time I backed up the python.org infrastructure, and I have quite a few thoughts on this. I also have a battle-tested rsync wrapper that I'll point to below. - Backups should be automatic, only requiring attention when it is needed. This script philosophy seems to be "Just do your best, mail a log file, and rely on the user to figure out if something didn't work". Even for home backups, this is just wrong. - As an example of the above: This script notes that it fails if a backup takes more than 24 hours. - The "look for other rsyncs running" part of the code is an odd way of approaching locking, but for a single personal "push" backup I guess it is ok. I've got an rsync wrapper that has been battle tested over a couple decades and hundreds of servers here: https://github.com/linsomniac/tummy-backup/blob/master/sbin/zfsharness https://github.com/linsomniac/tummy-backup/blob/master/sbin/... Features of it are: - As the filename implies, the goal is to rsync to a zfs destination, and it will take a zfs snapshot as part of this. It is easy to customize to another backup destination, I've had people report they have customized it for their own laptop backups, for example to an rsync.net destination. - It goes out of its way to detect when rsync has failed and log that. - It does do "inplace" rsyncs, which dramatically save space if you have large files that get appended to (logfiles, ZODB databases). - This is part of a larger system that manages the rsyncs of multiple systems, both local and remote. Things like alerting are done if a backup of a system has failed consistently for many days. - In the case that there are no failures, there is no e-mail sent, meaning the user only gets actionable e-mails. The hardlink trick only works for fairly small data sets. Issues include: Managing hard links takes a lot of overhead, especially on spinning discs. Large files being appended to use a ton of space (a 4GB file with 1K appended every day uses 128GB to store 14 dailies, 6 weeklies, and 12 monthlies). ZFS is a pretty good destination for rsync, as similar snapshots will use 4GB to store.
- sitkack 5y ago
  This SO post goes over the semantics of `--inplace` and `--no-whole-file` --inplace update destination files in-place --whole-file, -W copy files whole (w/o delta-xfer algorithm) It is very important to debug rsync scripts with a test corpus and not on live data, preferably on test vm or container. https://superuser.com/questions/576035/does-rsync-inplace-write-to-the-entire-file-or-just-to-the-parts-that-need-to https://superuser.com/questions/576035/does-rsync-inplace-wr... Also, when transferring from different systems, make sure that both rsyncs are of a high enough version, again preferably the same. MacOS ships with a really old version of rsync that doesn't support extended attributes at `/usr/bin/rsync` rsync version 2.6.9 protocol version 29 This is kinda a large footgun.
deleted 5y ago
[deleted]
justin_oaks 5y ago
I use Restic (https://restic.net/ https://restic.net/) for backups. Encrypted backup, incremental, deduplicating, and supports many storage backends. Most recent discussion on Hacker News: https://news.ycombinator.com/item?id=29209455 https://news.ycombinator.com/item?id=29209455
- beepbooptheory 5y ago
  I like restic, but doing rsync scripts like this always win out for me because I find myself wanting to stage the backups on the destination systems, and not the source/prod ones.
  - sneak 5y ago
    I really, really wanted to love restic, but it makes far, far too many tiny size files for my main remote backup use-case: backing up dozens of TB offsite to Glacier Deep Archive. Eagerly awaiting a configurable chunk size, if they decide to do that.
    - shaicoleman 5y ago
      Perhaps Kopia will work for your use case: https://kopia.io/ https://kopia.io/ https://kopia.io/docs/advanced/amazon-s3/ https://kopia.io/docs/advanced/amazon-s3/
      - remram 5y ago
        Kopia looks great! I am currently using Duplicati for my personal machines, but Mono and its dependencies have been less than reliable. borg/restic/... look good but seem like only part of a solution, having to write my own scripts and crontabs with monitoring etc on top seems counter-productive. edit: No web interface?
- mekster 5y ago
  Sorry to say none of these modern backup tools work reliably when backup volumes get large except borg. https://news.ycombinator.com/item?id=29210222 https://news.ycombinator.com/item?id=29210222 Using zfs snapshot is the best way to go but having a cloud destination wouldn't be easy for these tools unfortunately but there are services that accept borg and zfs send.
0xbadcafebee 5y ago
> # avoidng collisions with other rsync processes Use https://github.com/instacart/ohmycron https://github.com/instacart/ohmycron > MONTHROTATE=monthrotate # use DD instead of YYMMDD Use https://rotate-backups.readthedocs.io/en/latest/readme.html https://rotate-backups.readthedocs.io/en/latest/readme.html > $RSYNC -avR "$SOURCE" "${RSYNCCONF[@]}" Create a one-command script with the hardcoded rsync command you want to use and replace the directory to sync as a command-line argument, e.g. #!/bin/sh rsync -avR \ --delete \ --exclude=/Volumes/Raid/.DocumentRevisions-V100 \ --exclude=/Volumes/Raid/.TemporaryItems --exclude=/Volumes/Raid/.Trashes \ --exclude=/Volumes/Raid/.apdisc \ "$@" > $DATE >> $LOG Use logrotate > mail a report Use Cron's built-in mail feature
aborsy 5y ago
Restic is currently the best solution in my view. The backup script is much simpler, the repository is properly encrypted, takes snapshots, dedup, mounts the remote, integrates with backends, has clean output, and various useful features for working with repositories. Rsync over SSH is not even encrypted at rest.
SmellTheGlove 5y ago
You can already solve this quite trivially with Dropbox.
camnora 5y ago
Does anyone have experience with Duplicati [1] or recommend it? I am tentatively looking to make the switch to something a little more sophisticated than manually tar-balling + rsyncing backups. [1] https://www.duplicati.com/ https://www.duplicati.com/
- aborsy 5y ago
  I haven’t used it, but the comments on Reddit indicate there are issues. You may easily find them.
- wjnc 5y ago
  I’ve that in use on my personal pc (Ubuntu) for my personal files. Bringing in the pictures via Dropbox. It works flawlessly for about 1.5 yr and 150GB to Backblaze. What put me off initially is that the most important part of backups: testing the restore process was / is less documented. Got it tested by just using common sense and a fresh temporary install. (Any pointers to how to do automatic integrity tests greatly appreciated.)
- mekster 5y ago
  Do yourself a favor and use borg. https://news.ycombinator.com/item?id=29210222 https://news.ycombinator.com/item?id=29210222 If you have extra space, you could rclone the result of locally run borg to a cloud storage or find a service that accepts borg directly.
Sohcahtoa82 5y ago
About 10 years ago, I was earning my CS degree, and my Intro to Unix teacher was showing us how to use rsync. He somehow made an error that synced an empty directory to his home directory, deleting everything from his home directory. The lesson I learned in that class was to not use rsync, sadly.
- quesera 5y ago
  > The lesson I learned in that class was to not use rsync, sadly. This is a poor lesson! If your TA tripped on the stairs and broke their nose, would you swear off stairs forever? rsync is an essential tool for anyone who works with files in the UNIX world. There are other options, sometimes, but rsync is almost-always present, and does almost-everything you usually want. Any tool can be used incorrectly, but rsync is easy and follows the same pattern as most other UNIX file copy tools. If you can use cp, you can use rsync safely.
gunapologist99 5y ago
It's nice that the script works both ways, but keep in mind that it's always best to pull backups from a remote system, so that an attacker in the production system can't log into the backup system and delete the backups. If your prod system(s) is/are logging into the backup system, then you have a big problem because if any of them are compromised, they can wipe out / corrupt / etc all of their own backups, and possibly the other server's backups as well (if the same user account is used for all of them.) This problem goes away if your (isolated, hardened) backup server logs into the prod systems. Of course, the inverse problem is that if an attacker manages to get into your backup system, then they also have (at least read) access to all of your prod systems!
gunapologist99 5y ago
The script could use --rsync-path='sudo rsync', so that your rsync backup script can access root-owned files on the remote system. It looks like you can add that to the RSYNCCONF part of "subpart of part 2".
- uhradone 5y ago
  For security reasons the user "backup" on the linux acting backup system is not part of the sudo users. Having sudo rights will to my knowledge not help to excess root files on the remote system?
frays 5y ago
Very useful article and thread comments, thank you all.

199 points by bpasero 5y ago

Guess they never heard of rsnapshot?? [0] [0]: https://github.com/rsnapshot/rsnapshot https://github.com/rsnapshot/rsnapshot

idealmedtech 5y ago

I think there's a bug in this; the proc_on calculation doesn't account for multiple lines, it just calculates if the output of grep is greater than 1? grep -c may not be available on all systems, so a more portable way to do this is piping to 'wc -l'

rurban 5y ago

Perfect? Ha. Wrong ps grep usage. MacOS specific. No shell quoting. At least he doesn't advocate rm -rf / by an incorrect usage of --delete.

zeeZ 5y ago
A shell on a Mac is the "we have x at home" meme to me. Very frustrating.
BeefWellington 5y ago
> Perfect? Ha. The cynic in me believes this was intentional to drive engagement and discussion. It's that old adage of "if you want to learn something just say something wrong about it on the Internet" at work.
- lamontcg 5y ago
  There's a million blog posts out there with "the perfect <whatever> setup" which is this circle jerking way of saying that its the author's preferred configuration. I don't entirely know what the motivation is, but as a grumpy old GenXer, I wish it would stop.
uhradone 5y ago
You missed the word "almost" just before the word "perfect": meaning there is space for improvementes and any coding has, if you dig deeper, a never ending context. So, you see something "not so or not at all perfect" just name it to improve it. THX

throwaway81523 5y ago

Too complicated to even try to read. Rsync is great but I've switched to Borg for backups. Borg isn't perfect but it is a higher level approach to backups, as it were. Hetzner recently dropped the price of their Storage Box backup product to about 2 euro per TB per month, and Borg works nicely with it. Borg encrypts all the backup contents and conceals the metadata on the backup server, and yet you can (with the encryption passphrase) mount the backup archive as a read-only file system through FUSE and access it through normal file navigation. It is impressive.

slig 5y ago
Is it possible to use it as a read/write mounted drive on macOS?
- sandreas 5y ago
  I think restic[1] does provide this. [1] https://restic.net/ https://restic.net/
  - slig 5y ago
    Thank you!
lvass 5y ago
>Storage Box backup product The script here really only makes sense for servers you physically own. You wouldn't accept SSH access from a key located on a plain VPS, right? Also this script doesn't seem to encrypt the data at all!! Very dangerous on a VPS.
- throwaway81523 5y ago
  > You wouldn't accept SSH access from a key located on a plain VPS, right? With Borg, I use ssh -A from my laptop to start backups going. I can think of some other schemes like using multiple user accounts on a single Storage Box (Hetzner supports that and I believe offers an API) so that different VPS can't clobber each other's backups, that old backups become read-only, etc. It might be interesting to add some finer access controls on the server side. Borg supports an append-only mode but right now, that's only a config option rather than a security setting, I believe. I've only recently started using Borg so I'm not really familiar with its intricacies yet. There are some things I would change but it is mostly well thought out, imho.

nixcraft 5y ago

rsnapshot[1] is what I used on FreeBSD with a snapshot. It is like a well-tested version of the author's rsync and ssh blog post. I have a blog post here[2] describing my setup. It saved my bacon multiple times. Also, rsnapshot works in pull mode, so no client is needed on my Linux/macOS desktop or server except ssh and rsync. [1] https://github.com/rsnapshot/rsnapshot https://github.com/rsnapshot/rsnapshot [2] https://www.cyberciti.biz/faq/howto-install-rsnapshot-filesystem-snapshot-backup-utility-in-freebsd/ https://www.cyberciti.biz/faq/howto-install-rsnapshot-filesy...

ansible 5y ago
Yep. We use rsnapshot with btrfs snapshots on Debian / Ubuntu. You can set up multiple generations easily. Efficient storage using hard links. Very reliable.
- matja 5y ago
  AFAIK the crucial part of this, is rsync's `--link-dest=DIR` parameter, which hard-links an unmodified file to the respective place in "DIR", rather than making a copy.
  - ansible 5y ago
    The docs for my version of rsnapshot say not to use '--link-dest' if GNU cp is available, because the script will use 'cp -l' instead when copying "daily-1" to "daily-0" (for example). This is before it will overwrite "daily-0" with new files via rsync.
- xrisk 5y ago
  Could you elaborate on where the btrfs snapshots come into play? I use rsnapshot on a plain ext4 fs and it’s pretty slow (lots of small files).
  - ansible 5y ago
    The snapshots don't help with the speed. Instead, they make sure the files aren't changing / being deleted between when the rsync starts and finishes.

bluedino 5y ago

What seems like a long time ago, I was working at a place that used Linux, but not really any Linux experts. They had backup scripts that assumed (and did not check) that you would connect an external drive. If you didn't, the script would gladly copy files to /mnt/usbdrive, and if the server was full enough, backing up itself to itself would fill the drive and bad things would then happen. There was no email notification if the backup completed or failed, or even started to run. There was no notification of how long it took, how much data was backed up, etc. There were plenty of incidents of data loss.

DarylZero 5y ago

I love the irony of a document about a "perfect" script with a grammatical error in the very first word of the title.

koolba 5y ago

The first two lines of the script are already wrong; #!/bin/bash Should be: #!/usr/bin/env bash set -euo pipefail That’s table stakes for any bash script. With the first piece, exit on error, being critically important.

switch007 5y ago
Not using “env” isn’t “wrong”. It depends how many platforms they want to supply Bet yes pipefail and nounset should definitely be set.
- switch007 5y ago
  Support*
c0l0 5y ago
I wish this misguided notion regarding "inofficial strict mode" would go away, as `set -e` is suboptimal (and can be very tedious to deal with on top of that): https://mywiki.wooledge.org/BashFAQ/105 https://mywiki.wooledge.org/BashFAQ/105 It makes sense to opt into errexit for select blocks/sections in scripts and under certain circumstances, but having it default-on is a recipe for quite a bit of head-scratching in the future.
- koolba 5y ago
  Using -e isn't an excuse to not understand how it works and when it doesn't. It's so that the default in most situation is to exit on failure as that's likely what you want to do. That leads to more terse scripts which hopefully are easier to write correctly and understand. I'd argue that opting in to exit-on-failure for stand alone commands is the right default.
  - eurasiantiger 5y ago
    No, absolutely wrong. The ”right default” is to catch return codes and provide actionable, clear and consistent error messages through an error-exit function.

BeefWellington 5y ago

I wish more of these backup solutions (even simple scripts like this) would include zero trust features. I'd like to have a backup system that I didn't just kludge together myself over a weekend that accomplishes both delta change detection and rotation as well as allows me to keep my data encrypted with a key only I control.

kzrdude 5y ago

I use duplicity. Far from perfect, but I hope it's better than rsync and similar.

hkt 5y ago

Honestly, I just use duplicity. It is easy to test backups and has a nice GUI if you're that way inclined. I'm too old for bash scripts when the stakes are a lifetime of memories or critical data at work.

darrenf 5y ago

Can't help but throw a couple cents/pennies around whenever an "almost perfect" or "advanced" script makes my knee jerk so wildly. So here goes. For this part: pro_on=$(ps aux | grep -c rsync) # if someone is using rsync # grep is also producing one entry so -gt 1 `grep` can be excluded from the output by putting square brackets around one character of the search pattern, like so: pro_on=$(ps aux | grep -c rsyn[c]) Also, this: minutes=$(($minutes - 1)) Can be more easily written as: ((minutes--))

joosters 5y ago
> `grep` can be excluded from the output by putting square brackets around one character of the search pattern That's a great trick, thanks! I'm so used to adding '| grep -v grep' at the end of a 'ps | grep' command, your way is much nicer.
- Symbiote 5y ago
  On Linux, pgrep is probably what you need most of the time, or pkill.
  - varenc 5y ago
    I like the `psgrep` tool which behaves much more like greping ps that pgrep does. And `psgrep` is actually just a bash script itself.
uhradone 5y ago
THX the "almost" is getting closer to "almost"!

thg 5y ago

My go-to backup solution, that can also manage rsync over ssh (among plenty other things), is synbak[0]. Short wrapper script to automatically mount the backup medium and it's really simple to (automatically) run a backup job. I use encrypted (LUKS) USB drives for that at home. Highly recommend it. For encrypted backups to e.g. a NAS, Duplicity[1] is my go-to choice (full backup every month or so, with incremental backup every day inbetween). [0]: https://github.com/ugoviti/synbak https://github.com/ugoviti/synbak [1]: https://duplicity.gitlab.io/duplicity-web/ https://duplicity.gitlab.io/duplicity-web/

branon 5y ago

Haven't read the page yet but the header image struck me as interesting. "Molex to SATA, lose all your data" was drilled into my head at an early job and appears to be what's depicted there. If your disks are set up in a similar way, you might have a very immediate need for a good backup script :)

tjoff 5y ago
That's a weird mantra, molex to sata is perfectly fine.
- scottlamb 5y ago
  I'd never heard of it either but found this interesting youtube video: <https://www.youtube.com/watch?v=fAyy_WOSdVc https://www.youtube.com/watch?v=fAyy_WOSdVc>. tl;dw: many of these adapters are so poorly made they might catch on fire. They put the (insulated) wires in and then pour in something kind of like hot-melt glue around them (to hold them in place, and/or for extra insulation, not sure). Sometimes the wires are well separated within that medium, but sometimes they're touching, and possibly the wire's own insulation is melted by the glue. There also seems to be some contaminants in the glue, and possibly corrosion from the way it was soldered. At the end, he shows a better kind where crimped wires are put into hard plastic channels, which reliably keeps them in place without the same risk of compromising their insulation.
kobalsky 5y ago
could you explain why it's a bad idea to use molex to sata? did your guys use shitty injection molded connectors that melted or is there another reason?
- branon 5y ago
  I didn't really know why until I read scottlamb's comment, and yours. I think this is the only plausible explanation - someone had a bad experience with a really crappy connector and it stuck.
uhradone 5y ago

adrian_b 5y ago

Beware that this script only uses rsync with the "--archive" flag. This may be enough for some users, but "--archive" does not copy all file metadata, so it may cause surprises. rsync must be invoked with "--archive --xattrs --acls" to guarantee complete file copies. Unfortunately all command-line utilities for file copying that are available for UNIX-like operating systems use default options that do not copy most file metadata and they usually need many non-default flags to make complete file copies. Nevertheless, rsync is the best of all alternatives, because when invoked correctly it accurately copies all file metadata even between different file systems and different operating systems, in cases when other copying utilities may lose some file metadata without giving any errors or warnings for the users.

chronomex 5y ago
Rsync also doesn't track hardlinks by default! I use "-avPzHAXS" for my backups.
- Jenda_ 5y ago
  Also have a look at --numeric-ids, otherwise it translates UIDs back and forth according to real user names - which will end up in a terrible mess, especially when you are restoring the backup from a live "CD" (which has different UID ←→ username mapping than the target system).
  - amelius 5y ago
    I use getfacl to save permissions correctly, in case the backup server uses different user/group mappings. And setfacl after restoring. (I wish this was handled by rsync.)
- deleted 5y ago
  [deleted]
throwawayboise 5y ago
This also presumes that the target's rsync implementation and filesystem supports the same metadata.

sirius87 5y ago

For rsync tasks that require scheduling and need to ensure only a single instance executes, I personally prefer creating a systemd service (type=oneshot) and have that run under a systemd timer (set OnBootSec, OnUnitActiveSec) https://wiki.archlinux.org/title/Systemd/Timers https://wiki.archlinux.org/title/Systemd/Timers EDIT: I'd love to hear if anyone knows of any downsides, edge cases for running rsync under systemd timers.

lizknope 5y ago

I didn't read the whole article but the reason I like rsync is that I get files on a normal filesystem. I don't need any special backup software to read or access the files.

omniamutantur 5y ago
I totally agree. Do not underestimate the obstacle a family member will face when he needs to open a borg or restic repository to access a backup of your family photos. I use restic for most of my backups but hand-written rsync on an external drive with ntfs for the stuff that would be important to my family.

zoolook 5y ago

[dead]

linsomniac 5y ago

I've been doing rsync-based backups of close to a thousand systems for ~20 years, most notably for a long time I backed up the python.org infrastructure, and I have quite a few thoughts on this. I also have a battle-tested rsync wrapper that I'll point to below. - Backups should be automatic, only requiring attention when it is needed. This script philosophy seems to be "Just do your best, mail a log file, and rely on the user to figure out if something didn't work". Even for home backups, this is just wrong. - As an example of the above: This script notes that it fails if a backup takes more than 24 hours. - The "look for other rsyncs running" part of the code is an odd way of approaching locking, but for a single personal "push" backup I guess it is ok. I've got an rsync wrapper that has been battle tested over a couple decades and hundreds of servers here: https://github.com/linsomniac/tummy-backup/blob/master/sbin/zfsharness https://github.com/linsomniac/tummy-backup/blob/master/sbin/... Features of it are: - As the filename implies, the goal is to rsync to a zfs destination, and it will take a zfs snapshot as part of this. It is easy to customize to another backup destination, I've had people report they have customized it for their own laptop backups, for example to an rsync.net destination. - It goes out of its way to detect when rsync has failed and log that. - It does do "inplace" rsyncs, which dramatically save space if you have large files that get appended to (logfiles, ZODB databases). - This is part of a larger system that manages the rsyncs of multiple systems, both local and remote. Things like alerting are done if a backup of a system has failed consistently for many days. - In the case that there are no failures, there is no e-mail sent, meaning the user only gets actionable e-mails. The hardlink trick only works for fairly small data sets. Issues include: Managing hard links takes a lot of overhead, especially on spinning discs. Large files being appended to use a ton of space (a 4GB file with 1K appended every day uses 128GB to store 14 dailies, 6 weeklies, and 12 monthlies). ZFS is a pretty good destination for rsync, as similar snapshots will use 4GB to store.

sitkack 5y ago
This SO post goes over the semantics of `--inplace` and `--no-whole-file` --inplace update destination files in-place --whole-file, -W copy files whole (w/o delta-xfer algorithm) It is very important to debug rsync scripts with a test corpus and not on live data, preferably on test vm or container. https://superuser.com/questions/576035/does-rsync-inplace-write-to-the-entire-file-or-just-to-the-parts-that-need-to https://superuser.com/questions/576035/does-rsync-inplace-wr... Also, when transferring from different systems, make sure that both rsyncs are of a high enough version, again preferably the same. MacOS ships with a really old version of rsync that doesn't support extended attributes at `/usr/bin/rsync` rsync version 2.6.9 protocol version 29 This is kinda a large footgun.

deleted 5y ago

[deleted]

justin_oaks 5y ago

I use Restic (https://restic.net/ https://restic.net/) for backups. Encrypted backup, incremental, deduplicating, and supports many storage backends. Most recent discussion on Hacker News: https://news.ycombinator.com/item?id=29209455 https://news.ycombinator.com/item?id=29209455

beepbooptheory 5y ago
I like restic, but doing rsync scripts like this always win out for me because I find myself wanting to stage the backups on the destination systems, and not the source/prod ones.
- sneak 5y ago
  I really, really wanted to love restic, but it makes far, far too many tiny size files for my main remote backup use-case: backing up dozens of TB offsite to Glacier Deep Archive. Eagerly awaiting a configurable chunk size, if they decide to do that.
  - shaicoleman 5y ago
    Perhaps Kopia will work for your use case: https://kopia.io/ https://kopia.io/ https://kopia.io/docs/advanced/amazon-s3/ https://kopia.io/docs/advanced/amazon-s3/
    - remram 5y ago
      Kopia looks great! I am currently using Duplicati for my personal machines, but Mono and its dependencies have been less than reliable. borg/restic/... look good but seem like only part of a solution, having to write my own scripts and crontabs with monitoring etc on top seems counter-productive. edit: No web interface?
mekster 5y ago
Sorry to say none of these modern backup tools work reliably when backup volumes get large except borg. https://news.ycombinator.com/item?id=29210222 https://news.ycombinator.com/item?id=29210222 Using zfs snapshot is the best way to go but having a cloud destination wouldn't be easy for these tools unfortunately but there are services that accept borg and zfs send.

0xbadcafebee 5y ago

> # avoidng collisions with other rsync processes Use https://github.com/instacart/ohmycron https://github.com/instacart/ohmycron > MONTHROTATE=monthrotate # use DD instead of YYMMDD Use https://rotate-backups.readthedocs.io/en/latest/readme.html https://rotate-backups.readthedocs.io/en/latest/readme.html > $RSYNC -avR "$SOURCE" "${RSYNCCONF[@]}" Create a one-command script with the hardcoded rsync command you want to use and replace the directory to sync as a command-line argument, e.g. #!/bin/sh rsync -avR \ --delete \ --exclude=/Volumes/Raid/.DocumentRevisions-V100 \ --exclude=/Volumes/Raid/.TemporaryItems --exclude=/Volumes/Raid/.Trashes \ --exclude=/Volumes/Raid/.apdisc \ "$@" > $DATE >> $LOG Use logrotate > mail a report Use Cron's built-in mail feature

aborsy 5y ago

Restic is currently the best solution in my view. The backup script is much simpler, the repository is properly encrypted, takes snapshots, dedup, mounts the remote, integrates with backends, has clean output, and various useful features for working with repositories. Rsync over SSH is not even encrypted at rest.

SmellTheGlove 5y ago

You can already solve this quite trivially with Dropbox.

camnora 5y ago

Does anyone have experience with Duplicati [1] or recommend it? I am tentatively looking to make the switch to something a little more sophisticated than manually tar-balling + rsyncing backups. [1] https://www.duplicati.com/ https://www.duplicati.com/

aborsy 5y ago
I haven’t used it, but the comments on Reddit indicate there are issues. You may easily find them.
wjnc 5y ago
I’ve that in use on my personal pc (Ubuntu) for my personal files. Bringing in the pictures via Dropbox. It works flawlessly for about 1.5 yr and 150GB to Backblaze. What put me off initially is that the most important part of backups: testing the restore process was / is less documented. Got it tested by just using common sense and a fresh temporary install. (Any pointers to how to do automatic integrity tests greatly appreciated.)
mekster 5y ago
Do yourself a favor and use borg. https://news.ycombinator.com/item?id=29210222 https://news.ycombinator.com/item?id=29210222 If you have extra space, you could rclone the result of locally run borg to a cloud storage or find a service that accepts borg directly.

Sohcahtoa82 5y ago

About 10 years ago, I was earning my CS degree, and my Intro to Unix teacher was showing us how to use rsync. He somehow made an error that synced an empty directory to his home directory, deleting everything from his home directory. The lesson I learned in that class was to not use rsync, sadly.

quesera 5y ago
> The lesson I learned in that class was to not use rsync, sadly. This is a poor lesson! If your TA tripped on the stairs and broke their nose, would you swear off stairs forever? rsync is an essential tool for anyone who works with files in the UNIX world. There are other options, sometimes, but rsync is almost-always present, and does almost-everything you usually want. Any tool can be used incorrectly, but rsync is easy and follows the same pattern as most other UNIX file copy tools. If you can use cp, you can use rsync safely.

gunapologist99 5y ago

It's nice that the script works both ways, but keep in mind that it's always best to pull backups from a remote system, so that an attacker in the production system can't log into the backup system and delete the backups. If your prod system(s) is/are logging into the backup system, then you have a big problem because if any of them are compromised, they can wipe out / corrupt / etc all of their own backups, and possibly the other server's backups as well (if the same user account is used for all of them.) This problem goes away if your (isolated, hardened) backup server logs into the prod systems. Of course, the inverse problem is that if an attacker manages to get into your backup system, then they also have (at least read) access to all of your prod systems!

gunapologist99 5y ago

The script could use --rsync-path='sudo rsync', so that your rsync backup script can access root-owned files on the remote system. It looks like you can add that to the RSYNCCONF part of "subpart of part 2".

uhradone 5y ago
For security reasons the user "backup" on the linux acting backup system is not part of the sudo users. Having sudo rights will to my knowledge not help to excess root files on the remote system?

frays 5y ago

Very useful article and thread comments, thank you all.