As an aside, this started a decade ago as a personal backup script using rsync and hardlinks for a few personal machines. But hardlinks really start falling ap

As an aside, this started a decade ago as a personal backup script using rsync and hardlinks for a few personal machines. But hardlinks really start falling apart once you get a lot of machines, say more than 10, or large systems with lots of files. At one point we switched to BackupPC, but ended up switching back to this after around a year. BackupPC implemented its own rsync code, which (maybe this is fixed now) didn't support incremental file-data transfers in the rsync version 3 protocol, so large systems could take hours to build the file index and then hours to re-walk the file-system to send the data. Larger systems were taking longer than 24 hours and tons of IOPS to backup. It also wasn't very efficient, when we switched back to ZFS, we consolidated 4 Backup PC servers down to one with ZFS holding the same data. The biggest issue there was log and database files, big files that had small changes resulted in the whole file getting stored multiple times. Particularly Zope ZODB files killed us, they are append-only and we had users with 2+GB files that had small changes every day.