5 ms·

Somewhat offtopic but this reminds me: what's the best way to backup a Raspberry Pi? Mine has a 128GB SD card and that makes backing up the whole thing kind of

Somewhat offtopic but this reminds me: what's the best way to backup a Raspberry Pi? Mine has a 128GB SD card and that makes backing up the whole thing kind of silly. There has to be a better way. I've lost a few months of data twice because of my ignorance

wglb 5y ago
For small systems in my home lab, I use rsync from the small devices (e.g., Rpi, Intel NUC, etc) to the big system with a couple of 6TB drives.
- _wldu 5y ago
  I do this too. And you can combine rsync with gocryptfs (in reverse mode) to get strong encryption of your backups as well. This is especially important if you are storing the backups on a remote/untrusted device. https://wiki.archlinux.org/title/Gocryptfs#Usage https://wiki.archlinux.org/title/Gocryptfs#Usage
  - noir_lord 5y ago
    I use rsync to backup my desktop (which is mostly used for development) pretty much just rsync /home to a big enough external LUKS drive. Has saved my bacon a few times, I don't care about incremental snapshots or delta's (though I've used both rsnapshot and rdiff successfully in the past), I'm covering the "if the SSD blows up, how long to chuck a new one in and be back up" case not the "I might need that file from 6mths ago". I also have syncthing setup via /home/<user>/Shared/{Personal,Work}/ on every machine and important stuff I just chuck in Shared/ and forget about/it's available wherever I need it at any point. I've had really ornate bulletproof snapshot based backups but honestly for my particular use case they where more hassle than they were worth, rsync does what I want every time and has never let me down.
cliffwarden 5y ago
I like rdiff-backup. For me, it's a good middle-ground between rsync and a full backup solution. https://rdiff-backup.net/docs/features.html https://rdiff-backup.net/docs/features.html
- noir_lord 5y ago
  rdiff is good but if I had to pick between the two I'd still use rsnapshot.
- pengaru 5y ago
  Not unless they've improved it substantially in the past decade. A startup I worked for [0] shipped a backup/"DR" appliance using rdiff-backup behind the scenes that I had the pleasure of inheriting ownership of. What rdiff-backup was good for was creating the false impression that you had working backups you could restore from. But once your available disk space for backups filled up, which is kind of the whole goal of a backup system; accumulate as many revisions going back as far as you have space for, the thing paints itself into a corner you can't recover from without creating potentially huge amounts of free space first. Here's why: 1. The backup tree is modified in-place in the course of performing a backup. If the backup is prevented from finishing for any reason (admin/user cancelled, ENOSPC, power loss, backup source became unavailable, etc), the backup tree is left partially in the new revision and partially in the previous revision. Any subsequent operation, restore or backup, must first restore the backup tree to its previous version, using the same primitive restore algorithm rdiff-backup uses for general restores. 2. Unless you're restoring from the latest revision requiring no reassembly from differentials, the restore algorithm requires enough free space to store up to two additional copies of any given file having changes it's reassembling. This doesn't even include the final destination file, if restoring into the backup filesystem (as it does when recovering from an interrupted backup, mentioned in #1), you need space for the third copy too. Maybe they've fixed these problems since my time dealing with this, it's been years. I ended up writing a compatible replacement for my employer at the time which used hard link farms to facilitate transactional backups requiring no recovery process when interrupted. This also enabled remote replication to always have a consistent tree to copy offsite while backups were in-progress, something rdiff-backup's in-place modification interfered with. As-is you'd end up just propagating a partially updated backup offsite if it happened to overlap with an ongoing backup. My replacement also didn't require any temporary space for reassembling arbitrary versions of files from the differentials. So it could always perform a restore, even with no free space available. I even built a FUSE interface and versioned backup fs virtualization shim for QEMU+qcow2 atop those algorithms. But it was all proprietary and some of the stuff got patented unfortunately. I wouldn't consider rdiff-backup usable if it didn't at least have the ability to restore without free space yet. At least then it might still be able to do its rollback process when ~full, assuming it's still doing the in-place modification of the backup data. Edit: In case it's not clear from the above; it's particularly nefarious the way rdiff-backup would fail, since it was typically unattended automated backups that would fill the disks, leaving the backup tree in a rollback-required state to either run another backup OR restore. The customers usually discovered this situation when they urgently needed to restore something, and rdiff-backup couldn't perform any restore without first doing the rollback, which it couldn't do because there was no space available. Not that it could even perform a differential restore without free space, but the rollback-required state almost guaranteed a differential restore was required just to do the rollback. Back when I was implementing the replacement it was such an urgent crisis that I was logging into customer appliances to manually restore files from sets of differentials without needing temporary space, using unfinished test programs before I had even started on the integration glue to streamline that process. [0] https://www.crunchbase.com/organization/axcient https://www.crunchbase.com/organization/axcient
lucb1e 5y ago
Basically every back-up system does incremental backups, because indeed uploading or copying everything every time is kind of silly as you say. It's easy to go here "restic does that" but basically every other tool does that as well, so what "the best way" is depends on what other things you need, not on incrementalness. Personally I would use restic in your situation because I'm familiar with it already and it does what I need (I particularly like the encryption aspect), but that's not to say that borg, bup, rsync, or other tools couldn't also fit your needs.
xdfgh1112 5y ago
Use Borg, and copy the whole image. Borg will compress and deduplicate it, so you only backup the changes each time. You'll get point in time historical backups, so if you accidentally delete stuff, you can find it again. Useful because data loss isn't always just hardware failure!

wglb 5y ago

For small systems in my home lab, I use rsync from the small devices (e.g., Rpi, Intel NUC, etc) to the big system with a couple of 6TB drives.

_wldu 5y ago
I do this too. And you can combine rsync with gocryptfs (in reverse mode) to get strong encryption of your backups as well. This is especially important if you are storing the backups on a remote/untrusted device. https://wiki.archlinux.org/title/Gocryptfs#Usage https://wiki.archlinux.org/title/Gocryptfs#Usage
- noir_lord 5y ago
  I use rsync to backup my desktop (which is mostly used for development) pretty much just rsync /home to a big enough external LUKS drive. Has saved my bacon a few times, I don't care about incremental snapshots or delta's (though I've used both rsnapshot and rdiff successfully in the past), I'm covering the "if the SSD blows up, how long to chuck a new one in and be back up" case not the "I might need that file from 6mths ago". I also have syncthing setup via /home/<user>/Shared/{Personal,Work}/ on every machine and important stuff I just chuck in Shared/ and forget about/it's available wherever I need it at any point. I've had really ornate bulletproof snapshot based backups but honestly for my particular use case they where more hassle than they were worth, rsync does what I want every time and has never let me down.

cliffwarden 5y ago

I like rdiff-backup. For me, it's a good middle-ground between rsync and a full backup solution. https://rdiff-backup.net/docs/features.html https://rdiff-backup.net/docs/features.html

noir_lord 5y ago
rdiff is good but if I had to pick between the two I'd still use rsnapshot.
pengaru 5y ago
Not unless they've improved it substantially in the past decade. A startup I worked for [0] shipped a backup/"DR" appliance using rdiff-backup behind the scenes that I had the pleasure of inheriting ownership of. What rdiff-backup was good for was creating the false impression that you had working backups you could restore from. But once your available disk space for backups filled up, which is kind of the whole goal of a backup system; accumulate as many revisions going back as far as you have space for, the thing paints itself into a corner you can't recover from without creating potentially huge amounts of free space first. Here's why: 1. The backup tree is modified in-place in the course of performing a backup. If the backup is prevented from finishing for any reason (admin/user cancelled, ENOSPC, power loss, backup source became unavailable, etc), the backup tree is left partially in the new revision and partially in the previous revision. Any subsequent operation, restore or backup, must first restore the backup tree to its previous version, using the same primitive restore algorithm rdiff-backup uses for general restores. 2. Unless you're restoring from the latest revision requiring no reassembly from differentials, the restore algorithm requires enough free space to store up to two additional copies of any given file having changes it's reassembling. This doesn't even include the final destination file, if restoring into the backup filesystem (as it does when recovering from an interrupted backup, mentioned in #1), you need space for the third copy too. Maybe they've fixed these problems since my time dealing with this, it's been years. I ended up writing a compatible replacement for my employer at the time which used hard link farms to facilitate transactional backups requiring no recovery process when interrupted. This also enabled remote replication to always have a consistent tree to copy offsite while backups were in-progress, something rdiff-backup's in-place modification interfered with. As-is you'd end up just propagating a partially updated backup offsite if it happened to overlap with an ongoing backup. My replacement also didn't require any temporary space for reassembling arbitrary versions of files from the differentials. So it could always perform a restore, even with no free space available. I even built a FUSE interface and versioned backup fs virtualization shim for QEMU+qcow2 atop those algorithms. But it was all proprietary and some of the stuff got patented unfortunately. I wouldn't consider rdiff-backup usable if it didn't at least have the ability to restore without free space yet. At least then it might still be able to do its rollback process when ~full, assuming it's still doing the in-place modification of the backup data. Edit: In case it's not clear from the above; it's particularly nefarious the way rdiff-backup would fail, since it was typically unattended automated backups that would fill the disks, leaving the backup tree in a rollback-required state to either run another backup OR restore. The customers usually discovered this situation when they urgently needed to restore something, and rdiff-backup couldn't perform any restore without first doing the rollback, which it couldn't do because there was no space available. Not that it could even perform a differential restore without free space, but the rollback-required state almost guaranteed a differential restore was required just to do the rollback. Back when I was implementing the replacement it was such an urgent crisis that I was logging into customer appliances to manually restore files from sets of differentials without needing temporary space, using unfinished test programs before I had even started on the integration glue to streamline that process. [0] https://www.crunchbase.com/organization/axcient https://www.crunchbase.com/organization/axcient

lucb1e 5y ago

Basically every back-up system does incremental backups, because indeed uploading or copying everything every time is kind of silly as you say. It's easy to go here "restic does that" but basically every other tool does that as well, so what "the best way" is depends on what other things you need, not on incrementalness. Personally I would use restic in your situation because I'm familiar with it already and it does what I need (I particularly like the encryption aspect), but that's not to say that borg, bup, rsync, or other tools couldn't also fit your needs.

xdfgh1112 5y ago

Use Borg, and copy the whole image. Borg will compress and deduplicate it, so you only backup the changes each time. You'll get point in time historical backups, so if you accidentally delete stuff, you can find it again. Useful because data loss isn't always just hardware failure!