4 ms·

This article is about backups for servers. I think some people are missing that part. For my personal home machines, I was recently researching backup systems

This article is about backups for servers. I think some people are missing that part. For my personal home machines, I was recently researching backup systems. I decided against cloud backups and opted for the simple solution of a USB 3.0 drive. It's cheap, and if I ever need to get my backup quickly, I just pick it up and take it. For critical documents, I can't imagine most people have more than a few megabytes anyways. That stuff can be archived/encrypted periodically and sent to cloud email (gmail, yahoo) much cheaper and easier than worrying with S3. For the actual backups, I'm using Linux so I went with rsync. See here for roughly how it's done: http://www.mikerubel.org/computers/rsync_snapshots/ http://www.mikerubel.org/computers/rsync_snapshots/ The USB drive is encrypted, I then tell rsync to use hard links and it does the incremental backup. The nice thing about hard links is you can get at any file you want to, in any backup you want to while preserving space. Files in new backups that exist in prior backups are simply hard links. Then you just "rm" older backups when no longer needed. It's an incredibly elegant solution on Linux, especially if you combine it with LVM snapshots.

ineedtosleep 13y ago
I've researched this for an extremely long time as well and came to very similar conclusions: * Small-ish type stuff on SpiderOak and Dropbox (both on free tiers). I assume both of these are "compromised". * Data that I want to keep /private/ go into an encrypted USB flash drive and onto Tarsnap. I doubt this will ever get larger than 10-20MB. * I have ~5 full HDD images of Linux boxes that I need to keep for archival purposes and storing that on a cloud backup service would be a pain to upload and download. Those are on a Synology 213+ and on a larger capacity USB3 drive. I'm still evaluating git-annex, but right now I can't seem to find a workflow to my liking. I feel like it /could/ be something I'd use to replace most of the above, but I can't see it yet.
- edoloughlin 13y ago
  I'm still evaluating git-annex I thought git-annex would be a good solution but I can't seem to get it to stop locking files I'm working on without using its 'unsafe' mode. I can't seem to figure out the workflow it was designed to support.
Silhouette 13y ago
May I ask which model of USB drive you're using, if it works reliably with Linux? I was reading about these just last weekend, and surprised to hear that several of the current, big name brand USB hard drives seem to come with "clever" software, not necessarily easily removable, that means they don't work like a vanilla disk any more. The result is that operating systems without supplied drivers, such as Linux, won't necessarily be able to access the drives properly. I have no way to tell how much of this is real and how much is down to misunderstanding, but it was a recurring theme and applied to multiple major vendors, so as appealing as this simple approach is, I don't really feel like trusting my critical back-ups to any of these drives until I've got some facts.
- voltagex_ 13y ago
  Seagate and WD drives have worked fine for me, both on ARM and x86. WD drives detect a wd_ses device as well as the drive and I think that gives me LED control (great for turning off blinkenlights at night) My main problem with backup drives at the moment is getting hdparm/power management settings to stick so the drives will spin down after an amount of time. I've had more luck with USB than ESATA in this case. I'd recommend AGAINST any ESATA/USB "docks" - the USB to SATA bridges seem to have a very high failure rate (https://bugzilla.redhat.com/show_bug.cgi?id=895085 https://bugzilla.redhat.com/show_bug.cgi?id=895085 for the first one I saw on Google)
- aryastark 13y ago
  I use this, specifically: http://www.newegg.com/Product/Product.aspx?Item=N82E16822178110 http://www.newegg.com/Product/Product.aspx?Item=N82E16822178... It's a Seagate Backup Plus 3TB. I believe Newegg had it on sale for $100 when I picked it up. Been using it for maybe 3 months now. So far so good. The drive seems to have no problem spinning down. But I typically unmount and unplug when I'm not using it, just to be on the safe side. I also looked into NAS, which is what I originally wanted to go with. But my concern was none of the cheap ones seemed capable or fast enough to do encryption.
andreasvc 13y ago
I use a similar approach, rsnapshot. What I worry about though is that it seems very wasteful when backing up a whole system. Each time the backup is happing I hear the disk churning for half an hour, and I realize it's mostly recreating hard links to files that _haven't_ changed. So I wonder which backup method would require the least amount of disk activity.
- aryastark 13y ago
  Not sure. You might want to see if the --checksum option is enabled (or disabled?). Also, rsync man page says it does a checksum even without that option, to verify the file has been copied correctly. This may be what is causing a lot of I/O. I also forgot to mention that hashdeep/md5deep should probably be integrated into this sort of backup system (http://md5deep.sourceforge.net/ http://md5deep.sourceforge.net/). What hashdeep does is generates a list of checksums for a given backup. You would store this list with the backup, and then use hashdeep to "audit" the backup when you restore it, or to check if hardware is failing.
- unhammer 13y ago
  I just use plain rsync and then do a btrfs snapshot each time. Very fast and easy to set up. Probably just as easy with zfs if you prefer that. Of course I've got some setup around creating datestamped snapshots and keeping the N last ( http://sprunge.us/OKHb?bash http://sprunge.us/OKHb?bash for the bash function that does this), but in essence it's just $ rsync src trg/current $ btrfs subvolume snapshot -r trg/current trg/snapshot-$(date)
IgorPartola 13y ago
Caution: if you are using a single drive you are in danger. I also recently did a bunch of research on the best way to back up around 500 GB's of data (too expensive for my taste to put on something like S3 and I also needed a NAS). The conclusion I walked away with was that at least two drives and ZFS is a must. Silent drive corruption is a real problem. ZFS takes care of that much more intelligently than any hardware or software RAID and sure as hell better than no redundancy at all. IMO, the idea setup for a home backup system is two servers with ECC RAM with 2-3 drives each running ZFS and connected over at least a gigabit link. One of these would be the primary NAS server, and the other the backup. All important documents: your Documents folder, tax returns, home videos, pictures of kids, etc. go on the primary server. The backup retains months of copies of the primary server and exposes the files over SMB and AFP as read-only (unless you are all Linux, don't bother with NFS as nothing else supports it properly, even Macs). The backup server also has raw space for system-level backups, such as Time Machine, etc. Basically, space that's not backed up twice. My current setup is simpler: just one machine with no ECC (this makes me very nervous), with two drives running ZFS in a mirror setup. Since I am treating this array as reliable, I am using it for both primary storage and backup storage, though I will be changing this at some point soon when I get the time/money to build things out a bit. My biggest worry is physically separating primary and backup servers to different locations, at least within the house. As for software, I use netatalk for AFP (Mac file sharing), Samba for SMB (Windows), and rsnapshot for backups. rsnapshot is particularly nice because it's the least pain in the neck to browse backups. It is similar to Time Machine but FOSS.

ineedtosleep 13y ago

I've researched this for an extremely long time as well and came to very similar conclusions: * Small-ish type stuff on SpiderOak and Dropbox (both on free tiers). I assume both of these are "compromised". * Data that I want to keep /private/ go into an encrypted USB flash drive and onto Tarsnap. I doubt this will ever get larger than 10-20MB. * I have ~5 full HDD images of Linux boxes that I need to keep for archival purposes and storing that on a cloud backup service would be a pain to upload and download. Those are on a Synology 213+ and on a larger capacity USB3 drive. I'm still evaluating git-annex, but right now I can't seem to find a workflow to my liking. I feel like it /could/ be something I'd use to replace most of the above, but I can't see it yet.

edoloughlin 13y ago
I'm still evaluating git-annex I thought git-annex would be a good solution but I can't seem to get it to stop locking files I'm working on without using its 'unsafe' mode. I can't seem to figure out the workflow it was designed to support.

Silhouette 13y ago

May I ask which model of USB drive you're using, if it works reliably with Linux? I was reading about these just last weekend, and surprised to hear that several of the current, big name brand USB hard drives seem to come with "clever" software, not necessarily easily removable, that means they don't work like a vanilla disk any more. The result is that operating systems without supplied drivers, such as Linux, won't necessarily be able to access the drives properly. I have no way to tell how much of this is real and how much is down to misunderstanding, but it was a recurring theme and applied to multiple major vendors, so as appealing as this simple approach is, I don't really feel like trusting my critical back-ups to any of these drives until I've got some facts.

voltagex_ 13y ago
Seagate and WD drives have worked fine for me, both on ARM and x86. WD drives detect a wd_ses device as well as the drive and I think that gives me LED control (great for turning off blinkenlights at night) My main problem with backup drives at the moment is getting hdparm/power management settings to stick so the drives will spin down after an amount of time. I've had more luck with USB than ESATA in this case. I'd recommend AGAINST any ESATA/USB "docks" - the USB to SATA bridges seem to have a very high failure rate (https://bugzilla.redhat.com/show_bug.cgi?id=895085 https://bugzilla.redhat.com/show_bug.cgi?id=895085 for the first one I saw on Google)
aryastark 13y ago
I use this, specifically: http://www.newegg.com/Product/Product.aspx?Item=N82E16822178110 http://www.newegg.com/Product/Product.aspx?Item=N82E16822178... It's a Seagate Backup Plus 3TB. I believe Newegg had it on sale for $100 when I picked it up. Been using it for maybe 3 months now. So far so good. The drive seems to have no problem spinning down. But I typically unmount and unplug when I'm not using it, just to be on the safe side. I also looked into NAS, which is what I originally wanted to go with. But my concern was none of the cheap ones seemed capable or fast enough to do encryption.

andreasvc 13y ago

I use a similar approach, rsnapshot. What I worry about though is that it seems very wasteful when backing up a whole system. Each time the backup is happing I hear the disk churning for half an hour, and I realize it's mostly recreating hard links to files that _haven't_ changed. So I wonder which backup method would require the least amount of disk activity.

aryastark 13y ago
Not sure. You might want to see if the --checksum option is enabled (or disabled?). Also, rsync man page says it does a checksum even without that option, to verify the file has been copied correctly. This may be what is causing a lot of I/O. I also forgot to mention that hashdeep/md5deep should probably be integrated into this sort of backup system (http://md5deep.sourceforge.net/ http://md5deep.sourceforge.net/). What hashdeep does is generates a list of checksums for a given backup. You would store this list with the backup, and then use hashdeep to "audit" the backup when you restore it, or to check if hardware is failing.
unhammer 13y ago
I just use plain rsync and then do a btrfs snapshot each time. Very fast and easy to set up. Probably just as easy with zfs if you prefer that. Of course I've got some setup around creating datestamped snapshots and keeping the N last ( http://sprunge.us/OKHb?bash http://sprunge.us/OKHb?bash for the bash function that does this), but in essence it's just $ rsync src trg/current $ btrfs subvolume snapshot -r trg/current trg/snapshot-$(date)

IgorPartola 13y ago

Caution: if you are using a single drive you are in danger. I also recently did a bunch of research on the best way to back up around 500 GB's of data (too expensive for my taste to put on something like S3 and I also needed a NAS). The conclusion I walked away with was that at least two drives and ZFS is a must. Silent drive corruption is a real problem. ZFS takes care of that much more intelligently than any hardware or software RAID and sure as hell better than no redundancy at all. IMO, the idea setup for a home backup system is two servers with ECC RAM with 2-3 drives each running ZFS and connected over at least a gigabit link. One of these would be the primary NAS server, and the other the backup. All important documents: your Documents folder, tax returns, home videos, pictures of kids, etc. go on the primary server. The backup retains months of copies of the primary server and exposes the files over SMB and AFP as read-only (unless you are all Linux, don't bother with NFS as nothing else supports it properly, even Macs). The backup server also has raw space for system-level backups, such as Time Machine, etc. Basically, space that's not backed up twice. My current setup is simpler: just one machine with no ECC (this makes me very nervous), with two drives running ZFS in a mirror setup. Since I am treating this array as reliable, I am using it for both primary storage and backup storage, though I will be changing this at some point soon when I get the time/money to build things out a bit. My biggest worry is physically separating primary and backup servers to different locations, at least within the house. As for software, I use netatalk for AFP (Mac file sharing), Samba for SMB (Windows), and rsnapshot for backups. rsnapshot is particularly nice because it's the least pain in the neck to browse backups. It is similar to Time Machine but FOSS.