3 ms·

> If you didn't have ZFS, I would also caution anyone from using straight non-ZFS RAID as a backup substitute, because there's no checkpoints and no way to stop

by phil21 10y ago

> If you didn't have ZFS, I would also caution anyone from using straight non-ZFS RAID as a backup substitute, because there's no checkpoints and no way to stop data deletion from affecting you. The type of array redundancy (ZFS, mdadm, megaraid, etc.) does not matter. It is never a substitute for backups. End of discussion. These are two entirely different technologies meant to protect against entirely different problems. This is a common meme I've seen constantly through my entire career, and one of those common misconceptions you have to educate customers on quite often. RAID is not for data security. RAID is for availability. Snapshots, checksumming, etc. on ZFS are wonderful - I use it extensively. I also zfs send said snapshots to an off-site remote server, and if I didn't have that capability I'd rsync to an external hard disk or whatnot. Even if you could somehow guarantee ZFS or your hardware would never combine to lose your data (you can't) - it still doesn't protect from your house burning down. A substantial number of people completely neglect that failure mode until it's too late.

epistasis 10y ago
Agreed, it's never a full substitute for backups. It is a substitute for single drive failure. And ZFS is a substitute for some amount of data deletion (but not much). But second location, cold copy are absolutely necessary for a backup.
lmm 10y ago
It's a question of costs and benefits. RAID absolutely does reduce the risk of data loss. The extent to which you want to take further steps is a question of how valuable your data is and how much of it there is. It can be perfectly legitimate to decide that, given a particular risk profile and a finite budget, you can't afford both RAID and backups and the former is better value than the latter.
- throwawayish 10y ago
  If I guess right in what you mean here, then I'd say that in these instances backups are better value than RAID.
throwawayish 10y ago
(I'm biased here) > I also zfs send said snapshots to an off-site remote server, and if I didn't have that capability I'd rsync to an external hard disk or whatnot. I wouldn't consider a backup system any more that isn't able to establish full integrity of backups. It's also a kinda good idea to diversify the software; ZFS has a really good track record, but if I where using btrfs, for example, I'd make sure my backup software is not the same as my FS.
- phil21 10y ago
  Ah, a fellow paranoid sysadmin who's been around a while :) I do this for my "very important" stuff - e.g. documents and personal photos. That gets copied every year to a third off-site physical drive for safekeeping, and that is an entirely different filesystem. A lot of that stuff is also on google drive for the latest, so I'm not super concerned about the backup frequency there. The standard zfs send backs up about 25T of stuff I'd certainly not wish to lose, but could eventually mostly be re-created. For my risk level zfs send seemed to be the correct risk:ease of use tradeoff for me. Plus the nice diffing makes syncing a home NAS on a consumer broadband connection not entirely impossible - where even rsync can be difficult for some of my use-cases. Full integrity of backups is interesting - in a professional environment that means to me full on DR testing (e.g. restore your backups regularly to test them) among other things. For personal use I think automated regular zfs scrubs are about the best you can easily get.
- PhantomGremlin 10y ago
  I wouldn't consider a backup system any more that isn't able to establish full integrity of backups. This isn't very hard to do at all. It's very easy to DIY full integrity. Especially with ZFS, where you can take a read-only snapshot and not have the FS changing underneath you while you're traversing it. Here's a previous post of my on the subject: https://news.ycombinator.com/item?id=13129294 https://news.ycombinator.com/item?id=13129294 Something as simple as the following, done independently for source and destination, and results compared, should suffice: find -x \ . \ -type f -print0 \ | xargs -0 -n 100 -x md5 \ | sort > /tmp/manifest.source

by phil21 10y ago

epistasis 10y ago

Agreed, it's never a full substitute for backups. It is a substitute for single drive failure. And ZFS is a substitute for some amount of data deletion (but not much). But second location, cold copy are absolutely necessary for a backup.

lmm 10y ago

It's a question of costs and benefits. RAID absolutely does reduce the risk of data loss. The extent to which you want to take further steps is a question of how valuable your data is and how much of it there is. It can be perfectly legitimate to decide that, given a particular risk profile and a finite budget, you can't afford both RAID and backups and the former is better value than the latter.

throwawayish 10y ago
If I guess right in what you mean here, then I'd say that in these instances backups are better value than RAID.

throwawayish 10y ago

(I'm biased here) > I also zfs send said snapshots to an off-site remote server, and if I didn't have that capability I'd rsync to an external hard disk or whatnot. I wouldn't consider a backup system any more that isn't able to establish full integrity of backups. It's also a kinda good idea to diversify the software; ZFS has a really good track record, but if I where using btrfs, for example, I'd make sure my backup software is not the same as my FS.

phil21 10y ago
Ah, a fellow paranoid sysadmin who's been around a while :) I do this for my "very important" stuff - e.g. documents and personal photos. That gets copied every year to a third off-site physical drive for safekeeping, and that is an entirely different filesystem. A lot of that stuff is also on google drive for the latest, so I'm not super concerned about the backup frequency there. The standard zfs send backs up about 25T of stuff I'd certainly not wish to lose, but could eventually mostly be re-created. For my risk level zfs send seemed to be the correct risk:ease of use tradeoff for me. Plus the nice diffing makes syncing a home NAS on a consumer broadband connection not entirely impossible - where even rsync can be difficult for some of my use-cases. Full integrity of backups is interesting - in a professional environment that means to me full on DR testing (e.g. restore your backups regularly to test them) among other things. For personal use I think automated regular zfs scrubs are about the best you can easily get.
PhantomGremlin 10y ago
I wouldn't consider a backup system any more that isn't able to establish full integrity of backups. This isn't very hard to do at all. It's very easy to DIY full integrity. Especially with ZFS, where you can take a read-only snapshot and not have the FS changing underneath you while you're traversing it. Here's a previous post of my on the subject: https://news.ycombinator.com/item?id=13129294 https://news.ycombinator.com/item?id=13129294 Something as simple as the following, done independently for source and destination, and results compared, should suffice: find -x \ . \ -type f -print0 \ | xargs -0 -n 100 -x md5 \ | sort > /tmp/manifest.source