5 ms·

> Is that a good idea in terms of risk mitigation? It is definitely not. Snapshots + replication is a great 99.9% "hot data" backup, but they don't guard aga

> Is that a good idea in terms of risk mitigation? It is definitely not. Snapshots + replication is a great 99.9% "hot data" backup, but they don't guard against administrative error or massive pwnage (think ransomware). At least three copies, at least two locations, at least two different systems (media/backup device vendors/operating systems/etc.) is the full package, so to speak. EDIT: I don't want to crap on this project as it serves a great use case. But any filesystem-based replication should be backed up by a logical copy, e.g. a tar or a zip file or just an "aws s3 cp -r". I have seen even checksummed file systems replicate data incorrectly due to bugs, unlucky bit-flips, ECC errors, CPUs literally taking the wrong branch. And remember, at the base of everything, this stuff is written in C!

rjzzleep 6y ago
If the AWS IAM rules have no overwrite and delete permissions then both admin and ransomware aren't really an issue are they? Obviously putting all your buckets on AWS performing correctly is a potential risk by itself, but it's a different kind of risk.
rakoo 6y ago
rsync.net allows you to send zfs send your own filesystem, and they have provisions to make the data append-only so you can't fall victim to ransomware. Of course it doesn't protect against bugs in zfs itself, so it's still good to use another software depending on how critical your data is but I'd say for a very large majority of cases it should be enough
tomxor 6y ago
> they don't guard against administrative error or massive pwnage (think ransomware). It does if you fully automate it and also invert control and think of your backup servers as a place to deposit stuff which is then rotated out into immutable snapshots - this is how rsync.net works, but I couldn't use them due to location requirements so I built my own based on ZoL. > any filesystem-based replication should be backed up by a logical copy, e.g. a tar or a zip file or just an "aws s3 cp -r". I have seen even checksummed file systems replicate data incorrectly due to bugs, unlucky bit-flips, ECC errors, CPUs literally taking the wrong branch This is literally what ZFS was designed for, I would agree with you if it was any other file system, but ZFS is not any other file system - of course you do need to throw enough redundant hardware at it for it to be able to cope with this stuff though, a mirrored pair at minimum.
- ericbarrett 6y ago
  I certainly know what ZFS was designed for. I've been working with replicated checksummed file systems for decades, including WAFL, NetApp's file system, which ZFS was "inspired" by (see the Sun vs NetApp lawsuit). They had both block- and sector-based checksums in 2002, albeit designed for RAID-4. I wholeheartedly endorse checksummed file systems for maximum data integrity. ZFS will not protect you (by default) against bit flips in memory (please use ECC) or kernel bugs. If the disk buffer is corrupted in the kernel, on its way to the file system layer for dispatch, ZFS will happily checksum and perfectly store the bad data. Checksums are like Rust's borrow checker: a very good idea that makes your data safer, but not a panacea that prevents all errors. Finally, as with all complex file systems, ZFS is notorious for difficulty in recovering from multi-bit errors, often requiring "pool surgery" and multi-day disk scrubs to remount the file system. Perhaps your data is eventually recoverable, but do you have a week to finish the scrub to get it back online? If any this sounds unlikely to you, I assure you, it is! But it can happen, I've seen it happen, and for things that really matter, one of your backups should always always always be logical.
  - tomxor 6y ago
    > ZFS will not protect you (by default) against bit flips in memory (please use ECC) or kernel bugs. If the disk buffer is corrupted in the kernel, on its way to the file system layer for dispatch, ZFS will happily checksum and perfectly store the bad data. That's all very well but I don't see how your suggestions of doing archive backups make this any less of a problem, you are just operating on a higher level layer with the same problems. I'm also fairly sure most servers are built with ECC these days, I know my IaaS provider uses it. > Finally, as with all complex file systems, ZFS is notorious for difficulty in recovering from multi-bit errors, often requiring "pool surgery" and multi-day disk scrubs to remount the file system. Perhaps your data is eventually recoverable, but do you have a week to finish the scrub to get it back online? I cannot claim to have extensive ZFS war stories but I have had errors on a bad block device in a mirrored pool, but with this type of pool it does not affect mountability or performance. The only annoying performance issue affecting mountability I've had so far was when accidentally filling up a pool that had dedup enabled, since it struggles to function in the reserve area.
charcircuit 6y ago
Wouldn't a tar / zip / whatever suffer from the same exact problem of making a corrupted backup if the filesystem returns the wrong data?
- ericbarrett 6y ago
  Definitely, if the file was corrupted before entering ZFS's checksum regime—it won't help at all. However there are a few scenarios where making a logical copy, regardless of archive format, helps. (1) By reading the files you are "tickling" the file system to re-validate the checksums of the blocks retrieved. Note that ZFS doesn't by default do this for data in memory, and with modern memory sizes, buffer pages can stay around for a very long time. Of course if your memory pool is large enough and your data set small enough, your backup might not actually force disk access. You can be really paranoid and flush the page cache before the backup starts. (This used to be "echo 1 > /proc/sys/vm/drop_caches" for Linux, but kernel code changes quickly, so don't take my word on it.) This will make the backup job preen your original data and flush out any disk corruption which occurred between reads—quite useful if a lot of your data is cold. However it's typically a big performance hit and might be a little too paranoid even for me. (2) The act of reading the files' metadata and translating it into kernel data structures exercises quite a bit more code than copying a list of blocks onto a wire. I don't know ZFS's replication protocol[a], and the developers aren't dummies, so I'm sure they've thought of a lot of corner cases. But I have definitely seen filesystem-level replication copy some subtle underlying metadata issues which weren't detected until the files were read logically. Typically this is caused by code errors rather than hardware failure. The problem with these kinds of bugs is they've often rampaged behind the scenes for a few days by the time the first error shows up; logical reads will flush them out a lot faster. (3) The act of reading and translating the data to a new format will get your hands on it, and a lot of subtle systemic problems (e.g. application failures) can reveal themselves in the process. Filesystem replication is very seamless and quiet when it's working, and it's easy to lose track of what's actually going on. E.g. I once noticed my backups were broken because the nightly spike of bandwidth on our network graphs got a lot skinnier. There were notifications and alarms everywhere, none of which triggered. It turned out the database servers were writing to the wrong drive (shadow mounts). Replication of null data is, of course, not an error. [a] I would love to read a non-Oracle study of ZFS replication, if somebody has one—Google does not provide at the moment.

rjzzleep 6y ago

If the AWS IAM rules have no overwrite and delete permissions then both admin and ransomware aren't really an issue are they? Obviously putting all your buckets on AWS performing correctly is a potential risk by itself, but it's a different kind of risk.

rakoo 6y ago

rsync.net allows you to send zfs send your own filesystem, and they have provisions to make the data append-only so you can't fall victim to ransomware. Of course it doesn't protect against bugs in zfs itself, so it's still good to use another software depending on how critical your data is but I'd say for a very large majority of cases it should be enough

tomxor 6y ago

> they don't guard against administrative error or massive pwnage (think ransomware). It does if you fully automate it and also invert control and think of your backup servers as a place to deposit stuff which is then rotated out into immutable snapshots - this is how rsync.net works, but I couldn't use them due to location requirements so I built my own based on ZoL. > any filesystem-based replication should be backed up by a logical copy, e.g. a tar or a zip file or just an "aws s3 cp -r". I have seen even checksummed file systems replicate data incorrectly due to bugs, unlucky bit-flips, ECC errors, CPUs literally taking the wrong branch This is literally what ZFS was designed for, I would agree with you if it was any other file system, but ZFS is not any other file system - of course you do need to throw enough redundant hardware at it for it to be able to cope with this stuff though, a mirrored pair at minimum.

ericbarrett 6y ago
I certainly know what ZFS was designed for. I've been working with replicated checksummed file systems for decades, including WAFL, NetApp's file system, which ZFS was "inspired" by (see the Sun vs NetApp lawsuit). They had both block- and sector-based checksums in 2002, albeit designed for RAID-4. I wholeheartedly endorse checksummed file systems for maximum data integrity. ZFS will not protect you (by default) against bit flips in memory (please use ECC) or kernel bugs. If the disk buffer is corrupted in the kernel, on its way to the file system layer for dispatch, ZFS will happily checksum and perfectly store the bad data. Checksums are like Rust's borrow checker: a very good idea that makes your data safer, but not a panacea that prevents all errors. Finally, as with all complex file systems, ZFS is notorious for difficulty in recovering from multi-bit errors, often requiring "pool surgery" and multi-day disk scrubs to remount the file system. Perhaps your data is eventually recoverable, but do you have a week to finish the scrub to get it back online? If any this sounds unlikely to you, I assure you, it is! But it can happen, I've seen it happen, and for things that really matter, one of your backups should always always always be logical.
- tomxor 6y ago
  > ZFS will not protect you (by default) against bit flips in memory (please use ECC) or kernel bugs. If the disk buffer is corrupted in the kernel, on its way to the file system layer for dispatch, ZFS will happily checksum and perfectly store the bad data. That's all very well but I don't see how your suggestions of doing archive backups make this any less of a problem, you are just operating on a higher level layer with the same problems. I'm also fairly sure most servers are built with ECC these days, I know my IaaS provider uses it. > Finally, as with all complex file systems, ZFS is notorious for difficulty in recovering from multi-bit errors, often requiring "pool surgery" and multi-day disk scrubs to remount the file system. Perhaps your data is eventually recoverable, but do you have a week to finish the scrub to get it back online? I cannot claim to have extensive ZFS war stories but I have had errors on a bad block device in a mirrored pool, but with this type of pool it does not affect mountability or performance. The only annoying performance issue affecting mountability I've had so far was when accidentally filling up a pool that had dedup enabled, since it struggles to function in the reserve area.

charcircuit 6y ago

Wouldn't a tar / zip / whatever suffer from the same exact problem of making a corrupted backup if the filesystem returns the wrong data?

ericbarrett 6y ago
Definitely, if the file was corrupted before entering ZFS's checksum regime—it won't help at all. However there are a few scenarios where making a logical copy, regardless of archive format, helps. (1) By reading the files you are "tickling" the file system to re-validate the checksums of the blocks retrieved. Note that ZFS doesn't by default do this for data in memory, and with modern memory sizes, buffer pages can stay around for a very long time. Of course if your memory pool is large enough and your data set small enough, your backup might not actually force disk access. You can be really paranoid and flush the page cache before the backup starts. (This used to be "echo 1 > /proc/sys/vm/drop_caches" for Linux, but kernel code changes quickly, so don't take my word on it.) This will make the backup job preen your original data and flush out any disk corruption which occurred between reads—quite useful if a lot of your data is cold. However it's typically a big performance hit and might be a little too paranoid even for me. (2) The act of reading the files' metadata and translating it into kernel data structures exercises quite a bit more code than copying a list of blocks onto a wire. I don't know ZFS's replication protocol[a], and the developers aren't dummies, so I'm sure they've thought of a lot of corner cases. But I have definitely seen filesystem-level replication copy some subtle underlying metadata issues which weren't detected until the files were read logically. Typically this is caused by code errors rather than hardware failure. The problem with these kinds of bugs is they've often rampaged behind the scenes for a few days by the time the first error shows up; logical reads will flush them out a lot faster. (3) The act of reading and translating the data to a new format will get your hands on it, and a lot of subtle systemic problems (e.g. application failures) can reveal themselves in the process. Filesystem replication is very seamless and quiet when it's working, and it's easy to lose track of what's actually going on. E.g. I once noticed my backups were broken because the nightly spike of bandwidth on our network graphs got a lot skinnier. There were notifications and alarms everywhere, none of which triggered. It turned out the database servers were writing to the wrong drive (shadow mounts). Replication of null data is, of course, not an error. [a] I would love to read a non-Oracle study of ZFS replication, if somebody has one—Google does not provide at the moment.