3 ms·

Enforcing data integrity is great, but I don't see why it has to happen at the filesystem level. There are any number of ways you could solve it while still kee

Enforcing data integrity is great, but I don't see why it has to happen at the filesystem level. There are any number of ways you could solve it while still keeping the RAID functionality at the level of unstructured block devices. I can think of three just off the top of my head: #1: Do checksumming at the block level. Within each block, the RAID driver reserves a few bytes at the end for a checksum, and verifies them before returning data to the FS. (This would only work if the filesystem supports non-power-of-two block sizes.) #2: Similar to #1, but pack the checksums for multiple data blocks into a separate dedicated checksum block. (This adds some extra read latency in the worst case, but caching could mitigate it.) #3: Let the filesystem handle checksumming, but extend the block device API to provide feedback to the kernel if the checksum is invalid. I think you only really need two API calls -- one which says "read this block as fast as possible and I'll verify it", and one which says "the block you returned looks bad, try to reconstruct it and give me all available candidates".

klodolph 10y ago
You're right that it doesn't have to happen at the filesystem level, but going down to block or device level is the wrong direction. Almost all useful metadata is lost by the time you get to the block level. At the filesystem level, you can specify different redundancy policies for different types of data. You can mirror metadata but not file contents, for example, which is a pretty good policy available in BTRFS that you can't achieve at the block level. You can also do sensible things like mirror across three drives by putting each file on two of the three drives. But there's a better solution. The better solution is to handle this at a higher level. It's common to want data replicated across machines, at which point block-level redundancy really only sucks up disk space for no benefit. This is how cloud storage works, and it's how you set up an Exchange server these days, et cetera. RAID and block-level redundancy will still be around (somebody will always need it), but it's a dying technology, at least in the sense that new deployments are using it less and less frequently.
dap 10y ago
> Do checksumming at the block level. This doesn't handle phantom or misdirected writes. You can't really do that unless you keep the checksums elsewhere, and specifically in the place where you intend to reference the data. > Let the filesystem handle checksumming, but extend the block device API to provide feedback to the kernel if the checksum is invalid. I think by the time you get this API right, it's going to seem like just as bad a layering violation. ZFS has a coherent set of layers built into it. They're just not the ones that existed before it. But I don't think those previous layers were designed with many of the important failure modes of real hardware devices in mind. That's understandable given when they were designed, but we've come a long way since then in the understanding of those failure modes and our expectations of system integrity.

klodolph 10y ago

You're right that it doesn't have to happen at the filesystem level, but going down to block or device level is the wrong direction. Almost all useful metadata is lost by the time you get to the block level. At the filesystem level, you can specify different redundancy policies for different types of data. You can mirror metadata but not file contents, for example, which is a pretty good policy available in BTRFS that you can't achieve at the block level. You can also do sensible things like mirror across three drives by putting each file on two of the three drives. But there's a better solution. The better solution is to handle this at a higher level. It's common to want data replicated across machines, at which point block-level redundancy really only sucks up disk space for no benefit. This is how cloud storage works, and it's how you set up an Exchange server these days, et cetera. RAID and block-level redundancy will still be around (somebody will always need it), but it's a dying technology, at least in the sense that new deployments are using it less and less frequently.

dap 10y ago

> Do checksumming at the block level. This doesn't handle phantom or misdirected writes. You can't really do that unless you keep the checksums elsewhere, and specifically in the place where you intend to reference the data. > Let the filesystem handle checksumming, but extend the block device API to provide feedback to the kernel if the checksum is invalid. I think by the time you get this API right, it's going to seem like just as bad a layering violation. ZFS has a coherent set of layers built into it. They're just not the ones that existed before it. But I don't think those previous layers were designed with many of the important failure modes of real hardware devices in mind. That's understandable given when they were designed, but we've come a long way since then in the understanding of those failure modes and our expectations of system integrity.