4 ms·

I've never really understood the rationale behind integrating RAID into the filesystem. It seems like a giant layering violation, and unnecessary given that mda

• erkkie 10y ago
  Sometimes collapsing arbitrary layers gives you more context to work on. Judging from ZFS history, this is the correct path. (and if you've ever had to rebuild a mostly free, large, raid5-6 array, you will appreciate filesystem aware rebuilds)
• snuxoll 10y ago
  Integrating RAID into the filesystem solves a lot of problems, one big one being data integrity. When you have a dumb RAID layer unaware of the data on disk it will just replicate or shard it between disks, when you request that data back and you have replicas it has no way to verify that data is correct, if one replica is bad but the other is good your filesystem has no way to correct for it since it just sees one big block device. Moving storage pooling functionality to the filesystem is the right call for anything with server workloads, ZFS's self-healing functionality wouldn't work without it (and this is precisely why I use FreeNAS instead Unraid or other alternatives, I care about the long-term integrity of my data, especially my VMWare images and photo collection).
  • teraflop 10y ago
    Enforcing data integrity is great, but I don't see why it has to happen at the filesystem level. There are any number of ways you could solve it while still keeping the RAID functionality at the level of unstructured block devices. I can think of three just off the top of my head: #1: Do checksumming at the block level. Within each block, the RAID driver reserves a few bytes at the end for a checksum, and verifies them before returning data to the FS. (This would only work if the filesystem supports non-power-of-two block sizes.) #2: Similar to #1, but pack the checksums for multiple data blocks into a separate dedicated checksum block. (This adds some extra read latency in the worst case, but caching could mitigate it.) #3: Let the filesystem handle checksumming, but extend the block device API to provide feedback to the kernel if the checksum is invalid. I think you only really need two API calls -- one which says "read this block as fast as possible and I'll verify it", and one which says "the block you returned looks bad, try to reconstruct it and give me all available candidates".
    • klodolph 10y ago
      You're right that it doesn't have to happen at the filesystem level, but going down to block or device level is the wrong direction. Almost all useful metadata is lost by the time you get to the block level. At the filesystem level, you can specify different redundancy policies for different types of data. You can mirror metadata but not file contents, for example, which is a pretty good policy available in BTRFS that you can't achieve at the block level. You can also do sensible things like mirror across three drives by putting each file on two of the three drives. But there's a better solution. The better solution is to handle this at a higher level. It's common to want data replicated across machines, at which point block-level redundancy really only sucks up disk space for no benefit. This is how cloud storage works, and it's how you set up an Exchange server these days, et cetera. RAID and block-level redundancy will still be around (somebody will always need it), but it's a dying technology, at least in the sense that new deployments are using it less and less frequently.
    • dap 10y ago
      > Do checksumming at the block level. This doesn't handle phantom or misdirected writes. You can't really do that unless you keep the checksums elsewhere, and specifically in the place where you intend to reference the data. > Let the filesystem handle checksumming, but extend the block device API to provide feedback to the kernel if the checksum is invalid. I think by the time you get this API right, it's going to seem like just as bad a layering violation. ZFS has a coherent set of layers built into it. They're just not the ones that existed before it. But I don't think those previous layers were designed with many of the important failure modes of real hardware devices in mind. That's understandable given when they were designed, but we've come a long way since then in the understanding of those failure modes and our expectations of system integrity.
• rleigh 10y ago
  I used to think this was a horrible layering violation as well. I'd been using various filesystems on LVM on RAID for a good 15 years, and from a certain point of view these layers make a great deal of sense. But these layers also impede a number of important things needed for data integrity checking and efficient rebuilding on failure, as well as complicating administration when you need to rebuild bits of an array after failure or fiddle with logical volumes. From the point of view of a sysadmin, ZFS is a revelation. Datasets are simple; creating, resizing (quota), deleting are trivial and safe. Operations on the pool are logical and (relatively) safer than with mdraid or hardware RAID. Snapshots are safer--you are doing it at the filesystem level rather than the block device level, so it can never be inconsistent or get corrupted when your snapshot device is invalidated at some arbitrary future point when the block delta exceeds the device size. The layering is still there. It's just subtly different. If you're used to LVM, then a "pool" is basically a volume group. The "vdev"s making up the pool are the physical volumes, which might be RAID sets. The "datasets" or "zvols" are logical volumes. It's a reinvention of what we already had, but it's more powerful, more flexible, and vastly easier to administer. I was sceptical, but after learning about it (and there is a learning curve), doing some simple test installs (often repeated as I realised I'd not set up the pool with the right blocksize or partition alignment, or the optimal dataset structure) and then some actual deployments, I'm a convert. What it offers is currently unmatched.