14 ms·
Why ZFS is not good at growing and reshaping pools, or shrinking them
- bcwu 7y agoRAID-Z expansion is in the works... http://open-zfs.org/w/images/6/68/RAIDZ_Expansion_v2.pdf http://open-zfs.org/w/images/6/68/RAIDZ_Expansion_v2.pdf https://www.youtube.com/watch?v=Njt82e_3qVo https://www.youtube.com/watch?v=Njt82e_3qVo
- louwrentius 7y agoThat was announced about 2 years ago.
- tecleandor 7y agoThere's ongoing work on this PR: https://github.com/zfsonlinux/zfs/pull/8853 https://github.com/zfsonlinux/zfs/pull/8853
- dictionary_word 7y ago2 years in filesystem time isn't so long.
- anon9001 7y agoI might be misguided here (please correct me), but I feel like ZFS and hardware RAID had their place pre-SSD, and now they don't make much sense. If you want an array of disks with a mutable configuration, I think the right choice is mdadm and LVM. They're reliable, relatively easy to use, let you move disks between machines, don't rely on hardware RAID, and support all file systems. If you need something faster and more optimized like ZFS or btrfs, you're probably better off spending on SSDs anyway. Tell me why I'm wrong, HN :)
- the8472 7y agomdadm and lvm don't provide data checksumming.
- zozbot234 7y agoDM does, via dm-integrity.
- the8472 7y agodoes that talk to the raid part so it can restore the correct sector when two raid1 mirrors disagree? And it seems to come with its own journal, so stacking several journaled systems could lead to write amplification, no?
- cthalupa 7y agoYou would use dm-integrity on each individual disk below the RAID, and when it detected the corrupted file it would return an error, which would then be used as a signal to the RAID system to read from the other disk. You have to manually set up dm-integrity on the disks, however - it is not used by default when creating an mdadm/lvm setup.
- the8472 7y agoThat setup would mean more overhead. Separate journals for each disk, one checksum copy on each disk instead of only up to your redundancy level, more disk writes.
- cthalupa 7y agoFor sure. I'm a ZFS kind of guy myself. But you can start hacking together a lot of the functionality the more dm modules you start using.
- zozbot234 7y agoIt's not "more" overhead, just a different distribution of it. You can actually use erasure coding instead of conventional RAID once the lower-level blocks are protected by their own checksums.
- lunixbochs 7y agoPrice per gigabyte. Far more economical to use ZFS on spinning disks with a huge NVME + RAM cache than it is to buy 80TB of NVME. As long as you don’t need fast random access to the whole dataset all-SSD is wasted.
- Wowfunhappy 7y agoI think most people use ZFS for data integrity, not performance. I make backups regularly (manually, because I want them to be offline), but I'd rather not have to use them if one of my drives die. Unless I literally just finished making the backup, I'm going to lose some amount of work. My mirrored ZFS pool means that (A) a drive can die and I won't lose _any_ data, no matter how recent, and (B) I'm confident none of the data on either drive is corrupted, because ZFS checks it regularly (every time it's accessed, and during weekly scrubs).
- kyuudou 7y agoYep I'd use ZFS for archival and tier-2 storage and use expensive SSD for I/O that needs to be fast and nimble.
- stiray 7y agoExactly. Anyway, my home server is having 6Tb redundant storage (3xToshiba 3Tb) and 10Tb Hitachi drive, non redundant. What would the cost of SSDs be? Over $1000 not to mention the controlle 4 SATA port vs 19+ port controller is a huge difference in price. I do use SSDs, where I need speed - L2ARC ;), far better use than buying 19Tb of SSDs, not to mention the price. Anyway, I do use ZFS on laptop for linux root. Reason? Snapshots. Kvm + ZFS, perfect match. /.zfs/snapshots? Fabulous.
- nickik 7y agoThere is this fun backup tool, that have yet to set up. https://www.znapzend.org/ https://www.znapzend.org/
- deleted 7y ago[deleted]
- rbanffy 7y agoRAID that knows about which blocks are occupied and which are not will take a shorter time to rebuild because it'll need to read and write fewer blocks. For `mdadm` to do that, it'd need to know a lot about the internals of the filesystems that are mapped to the block device. Filesystems like BtrFS and ZFS know what is being used and what is not and won't waste time reading or writing to blocks that don't hold any currently used data.
- zozbot234 7y ago> For `mdadm` to do that, it'd need to know a lot about the internals of the filesystems that are mapped to the block device. That's taken care of by TRIM/DISCARD, which all modern filesystems support.
- greenshackle2 7y agoZFS is optimized first for data integrity, not speed. It's like you're saying, why do we need ECC when there is fast DDR4 memory? ZFS is a filesystem not just a layer like LVM. You have to compare it to LVM + some filesystem. Which filesystem would you suggest using with LVM to get ZFS-level integrity? (Technically you can use ZFS on top of LVM but that would be a weird thing to do.)
- anon9001 7y agoI use ECC and mdadm+LVM+ext4. I haven't had any issues yet, but this thread is making me wonder if there's some data integrity problem that I'm setting myself up for. With RAID6 I can have 2 drives fail without data loss. It does take several days to rebuild when a drive dies, so maybe that's the ZFS feature I'm missing? If the concerns about "data integrity" are really about time until a disk is replaced, then it makes sense. Otherwise, what's the data integrity concern? Ext4 is a journaled file system, which I thought was the equivalent of ECC for filesystems.
- cthalupa 7y ago>I use ECC and mdadm+LVM+ext4. I haven't had any issues yet, but this thread is making me wonder if there's some data integrity problem that I'm setting myself up for. Nothing in your setup there protects you from bitrot. >With RAID6 I can have 2 drives fail without data loss. It does take several days to rebuild when a drive dies, so maybe that's the ZFS feature I'm missing? Drive rebuild time is primarily a function of a variety of things - speed of the disks, processing power involved, etc. There's nothing magic about any solution here. ZFS does get a leg up in that it knows what blocks are in use, but hardware raid and mdadm are quite fast when they find empty blocks too, so it's not some overwhelmingly massive improvement. >Otherwise, what's the data integrity concern? Ext4 is a journaled file system, which I thought was the equivalent of ECC for filesystems. I think you're fundamentally confused about what journaling provides a filesystem. A journal helps you in situations where your system crashes, loses power, etc. and you had not finished committing your changes to the filesystem. It does not protect you from data integrity issues that are unrelated to this sort of situation - a bit flipping in an HDD from electromagnetic radiation or an SSD from electron leakage will not be caught by a journal.
- briffle 7y agossd's still can become corrupt (bitrot) and can still die. File systems like zfs and btrfs checksum every block to ensure that if they do become corrupt, they can get repaired. (otherwise, if you had a mirror of disks, and one disk says the value is 0010 and the other says its 0001, wich one is right?) Additionally, the 'raid' concept is not just about speed. Its also about capacity, and how many disks can fail. I might have 48 drives in a single server. I certainly don't want an LVM stripe across all drives and lose data. I also don't want raid1, losing the capacity of 24 of the drives. I want it so that any 3 drives can fail in that system, and I can still recover the data.
- thedance 7y agoSSDs are always corrupt. Their raw bit error rate can be 90%+ in normal operation. The only reason SSDs work at all is because of awesome math.
- throwaway-9320 7y agoYou might be interested in this article which does mention valid points for choosing mirrored data instead of any RAIDZ levels in ZFS: https://jrs-s.net/2015/02/06/zfs-you-should-use-mirror-vdevs-not-raidz/ https://jrs-s.net/2015/02/06/zfs-you-should-use-mirror-vdevs... It certainly made me reconsider my ZFS practices.
- rhinoceraptor 7y agoZFS might not have the single disk, raw synthetic benchmark performance of a stupid file system with no correctness/integrity built in, but that use case isn't really what anyone wants or needs. ZFS isn't really a filesystem. It's more like a storage backend, that happens to support the POSIX interface. The original intent of ZFS was to completely rethink the volume management/filesystem layering nonsense, and to provide for storage what the MMU/VM system provides for memory. So it can do much more interesting things like memory backed read caching (via ARC), SSD backed read caching (via L2ARC), and SSD backed write buffering (via ZIL/SLOG), as well as the normal RAID stuff.
- ses1984 7y agoI have lost data with mdadm. Never lost data with ZFS yet. Granted I was a much nooblier noob when using mdadm.
- matheusmoreira 7y agoLVM software RAID has the same problems as ZFS's implementation and hardware RAID. When drives fail, high capacity arrays will take over 24 hours to rebuild themselves and are likely to run into additional read errors due to the heavy load. Expanding the capacity of the array is an incredibly complex process: https://raid.wiki.kernel.org/index.php/Growing https://raid.wiki.kernel.org/index.php/Growing It should be possible to buy disks of any size, add them to the computer and tell the file system to use them. Drive failures shouldn't increase the risk of additional failures.
- rodgerd 7y agoIf you think LVM snapshots are an adequate replacement for ZFS snapshots, I can only assume you've not used either in anger.
- cmurf 7y agoThere are two kinds of LVM snapshots: thick and thin. You're probably thinking of conventional (thick) snapshots, rather than the newer thinly provisioned snapshots. Snapshotting an LV will FIFREEZE the file system, then snapshot, then FITHAW the file system. So depending on how dirty things are, it might take a while for the snapshot to happen, but otherwise they are nearly instant to create and destroy.
- rodgerd 7y agoI am aware of the difference - and even the much-touted thins are very poor by comparison, as anyone who has had the displeasure of trying to use them in anger can attest.
- cmurf 7y agoI have but only small scale: single 2T drive and data, with maybe a dozen snapshots. I use Btrfs for most everything these days, with ZFS archive. The biggest issue I had, a while back now, with thinp is if the metadata volume gets full...kaboom. And it's non-trivial to repair.
- the8472 7y agoIsn't that a bit of rewriting the history? Originally the roadmap was that these kinds of features would be added based on the mythical block pointer rewrite. Eventually BPR and thus all features requiring it were put on ice.
- kbenson 7y agoWhat specifically are you replying to? I'm not sure I recognize something in the article this applies to (but that could just be me interpreting what was presented differently than you).
- the8472 7y ago> One fundamental reason is that ZFS is philosophically and practically opposed to rewriting existing data on disk; once written, it wants everything to be completely immutable ZFS was not philosophically opposed to those ideas. They were just dropped at some point.
- nickik 7y agoThey were dropped because getting them fully correct and implemented was really hard and required a lot of work that nobody really wanted to pay for.
- phs2501 7y agoWhat's funny is now ZFS is used on SSDs, where that kind of rewriting is constantly happening behind the scenes (even if you're not writing it's constantly doing housekeeping for degraded blocks, especially on modern denser flash).
- boomboomsubban 7y agoI wouldn't call it rewriting history. ZFS had plans to eventually develop the more complicated code to handle reshaping pools, but that never happened and the described system is the one still in place.
- sliken 7y agoOne thing I like about btrfs is offline deduplication. So whenever you want you can search for duplications and then tell the filesystem about what you find. This avoids many of the performance and ram impacts of ZFS deduplication.
- boomboomsubban 7y agoAre you really saving that much space doing that on a block level rather than a file level?
- secabeen 7y agoIf you are doing it on disk images for VMs or virtual desktops, yes.
- sliken 7y agoAgreed, but otherwise no.
- the8472 7y agoYes. For example when you're using snapshots but download new versions of something on the live volume then most installer software just overwrites the old files with nearly or fully identical new files. Running an offline dedup will then go back and reintegrate the new files with the dupes on the snapshots. Lots of software is not as CoW-friendly as it could be, offline dedup fixes that with relatively low overhead.
- boomboomsubban 7y ago>. Running an offline dedup will then go back and reintegrate the new files with the dupes on the snapshots. Running a dedupe based on file rather than block would do the same thing. You don't gain much if anything in your example using btrfs.
- washadjeffmad 7y ago
- williesleg 7y agoThanks Larry Ellison.
- tobias3 7y agoAs far as I know it is a fundamental design difference between btrfs and ZFS. Btrfs has extent back references. Given a location on disk btrfs has an efficient data structure that gets from the disk location to the items using that disk location. E.g. if a sector on disk gets corrupted it has a command that lets you list all the the files using that sector. With ZFS you'd have to iterate through all files, list block locations and check if one of them is affected. Of course such an additional index comes with a performance cost for managing that index (any kind of write or delete operation plus more metadata space used). On the top of my head this is used for rebalance, quota groups, device removal, send/receive (finding reflinks) and file system shrinking. For a similar trade off compare hard links on Windows vs. ext4 and co. on Linux. On Linux there is no command to list all hard links to a file while Windows (NTFS) has one (FindFirstFileName etc.). This is why creating hard links on Windows is much slower and there is a limit of 1024 hard links to a file. To find all hard links to a file on Linux one would need to iterate over all files and check if they have the same inode number, however.
- craftkiller 7y agoCouldn't such an index simply be generated at the start of a grow/shrink/reshape operation and then thrown away after?
- djsumdog 7y agoI thought most *nix filesystems were like how you describe ZFS (and ext2/3/4) where all the metadata is at the beginning of the disk. At least with the ext family, doesn't the metadata reference the first 10 blocks of a file (where the last two blocks are pointers to double linked lists and the last a reference to a triple linked list). ... It's been a long time since I took that filesystems class, but I recall it's something like that?
- Dylan16807 7y agoThat's about right. But with ext2/3/4 there are no snapshots and when you have a reference to a block you know that's the only reference, so it's not a problem there. https://en.wikipedia.org/wiki/Inode_pointer_structure https://en.wikipedia.org/wiki/Inode_pointer_structure has a summary https://docs.microsoft.com/en-us/archive/blogs/askcore/the-four-stages-of-ntfs-file-growth https://docs.microsoft.com/en-us/archive/blogs/askcore/the-f... talks about the NTFS equivalent
- microcolonel 7y agobcachefs is looking promising to me. I hope that it doesn't unlearn any of the lessons we've learned over the years from ZFS, which is itself excellent in many ways.
- cmurf 7y agoThe indirection that exists in Btrfs is the internal virtual address space and block groups. Metadata and data extents are referenced in virtual addresses, bytes. That address is translated by the chunk (block group) tree into two things: a physical device, and location on that device. Conceptually it's not altogether different from ext4 on LVM, and hence the old critique that Btrfs is a layering violation. But these things are integrated on Btrfs, and the chunk is what makes so many things flexible, as well as a PITA when it comes to ENOSPC (mostly solved these days).
- cryptonector 7y ago> One fundamental reason is that ZFS is philosophically and practically opposed to rewriting existing data on disk; ... That's not quite the issue. The issue is that PHYSICAL and HASH block addresses are stored together, intermingled, which means that the PHYSICAL addresses get hashed into the copy-on-write tree, which means any block can't be relocated (i.e., have its physical address(es) changed) without rewriting the path to the block containing a pointer to it as well. I.e., ZFS is not content-addressed storage. In CAS the hash of a block is its pointer, and the physical addresses are stored separately (not intermingled with hashed data) and not hashed into the tree. IF instead ZFS had used ONLY the hash value as the blkptr_t, and then had stored the physical addresses at the end of metadata blocks and then NOT hashed the physical addresses into the tree, THEN relocating would be a lot easier. Not trivial, mind you, just easier. Not trivial because of snapshots and dedup: you can find a singular pointer to any data block in the absence of snapshots and dedup, and rewrite the physical address stored nearby, but you can't easily find duplicate pointers, so to relocate blocks in CAS w/ snapshots and dedup you have to leave behind forwardings, or have a database of relocations (which is costly). > (In the grand tradition of computer science we can sort of solve this problem with a layer of indirection, where the top layer stays immutable but the bottom layer mutates. This is awkward and doesn't entirely satisfy either side, and is in fact how ZFS's relatively new pool shrinking works.) Yes, and proper CAS also has a layer of indirection, and some awkwardness around snapshots and dedup (see above). For mirror drive evacuation there's nothing to do. For shrinking operations you only need to relocate blocks whose physical addresses are beyond the end of the new device (but you'll need a fair bit of swing storage for the interim relocation or to use as forwardings until you've found and fixed all pointers to any one relocated block). I've said this many times in the past, and always someone complains that not hashing the physical addresses is unsafe, as if the hash of the contents is not safe but hash of contents and physical address could somehow be so much safer (it can't be much safer, if at all). And again, see CAS.
- hinkley 7y agoEven before ZFS I wanted there to exist a RAID array where you could just swap out a disk to increase capacity. I wanted it in part for me, but I also wanted it for all the people in my life who were doing a terrible job of taking care of their important data. “Don’t worry,” I wanted to tell them. “Get one of these, slap some drives in it. When you need some more space, buy a new drive, take the smallest one out and put the new one in.” Some of the early PR for ZFS was unclear on whether that was possible, but I was excited. This was by far the closest we’d gotten. Later it was a feature we’d have soon. I waited to adopt. Apple was going to support it, here we come. Apple was not going to support it. Okay, NAS could still happen. Sun craters and I lose track, but check in every so often, and still nothing. In the grand scheme of things, it is not the biggest tech disappointment I’ve suffered. But it feels like it should be in the top ten, even if it’s #10.
- tatersolid 7y agoWindows Storage Spaces and Drobo both have the “grow proportionally onto bigger or differently sized disks” feature
- deleted 7y ago[deleted]