4 ms·
> there is no defrag for ZFS Why not? Seems like an important thing to have.
by matheusmoreira 4y ago
> there is no defrag for ZFS
Why not? Seems like an important thing to have.
- cryptonector 4y agoBecause ZFS is not a content-addressed store (CAS), so block pointer (BP) rewrite is extremely expensive and has not been implemented. And defragmentation requires BP rewrite, or what GP says they do: zfs send, then swap the old and new datasets, but that can't be done atomically and transparently with a script, so that kinda sucks. The fundamental problem is that ZFS is not content addressed. It's almost CAS, but not quite. In ZFS block pointers have physical addresses in the pointer. That means that physical block locations are inexorably part of the ZFS Merkle hash tree. And that means that any change to the location of any block necessitates that the pointers to it must change and so be rewritten, but now you have to find all those pointers -- even if there can be only one pointer, you still have to change it, and when you change it, you change the Merkle hash tree of the dataset. The solution, IMO, is to split all znodes and any interior nodes that have block pointers into two halves. One half should have only logical block pointers free of any physical location pointers, thus the pointers in this half should be some metadata + hashes of the pointed-to blocks. The second half should be a "cache" of physical locations corresponding to the logical block pointers in the first half, plus a hash of just those physical locations. The key is that the Merkle hash tree should not bind physical locations so that changing those locations does not alter the Merkle hash tree. That leaves only the task of updating those second block halves that carry physical locations. So when traversing the tree the filesystem could optimistically read pointed-to blocks from the cached locations, and if the hash of the block read does not match the logical pointer, then go looking in a log of recent block moves. This way a BP rewrite system could simply traverse a dataset looking for blocks to move, copy the blocks to new locations, update the cached locations where the block pointer was found, log the move, then add the old location to the freelist. But ZFS is not like that. ZFS is not a content-addressed store, but it's a Merkle hash tree all the same. That makes BP rewrite insanely too difficult. I do wonder how much effort it would take to make a ZZFS that is based on ZFS but with the CAS redesign sketched above. This is not the first time I've written about this. I think I first proposed this back in the early days of Illumos, and one or two people thought that not hashing the physical locations into the Merkle hash tree would be folly -- I think they're wrong, because either you trust your hash function or you don't. Granted, to make ZFS into a proper CAS does require a strong cryptographic hash function, but ZFS uses one so...
- rsync 4y agoThank you very much for this explanation - I was about to respond with "I hear it's really, really hard ..." ... and this was much more useful and interesting :)
- cryptonector 4y agoBecause making such a change seems ETOOHARD, I don't expect anyone to be interested in making it in the OpenZFS community. It would take a fork, and no one will want that either. Instead I think the OpenZFS community could maybe develop an automatic and transparent feature like what you do: internally "zfs send" a dataset, and then apply any transactions that took place at the origin while doing that, then lock the origin dataset, do one more round of transaction copying to the new dataset, then atomically swap the old and new, unlock, then destroy the old. But I'm very curious what filesystems devs think of the CAS idea. It's not original, mind you. CAS is a well researched topic. I'm dead certain there's nothing wrong with a CAS design for a filesystem -- the key is to make the logical -> physical block pointer mapping fast, which is why I'd have a cache right next to every interior block, with the cache excluded from the Merkle hash tree.
- fomine3 4y agoRewriting data on ZFS is also a needed feature to complement RAIDZ expansion.
- cryptonector 4y agoI wonder if a company like rsync.net would fund work to make that "zfs send and swap" thing automatic and transparent. Or to fund a ZZFS. If for you the defragmentation need is critical... Not that I'm offering to do any of that work, mind you, but I am curious how much it would cost, and whether it'd be worth anyone's while.
- rincebrain 4y agoSubsets of BPR have gone in...that's more or less what vdev evacuation does, for example. I would be curious to see how practical someone writing something like this would be without requiring one make an entirely new pool or do offline conversion...
- toast0 4y agoI'm not a ZFS contributor or anything, but there's a lot of 'missing' features that would be nice but require moving or altering data that's stored on disk, so I think there's a general apprehension to that. ZFS is trying to be a lot of things, one of which is kind of archival quality storage; fiddling with things that were stored just fine on disk makes that harder to do. A list of 'missing' features that need data moving off the top of my head: defrag (since we're talking about it), shrinking partitions (has some overlap with defrag), compression of existing data, dedup of existing data, moving data between filesystems on the same pool without having to read the data and write it a second time. I think there's maybe one other gotcha I can't remember. All these things might be nice, but are tricky to get right, and users can figure out other ways to do them, so there you go. Limited resources and all.
- cryptonector 4y agoAdd to that list changing compression algorithms.
- rincebrain 4y agoThat one's only a little bad if you do it - you break uncompressed ARC with compressed L2ARC, nopwrite, and dedup, but all three of those are uncommon cases, and you can add helpful glue to make it work without breaking those with only a little work. For example, we've had three different zlibs compressing gzip streams for a while, and nobody really complained about it. (Linux ships 1.1.x; FBSD and everyone else ships 1.2.x; Intel QAT produces identical output to neither.) For a while, zstd was outputting different results on BE and LE systems and nobody noticed. Most of the reason we haven't updated compressors nowadays is that nobody's convinced anyone it's got enough benefit to be worth the added complexity.