> One fundamental reason is that ZFS is philosophically and practically opposed to rewriting existing data on disk; ... That's not quite the issue. The issue

> One fundamental reason is that ZFS is philosophically and practically opposed to rewriting existing data on disk; ... That's not quite the issue. The issue is that PHYSICAL and HASH block addresses are stored together, intermingled, which means that the PHYSICAL addresses get hashed into the copy-on-write tree, which means any block can't be relocated (i.e., have its physical address(es) changed) without rewriting the path to the block containing a pointer to it as well. I.e., ZFS is not content-addressed storage. In CAS the hash of a block is its pointer, and the physical addresses are stored separately (not intermingled with hashed data) and not hashed into the tree. IF instead ZFS had used ONLY the hash value as the blkptr_t, and then had stored the physical addresses at the end of metadata blocks and then NOT hashed the physical addresses into the tree, THEN relocating would be a lot easier. Not trivial, mind you, just easier. Not trivial because of snapshots and dedup: you can find a singular pointer to any data block in the absence of snapshots and dedup, and rewrite the physical address stored nearby, but you can't easily find duplicate pointers, so to relocate blocks in CAS w/ snapshots and dedup you have to leave behind forwardings, or have a database of relocations (which is costly). > (In the grand tradition of computer science we can sort of solve this problem with a layer of indirection, where the top layer stays immutable but the bottom layer mutates. This is awkward and doesn't entirely satisfy either side, and is in fact how ZFS's relatively new pool shrinking works.) Yes, and proper CAS also has a layer of indirection, and some awkwardness around snapshots and dedup (see above). For mirror drive evacuation there's nothing to do. For shrinking operations you only need to relocate blocks whose physical addresses are beyond the end of the new device (but you'll need a fair bit of swing storage for the interim relocation or to use as forwardings until you've found and fixed all pointers to any one relocated block). I've said this many times in the past, and always someone complains that not hashing the physical addresses is unsafe, as if the hash of the contents is not safe but hash of contents and physical address could somehow be so much safer (it can't be much safer, if at all). And again, see CAS.