4 ms·
Illumos has a way to expand pools, FYI. IDK if that's in OpenZFS yet. It works thusly: ZFS creates a vdev inside the new larger vdev, then moves all the data
by cryptonector 9y ago
Illumos has a way to expand pools, FYI. IDK if that's in OpenZFS yet.
It works thusly: ZFS creates a vdev inside the new larger vdev, then moves all the data from the old vdev to the new vdev, then when all these moves are done the nested vdevs are enlarged.
What should originally have happened is this: ZFS should have been closer to a pure CAS FS. I.e., physical block addresses should never have been part of the ZFS Merkle hash tree, thus allowing physical addresses to change without having to rewrite every block from the root down.
Now, the question then becomes "how do you get the physical address of a block given just its hash?". And the answer is simple: you store the physical addresses near the logical (CAS) block pointers, and you scribble over those if you move a block. To move a block you'd first write a new copy at the new location, then overwrite the previous "cached" address. This would require some machinery to recover from failures to overwrite cached addresses: a table of in-progress moves, and even a forwarding entry format to write into the moved block's old location. A forwarding entry format would have a checksum, naturally, and would link back into the in-progress-move / move-history table.
During a move (e.g., after a crash during a move) one can recover in several ways: you can go use the in-progress-moves table as journal to replay, or you can simply deref block addresses as usual and on checksum mismatch check if you read a forwarding entry or else check the in-progress-moves table.
For example, an indirect block should be not an array of zfs_blkptr_t but two arrays, one of logical block pointers (just a checksum and misc metadata), and one of physical locations corresponding to blocks referenced by the first array entries. When computing the checksum of an indirect block, only the array of logical block pointers would be checksummed, thus the Merkle hash tree would never bind physical addresses. The same would apply to znodes, since they contain some block pointers, which would then have three parts: non-blockpointer metadata, an array of logical block pointers, and an array of physical block pointers.
The main issue with such a design now is that it's much too hard to retrofit it into ZFS. It would have to be a new filesystem.
- raattgift 9y ago> Illumos has a way to expand pools ... ZFS creates a vdev inside the new larger vdev Huh? > IDK if that's in OpenZFS yet. The openzfs tree (on github) is virtually identical to illumos-gate (on github). > physical block addresses should never have been part of the ZFS Merkle hash tree, thus allowing physical addresses to change without having to rewrite every block from the root down. mahrens deals with this (and block pointer rewriting) here: https://www.youtube.com/watch?v=G2vIdPmsnTI#t=44m53s https://www.youtube.com/watch?v=G2vIdPmsnTI#t=44m53s Even with SSDs IOPS are precious. On rotating media, burning track-to-track seeks in reading and updating a large hash table is a bad plan (cf. the deduplication table).