4 ms·

FWIW, we (rsync.net) defrag our zpools. But, of course, there is no defrag for ZFS and so we do it the trailer-park way - we add free space to a zpool when add

by rsync 4y ago

FWIW, we (rsync.net) defrag our zpools. But, of course, there is no defrag for ZFS and so we do it the trailer-park way - we add free space to a zpool when adding a vdev, and then we 'zfs send' datasets to ourselves on that same zpool thus allowing ZFS to lay down those bytes in an efficient and orderly fashion ... as opposed to the inefficient way they were laid down over time. This works.

fire 4y ago
do you have more info on this? I'd love to read a blog post about it. Are there any performance implications to doing ( or not doing ) this?
- rsync 4y ago
  We've started publishing "technical notes" - some of which are always ZFS related: https://rsync.net/resources/notes/ https://rsync.net/resources/notes/ ... we've not disussed this item yet but I am finalizing the most recent "notes" and will include it. If you don't have busy filesystems filling beyond ~90% (used to be 80%...) then I wouldn't ever worry about this. OTOH, if you have very busy filesystems with lots of different consumers and hundreds of millions (or billions) of inodes then a recent dataset might be scattered all over your zpool. There are absolutely performance ramifications to this. So you expand that zpool (you have to expand it for this to work) by adding a vdev, and then you 'zfs send' that dataset onto the same zpool and it will be laid down nice and orderly onto the new free space that was added to the zpool. That dataset will then be more performant. This is analogous to "defrag" as we think of it but, again, in a duct-taped-engineering kind of way.
  - kernelbugs 4y ago
    Is there a way to subscribe to these technical notes postings? I (and I assume others) would be quite interested in being alerted when a new one is posted!
    - rsync 4y ago
      I think the best method would be to follow us (@rsyncnet) on twitter - the publication of these tech notes is one of the very few things we post on that platform ...
matheusmoreira 4y ago
> there is no defrag for ZFS Why not? Seems like an important thing to have.
- cryptonector 4y ago
  Because ZFS is not a content-addressed store (CAS), so block pointer (BP) rewrite is extremely expensive and has not been implemented. And defragmentation requires BP rewrite, or what GP says they do: zfs send, then swap the old and new datasets, but that can't be done atomically and transparently with a script, so that kinda sucks. The fundamental problem is that ZFS is not content addressed. It's almost CAS, but not quite. In ZFS block pointers have physical addresses in the pointer. That means that physical block locations are inexorably part of the ZFS Merkle hash tree. And that means that any change to the location of any block necessitates that the pointers to it must change and so be rewritten, but now you have to find all those pointers -- even if there can be only one pointer, you still have to change it, and when you change it, you change the Merkle hash tree of the dataset. The solution, IMO, is to split all znodes and any interior nodes that have block pointers into two halves. One half should have only logical block pointers free of any physical location pointers, thus the pointers in this half should be some metadata + hashes of the pointed-to blocks. The second half should be a "cache" of physical locations corresponding to the logical block pointers in the first half, plus a hash of just those physical locations. The key is that the Merkle hash tree should not bind physical locations so that changing those locations does not alter the Merkle hash tree. That leaves only the task of updating those second block halves that carry physical locations. So when traversing the tree the filesystem could optimistically read pointed-to blocks from the cached locations, and if the hash of the block read does not match the logical pointer, then go looking in a log of recent block moves. This way a BP rewrite system could simply traverse a dataset looking for blocks to move, copy the blocks to new locations, update the cached locations where the block pointer was found, log the move, then add the old location to the freelist. But ZFS is not like that. ZFS is not a content-addressed store, but it's a Merkle hash tree all the same. That makes BP rewrite insanely too difficult. I do wonder how much effort it would take to make a ZZFS that is based on ZFS but with the CAS redesign sketched above. This is not the first time I've written about this. I think I first proposed this back in the early days of Illumos, and one or two people thought that not hashing the physical locations into the Merkle hash tree would be folly -- I think they're wrong, because either you trust your hash function or you don't. Granted, to make ZFS into a proper CAS does require a strong cryptographic hash function, but ZFS uses one so...
tjoff 4y ago
This requires you to have at least 50% free space, right? At which point fragmentation usually isn't that big of a deal. But sure, if you add enough new storage it works. It's more of a problem for individuals. That and that you can't increase the size of a vdev is (in the works but with lots of caveats) really dampens my enthusiasm as a hobbyist.
- cryptonector 4y ago
  You can add a drive, do vdev evacuation of one to another, to grow your pool.
  - tjoff 4y ago
    But not when having any form of parity? Doesn't seem particularly useful if so.

by rsync 4y ago

fire 4y ago

do you have more info on this? I'd love to read a blog post about it. Are there any performance implications to doing ( or not doing ) this?

rsync 4y ago
We've started publishing "technical notes" - some of which are always ZFS related: https://rsync.net/resources/notes/ https://rsync.net/resources/notes/ ... we've not disussed this item yet but I am finalizing the most recent "notes" and will include it. If you don't have busy filesystems filling beyond ~90% (used to be 80%...) then I wouldn't ever worry about this. OTOH, if you have very busy filesystems with lots of different consumers and hundreds of millions (or billions) of inodes then a recent dataset might be scattered all over your zpool. There are absolutely performance ramifications to this. So you expand that zpool (you have to expand it for this to work) by adding a vdev, and then you 'zfs send' that dataset onto the same zpool and it will be laid down nice and orderly onto the new free space that was added to the zpool. That dataset will then be more performant. This is analogous to "defrag" as we think of it but, again, in a duct-taped-engineering kind of way.
- kernelbugs 4y ago
  Is there a way to subscribe to these technical notes postings? I (and I assume others) would be quite interested in being alerted when a new one is posted!
  - rsync 4y ago
    I think the best method would be to follow us (@rsyncnet) on twitter - the publication of these tech notes is one of the very few things we post on that platform ...

matheusmoreira 4y ago

> there is no defrag for ZFS Why not? Seems like an important thing to have.

cryptonector 4y ago
Because ZFS is not a content-addressed store (CAS), so block pointer (BP) rewrite is extremely expensive and has not been implemented. And defragmentation requires BP rewrite, or what GP says they do: zfs send, then swap the old and new datasets, but that can't be done atomically and transparently with a script, so that kinda sucks. The fundamental problem is that ZFS is not content addressed. It's almost CAS, but not quite. In ZFS block pointers have physical addresses in the pointer. That means that physical block locations are inexorably part of the ZFS Merkle hash tree. And that means that any change to the location of any block necessitates that the pointers to it must change and so be rewritten, but now you have to find all those pointers -- even if there can be only one pointer, you still have to change it, and when you change it, you change the Merkle hash tree of the dataset. The solution, IMO, is to split all znodes and any interior nodes that have block pointers into two halves. One half should have only logical block pointers free of any physical location pointers, thus the pointers in this half should be some metadata + hashes of the pointed-to blocks. The second half should be a "cache" of physical locations corresponding to the logical block pointers in the first half, plus a hash of just those physical locations. The key is that the Merkle hash tree should not bind physical locations so that changing those locations does not alter the Merkle hash tree. That leaves only the task of updating those second block halves that carry physical locations. So when traversing the tree the filesystem could optimistically read pointed-to blocks from the cached locations, and if the hash of the block read does not match the logical pointer, then go looking in a log of recent block moves. This way a BP rewrite system could simply traverse a dataset looking for blocks to move, copy the blocks to new locations, update the cached locations where the block pointer was found, log the move, then add the old location to the freelist. But ZFS is not like that. ZFS is not a content-addressed store, but it's a Merkle hash tree all the same. That makes BP rewrite insanely too difficult. I do wonder how much effort it would take to make a ZZFS that is based on ZFS but with the CAS redesign sketched above. This is not the first time I've written about this. I think I first proposed this back in the early days of Illumos, and one or two people thought that not hashing the physical locations into the Merkle hash tree would be folly -- I think they're wrong, because either you trust your hash function or you don't. Granted, to make ZFS into a proper CAS does require a strong cryptographic hash function, but ZFS uses one so...

tjoff 4y ago

This requires you to have at least 50% free space, right? At which point fragmentation usually isn't that big of a deal. But sure, if you add enough new storage it works. It's more of a problem for individuals. That and that you can't increase the size of a vdev is (in the works but with lots of caveats) really dampens my enthusiasm as a hobbyist.

cryptonector 4y ago
You can add a drive, do vdev evacuation of one to another, to grow your pool.
- tjoff 4y ago
  But not when having any form of parity? Doesn't seem particularly useful if so.