7 ms·
"To be fair, here's some issues with ZFS:" Let me add: 3) Do not allow a pool to exceed 90% capacity ... and probably don't let it exceed 85%. ZFS does not h
by rsync 11y ago
"To be fair, here's some issues with ZFS:"
Let me add:
3) Do not allow a pool to exceed 90% capacity ... and probably don't let it exceed 85%.
ZFS does not have a defrag utility and it badly needs one. You can permanently wreck zpool performance by running it up past 90% capacity - even if you later reduce capacity back down to 75-80%. You can sort of fix it if you add additional top level vdevs to the zpool, thus farming out some IO to the new set of disks, but it's still going to be performance constrained forever. The only solution is to create a new pool and export the data to it.
This is unacceptable, by the way.
It is not at all reasonable to require a filesystem to stay below 80% capacity (our target "full" number at rsync.net) nor is it acceptable that hitting 90% is a (performance) death sentence.
When you consider that you might have already sacrificed 25% of your physical disk just for the raidz2/raidz3 overhead, being constrained to 80% means you're only using 60% or so of your physical disks that you bought.
ZFS needs defrag. Badly.
- ryao 11y agoIf gang blocks are generated, you get more I/Os than necessary and an extra level of indirection, but the ZFS code base tries very hard to avoid that situation by switching to the allocator behavior at the metaslab level to best fit, such that most data written to the pool would not have gang blocks used at all. This is understandably terrible for IOPS, but it is likely the source of the performance degradation that you saw. You can probe for the zio_gang_* functions during a scrub to see if any gang blocks exist. On my pool that has exceeded 90% on multiple ocasioks, there are zero gang blocks and consequently, no permanent degradation from them. The only other problem that you might have (which tends to be caused by the order of writes rather than the fullness of the pool) is lower sequential performance from nonlinear block placement (one ZoL user measured this as cutting sequential reads in half on a pool filled with files made by bit torrent), but that is a much less severe problem, especially on solid state drives. If you want to fix placement, you can do a file copy or send/recv. The new locations should have blocks picked in sequence whenever possible.
- l1ambda 11y agoThe Reiser guys were talking about this too. I think they needed defrag and were trying to get DARPA money to implement it, but it never happened.
- cthalupa 11y agoFor ZFS to get defrag, someone with enough ability and knowledge would need to bite the bullet and go through the fairly massive and difficult task of adding in block pointer rewrite, which doesn't seem to be something anyone has been willing to do, and I've seen a lot of concerns about the actual feasibility of it from some very smart people that are knowledgeable with the codebase.
- XorNot 11y agoWhich also incidentally would solve the pool shrinking problem as well.
- voltagex_ 11y agoI wonder how much money you'd need to pay a very talented engineer to do the work. This is yet another thing I didn't read about before building a home NAS on ZFS.
- aduitsis 11y agoMaybe off topic, but Alex Reece from Delphix had a blog post about a year ago about the ability to remove a disk from a pool. http://blog.delphix.com/alex/2015/01/15/openzfs-device-removal/ http://blog.delphix.com/alex/2015/01/15/openzfs-device-remov... This would imply some kind of rebalancing, if I understand correctly. Which maybe could be considered a form of indirect defragmentation.
- aidenn0 11y agoThat's a good point about the 90% capacity. Also the same "technically challenging" problem that needs to be solved for shrinking pools would also make defrag possible.
- binarycrusader 11y agoNot all of the performance issues encountered when a pool exceeds 90% capacity are related to fragmentation. Some implementers have been able to significantly reduce the pain of those situations.
- ryao 11y agoThe main factor seems to be the best-fit allocator, which tends to go into action sooner from metaslabs crossing the 94% threshold earlier than the pool itself and still be selected due to LBA weighting, which is a trick to increase throughput on rotational media. This ought to help prevent best-fit allocation from occurring earlier than necessary on SSDs: https://github.com/zfsonlinux/zfs/commit/fb40095f5f0853946f8150481ca22602d1334dfe https://github.com/zfsonlinux/zfs/commit/fb40095f5f0853946f8... That said, Delphix made changes to consider on fragmentation rather than space usage when the histogram is in use, so the performance before best-fit behavior goes into effect is better than outright selection of metaslabs by free space.
- po 11y agoI had this happen to me... I accidentally allowed a pool to fill up to capacity and couldn't do anything with it because deleting files wouldn't free space due to snapshots and the commands to delete snapshots wouldn't work. Then added a disk to it to try to recover. That worked, but only after adding a disk did I realize that I couldn't shrink the pool down again. I ended up moving the whole thing off to a new disk cluster and back again. Really painful.