4 ms·

My home server is hardly production, but it has been running a ZFS raidz+1 for more than 5 years, and survived a disk failure. (Nb, avoid SMR hard drives - the

My home server is hardly production, but it has been running a ZFS raidz+1 for more than 5 years, and survived a disk failure. (Nb, avoid SMR hard drives - the rebuild took more than a week!)

StavrosK 6y ago
Mine is RAIDZ too, and has become unusably slow since a few years back. Now I need to find a way to recreate the pool, but it's too much data to store on another disk... Not in love with ZFS so far.
- magicalhippo 6y ago
  ZFS is great for storing data. It's very unlikely it will lose your data when using ZFS. It does however require a bit of care to maintain performance, and you need to know your expected workload going in. Otherwise you can find yourself in a situation with a very poorly performing pool where the only realistic route to recovery is a send/receive to a fresh pool and back. If you do not have a lot of sync workload (VMs, DBs) and don't have super-high performance needs, say a home NAS, then mainly you just need to think about not filling up the pool too much. A nice way to do this is to create a root dataset where you set a quota to say 75% of capacity, and then create all other datasets below this one. You should not go above 85% space usage, as ZFS switches allocation strategy then to one which can significantly increase fragmentation. If you do have a lot of sync writes, a SLOG device is basically mandatory. The SLOG device does not have to be large, it only stores about 5-10 seconds worth of writes, so 10-20GB can be plenty. I've partitioned up my SSDs and created a mirror out of two small partitions, using the remaining SSD space for other things. One thing to keep in mind is that while an L2ARC device sounds like a great thing, depending on your configuration you can actually slow things down with one. A bunch of disks has a lot more bandwidth than a single SATA SSD. An L2ARC device also requires some memory overhead, so reduces your primary ARC. Again depending on load this can be detrimental. And finally, don't ever think about using deduplication, unless you've read about the consequences, measured the performance benefits and ensured the memory overhead is acceptable. It sounds great on paper but has a lot of associated downsides that can ruin pool performance, and disabling it does not make it go away. At least that's what I've picked up so far.
  - StavrosK 6y ago
    This is a home NAS with 50% free space, I rarely write stuff and don't even read much, yet it still managed to get super slow :( I don't use deduplication or compression, and the good people over at #zfs have failed to find anything wrong multiple times. I think recreating the entire thing is about my only option at the moment.
    - magicalhippo 6y ago
      That sucks, but also very weird. My pool, 2 vdevs each a 4-way RAID-Z1, has been used and abused for almost 7 years now. I've gone over the 85% mark but it still works fine, performance too. Some VM stuff but mostly media and similar. Do you mean slow IOPS or also sequential performance?
      - StavrosK 6y ago
        It's mostly sequential performance, but it's also very odd. Deleting a single 2 GB file sometimes takes tens of seconds, or copying files from one volume to another will again be extremely slow.. .
        magicalhippo 6y ago
        It does sound like fragmentation, which can still happen with a pool that's not filled under the "right" circumstances (usually related to workload vs configuration). But yeah sadly that's the one area where ZFS is less stellar. Once it's fragmented it's hard to fix. Easiest is to send/receive to another pool, but as you note that is not always feasible.
        StavrosK 6y ago
        Is there no defragmentation tool? I do have lots of free space.
        magicalhippo 6y ago

My home server is hardly production, but it has been running a ZFS raidz+1 for more than 5 years, and survived a disk failure. (Nb, avoid SMR hard drives - the rebuild took more than a week!)

StavrosK 6y ago

Mine is RAIDZ too, and has become unusably slow since a few years back. Now I need to find a way to recreate the pool, but it's too much data to store on another disk... Not in love with ZFS so far.

magicalhippo 6y ago
ZFS is great for storing data. It's very unlikely it will lose your data when using ZFS. It does however require a bit of care to maintain performance, and you need to know your expected workload going in. Otherwise you can find yourself in a situation with a very poorly performing pool where the only realistic route to recovery is a send/receive to a fresh pool and back. If you do not have a lot of sync workload (VMs, DBs) and don't have super-high performance needs, say a home NAS, then mainly you just need to think about not filling up the pool too much. A nice way to do this is to create a root dataset where you set a quota to say 75% of capacity, and then create all other datasets below this one. You should not go above 85% space usage, as ZFS switches allocation strategy then to one which can significantly increase fragmentation. If you do have a lot of sync writes, a SLOG device is basically mandatory. The SLOG device does not have to be large, it only stores about 5-10 seconds worth of writes, so 10-20GB can be plenty. I've partitioned up my SSDs and created a mirror out of two small partitions, using the remaining SSD space for other things. One thing to keep in mind is that while an L2ARC device sounds like a great thing, depending on your configuration you can actually slow things down with one. A bunch of disks has a lot more bandwidth than a single SATA SSD. An L2ARC device also requires some memory overhead, so reduces your primary ARC. Again depending on load this can be detrimental. And finally, don't ever think about using deduplication, unless you've read about the consequences, measured the performance benefits and ensured the memory overhead is acceptable. It sounds great on paper but has a lot of associated downsides that can ruin pool performance, and disabling it does not make it go away. At least that's what I've picked up so far.
- StavrosK 6y ago
  This is a home NAS with 50% free space, I rarely write stuff and don't even read much, yet it still managed to get super slow :( I don't use deduplication or compression, and the good people over at #zfs have failed to find anything wrong multiple times. I think recreating the entire thing is about my only option at the moment.
  - magicalhippo 6y ago
    That sucks, but also very weird. My pool, 2 vdevs each a 4-way RAID-Z1, has been used and abused for almost 7 years now. I've gone over the 85% mark but it still works fine, performance too. Some VM stuff but mostly media and similar. Do you mean slow IOPS or also sequential performance?
    - StavrosK 6y ago
      It's mostly sequential performance, but it's also very odd. Deleting a single 2 GB file sometimes takes tens of seconds, or copying files from one volume to another will again be extremely slow.. .
      - magicalhippo 6y ago
        It does sound like fragmentation, which can still happen with a pool that's not filled under the "right" circumstances (usually related to workload vs configuration). But yeah sadly that's the one area where ZFS is less stellar. Once it's fragmented it's hard to fix. Easiest is to send/receive to another pool, but as you note that is not always feasible.
        StavrosK 6y ago
        Is there no defragmentation tool? I do have lots of free space.
        magicalhippo 6y ago