3 ms·

I’m running an 8-drive ZFS RAIDZ2 pool. I’m wondering if you know — are the free space recommendations around ZFS cargo or real? Like I’m already giving up tw

I’m running an 8-drive ZFS RAIDZ2 pool. I’m wondering if you know — are the free space recommendations around ZFS cargo or real? Like I’m already giving up two full drives for redundancy (which saved my ass - I recently had two drives fail on me in quick succession — both SSDs from what looks like an identical batch) but then the advice is kinda saying I need to keep at least another drive worth of space free for the pool to perform well and not crap itself. That hurts with current prices for sure.

gpt5 3mo ago
ZFS will auto-degrade performance if there isn't enough headroom. In addition, if you use SSD, you also want the headroom, because otherwise you end up writing and rewriting on the same small empty space which kills SSD (unless you have enterprise SSD, which have built-in headroom).
- gpt5 3mo ago
  Not sure why I'm being downvoted for being factually correct (and also, why I can't edit my above comment). ZFS will definitely degrade write performance gradually from 90% utilization, and will hit a stronger cliff at 95%+. Same with SSD, using a consumer SSD above 90%+ utilization would rapidly degrade its lifetime. The effect will be smaller for very large pools and very large files, but the effect is stil there. I'm repeating this so that people who set up these drives know what is actually going to happen.
  - 6581 3mo ago
    > Not sure why I'm being downvoted for being factually correct Maybe because your statement about SSDs is incorrect. Consumer SSDs have spare blocks too, and wear leveling prevents the scenario you're describing.
    - gpt5 3mo ago
      I appreciate the reply, but my warning consumer SSD is not incorrect. Perhaps I should have clarified that While they typically have a small overprovisioning of 7%, enterprise SSDs have close to 4 times as much, exactly to allow more writes (which is very relevant if running a ZFS raid on the drive). Keeping 10-20% headroom will prolong your SSD's life significantly, and increase performance of ZFS.
      - seniorThrowaway 3mo ago
        Go ahead and let them try consumer SSD's. It seems to be a lesson everyone has to learn with ZFS once.
Confiks 3mo ago
> I’m running an 8-drive ZFS RAIDZ2 pool. I’m wondering if you know — are the free space recommendations around ZFS cargo or real? I'm not entirely sure, but it seems to me that free space (and the 20% reservation) is mostly a proxy for fragmentation, and you can therefore better look at fragmentation directly. That would mean that if you mostly store large files, there shouldn't be a lot of fragmentation even at high utilization. The whole "ZFS changes allocation algorithm from 80% usage on" is something of 10+ years distant past, and lots of things around the allocator have been improved. It's also something that probably isn't too different from the performance of other filesystems at high utilization, so it shouldn't be exaggerated.
- Dylan16807 3mo ago
  > It's also something that probably isn't too different from the performance of other filesystems at high utilization, so it shouldn't be exaggerated. Well, other filesystems can defragment.
vladvasiliu 3mo ago
I don't think they are. I've already filled a ZFS pool almost to the brim with no adverse effects.
- barrkel 3mo ago
  My experience has been escalating fragmentation past 90%, along with lower performance owing to the fragmentation.
cm2187 3mo ago
For home usage, if you have backups raidz1 is fine (just do an incremental backup at the first sign of trouble). If you don’t have backups, then you probably shouldn’t be running a NAS in the first place.
- layer8 3mo ago
  This isn’t black and white. You might have huge amounts of non-essential data that aren’t worth the cost of off-site backups, but worth the cost of an extra disk of redundancy to lessen the risk. Even when you do have backups, it will reduce the risk of extended downtime (and possibly egress costs) caused by having to restore large amounts of data from backups.
- bluGill 3mo ago
  For most people snapshots are enough backup. It is still useful to have an offsite backup, but realistically fire is rare enough that you can risk it, and that is about the only risk most people have (you can have your NAS in a location likely far from where a fire might break out to reduce the risk farther).
  - cm2187 3mo ago
    Fire, malware, accidental deletions, capricious RAID controller (pre-ZFS). And that’s only the stuff that happened to me. Add power surge, theft, correlation in SSD failures (eg power on counter overflow firmware bug), damaging the array while moving, etc.
    - bluGill 3mo ago
      Malware, accidental deletion, RAID controller issues - all things ZFS with snapshots are immune to. You can decide if the other worry you or not - the risk is not zero, but it may be acceptable. Your backups are all subject to similar issues.
      - cm2187 3mo ago
        Snapshots don’t protect you from malware. I bet 99% of home users use the same credentials on all their machines, once a malware compromised one, the others are compromised within seconds. [edit] also snapshots aren’t really workable for large files. Remux a movie file and now it occupies twice the space.

gpt5 3mo ago

ZFS will auto-degrade performance if there isn't enough headroom. In addition, if you use SSD, you also want the headroom, because otherwise you end up writing and rewriting on the same small empty space which kills SSD (unless you have enterprise SSD, which have built-in headroom).

gpt5 3mo ago
Not sure why I'm being downvoted for being factually correct (and also, why I can't edit my above comment). ZFS will definitely degrade write performance gradually from 90% utilization, and will hit a stronger cliff at 95%+. Same with SSD, using a consumer SSD above 90%+ utilization would rapidly degrade its lifetime. The effect will be smaller for very large pools and very large files, but the effect is stil there. I'm repeating this so that people who set up these drives know what is actually going to happen.
- 6581 3mo ago
  > Not sure why I'm being downvoted for being factually correct Maybe because your statement about SSDs is incorrect. Consumer SSDs have spare blocks too, and wear leveling prevents the scenario you're describing.
  - gpt5 3mo ago
    I appreciate the reply, but my warning consumer SSD is not incorrect. Perhaps I should have clarified that While they typically have a small overprovisioning of 7%, enterprise SSDs have close to 4 times as much, exactly to allow more writes (which is very relevant if running a ZFS raid on the drive). Keeping 10-20% headroom will prolong your SSD's life significantly, and increase performance of ZFS.
    - seniorThrowaway 3mo ago
      Go ahead and let them try consumer SSD's. It seems to be a lesson everyone has to learn with ZFS once.

Confiks 3mo ago

> I’m running an 8-drive ZFS RAIDZ2 pool. I’m wondering if you know — are the free space recommendations around ZFS cargo or real? I'm not entirely sure, but it seems to me that free space (and the 20% reservation) is mostly a proxy for fragmentation, and you can therefore better look at fragmentation directly. That would mean that if you mostly store large files, there shouldn't be a lot of fragmentation even at high utilization. The whole "ZFS changes allocation algorithm from 80% usage on" is something of 10+ years distant past, and lots of things around the allocator have been improved. It's also something that probably isn't too different from the performance of other filesystems at high utilization, so it shouldn't be exaggerated.

Dylan16807 3mo ago
> It's also something that probably isn't too different from the performance of other filesystems at high utilization, so it shouldn't be exaggerated. Well, other filesystems can defragment.

vladvasiliu 3mo ago

I don't think they are. I've already filled a ZFS pool almost to the brim with no adverse effects.

barrkel 3mo ago
My experience has been escalating fragmentation past 90%, along with lower performance owing to the fragmentation.

cm2187 3mo ago

For home usage, if you have backups raidz1 is fine (just do an incremental backup at the first sign of trouble). If you don’t have backups, then you probably shouldn’t be running a NAS in the first place.

layer8 3mo ago
This isn’t black and white. You might have huge amounts of non-essential data that aren’t worth the cost of off-site backups, but worth the cost of an extra disk of redundancy to lessen the risk. Even when you do have backups, it will reduce the risk of extended downtime (and possibly egress costs) caused by having to restore large amounts of data from backups.
bluGill 3mo ago
For most people snapshots are enough backup. It is still useful to have an offsite backup, but realistically fire is rare enough that you can risk it, and that is about the only risk most people have (you can have your NAS in a location likely far from where a fire might break out to reduce the risk farther).
- cm2187 3mo ago
  Fire, malware, accidental deletions, capricious RAID controller (pre-ZFS). And that’s only the stuff that happened to me. Add power surge, theft, correlation in SSD failures (eg power on counter overflow firmware bug), damaging the array while moving, etc.
  - bluGill 3mo ago
    Malware, accidental deletion, RAID controller issues - all things ZFS with snapshots are immune to. You can decide if the other worry you or not - the risk is not zero, but it may be acceptable. Your backups are all subject to similar issues.
    - cm2187 3mo ago
      Snapshots don’t protect you from malware. I bet 99% of home users use the same credentials on all their machines, once a malware compromised one, the others are compromised within seconds. [edit] also snapshots aren’t really workable for large files. Remux a movie file and now it occupies twice the space.