5 ms·

We run ZFS at my company, Datto, across hundreds of machines, including devices out in the field and servers in our data centers. It has a few warts, but overal

by wspeirs 10y ago

We run ZFS at my company, Datto, across hundreds of machines, including devices out in the field and servers in our data centers. It has a few warts, but overall very stable and the feature set has allowed us to deliver world-class features to our customers.

NovaS1X 10y ago
What are the few warts you encounter? I'm a Sysadmin/Engineer at a VFX company. We have hundreds of machines, render, and servers all running Linux but we're typically on ext4 with the notable exception of our Isilon cluster. What does your environment look like?
- e40 10y ago
  One that I've heard often repeated (paging user rsync) is that you should never fill your filesystems past 80% (or so). Things get slow otherwise.
  - oofabz 10y ago
    ZFS certainly slows down as it fills up, but only enough to affect high-IO, write-intensive loads like databases. A light-weight application like a home or office file server can still saturate gigabit ethernet even while 99% full.
  - NovaS1X 10y ago
    This is also true of other filesystems however. I don't have experience with high IO on ZFS at those rumored 80%+ capacities however once a core filer fills up to 90% you start seeing the tickets rolling in and application performance tank. 80% if usually fine however. I wonder to what degree the performance degradation is on OpenZFS.
    - throwaway7767 10y ago
      The problem is much more severe on ZFS and other CoW filesystems, hence the common warning. Filling an ext4 filesystem up to 80%+ and then removing files might increase fragmentation a bit but it'll not have that much effect on performance. On ZFS, filling the fileystem up once can kill performance permanently on that pool.
  - Alupis 10y ago
    > you should never fill your filesystems past 80% (or so). Things get slow otherwise. That's a ZFS-in-general complaint, not a ZFS-on-Linux complaint. However, it's worth mentioning most (all?) filesystems drop sharply in performance as it nears capacity.
  - rsync 10y ago
    "One that I've heard often repeated (paging user rsync) is that you should never fill your filesystems past 80% (or so). Things get slow otherwise." Hi. I've said a lot about this in HN comments, so I encourage you, future reader, to search for that. It is indeed the case. Recently it has been suggested that the presence of a fast write cache (a SLOG, in ZFS parlance) minimizes this problem and allows you to run up to, and around, 90% without breaking the filesystem. We haven't tested this, intentionally or otherwise, but it sort of makes sense ... Make no mistake, however - if you fill up a ZFS filesystem and run it for a while in that state, it will be permanently broken, at least in terms of performance. A ZFS defrag utility would solve this problem, or at least provide a way out if you fall into this trap, but it has been related to me that ZFS defrag would be extremely complicated to implement. Yes, we do indeed see this on other filesystems ... even a UFS2 filesystem with NO snapshots enabled can be effectively "broken" is you set minfree down to 0%, fill it all the way up, and then run it like that for a while. Freeing up the space and resetting minfree back to 6 or 8%, etc., doesn't fix it.
  - Annatar 10y ago
    Yes, because after 80% ZFS will consume CPU(s) in an attempt to find enough contiguous space for the new branch of blocks in order to prevent fragmentation. The fix is to either add more vdevs or to change the kernel tunable which controls the free contiguous space allocation algorithm. The latter approach is very likely to cause fragmentation where there would otherwise be none. As I don't remember the name of the tunable, you can search my comments and in one of them on the subject there will be a link to the procedure. However, I strongly recommend adding more logical units or physical disks as vdevs to the zpool instead.
- wspeirs 10y ago
  As a few others mentioned, the 80% issue is one. We try and keep all machines below that so performance doesn't degrade. With the number of drives we have (200+ PB total space) we also run into issues where there are "holes" in the ZFS array after replacing a disk. We've always had enough redundancy work around any issues we find. Overall, I cannot speak highly enough of ZFS!
tobias3 10y ago
But you are only using the zpool part (zvol), not the zfs part. Correct?

by wspeirs 10y ago

NovaS1X 10y ago

What are the few warts you encounter? I'm a Sysadmin/Engineer at a VFX company. We have hundreds of machines, render, and servers all running Linux but we're typically on ext4 with the notable exception of our Isilon cluster. What does your environment look like?

e40 10y ago
One that I've heard often repeated (paging user rsync) is that you should never fill your filesystems past 80% (or so). Things get slow otherwise.
- oofabz 10y ago
  ZFS certainly slows down as it fills up, but only enough to affect high-IO, write-intensive loads like databases. A light-weight application like a home or office file server can still saturate gigabit ethernet even while 99% full.
- NovaS1X 10y ago
  This is also true of other filesystems however. I don't have experience with high IO on ZFS at those rumored 80%+ capacities however once a core filer fills up to 90% you start seeing the tickets rolling in and application performance tank. 80% if usually fine however. I wonder to what degree the performance degradation is on OpenZFS.
  - throwaway7767 10y ago
    The problem is much more severe on ZFS and other CoW filesystems, hence the common warning. Filling an ext4 filesystem up to 80%+ and then removing files might increase fragmentation a bit but it'll not have that much effect on performance. On ZFS, filling the fileystem up once can kill performance permanently on that pool.
- Alupis 10y ago
  > you should never fill your filesystems past 80% (or so). Things get slow otherwise. That's a ZFS-in-general complaint, not a ZFS-on-Linux complaint. However, it's worth mentioning most (all?) filesystems drop sharply in performance as it nears capacity.
- rsync 10y ago
  "One that I've heard often repeated (paging user rsync) is that you should never fill your filesystems past 80% (or so). Things get slow otherwise." Hi. I've said a lot about this in HN comments, so I encourage you, future reader, to search for that. It is indeed the case. Recently it has been suggested that the presence of a fast write cache (a SLOG, in ZFS parlance) minimizes this problem and allows you to run up to, and around, 90% without breaking the filesystem. We haven't tested this, intentionally or otherwise, but it sort of makes sense ... Make no mistake, however - if you fill up a ZFS filesystem and run it for a while in that state, it will be permanently broken, at least in terms of performance. A ZFS defrag utility would solve this problem, or at least provide a way out if you fall into this trap, but it has been related to me that ZFS defrag would be extremely complicated to implement. Yes, we do indeed see this on other filesystems ... even a UFS2 filesystem with NO snapshots enabled can be effectively "broken" is you set minfree down to 0%, fill it all the way up, and then run it like that for a while. Freeing up the space and resetting minfree back to 6 or 8%, etc., doesn't fix it.
- Annatar 10y ago
  Yes, because after 80% ZFS will consume CPU(s) in an attempt to find enough contiguous space for the new branch of blocks in order to prevent fragmentation. The fix is to either add more vdevs or to change the kernel tunable which controls the free contiguous space allocation algorithm. The latter approach is very likely to cause fragmentation where there would otherwise be none. As I don't remember the name of the tunable, you can search my comments and in one of them on the subject there will be a link to the procedure. However, I strongly recommend adding more logical units or physical disks as vdevs to the zpool instead.
wspeirs 10y ago
As a few others mentioned, the 80% issue is one. We try and keep all machines below that so performance doesn't degrade. With the number of drives we have (200+ PB total space) we also run into issues where there are "holes" in the ZFS array after replacing a disk. We've always had enough redundancy work around any issues we find. Overall, I cannot speak highly enough of ZFS!

tobias3 10y ago

But you are only using the zpool part (zvol), not the zfs part. Correct?