7 ms·

Relevant bits: "Don't use ZFS. It's that simple. It was always more of a buzzword than anything else, I feel, and the licensing issues just make it a non-start

Relevant bits: "Don't use ZFS. It's that simple. It was always more of a buzzword than anything else, I feel, and the licensing issues just make it a non-starter for me. The benchmarks I've seen do not make ZFS look all that great. And as far as I can tell, it has no real maintenance behind it either any more, so from a long-term stability standpoint, why would you ever want to use it in the first place?"

rleigh 7y ago
Not sure where that belief comes from. But it might be that many benchmarks are naive and compare it against other filesystems in single-disc setups with zero tuning. Since its metadata overheads are higher, it's definitely slower in this scenario. However, put a pool onto an array of discs and tune it a little, and the performance scales up and up leaving all Linux-native filesystems, and LVM/dm/mdraid, well behind. It's a shame that Linux has nothing compelling to do better than this.
- Jonnax 7y ago
  Have you seen any benchmarks for the scenario you've described?
- voltagex_ 7y ago
  Have you got any info on how to do the required tuning that's geared towards a home NAS?
  - HorstG 7y ago
    Group your disks in bunches of 4 or 5 per Raidz, no more. And have them on the same controller or SAS-expander per bunch. Use striping over the bunches. Don't use hotspares, for performance maybe avoid RAIDz6. Try out and benchmark a lot. Get more RAM, lots more RAM.
    - timc3 7y ago
      Back when I setup my last ZFS running on OmniOS 5 disks was not optimal, though i am running RAIDZ2 But yes, lots of RAM
      - HorstG 7y ago
        I think the optimal number of RAIDz5 disks is 3, if you just want performance. But this wastes lots of space of course. Also, the number of SAS/SATA-channels per controller and the topology of expanders is important. Thats why I don't think there is a recipy, you have to try it out for each new kind of hardware. And as another thread pointed out, stripe size is also an important parameter.
        sokoloff 7y ago
        I think you mean RAIDZ1, not 5.
        HorstG 7y ago
        Yes. RAIDz (without the "1") was the original RAID5-equivalent, RAIDz2 is equivalent to RAID6. However since nobody really knows what the hell z1 and z2 is and z1 is easy to mix up with RAID1 for nonZFS people, calling it z5 and z6 is far less confusing.
        chungy 7y ago
        It's the number of parity disks, pretty simple. There has been occasional talk of making the number arbitrary, though presently only raidz1, raidz2, and raidz3 exist.
- montjoy 7y ago
  Last time I used ZFS write performance was terrible compared to an ordinary RAID5. IIRC Writes in a raidz are always limited to a single disk’s performance. The only way to get better write speed is to combine multiple raidzs - which means you need a boatload if disks.
  - HorstG 7y ago
    We had a bunch of Thumpers (SunFire X4200) with 48 disks at work, running ZFS on Solaris. It was dog slow and awful, tuning performance was complicated and took ages. One had to use just the right disks in just the right order in RaidZs with striping over them. Swap in a hotspare: things slow to a crawl (i.e. not even Gbit/s). After EoL a colleague installed Linux with dmraid, LVM and xfs on the same hardware: much faster, more robust. Sorry, don't have numbers around anymore, stuff has been trashed since. Oh, and btw., snapshots and larger numbers of filesystems (which Sun recommended instead of the missing Quota support) also slow things down to a crawl. ZFS is nice on paper and maybe nice to play with. Definitely simpler to use than anything else. But performance-wise it sucked big time, at least on Solaris.
    - Annatar 7y ago
      "After EoL a colleague installed Linux with dmraid, LVM and xfs on the same hardware: much faster, more robust." Please let me know which company this is, so I can ensure that I never end up working there by accident. Much obliged in advance, thank you kindly.
      - HorstG 7y ago
        Why? What is bad about playing around with leftover hardware?
        Annatar 7y ago
        Nothing at all; it's what was done to that hardware that's the travesty here. It takes an extraordinary level of incompetence and ignorance to even get the idea to slap Linux with dmraid and LVM on that hardware and then claim that it was faster and more robust without understanding how unreliable and fragile that constelation is and that it was faster because all the reliability was gone.
usefulcat 7y ago
I think speed is not the primary reason many (most?) people use ZFS; I think it's mostly about stability, reliability and maintainability.
Seenso 7y ago
> The benchmarks I've seen do not make ZFS look all that great. The thing about ZFS that actually appeals to me is how much error-checking it does. Checksums/hashes are kept of both data and metadata, and those checksums are regularly checked to detect and fix corruption. As far as I know it (and filesystems with similar architectures) are the only ones that can actually protect against bit rot. https://github.com/zfsonlinux/zfs/wiki/Checksums https://github.com/zfsonlinux/zfs/wiki/Checksums > And as far as I can tell, it has no real maintenance behind it either any more, so from a long-term stability standpoint, why would you ever want to use it in the first place?" It has as much maintenance as any open source project: http://open-zfs.org/ http://open-zfs.org/. IIRC, it has more development momentum behind it than the competing btrfs project.
- cogman10 7y ago
  > those checksums are regularly checked to detect and fix corruption. I don't believe that's true. They are checked on access, but if left alone, nothing will verify them. From what I've read, you need to setup a cron job that runs scrubbing on some regular schedule.
  - secabeen 7y ago
    Yes. Those cron jobs are installed by default by all major vendors that supply/support ZFS.
  - deadbunny 7y ago
    The setup instructions for ZFS always include the "how to setup regular scrubs" step.
__jal 7y ago
Linus is just wrong as far as maintenance, as a look at the linux-zfs lists would show. From my perspective, it has no real competitor under linux, which is why I use it. I don't consider brtfs mature enough for critical data. (Others can reasonably disagree, I have intentionally high standards for data durability.) Aside from legal issues, he's talking out of his ass.
- pnako 7y ago
  I don't care about my data, so I use ext4, and like most non-ZFS peasants I lose files every other day.
  - Erlich_Bachman 7y ago
    Bitrot is a real thing and deduplication is actually very useful for many usecases, so your sarcasm is ill-advised. ZFS has legitimate useful features that ext4 does not.

rleigh 7y ago

Not sure where that belief comes from. But it might be that many benchmarks are naive and compare it against other filesystems in single-disc setups with zero tuning. Since its metadata overheads are higher, it's definitely slower in this scenario. However, put a pool onto an array of discs and tune it a little, and the performance scales up and up leaving all Linux-native filesystems, and LVM/dm/mdraid, well behind. It's a shame that Linux has nothing compelling to do better than this.

Jonnax 7y ago
Have you seen any benchmarks for the scenario you've described?
voltagex_ 7y ago
Have you got any info on how to do the required tuning that's geared towards a home NAS?
- HorstG 7y ago
  Group your disks in bunches of 4 or 5 per Raidz, no more. And have them on the same controller or SAS-expander per bunch. Use striping over the bunches. Don't use hotspares, for performance maybe avoid RAIDz6. Try out and benchmark a lot. Get more RAM, lots more RAM.
  - timc3 7y ago
    Back when I setup my last ZFS running on OmniOS 5 disks was not optimal, though i am running RAIDZ2 But yes, lots of RAM
    - HorstG 7y ago
      I think the optimal number of RAIDz5 disks is 3, if you just want performance. But this wastes lots of space of course. Also, the number of SAS/SATA-channels per controller and the topology of expanders is important. Thats why I don't think there is a recipy, you have to try it out for each new kind of hardware. And as another thread pointed out, stripe size is also an important parameter.
      - sokoloff 7y ago
        I think you mean RAIDZ1, not 5.
        HorstG 7y ago
        Yes. RAIDz (without the "1") was the original RAID5-equivalent, RAIDz2 is equivalent to RAID6. However since nobody really knows what the hell z1 and z2 is and z1 is easy to mix up with RAID1 for nonZFS people, calling it z5 and z6 is far less confusing.
        chungy 7y ago
        It's the number of parity disks, pretty simple. There has been occasional talk of making the number arbitrary, though presently only raidz1, raidz2, and raidz3 exist.
montjoy 7y ago
Last time I used ZFS write performance was terrible compared to an ordinary RAID5. IIRC Writes in a raidz are always limited to a single disk’s performance. The only way to get better write speed is to combine multiple raidzs - which means you need a boatload if disks.
- HorstG 7y ago
  We had a bunch of Thumpers (SunFire X4200) with 48 disks at work, running ZFS on Solaris. It was dog slow and awful, tuning performance was complicated and took ages. One had to use just the right disks in just the right order in RaidZs with striping over them. Swap in a hotspare: things slow to a crawl (i.e. not even Gbit/s). After EoL a colleague installed Linux with dmraid, LVM and xfs on the same hardware: much faster, more robust. Sorry, don't have numbers around anymore, stuff has been trashed since. Oh, and btw., snapshots and larger numbers of filesystems (which Sun recommended instead of the missing Quota support) also slow things down to a crawl. ZFS is nice on paper and maybe nice to play with. Definitely simpler to use than anything else. But performance-wise it sucked big time, at least on Solaris.
  - Annatar 7y ago
    "After EoL a colleague installed Linux with dmraid, LVM and xfs on the same hardware: much faster, more robust." Please let me know which company this is, so I can ensure that I never end up working there by accident. Much obliged in advance, thank you kindly.
    - HorstG 7y ago
      Why? What is bad about playing around with leftover hardware?
      - Annatar 7y ago
        Nothing at all; it's what was done to that hardware that's the travesty here. It takes an extraordinary level of incompetence and ignorance to even get the idea to slap Linux with dmraid and LVM on that hardware and then claim that it was faster and more robust without understanding how unreliable and fragile that constelation is and that it was faster because all the reliability was gone.

usefulcat 7y ago

I think speed is not the primary reason many (most?) people use ZFS; I think it's mostly about stability, reliability and maintainability.

Seenso 7y ago

> The benchmarks I've seen do not make ZFS look all that great. The thing about ZFS that actually appeals to me is how much error-checking it does. Checksums/hashes are kept of both data and metadata, and those checksums are regularly checked to detect and fix corruption. As far as I know it (and filesystems with similar architectures) are the only ones that can actually protect against bit rot. https://github.com/zfsonlinux/zfs/wiki/Checksums https://github.com/zfsonlinux/zfs/wiki/Checksums > And as far as I can tell, it has no real maintenance behind it either any more, so from a long-term stability standpoint, why would you ever want to use it in the first place?" It has as much maintenance as any open source project: http://open-zfs.org/ http://open-zfs.org/. IIRC, it has more development momentum behind it than the competing btrfs project.

cogman10 7y ago
> those checksums are regularly checked to detect and fix corruption. I don't believe that's true. They are checked on access, but if left alone, nothing will verify them. From what I've read, you need to setup a cron job that runs scrubbing on some regular schedule.
- secabeen 7y ago
  Yes. Those cron jobs are installed by default by all major vendors that supply/support ZFS.
- deadbunny 7y ago
  The setup instructions for ZFS always include the "how to setup regular scrubs" step.

__jal 7y ago

Linus is just wrong as far as maintenance, as a look at the linux-zfs lists would show. From my perspective, it has no real competitor under linux, which is why I use it. I don't consider brtfs mature enough for critical data. (Others can reasonably disagree, I have intentionally high standards for data durability.) Aside from legal issues, he's talking out of his ass.

pnako 7y ago
I don't care about my data, so I use ext4, and like most non-ZFS peasants I lose files every other day.
- Erlich_Bachman 7y ago
  Bitrot is a real thing and deduplication is actually very useful for many usecases, so your sarcasm is ill-advised. ZFS has legitimate useful features that ext4 does not.