7 ms·

ZFS is an awesome filesystem, this article gives a good overview. Very interesting for home users. I run it on my NAS. It is very important to realize that you

ZFS is an awesome filesystem, this article gives a good overview. Very interesting for home users. I run it on my NAS. It is very important to realize that you can’t expand VDEVS. You can only add VDEVS. This makes expanding storage less flexible than regular MDADM RAID. Going for Mirrors is the most flexible but you lose 50% of capacity. You also get the random IOPs performance of a single drive per VDEV. Sequential performance does scale within a VDEV. You scale random I/O performance by adding VDEVS. For home usage, as a NAS, you don’t need to use SSDs for a SLOG, unless you have write-intensive random I/O workloads.

bonestamp2 6y ago
Which OS are you using for your NAS? I ruled out Unraid and I'm looking at TrueNAS Core (FreeNAS). I want it mostly for scalable backup storage where I can easily add/remove drives from the array as I need more storage. Any others that I should consider?
- louwrentius 6y ago
  I am using Debian with ZoL, but it's now ancient. Just ZFS + NFS & SMB. https://louwrentius.com/71-tib-diy-nas-based-on-zfs-on-linux.html https://louwrentius.com/71-tib-diy-nas-based-on-zfs-on-linux... I would now go for Ubuntu + ZoL myself. I bought all capacity up front and paid the ZFS tax. So I don't need to expand as I go. But if you want to expand as you go, Linux + MDADM are still fine in my opinion. It's a tradeoff. Do you want to 'pay' the ZFS tax and expand at a cost, or do you want a bit more risk (on paper) but more flexibility? I ran a RAID6 of 20 drives before that using Linux + MDADM and that worked fine. And MDADM allows you to expand as you go, exactly as you want. I think the risks ZFS protect against are very small. https://louwrentius.com/what-home-nas-builders-should-understand-about-silent-data-corruption.html https://louwrentius.com/what-home-nas-builders-should-unders...
- tinco 6y ago
  Unraid is the only one that allows you to freely add/remove drives. And only if you're OK with taking cluster down for a minute as you add it. If you true freedom you'd need to go Ceph(FS) based, but there's no GUI to manage a Ceph cluster that I'm aware of, and they're so focused on cloud usage that you'll find little guidance on single-node clusters. edit: Forgot to read Louwrentius' comment, of course MDADM with raid 6 allows you to add disks in pairs of two, so that's definitely also an option. Don't forget to use LVM, it can get real awkward without it.
  - nix23 6y ago
    Don't forget GlusterFS, example: Every single disk is a ZFS-pool on top of that you run your GlusterFS-Cluster, GlusterFS then looks for the redundancy and underneath you can export/import your pools aka Disks and or vdevs (if you need more performance). >but there's no GUI to manage a Ceph cluster that I'm aware of There are two, Open Attic and Calamari ,but Open Attic is in maintenance mode, so all the work goes now into Ceph Dashboard: https://docs.ceph.com/en/latest/mgr/dashboard/ https://docs.ceph.com/en/latest/mgr/dashboard/
  - magicalhippo 6y ago
    > Unraid is the only one that allows you to freely add/remove drives. I think technically you can do something similar to Synology's Hybrid RAID[1] with ZFS, using partitions and creating mirror VDEVs from the partitions. But I haven't gone through all the details so could be there are some risks I overlooked and in any case you'd have to manage it from the command line so would be a bit tedious. But I did set up a proof of concept in a VM. [1]: https://nascompares.com/guide/shr-synology-hybrid-raid/ https://nascompares.com/guide/shr-synology-hybrid-raid/
MarkSweep 6y ago
Strictly speaking you can expand a VDEV. If you replace all the leaf drives in a VDEV with larger devices, you can extend the VDEV to take the extra space. There is work ongoing to allow adding drives to a RAIDZ VED[1]. And VDEV Removal gives some flexibility (at the cost of indirection on the read path). But yeah, ZFS is not super flexible for adding and removing random drives. [1]: https://github.com/openzfs/zfs/pull/8853 https://github.com/openzfs/zfs/pull/8853
- louwrentius 6y ago
  Yes that's totally true, but it is very time consuming and cumbersome in my view. I've detailed this also here. https://louwrentius.com/the-hidden-cost-of-using-zfs-for-your-home-nas.html https://louwrentius.com/the-hidden-cost-of-using-zfs-for-you... You may also realise from the top of my article that expanding VDEVS is a topic since 2017 but still no show.
  - iooi 6y ago
    I just did this operation on moving from 12x 4TB drives to 12x 16TB drives. I did it over 12 days (each resilver was ~24hours) and it wasn't too cumbersome. Considering how often you're going to grow your pool, it doesn't seem like a big drawback.
    - louwrentius 6y ago
      Fair enough
ComputerGuru 6y ago
> You also get the random IOPs performance of a single drive per VDEV This isn't strictly true. It is true for RAIDZ-n and for writing to a mirror, but ZFS can accelerate random reads with a mirror by distributing the read requests to the underlying disks.

bonestamp2 6y ago

Which OS are you using for your NAS? I ruled out Unraid and I'm looking at TrueNAS Core (FreeNAS). I want it mostly for scalable backup storage where I can easily add/remove drives from the array as I need more storage. Any others that I should consider?

louwrentius 6y ago
I am using Debian with ZoL, but it's now ancient. Just ZFS + NFS & SMB. https://louwrentius.com/71-tib-diy-nas-based-on-zfs-on-linux.html https://louwrentius.com/71-tib-diy-nas-based-on-zfs-on-linux... I would now go for Ubuntu + ZoL myself. I bought all capacity up front and paid the ZFS tax. So I don't need to expand as I go. But if you want to expand as you go, Linux + MDADM are still fine in my opinion. It's a tradeoff. Do you want to 'pay' the ZFS tax and expand at a cost, or do you want a bit more risk (on paper) but more flexibility? I ran a RAID6 of 20 drives before that using Linux + MDADM and that worked fine. And MDADM allows you to expand as you go, exactly as you want. I think the risks ZFS protect against are very small. https://louwrentius.com/what-home-nas-builders-should-understand-about-silent-data-corruption.html https://louwrentius.com/what-home-nas-builders-should-unders...
tinco 6y ago
Unraid is the only one that allows you to freely add/remove drives. And only if you're OK with taking cluster down for a minute as you add it. If you true freedom you'd need to go Ceph(FS) based, but there's no GUI to manage a Ceph cluster that I'm aware of, and they're so focused on cloud usage that you'll find little guidance on single-node clusters. edit: Forgot to read Louwrentius' comment, of course MDADM with raid 6 allows you to add disks in pairs of two, so that's definitely also an option. Don't forget to use LVM, it can get real awkward without it.
- nix23 6y ago
  Don't forget GlusterFS, example: Every single disk is a ZFS-pool on top of that you run your GlusterFS-Cluster, GlusterFS then looks for the redundancy and underneath you can export/import your pools aka Disks and or vdevs (if you need more performance). >but there's no GUI to manage a Ceph cluster that I'm aware of There are two, Open Attic and Calamari ,but Open Attic is in maintenance mode, so all the work goes now into Ceph Dashboard: https://docs.ceph.com/en/latest/mgr/dashboard/ https://docs.ceph.com/en/latest/mgr/dashboard/
- magicalhippo 6y ago
  > Unraid is the only one that allows you to freely add/remove drives. I think technically you can do something similar to Synology's Hybrid RAID[1] with ZFS, using partitions and creating mirror VDEVs from the partitions. But I haven't gone through all the details so could be there are some risks I overlooked and in any case you'd have to manage it from the command line so would be a bit tedious. But I did set up a proof of concept in a VM. [1]: https://nascompares.com/guide/shr-synology-hybrid-raid/ https://nascompares.com/guide/shr-synology-hybrid-raid/

MarkSweep 6y ago

Strictly speaking you can expand a VDEV. If you replace all the leaf drives in a VDEV with larger devices, you can extend the VDEV to take the extra space. There is work ongoing to allow adding drives to a RAIDZ VED[1]. And VDEV Removal gives some flexibility (at the cost of indirection on the read path). But yeah, ZFS is not super flexible for adding and removing random drives. [1]: https://github.com/openzfs/zfs/pull/8853 https://github.com/openzfs/zfs/pull/8853

louwrentius 6y ago
Yes that's totally true, but it is very time consuming and cumbersome in my view. I've detailed this also here. https://louwrentius.com/the-hidden-cost-of-using-zfs-for-your-home-nas.html https://louwrentius.com/the-hidden-cost-of-using-zfs-for-you... You may also realise from the top of my article that expanding VDEVS is a topic since 2017 but still no show.
- iooi 6y ago
  I just did this operation on moving from 12x 4TB drives to 12x 16TB drives. I did it over 12 days (each resilver was ~24hours) and it wasn't too cumbersome. Considering how often you're going to grow your pool, it doesn't seem like a big drawback.
  - louwrentius 6y ago
    Fair enough

ComputerGuru 6y ago

> You also get the random IOPs performance of a single drive per VDEV This isn't strictly true. It is true for RAIDZ-n and for writing to a mirror, but ZFS can accelerate random reads with a mirror by distributing the read requests to the underlying disks.