5 ms·

Perfect NAS involves drives in ZFS mirror and ECC memory. It is very cool that nvme drives are now cheap. It looks like pricing is linear up to 4TB now. That p

Perfect NAS involves drives in ZFS mirror and ECC memory. It is very cool that nvme drives are now cheap. It looks like pricing is linear up to 4TB now. That probably means the time for 10gb copper at home is finally here. (edit 10gb RJ45 switches also appear to be sufficiently available now)

vermaden 2y ago
I agree with You. I believe there would be a 'follow up' to this article when 8 TB (or at least 6 TB) NVMe SSD modules will be available on 2280 and 2242 sizes :) Then I would make ZFS mirror of them ... and hopefully with ECC RAM.
- codetrotter 2y ago
  I’ve had unrecoverable data errors in mirrored ZFS pools before. I had remote backup of course, but now instead of mirror I use ZFS pools with many disks and set up with groups of raidz-3.
theshrike79 2y ago
ZFS is good when someone else pays for the drives. It's a pain in the butt when you have a mismatch of random drives found during sales and want to create a single unified storage pool of them. Synology's SHR and Unraid do it easily.
- vermaden 2y ago
  Lets assume You got 4TB 6TB 5TB and 8TB drives ... just do a 4TB 1st primary partition on each of them and create a ZFS pool from that - I do not understand where the 'pain' is ... You can then add 1TB 2nd partition on 6TB and 5TB and 8TB drives and add another ZFS VDEV to the same pool - or create another pool. ... and You can also add 3rd partition. Also its really easy to get used drives in the same or similar size ... unless you live in some really 'hard to live' part of the world.
  - theshrike79 2y ago
    "just" :) As far as I know you can't have mismatching drives as a single contiguous space in ZFS. Maybe it has changed. And when I want to add a new drive of a random size to extend the size of the same directory it's again a pain of adding and extending stuff.
    - vermaden 2y ago
      > As far as I know you can't have mismatching drives as a single contiguous space in ZFS. Maybe it has changed. Yes you can, example below: vermaden ~ % truncate -s 1G FILE.0 vermaden ~ % truncate -s 2G FILE.1 vermaden ~ % mdconfig.sh -c FILE.0 IN: created vnode at /dev/md0 vermaden ~ % mdconfig.sh -c FILE.1 IN: created vnode at /dev/md1 vermaden ~ % lsblk md0 DEVICE MAJ:MIN SIZE TYPE LABEL MOUNT md0 2:17 1.0G zfs - - vermaden ~ % lsblk md1 DEVICE MAJ:MIN SIZE TYPE LABEL MOUNT md1 2:18 2.0G zfs - - vermaden ~ % doas zpool create test md0 md1 vermaden ~ % zpool status test pool: test state: ONLINE config: NAME STATE READ WRITE CKSUM test ONLINE 0 0 0 md0 ONLINE 0 0 0 md1 ONLINE 0 0 0 errors: No known data errors I do not know how you buy your disks but when I need to buy storage I know exactly what SIZE I want to buy - I do not just get ANY disk and then think 'how' to utilize it :) But with ZFS you have plenty of options to use disks of different sizes. Example below, RAIDZ from 4 disks of 2TB and added a mirror of two 5TB disks to the same ZFS pool. vermaden ~ % doas zpool create test raidz md{0,1,2,3} vermaden ~ % zpool status test pool: test state: ONLINE config: NAME STATE READ WRITE CKSUM test ONLINE 0 0 0 raidz1-0 ONLINE 0 0 0 md0 ONLINE 0 0 0 md1 ONLINE 0 0 0 md2 ONLINE 0 0 0 md3 ONLINE 0 0 0 errors: No known data errors vermaden ~ % doas zpool add test mirror md4 md5 vermaden ~ % zpool status test pool: test state: ONLINE config: NAME STATE READ WRITE CKSUM test ONLINE 0 0 0 raidz1-0 ONLINE 0 0 0 md0 ONLINE 0 0 0 md1 ONLINE 0 0 0 md2 ONLINE 0 0 0 md3 ONLINE 0 0 0 mirror-1 ONLINE 0 0 0 md4 ONLINE 0 0 0 md5 ONLINE 0 0 0 errors: No known data errors Hope that helps. Regards, vermaden
mgarfias 2y ago
That was my initial thought too. I clicked hoping for some new chassis/hardware that would make things easier. Like maybe a rack mounted low power quad core with lots of memory slots, 6 or so 3.5” spinny slots, and some hot swap nvme bays for an affordable price. I dont need exotic hw, I just want the ability to have some big slow storage mixed with some fast storage. My current set is a 1U atom setup with an esas card hooked up to a 1U drive shelf. With dual bonded 10gbps uplink.
packetlost 2y ago
Yup, my personal setup involves 2 ZFS pools of 3 drives each in RAIDZ1, 2 NVMe drives partitioned up using LVM. Each pool gets a partition on each of the NVMe drives for a read cache. The primary/larger pool gets a partition that is mirrored across the 2 NVMe drives as a log/write cache. The system has 64GB of ECC DDR4 and a 10Gbe NIC. As far as I could tell, there was exactly 1 motherboard that actually allowed for this setup (2 full speed NVMe, ECC support, AMD4 socket), so it was not easy, but you can do it! This is my 4th NAS iteration, so I've worked out exactly how I want it over the course of about 10 years. Partial parts list here: https://pcpartpicker.com/user/chiefnoah/saved/CxjNyc https://pcpartpicker.com/user/chiefnoah/saved/CxjNyc

vermaden 2y ago

I agree with You. I believe there would be a 'follow up' to this article when 8 TB (or at least 6 TB) NVMe SSD modules will be available on 2280 and 2242 sizes :) Then I would make ZFS mirror of them ... and hopefully with ECC RAM.

codetrotter 2y ago
I’ve had unrecoverable data errors in mirrored ZFS pools before. I had remote backup of course, but now instead of mirror I use ZFS pools with many disks and set up with groups of raidz-3.

theshrike79 2y ago

ZFS is good when someone else pays for the drives. It's a pain in the butt when you have a mismatch of random drives found during sales and want to create a single unified storage pool of them. Synology's SHR and Unraid do it easily.

vermaden 2y ago
Lets assume You got 4TB 6TB 5TB and 8TB drives ... just do a 4TB 1st primary partition on each of them and create a ZFS pool from that - I do not understand where the 'pain' is ... You can then add 1TB 2nd partition on 6TB and 5TB and 8TB drives and add another ZFS VDEV to the same pool - or create another pool. ... and You can also add 3rd partition. Also its really easy to get used drives in the same or similar size ... unless you live in some really 'hard to live' part of the world.
- theshrike79 2y ago
  "just" :) As far as I know you can't have mismatching drives as a single contiguous space in ZFS. Maybe it has changed. And when I want to add a new drive of a random size to extend the size of the same directory it's again a pain of adding and extending stuff.
  - vermaden 2y ago
    > As far as I know you can't have mismatching drives as a single contiguous space in ZFS. Maybe it has changed. Yes you can, example below: vermaden ~ % truncate -s 1G FILE.0 vermaden ~ % truncate -s 2G FILE.1 vermaden ~ % mdconfig.sh -c FILE.0 IN: created vnode at /dev/md0 vermaden ~ % mdconfig.sh -c FILE.1 IN: created vnode at /dev/md1 vermaden ~ % lsblk md0 DEVICE MAJ:MIN SIZE TYPE LABEL MOUNT md0 2:17 1.0G zfs - - vermaden ~ % lsblk md1 DEVICE MAJ:MIN SIZE TYPE LABEL MOUNT md1 2:18 2.0G zfs - - vermaden ~ % doas zpool create test md0 md1 vermaden ~ % zpool status test pool: test state: ONLINE config: NAME STATE READ WRITE CKSUM test ONLINE 0 0 0 md0 ONLINE 0 0 0 md1 ONLINE 0 0 0 errors: No known data errors I do not know how you buy your disks but when I need to buy storage I know exactly what SIZE I want to buy - I do not just get ANY disk and then think 'how' to utilize it :) But with ZFS you have plenty of options to use disks of different sizes. Example below, RAIDZ from 4 disks of 2TB and added a mirror of two 5TB disks to the same ZFS pool. vermaden ~ % doas zpool create test raidz md{0,1,2,3} vermaden ~ % zpool status test pool: test state: ONLINE config: NAME STATE READ WRITE CKSUM test ONLINE 0 0 0 raidz1-0 ONLINE 0 0 0 md0 ONLINE 0 0 0 md1 ONLINE 0 0 0 md2 ONLINE 0 0 0 md3 ONLINE 0 0 0 errors: No known data errors vermaden ~ % doas zpool add test mirror md4 md5 vermaden ~ % zpool status test pool: test state: ONLINE config: NAME STATE READ WRITE CKSUM test ONLINE 0 0 0 raidz1-0 ONLINE 0 0 0 md0 ONLINE 0 0 0 md1 ONLINE 0 0 0 md2 ONLINE 0 0 0 md3 ONLINE 0 0 0 mirror-1 ONLINE 0 0 0 md4 ONLINE 0 0 0 md5 ONLINE 0 0 0 errors: No known data errors Hope that helps. Regards, vermaden

mgarfias 2y ago

That was my initial thought too. I clicked hoping for some new chassis/hardware that would make things easier. Like maybe a rack mounted low power quad core with lots of memory slots, 6 or so 3.5” spinny slots, and some hot swap nvme bays for an affordable price. I dont need exotic hw, I just want the ability to have some big slow storage mixed with some fast storage. My current set is a 1U atom setup with an esas card hooked up to a 1U drive shelf. With dual bonded 10gbps uplink.

packetlost 2y ago

Yup, my personal setup involves 2 ZFS pools of 3 drives each in RAIDZ1, 2 NVMe drives partitioned up using LVM. Each pool gets a partition on each of the NVMe drives for a read cache. The primary/larger pool gets a partition that is mirrored across the 2 NVMe drives as a log/write cache. The system has 64GB of ECC DDR4 and a 10Gbe NIC. As far as I could tell, there was exactly 1 motherboard that actually allowed for this setup (2 full speed NVMe, ECC support, AMD4 socket), so it was not easy, but you can do it! This is my 4th NAS iteration, so I've worked out exactly how I want it over the course of about 10 years. Partial parts list here: https://pcpartpicker.com/user/chiefnoah/saved/CxjNyc https://pcpartpicker.com/user/chiefnoah/saved/CxjNyc