5 ms·

Maybe I need to try ZFS again now that I have different drives. I used to run it on 6x 3TB WD Red drives (raid-z2) and could never get ZFS to perform anywhere n

Maybe I need to try ZFS again now that I have different drives. I used to run it on 6x 3TB WD Red drives (raid-z2) and could never get ZFS to perform anywhere near what I expected. Ended up going with LVM on mdraid (raid 6) on dm-integrity and got tremendously better performance for my "home nas / vm image host" machine.

nix23 6y ago
Yes you should look at that numbers: https://arstechnica.com/gadgets/2020/05/zfs-versus-raid-eight-ironwolf-disks-two-filesystems-one-winner/ https://arstechnica.com/gadgets/2020/05/zfs-versus-raid-eigh...
jhoechtl 6y ago
Use Btrfs. Seriously. It's a tremendous fs. Read the caveats concerning raid56 and success.
- Teknoman117 6y ago
  My stack at the moment is 6x [10 TB WD Red Pros -> dm-integrity] -> mdadm raid6 -> dm-crypt -> LVM. One of my LVM volumes is btrfs. I only use LVM because I've had terrible performance for VM images stored in btrfs volumes. You can always use fallocate and disable CoW for the image but that prevents taking snapshots. I'd if there was a zvol equivalent in btrfs.
  - cmurf 6y ago
    Neither fallocate nor nodatacow will prevent taking snapshots. The result of writing to nodatacow shared extents is that the write must be COW, not an overwrite. Same as a reflink copy of a file on XFS. New "overwrites" are COW, subsequent overwrites are in fact overwrites.
    - deleted 6y ago
      [deleted]
- ansible 6y ago
  We've phasing out using btrfs for RAID at work. It has been just a lot more straight-forward and easier to manage MD for the RAID-1 on our servers. We still use btrfs for container filesystems, subvolume snapshots are so handy. But that is btrfs on top of a MD RAID array.
- bromonkey 6y ago
  Oh I'll get right on downloading that to my FreeBSD storage boxes, oh wait a minute.....
firethief 6y ago
Measuring the performance of a monolithic RAID-Z array for a VM image workload is like measuring the 0-60 time of a semi tractor. That's not what that configuration is optimized for. Especially for a high IOPS workload, you should see much better performance with a pool of mirrors. Your requirements sound similar to my home network. In my case how I orchestrated the spinning rust ended up being a lot less important than getting the random access parts of my workload on an NVMe SSD. Best $40 upgrade I ever made.

nix23 6y ago

Yes you should look at that numbers: https://arstechnica.com/gadgets/2020/05/zfs-versus-raid-eight-ironwolf-disks-two-filesystems-one-winner/ https://arstechnica.com/gadgets/2020/05/zfs-versus-raid-eigh...

jhoechtl 6y ago

Use Btrfs. Seriously. It's a tremendous fs. Read the caveats concerning raid56 and success.

Teknoman117 6y ago
My stack at the moment is 6x [10 TB WD Red Pros -> dm-integrity] -> mdadm raid6 -> dm-crypt -> LVM. One of my LVM volumes is btrfs. I only use LVM because I've had terrible performance for VM images stored in btrfs volumes. You can always use fallocate and disable CoW for the image but that prevents taking snapshots. I'd if there was a zvol equivalent in btrfs.
- cmurf 6y ago
  Neither fallocate nor nodatacow will prevent taking snapshots. The result of writing to nodatacow shared extents is that the write must be COW, not an overwrite. Same as a reflink copy of a file on XFS. New "overwrites" are COW, subsequent overwrites are in fact overwrites.
  - deleted 6y ago
    [deleted]
ansible 6y ago
We've phasing out using btrfs for RAID at work. It has been just a lot more straight-forward and easier to manage MD for the RAID-1 on our servers. We still use btrfs for container filesystems, subvolume snapshots are so handy. But that is btrfs on top of a MD RAID array.
bromonkey 6y ago
Oh I'll get right on downloading that to my FreeBSD storage boxes, oh wait a minute.....

firethief 6y ago

Measuring the performance of a monolithic RAID-Z array for a VM image workload is like measuring the 0-60 time of a semi tractor. That's not what that configuration is optimized for. Especially for a high IOPS workload, you should see much better performance with a pool of mirrors. Your requirements sound similar to my home network. In my case how I orchestrated the spinning rust ended up being a lot less important than getting the random access parts of my workload on an NVMe SSD. Best $40 upgrade I ever made.