4 ms·

Have you guys tried hardware RAID? I've been a hardware RAID user for about 0.5PB. Then I built a your box from your kit kit and was shocked by the terrible per

Have you guys tried hardware RAID? I've been a hardware RAID user for about 0.5PB. Then I built a your box from your kit kit and was shocked by the terrible performance of ZFS. About half the speed of my Adaptec builds, and much worse for other loads.

jamesblonde 9y ago
ZFS needs a lot of RAM for good performance. Like 10s of GBs. How were your boxes configured?
- frozenport 9y ago
  Storage Pod 6.0, ~190 GB of RAM + ~48 cores. 30 drives gets like 600 MB/s on ZFS, compared to 1200 MB/s with Adaptec hardware RAID. [ps] Also got a lot of bad stuff to say about backuppods. Mostly that they don't sell the correct number of wire harnesses. (They need to sell two but they're only selling one, I assume I'm the only person stupid enough to buy from them...)
  - olavgg 9y ago
    First ZFS don't need much RAM, you are fint with as little as 2GB. You can still use hardware raid with ZFS if you export each disk as its own device and assemble the raid with ZFS. It is not the parity calculations that are slow, it is the additional io operations that are required and ZFS writes these directly to disk. A hardware raid controller writes these io operations to memory. Another advantage with ZFS is if your data compress well, you will get additional speed benefit from that.
    - frozenport 9y ago
      Frankly, I suspect the ZFS community is in denial, or optimizing for cost. I got what I think is a correctly configured ZFS system and its getting the same speeds as most people report online. NOW the thing is that the ZFS system doing about 2x worse for the dense sequential write workloads compared to the RAID6 HW solution. Not to mention eating a dozen or so GBs of RAM.
      - olavgg 9y ago
        ZFS autotunes memory allocation based on how much ram you have, if you have 128GB ram free, ZFS will use that. In fact all operating systems eat GB's of free RAM just like ZFS. Software raid has a larger write penalty, that is why you see slower write performance versus using hardware raid. As I said earlier, writing directly to memory helps. Recommended read: http://rickardnobel.se/raid-5-write-penalty/ http://rickardnobel.se/raid-5-write-penalty/
        frozenport 9y ago
        So, basically, if you're building a $15,000 storage box, cough up $500 for a hardware RAID solution.
- jonafato 9y ago
  I'm by no means a ZFS expert, but much of what I've read from "authoritative" sources [1] suggests that this is a myth. The 1GB of RAM per TB of disk is largely a suggestion from the FreeNAS developers a while back that was specific to FreeNAS (not ZFS in general) and more of a gut feeling than something backed by measurements. ZFS deduplication can be memory-hungry, but it's more "adding memory helps" than "not having lots of memory is catastrophic". [1] https://linustechtips.com/main/topic/738402-zfs-memory-requirements/ https://linustechtips.com/main/topic/738402-zfs-memory-requi...

jamesblonde 9y ago

ZFS needs a lot of RAM for good performance. Like 10s of GBs. How were your boxes configured?

frozenport 9y ago
Storage Pod 6.0, ~190 GB of RAM + ~48 cores. 30 drives gets like 600 MB/s on ZFS, compared to 1200 MB/s with Adaptec hardware RAID. [ps] Also got a lot of bad stuff to say about backuppods. Mostly that they don't sell the correct number of wire harnesses. (They need to sell two but they're only selling one, I assume I'm the only person stupid enough to buy from them...)
- olavgg 9y ago
  First ZFS don't need much RAM, you are fint with as little as 2GB. You can still use hardware raid with ZFS if you export each disk as its own device and assemble the raid with ZFS. It is not the parity calculations that are slow, it is the additional io operations that are required and ZFS writes these directly to disk. A hardware raid controller writes these io operations to memory. Another advantage with ZFS is if your data compress well, you will get additional speed benefit from that.
  - frozenport 9y ago
    Frankly, I suspect the ZFS community is in denial, or optimizing for cost. I got what I think is a correctly configured ZFS system and its getting the same speeds as most people report online. NOW the thing is that the ZFS system doing about 2x worse for the dense sequential write workloads compared to the RAID6 HW solution. Not to mention eating a dozen or so GBs of RAM.
    - olavgg 9y ago
      ZFS autotunes memory allocation based on how much ram you have, if you have 128GB ram free, ZFS will use that. In fact all operating systems eat GB's of free RAM just like ZFS. Software raid has a larger write penalty, that is why you see slower write performance versus using hardware raid. As I said earlier, writing directly to memory helps. Recommended read: http://rickardnobel.se/raid-5-write-penalty/ http://rickardnobel.se/raid-5-write-penalty/
      - frozenport 9y ago
        So, basically, if you're building a $15,000 storage box, cough up $500 for a hardware RAID solution.
jonafato 9y ago
I'm by no means a ZFS expert, but much of what I've read from "authoritative" sources [1] suggests that this is a myth. The 1GB of RAM per TB of disk is largely a suggestion from the FreeNAS developers a while back that was specific to FreeNAS (not ZFS in general) and more of a gut feeling than something backed by measurements. ZFS deduplication can be memory-hungry, but it's more "adding memory helps" than "not having lots of memory is catastrophic". [1] https://linustechtips.com/main/topic/738402-zfs-memory-requirements/ https://linustechtips.com/main/topic/738402-zfs-memory-requi...