5 ms·

Is your NAS NVME based if you are using spinning metal you won't reach speeds to saturate 10GbE especially with small file sizes?

mrb 5y ago
I have a Linux NAS with 6 7200RPM HDDs in a raidz2 ZFS pool, and the local fs read speed on large files is 7 times faster than GbE, about ~930 MB/s. So while this wouldn't saturate a 10GbE link, it would still be greatly beneficial to multiply the network read speed by 7-fold by upgrading from GbE to 10GbE.
- csdvrx 5y ago
  How did you bench your ZFS pool?
  - mrb 5y ago
    With dd. And normally when I do this I drop the caches with "echo 3 > /proc/sys/vm/drop_caches" but this doesn't work with ZFSOnLinux as it has a separate ARC cache that doesn't get cleared by that command (and I don't believe it's possible to clear it, other than by exporting and reimporting the zpool.) So I read a 10GB chunk of a file that I know is not cached: $ dd if=/some/large/file of=/dev/null bs=1M count=10240 10240+0 records in 10240+0 records out 10737418240 bytes (11 GB, 10 GiB) copied, 11.3378 s, 947 MB/s The 947 MB/s figure printed by dd matches the speed reported by a "zpool iostat tank 1" running during the benchmarking, which confirms I'm not reading from the cache. When I repeat the same command, dd completes almost immediately reporting a speed of about 6-8 GB/s on my machine, while the zpool iostat command shows zero read bandwidth.
pdimitar 5y ago
It's a ZFS-based NAS with an SSD cache. Right now it's only 1x SATA3 SSD which means it will top at 500MB/s (so slightly less than a half of 10GbE) and I am planning to add one more which will make it almost saturate a 10GbE link. Additionally yep, I plan to have one NVMe dataset sometime this year. And even further, I plan to add a few enterprise HDDs in stripe and mirror. If you have 8 (with capacity of 4 for full mirroring) and each is at 260MB/s then et voila, you get 10GbE again.
jjoonathan 5y ago
If somebody knows why they should pay a premium for 10GbE, they also know how bad spinning rust is.
- seized 5y ago
  Those same people should also know that spinning rust can go very fast as well... My 8 and 12 drive spinning rust pools can read at 500MB/sec easily and get close to GB/sec. But the 10Gbit network is there for the flash pool.
  - aseipp 5y ago
    Yeah, but... The noise! I just can't handle spinning rust anymore for this alone even if the $/GB ratio is still so much better. Consumer SATA SSDs are plentiful enough that I'm just going all in on flash when I revamp my server and do an 8 drive (+1 hotswap) RAID10 build or whatever. I don't need +60TB or anything and 8-9 drives is about the limit you can find on high end consumer mobos I think...
    - justsomehnguy 5y ago
      > The noise It is not required for the NAS to be in the same room where you sleep. With a proper drives (and maybe some APM tinkering) you can make a very quiet one. Also using 2.5" drives is an option too. Though if you can throw money at the problem than of course you can just buy a bunch of TB SSD drives.
  - fomine3 5y ago
    Or say 1Gbps is too low bar. Even single rust drive can fill 1GbE in simple workload.
- pdimitar 5y ago
  Put 8 high-quality HDDs in a stripe & mirror config. Thus you can read from 4 simultaneously. They each top at 260MB/s. That amounts to little over 1GB/s -- which is 10GbE.

mrb 5y ago

I have a Linux NAS with 6 7200RPM HDDs in a raidz2 ZFS pool, and the local fs read speed on large files is 7 times faster than GbE, about ~930 MB/s. So while this wouldn't saturate a 10GbE link, it would still be greatly beneficial to multiply the network read speed by 7-fold by upgrading from GbE to 10GbE.

csdvrx 5y ago
How did you bench your ZFS pool?
- mrb 5y ago
  With dd. And normally when I do this I drop the caches with "echo 3 > /proc/sys/vm/drop_caches" but this doesn't work with ZFSOnLinux as it has a separate ARC cache that doesn't get cleared by that command (and I don't believe it's possible to clear it, other than by exporting and reimporting the zpool.) So I read a 10GB chunk of a file that I know is not cached: $ dd if=/some/large/file of=/dev/null bs=1M count=10240 10240+0 records in 10240+0 records out 10737418240 bytes (11 GB, 10 GiB) copied, 11.3378 s, 947 MB/s The 947 MB/s figure printed by dd matches the speed reported by a "zpool iostat tank 1" running during the benchmarking, which confirms I'm not reading from the cache. When I repeat the same command, dd completes almost immediately reporting a speed of about 6-8 GB/s on my machine, while the zpool iostat command shows zero read bandwidth.

pdimitar 5y ago

It's a ZFS-based NAS with an SSD cache. Right now it's only 1x SATA3 SSD which means it will top at 500MB/s (so slightly less than a half of 10GbE) and I am planning to add one more which will make it almost saturate a 10GbE link. Additionally yep, I plan to have one NVMe dataset sometime this year. And even further, I plan to add a few enterprise HDDs in stripe and mirror. If you have 8 (with capacity of 4 for full mirroring) and each is at 260MB/s then et voila, you get 10GbE again.

jjoonathan 5y ago

If somebody knows why they should pay a premium for 10GbE, they also know how bad spinning rust is.

seized 5y ago
Those same people should also know that spinning rust can go very fast as well... My 8 and 12 drive spinning rust pools can read at 500MB/sec easily and get close to GB/sec. But the 10Gbit network is there for the flash pool.
- aseipp 5y ago
  Yeah, but... The noise! I just can't handle spinning rust anymore for this alone even if the $/GB ratio is still so much better. Consumer SATA SSDs are plentiful enough that I'm just going all in on flash when I revamp my server and do an 8 drive (+1 hotswap) RAID10 build or whatever. I don't need +60TB or anything and 8-9 drives is about the limit you can find on high end consumer mobos I think...
  - justsomehnguy 5y ago
    > The noise It is not required for the NAS to be in the same room where you sleep. With a proper drives (and maybe some APM tinkering) you can make a very quiet one. Also using 2.5" drives is an option too. Though if you can throw money at the problem than of course you can just buy a bunch of TB SSD drives.
- fomine3 5y ago
  Or say 1Gbps is too low bar. Even single rust drive can fill 1GbE in simple workload.
pdimitar 5y ago
Put 8 high-quality HDDs in a stripe & mirror config. Thus you can read from 4 simultaneously. They each top at 260MB/s. That amounts to little over 1GB/s -- which is 10GbE.