43 ms·
I have a Linux NAS with 6 7200RPM HDDs in a raidz2 ZFS pool, and the local fs read speed on large files is 7 times faster than GbE, about ~930 MB/s. So while th
by mrb 5y ago
I have a Linux NAS with 6 7200RPM HDDs in a raidz2 ZFS pool, and the local fs read speed on large files is 7 times faster than GbE, about ~930 MB/s. So while this wouldn't saturate a 10GbE link, it would still be greatly beneficial to multiply the network read speed by 7-fold by upgrading from GbE to 10GbE.
- csdvrx 5y agoHow did you bench your ZFS pool?
- mrb 5y agoWith dd. And normally when I do this I drop the caches with "echo 3 > /proc/sys/vm/drop_caches" but this doesn't work with ZFSOnLinux as it has a separate ARC cache that doesn't get cleared by that command (and I don't believe it's possible to clear it, other than by exporting and reimporting the zpool.) So I read a 10GB chunk of a file that I know is not cached: $ dd if=/some/large/file of=/dev/null bs=1M count=10240 10240+0 records in 10240+0 records out 10737418240 bytes (11 GB, 10 GiB) copied, 11.3378 s, 947 MB/s The 947 MB/s figure printed by dd matches the speed reported by a "zpool iostat tank 1" running during the benchmarking, which confirms I'm not reading from the cache. When I repeat the same command, dd completes almost immediately reporting a speed of about 6-8 GB/s on my machine, while the zpool iostat command shows zero read bandwidth.