4 ms·

"I was able to get consistent performance using a striped RAIDZ mirror pool with four SSDs and a hot spare" That doesn't make sense. The author is not using ra

by mrb 4y ago

"I was able to get consistent performance using a striped RAIDZ mirror pool with four SSDs and a hot spare" That doesn't make sense. The author is not using raidz. In ZFS terminology, he's mirroring two striped vdevs (like raid10), plus using a hotspare. And that is a bad choice, as he gets only 16 TB usable, and the mirrored stripes could fail with certain combinations of two drives failures. Instead, he should have set up a raidz2 across 5 drives: 50% more usable space (24 TB), can tolerate any two drive failing, and it would give him higher performance on sequential I/O. I have a raidz2 pool across 6 spinning rust 18TB hard drives, and my server can handle 900 MB/s sequential reads, and 700 MB/s sequential writes (benchmarked locally). It it was built on 5 SAMSUNG 870 QVO SATA III SSD drives like the author, it would certainly get to 1.5+ GB/s in both sequential reads and writes. In other words the bottleneck would shift to the 10 Gb/s network (1.25 GB/s). For comparison the author discloses his writes are limited to 700 MB/s (over the network, so his bottleneck is not the network link but local I/O contention). That's right, a 6-HDD raidz2 matches the write performance of his 5-SSD raid10 setup. And that's because the write overhead of raid10 is 100%, while the write overhead of a 6-drive raidz2 is only 50%. Edit: Oh and the strange performance issue that he noticed "disappeared" after a while is most likely due to cache recovery. The drive can sustain writes of about 490 MB/s for a little while then it drops to 170 MB/s, but after idling 5 minutes it can recover the initial speed. See https://www.tomshardware.com/reviews/samsung-870-qvo-sata-ssd/2 https://www.tomshardware.com/reviews/samsung-870-qvo-sata-ss...: "As we noticed with the 1TB model, the 8TB model’s cache recovery mechanisms work similarly. After letting the drive rest at idle for 30 seconds, the 870 QVO gains back 6GB of its cache. It recovers fully with 5 minutes of idle time. "

geerlingguy 4y ago
Note that the 8TB model has about 50TB of cache (quite a bit more than the 1TB model—it doubles in size with each doubling of storage capacity). One concern Wendell had with RAIDZ2 was the potential for the older Xeon to be a bottleneck for writes. It probably wouldn't be, but I didn't do too much testing with that layout. I might still, we'll see.
- mrb 4y ago
  For reference, my server CPU is an AMD Ryzen 7 PRO 5750G (65W TDP, 3.8-4.6GHz, 8-core, 16-threads) and CPU usage is around 8% when writing at 700 MB/s on my raidz2. In other words, only a little more than 1 of the 16 hardware threads is utilized to handle this workload. Your older Xeon D-1521 should be able to handle that, but it's true that being an older processor, it is pushing it closer to its limits. Handling concurrently 10 Gb/s of network traffic would likely push it to more than 50% utilization. That's one reason I like to not cheap out too much on the CPU, especially for a 10GbE ZFS NAS.

by mrb 4y ago

geerlingguy 4y ago

Note that the 8TB model has about 50TB of cache (quite a bit more than the 1TB model—it doubles in size with each doubling of storage capacity). One concern Wendell had with RAIDZ2 was the potential for the older Xeon to be a bottleneck for writes. It probably wouldn't be, but I didn't do too much testing with that layout. I might still, we'll see.

mrb 4y ago
For reference, my server CPU is an AMD Ryzen 7 PRO 5750G (65W TDP, 3.8-4.6GHz, 8-core, 16-threads) and CPU usage is around 8% when writing at 700 MB/s on my raidz2. In other words, only a little more than 1 of the 16 hardware threads is utilized to handle this workload. Your older Xeon D-1521 should be able to handle that, but it's true that being an older processor, it is pushing it closer to its limits. Handling concurrently 10 Gb/s of network traffic would likely push it to more than 50% utilization. That's one reason I like to not cheap out too much on the CPU, especially for a 10GbE ZFS NAS.