4 ms·

Something seems wrong here. No doubt this configuration rocks, but it should be physically impossible to get better bandwidth than 8x 3.5GB/s = ~28GB/s that the

by fargle 5y ago

Something seems wrong here. No doubt this configuration rocks, but it should be physically impossible to get better bandwidth than 8x 3.5GB/s = ~28GB/s that the raw drives offer, which is also well matched to the PCIe x4 slots bandwidth. Maybe fio is producing trivially compressible data? Dedup? Directio not working? What am I missing?

tanelpoder 5y ago
FYI I got 66 GiB/s using 10 Samsung 980 Pro PCIe 4.0 SSDs on a ThreadRipper Pro workstation: https://news.ycombinator.com/item?id=25956670 https://news.ycombinator.com/item?id=25956670 Had to use direct I/O and use random large IO patterns to avoid various caching, buffering and prefetching effects.
- ksec 5y ago
  You are using a PCI 4.0 SSD compared to the SSD used in this RAIDZ2 with PCIE 3.0 10 PCIe 4.0 SSDs give you maximum of around 70GB/s. The test show sustained throughput of 80GB/s without compression. Which is still higher than the maximum allowed by 8 x 4 PCIE 3.0 slot at total of 32GB/s. Waiting for some ZFS expert to give a plausible explanation.
neilpanchal 5y ago
I am not sure, I think some of it is cached in the RAM. I did a longer sustained test and I am getting about 70 GB/s with a 200GB file - this is definitely larger than 64GB RAM. Is it to do with lz4 compression? Fetching smaller blocks from disk and then decompressing it, thus inflating the bandwidth? The initial file was created with fio rw=randwrite, so the dummy data is random. I think its even more impressive that ZFS can do this. I verified with ext4 fs on a single NVMe drive and no matter what knobs I turn, I am getting around around 3.5 GB/s. Edit: See note 2 at the end of the article, I posted the raw results of another 100GB file test. Edit 2: See note 3. I rebooted the VM with 2GB ram (instead of 64GB) and it matches almost exactly as per your prediction ~ 20GB/s. But, it still doesn't explain the absurd 80GB/s sustained reads of a 100GB file with only 64GB ram.
- fargle 5y ago
  I do think it sounds like compression might be part of it. I'm thinking that a) fio might create test data that is "trivially" compressible and b) this might occur before directio says "ignore the buffer cache and write directly to the drive", but then what gets written is 10x or 100x smaller. Another possibility is that somehow directio doesn't work when zfs+compression is turned on? I'd do a test where everything else is the same, but turn lz4 compression off. In theory, with a real world dataset, compression will cause CPU load to go up, but I would not expect to see the datasize at the disk go down by more than 2x to 3x, and therefore the bandwidth should not go up my anymore than that, and only if the CPU isn't the limiting factor. In most cases, compressed (or encrypted) filesystems should be slower, not faster because of this. So a benchmark without compression would be the first step. Then maybe as a second step, a "real world" benchmark - copy 10-100GB of representative data or something like that - with and without compression to see if bandwidth goes up or down and how bad CPU utilization goes up. Another estimate is to look at the stats that tell you what the compression ratio is. If it's really, really high, that just means fio created all-zeros, really fast. ???
  - neilpanchal 5y ago
    Interesting. I turned off lz4 compression and it didn't make any difference - still getting around 80 GB/s on a 30 second test. I also checked the content of the dummy file, it is random and takes about the same time as using `dd` to pipe data from `/dev/random`. It's a mystery and it's related to RAM (see note 3). I think this would be an interesting question to ask on SO! See note 4: https://neil.computer/notes/zfs-raidz2/#note-4 https://neil.computer/notes/zfs-raidz2/#note-4
    - fargle 5y ago
      Well that's mysterious. I have a similar but smaller system - two EVO 970s on a 4x4x4x4 PCI card just like you have (I planned on adding more if it went well). I went through similar issues benchmarking it about a year ago. At the end of the day, I found that with ZFS it was just very difficult to bypass, avoid, turn-off, etc. all the caching. The machine I was using has a lot of RAM. I had to reduce the RAM to like 64GB with a boot option to even see what was happening. I was still never thrilled with the numbers I got. Then, finally it dawned on me: with enough RAM, and an efficient filesystem like ZFS, the underlying storage is only a small part of the equation. The important thing is that your benchmark is accurate to the real world workload. If the real world can use a lot of buffer cache and RAM, and it makes it go fast - then good, use it like that. Which I did, and was happy, and forgot all about it. What we want to avoid is fio saying you have 100GB/s and then in the real world you don't see nearly that and are disappointed. To me benchmarks serve two purposes: comparison and prediction of performance with a reproducible test case, for example compare the effect of ARC cache size setting. Second, diagnosing whether the hardware is working correctly and as close to ideal. So I did see some stuff vaguely like you are seeing. I remember being impressed at ZFS and how hard it is to benchmark because it tries so hard to cache efficiently (or whatever). Unfortunately it was like a year ago so I don't remember more details.

by fargle 5y ago

tanelpoder 5y ago

FYI I got 66 GiB/s using 10 Samsung 980 Pro PCIe 4.0 SSDs on a ThreadRipper Pro workstation: https://news.ycombinator.com/item?id=25956670 https://news.ycombinator.com/item?id=25956670 Had to use direct I/O and use random large IO patterns to avoid various caching, buffering and prefetching effects.

ksec 5y ago
You are using a PCI 4.0 SSD compared to the SSD used in this RAIDZ2 with PCIE 3.0 10 PCIe 4.0 SSDs give you maximum of around 70GB/s. The test show sustained throughput of 80GB/s without compression. Which is still higher than the maximum allowed by 8 x 4 PCIE 3.0 slot at total of 32GB/s. Waiting for some ZFS expert to give a plausible explanation.

neilpanchal 5y ago

I am not sure, I think some of it is cached in the RAM. I did a longer sustained test and I am getting about 70 GB/s with a 200GB file - this is definitely larger than 64GB RAM. Is it to do with lz4 compression? Fetching smaller blocks from disk and then decompressing it, thus inflating the bandwidth? The initial file was created with fio rw=randwrite, so the dummy data is random. I think its even more impressive that ZFS can do this. I verified with ext4 fs on a single NVMe drive and no matter what knobs I turn, I am getting around around 3.5 GB/s. Edit: See note 2 at the end of the article, I posted the raw results of another 100GB file test. Edit 2: See note 3. I rebooted the VM with 2GB ram (instead of 64GB) and it matches almost exactly as per your prediction ~ 20GB/s. But, it still doesn't explain the absurd 80GB/s sustained reads of a 100GB file with only 64GB ram.

fargle 5y ago
I do think it sounds like compression might be part of it. I'm thinking that a) fio might create test data that is "trivially" compressible and b) this might occur before directio says "ignore the buffer cache and write directly to the drive", but then what gets written is 10x or 100x smaller. Another possibility is that somehow directio doesn't work when zfs+compression is turned on? I'd do a test where everything else is the same, but turn lz4 compression off. In theory, with a real world dataset, compression will cause CPU load to go up, but I would not expect to see the datasize at the disk go down by more than 2x to 3x, and therefore the bandwidth should not go up my anymore than that, and only if the CPU isn't the limiting factor. In most cases, compressed (or encrypted) filesystems should be slower, not faster because of this. So a benchmark without compression would be the first step. Then maybe as a second step, a "real world" benchmark - copy 10-100GB of representative data or something like that - with and without compression to see if bandwidth goes up or down and how bad CPU utilization goes up. Another estimate is to look at the stats that tell you what the compression ratio is. If it's really, really high, that just means fio created all-zeros, really fast. ???
- neilpanchal 5y ago
  Interesting. I turned off lz4 compression and it didn't make any difference - still getting around 80 GB/s on a 30 second test. I also checked the content of the dummy file, it is random and takes about the same time as using `dd` to pipe data from `/dev/random`. It's a mystery and it's related to RAM (see note 3). I think this would be an interesting question to ask on SO! See note 4: https://neil.computer/notes/zfs-raidz2/#note-4 https://neil.computer/notes/zfs-raidz2/#note-4
  - fargle 5y ago
    Well that's mysterious. I have a similar but smaller system - two EVO 970s on a 4x4x4x4 PCI card just like you have (I planned on adding more if it went well). I went through similar issues benchmarking it about a year ago. At the end of the day, I found that with ZFS it was just very difficult to bypass, avoid, turn-off, etc. all the caching. The machine I was using has a lot of RAM. I had to reduce the RAM to like 64GB with a boot option to even see what was happening. I was still never thrilled with the numbers I got. Then, finally it dawned on me: with enough RAM, and an efficient filesystem like ZFS, the underlying storage is only a small part of the equation. The important thing is that your benchmark is accurate to the real world workload. If the real world can use a lot of buffer cache and RAM, and it makes it go fast - then good, use it like that. Which I did, and was happy, and forgot all about it. What we want to avoid is fio saying you have 100GB/s and then in the real world you don't see nearly that and are disappointed. To me benchmarks serve two purposes: comparison and prediction of performance with a reproducible test case, for example compare the effect of ARC cache size setting. Second, diagnosing whether the hardware is working correctly and as close to ideal. So I did see some stuff vaguely like you are seeing. I remember being impressed at ZFS and how hard it is to benchmark because it tries so hard to cache efficiently (or whatever). Unfortunately it was like a year ago so I don't remember more details.