4 ms·

I am not sure, I think some of it is cached in the RAM. I did a longer sustained test and I am getting about 70 GB/s with a 200GB file - this is definitely larg

I am not sure, I think some of it is cached in the RAM. I did a longer sustained test and I am getting about 70 GB/s with a 200GB file - this is definitely larger than 64GB RAM. Is it to do with lz4 compression? Fetching smaller blocks from disk and then decompressing it, thus inflating the bandwidth? The initial file was created with fio rw=randwrite, so the dummy data is random. I think its even more impressive that ZFS can do this. I verified with ext4 fs on a single NVMe drive and no matter what knobs I turn, I am getting around around 3.5 GB/s. Edit: See note 2 at the end of the article, I posted the raw results of another 100GB file test. Edit 2: See note 3. I rebooted the VM with 2GB ram (instead of 64GB) and it matches almost exactly as per your prediction ~ 20GB/s. But, it still doesn't explain the absurd 80GB/s sustained reads of a 100GB file with only 64GB ram.

fargle 5y ago
I do think it sounds like compression might be part of it. I'm thinking that a) fio might create test data that is "trivially" compressible and b) this might occur before directio says "ignore the buffer cache and write directly to the drive", but then what gets written is 10x or 100x smaller. Another possibility is that somehow directio doesn't work when zfs+compression is turned on? I'd do a test where everything else is the same, but turn lz4 compression off. In theory, with a real world dataset, compression will cause CPU load to go up, but I would not expect to see the datasize at the disk go down by more than 2x to 3x, and therefore the bandwidth should not go up my anymore than that, and only if the CPU isn't the limiting factor. In most cases, compressed (or encrypted) filesystems should be slower, not faster because of this. So a benchmark without compression would be the first step. Then maybe as a second step, a "real world" benchmark - copy 10-100GB of representative data or something like that - with and without compression to see if bandwidth goes up or down and how bad CPU utilization goes up. Another estimate is to look at the stats that tell you what the compression ratio is. If it's really, really high, that just means fio created all-zeros, really fast. ???
- neilpanchal 5y ago
  Interesting. I turned off lz4 compression and it didn't make any difference - still getting around 80 GB/s on a 30 second test. I also checked the content of the dummy file, it is random and takes about the same time as using `dd` to pipe data from `/dev/random`. It's a mystery and it's related to RAM (see note 3). I think this would be an interesting question to ask on SO! See note 4: https://neil.computer/notes/zfs-raidz2/#note-4 https://neil.computer/notes/zfs-raidz2/#note-4
  - fargle 5y ago
    Well that's mysterious. I have a similar but smaller system - two EVO 970s on a 4x4x4x4 PCI card just like you have (I planned on adding more if it went well). I went through similar issues benchmarking it about a year ago. At the end of the day, I found that with ZFS it was just very difficult to bypass, avoid, turn-off, etc. all the caching. The machine I was using has a lot of RAM. I had to reduce the RAM to like 64GB with a boot option to even see what was happening. I was still never thrilled with the numbers I got. Then, finally it dawned on me: with enough RAM, and an efficient filesystem like ZFS, the underlying storage is only a small part of the equation. The important thing is that your benchmark is accurate to the real world workload. If the real world can use a lot of buffer cache and RAM, and it makes it go fast - then good, use it like that. Which I did, and was happy, and forgot all about it. What we want to avoid is fio saying you have 100GB/s and then in the real world you don't see nearly that and are disappointed. To me benchmarks serve two purposes: comparison and prediction of performance with a reproducible test case, for example compare the effect of ARC cache size setting. Second, diagnosing whether the hardware is working correctly and as close to ideal. So I did see some stuff vaguely like you are seeing. I remember being impressed at ZFS and how hard it is to benchmark because it tries so hard to cache efficiently (or whatever). Unfortunately it was like a year ago so I don't remember more details.
    - fargle 5y ago
      OK, I found part of the problem. I resurrected my old setup and switched to using fio (was using iozone and some others). When you specify --filename= in fio but have multiple jobs, they are all sharing the same file (https://fio.readthedocs.io/en/latest/fio_doc.html https://fio.readthedocs.io/en/latest/fio_doc.html). So instead of 28 jobs reading 28 distinct files, you have reading the same file 28 times. So even when dropping the buffer-cache, reboot clean, etc. etc. before the benchmark, this is 27 times a test of the read caches and 1 time a test of the actual disk, which will obviously seriously inflate the results. I'm thinking --directory= would be the best option. I also did the following to drop (most) cached data before the test: sync; echo 3 > /proc/sys/vm/drop_caches; zpool export $pool; zpool import $pool

fargle 5y ago

I do think it sounds like compression might be part of it. I'm thinking that a) fio might create test data that is "trivially" compressible and b) this might occur before directio says "ignore the buffer cache and write directly to the drive", but then what gets written is 10x or 100x smaller. Another possibility is that somehow directio doesn't work when zfs+compression is turned on? I'd do a test where everything else is the same, but turn lz4 compression off. In theory, with a real world dataset, compression will cause CPU load to go up, but I would not expect to see the datasize at the disk go down by more than 2x to 3x, and therefore the bandwidth should not go up my anymore than that, and only if the CPU isn't the limiting factor. In most cases, compressed (or encrypted) filesystems should be slower, not faster because of this. So a benchmark without compression would be the first step. Then maybe as a second step, a "real world" benchmark - copy 10-100GB of representative data or something like that - with and without compression to see if bandwidth goes up or down and how bad CPU utilization goes up. Another estimate is to look at the stats that tell you what the compression ratio is. If it's really, really high, that just means fio created all-zeros, really fast. ???

neilpanchal 5y ago
Interesting. I turned off lz4 compression and it didn't make any difference - still getting around 80 GB/s on a 30 second test. I also checked the content of the dummy file, it is random and takes about the same time as using `dd` to pipe data from `/dev/random`. It's a mystery and it's related to RAM (see note 3). I think this would be an interesting question to ask on SO! See note 4: https://neil.computer/notes/zfs-raidz2/#note-4 https://neil.computer/notes/zfs-raidz2/#note-4
- fargle 5y ago
  Well that's mysterious. I have a similar but smaller system - two EVO 970s on a 4x4x4x4 PCI card just like you have (I planned on adding more if it went well). I went through similar issues benchmarking it about a year ago. At the end of the day, I found that with ZFS it was just very difficult to bypass, avoid, turn-off, etc. all the caching. The machine I was using has a lot of RAM. I had to reduce the RAM to like 64GB with a boot option to even see what was happening. I was still never thrilled with the numbers I got. Then, finally it dawned on me: with enough RAM, and an efficient filesystem like ZFS, the underlying storage is only a small part of the equation. The important thing is that your benchmark is accurate to the real world workload. If the real world can use a lot of buffer cache and RAM, and it makes it go fast - then good, use it like that. Which I did, and was happy, and forgot all about it. What we want to avoid is fio saying you have 100GB/s and then in the real world you don't see nearly that and are disappointed. To me benchmarks serve two purposes: comparison and prediction of performance with a reproducible test case, for example compare the effect of ARC cache size setting. Second, diagnosing whether the hardware is working correctly and as close to ideal. So I did see some stuff vaguely like you are seeing. I remember being impressed at ZFS and how hard it is to benchmark because it tries so hard to cache efficiently (or whatever). Unfortunately it was like a year ago so I don't remember more details.
  - fargle 5y ago
    OK, I found part of the problem. I resurrected my old setup and switched to using fio (was using iozone and some others). When you specify --filename= in fio but have multiple jobs, they are all sharing the same file (https://fio.readthedocs.io/en/latest/fio_doc.html https://fio.readthedocs.io/en/latest/fio_doc.html). So instead of 28 jobs reading 28 distinct files, you have reading the same file 28 times. So even when dropping the buffer-cache, reboot clean, etc. etc. before the benchmark, this is 27 times a test of the read caches and 1 time a test of the actual disk, which will obviously seriously inflate the results. I'm thinking --directory= would be the best option. I also did the following to drop (most) cached data before the test: sync; echo 3 > /proc/sys/vm/drop_caches; zpool export $pool; zpool import $pool