4 ms·

Have you checked if using the fio options (--iodepth_batch_*) to batch submissions helps? Fio doesn't do that by default, and I found that that can be a signifi

Have you checked if using the fio options (--iodepth_batch_*) to batch submissions helps? Fio doesn't do that by default, and I found that that can be a significant benefit. Particularly submitting multiple up requests can amortize the cost of setting the nvme doorbell (the expensive part as far as I understand it) across multiple requests.

tanelpoder 6y ago
I tested various fio options, but didn't notice this one - I'll check it out! It might explain why I still kept seeing lots of interrupts raised even though I had enabled the I/O completion polling instead, with io_uring's --hipri option. edit: I ran a quick test with various IO batch sizes and it didn't make a difference - I guess because thanks to using io_uring, my bottleneck is not in IO submission, but deeper in the block IO stack...
- wtallis 6y ago
  I think on recent kernels, using the hipri option doesn't get you interrupt-free polled IO unless you've configured the nvme driver to allocate some queues specifically for polled IO. Since these Samsung drives support 128 queues and you're only using a 16C/32T processor, you have more than enough for each drive to have one poll queue and one regular IO queue allocated to each (virtual) CPU core.
  - tanelpoder 6y ago
    That would explain it. Do you recommend any docs/links I should read about allocating queues for polled IO?
    - anarazel 6y ago
      It's terribly documented :(. You need to set the nvme.poll_queues to the number of queues you want, before the disks are attached. I.e. either at boot, or you need to set the parameter and then cause the NVMe to be rescanned (you can do that in sysfs, but I can't immediately recall the steps with high confidence).
      - anarazel 6y ago
        Ah, yes, shell history ftw. Of course you should ensure no filesystem is mounted or such: root@awork3:~# echo 4 > /sys/module/nvme/parameters/poll_queues root@awork3:~# echo 1 > /sys/block/nvme1n1/device/reset_controller root@awork3:~# dmesg -c [749717.253101] nvme nvme1: 12/0/4 default/read/poll queues root@awork3:~# echo 8 > /sys/module/nvme/parameters/poll_queues root@awork3:~# dmesg -c root@awork3:~# echo 1 > /sys/block/nvme1n1/device/reset_controller root@awork3:~# dmesg -c [749736.513102] nvme nvme1: 8/0/8 default/read/poll queues
        tanelpoder 6y ago
        Thanks for the pointers, I'll bookmark this and will try it out someday.
- anarazel 6y ago
  > I tested various fio options, but didn't notice this one - I'll check it out! It might explain why I still kept seeing lots of interrupts raised even though I had enabled the I/O completion polling instead, with io_uring's --hipri option. I think that should be independent. > edit: I ran a quick test with various IO batch sizes and it didn't make a difference - I guess because thanks to using io_uring, my bottleneck is not in IO submission, but deeper in the block IO stack... It probably won't get you drastically higher speeds in an isolated test - but it should help reduce CPU overhead. E.g. on one of my SSDs fio --ioengine io_uring --rw randread --filesize 50GB --invalidate=0 --name=test --direct=1 --bs=4k --numjobs=1 --registerfiles --fixedbufs --gtod_reduce=1 --iodepth 48 uses about 25% more CPU than when I add --iodepth_batch_submit=0 --iodepth_batch_complete_max=0. But the resulting iops are nearly the same as long as there are enough cycles available. This is via filesystem, so ymmv, but the mechanism should be mostly independent.

tanelpoder 6y ago

I tested various fio options, but didn't notice this one - I'll check it out! It might explain why I still kept seeing lots of interrupts raised even though I had enabled the I/O completion polling instead, with io_uring's --hipri option. edit: I ran a quick test with various IO batch sizes and it didn't make a difference - I guess because thanks to using io_uring, my bottleneck is not in IO submission, but deeper in the block IO stack...

wtallis 6y ago
I think on recent kernels, using the hipri option doesn't get you interrupt-free polled IO unless you've configured the nvme driver to allocate some queues specifically for polled IO. Since these Samsung drives support 128 queues and you're only using a 16C/32T processor, you have more than enough for each drive to have one poll queue and one regular IO queue allocated to each (virtual) CPU core.
- tanelpoder 6y ago
  That would explain it. Do you recommend any docs/links I should read about allocating queues for polled IO?
  - anarazel 6y ago
    It's terribly documented :(. You need to set the nvme.poll_queues to the number of queues you want, before the disks are attached. I.e. either at boot, or you need to set the parameter and then cause the NVMe to be rescanned (you can do that in sysfs, but I can't immediately recall the steps with high confidence).
    - anarazel 6y ago
      Ah, yes, shell history ftw. Of course you should ensure no filesystem is mounted or such: root@awork3:~# echo 4 > /sys/module/nvme/parameters/poll_queues root@awork3:~# echo 1 > /sys/block/nvme1n1/device/reset_controller root@awork3:~# dmesg -c [749717.253101] nvme nvme1: 12/0/4 default/read/poll queues root@awork3:~# echo 8 > /sys/module/nvme/parameters/poll_queues root@awork3:~# dmesg -c root@awork3:~# echo 1 > /sys/block/nvme1n1/device/reset_controller root@awork3:~# dmesg -c [749736.513102] nvme nvme1: 8/0/8 default/read/poll queues
      - tanelpoder 6y ago
        Thanks for the pointers, I'll bookmark this and will try it out someday.
anarazel 6y ago
> I tested various fio options, but didn't notice this one - I'll check it out! It might explain why I still kept seeing lots of interrupts raised even though I had enabled the I/O completion polling instead, with io_uring's --hipri option. I think that should be independent. > edit: I ran a quick test with various IO batch sizes and it didn't make a difference - I guess because thanks to using io_uring, my bottleneck is not in IO submission, but deeper in the block IO stack... It probably won't get you drastically higher speeds in an isolated test - but it should help reduce CPU overhead. E.g. on one of my SSDs fio --ioengine io_uring --rw randread --filesize 50GB --invalidate=0 --name=test --direct=1 --bs=4k --numjobs=1 --registerfiles --fixedbufs --gtod_reduce=1 --iodepth 48 uses about 25% more CPU than when I add --iodepth_batch_submit=0 --iodepth_batch_complete_max=0. But the resulting iops are nearly the same as long as there are enough cycles available. This is via filesystem, so ymmv, but the mechanism should be mostly independent.