3 ms·

The limitation of "0x10000 operations" (64k) suprises me. Is that what can be submitted at once to the kernel, or also the amount of operations in-flight? Ther

The limitation of "0x10000 operations" (64k) suprises me. Is that what can be submitted at once to the kernel, or also the amount of operations in-flight? There's certainly a lot of servers out there which have more than 100k connections with low activity (keep-alive between HTTP requests, running messaging protocols, etc), where one read() operation would need to be enqueued for each connection for the case new data arrives or the connection closes. Is this supported or not?

wtallis 5y ago
My understanding of io_uring on Linux is that the submission queue only holds the recently-submitted entries that the kernel has not started processing. Once the kernel reads a Submission Queue Entry, it consumes that SQE and keeps track of command status using private memory, and a Completion Queue entry will not appear until the kernel is done handling that IO operation. So the Submission Queue size is not a limitation on how many in-flight/pending IOs there can be, just on how big a batch you can dump on the kernel at once. Of course, the kernel's internal limits on operation state tracking might be smaller or larger than the Submission Queue size.
- Matthias247 5y ago
  Right. That would make sense. Would be interesting to know what the limit for submitted operations is (if any)
IvyMike 5y ago
It's unclear if there's a global limitation, but my reading is that the limit of 64k operations is per IORING_OBJECT. There's nothing that says you can't have multiple submission queues; for example, the NVMe spec allows 64k commands per queue on up to 64k queues. (On the other hand, there's nothing that says that multiple submission queues is a good match for any given application either, but in the storage world it often is). In any case, given how popular NVMe is, I assume IORING_OBJECT would support this model. https://blog.westerndigital.com/nvme-queues-explained/ https://blog.westerndigital.com/nvme-queues-explained/
- yShafir 5y ago
  That is correct - the 64K limitation is per IORING_OBJECT. Nothing stopping a process from creating multiple objects, as many as its handle table allows actually, so very similar to NVMe.

wtallis 5y ago

My understanding of io_uring on Linux is that the submission queue only holds the recently-submitted entries that the kernel has not started processing. Once the kernel reads a Submission Queue Entry, it consumes that SQE and keeps track of command status using private memory, and a Completion Queue entry will not appear until the kernel is done handling that IO operation. So the Submission Queue size is not a limitation on how many in-flight/pending IOs there can be, just on how big a batch you can dump on the kernel at once. Of course, the kernel's internal limits on operation state tracking might be smaller or larger than the Submission Queue size.

Matthias247 5y ago
Right. That would make sense. Would be interesting to know what the limit for submitted operations is (if any)

IvyMike 5y ago

It's unclear if there's a global limitation, but my reading is that the limit of 64k operations is per IORING_OBJECT. There's nothing that says you can't have multiple submission queues; for example, the NVMe spec allows 64k commands per queue on up to 64k queues. (On the other hand, there's nothing that says that multiple submission queues is a good match for any given application either, but in the storage world it often is). In any case, given how popular NVMe is, I assume IORING_OBJECT would support this model. https://blog.westerndigital.com/nvme-queues-explained/ https://blog.westerndigital.com/nvme-queues-explained/

yShafir 5y ago
That is correct - the 64K limitation is per IORING_OBJECT. Nothing stopping a process from creating multiple objects, as many as its handle table allows actually, so very similar to NVMe.