4 ms·

It's one thread per ring, not one thread per socket or FileDescriptor that dispatches events to the ring.

Const-me 7y ago
How many rings should I create on a server with 100GB RAM, 64 cores, that handles 100k connections? Even if there's a definite answer to that, I still don't understand why threads are necessary? While CPU waits for completed IO, it doesn't poll anything. The CPU fires a DMA request at hardware, and forgets about it until it gets the response.
- geocar 7y ago
  It depends on your application, so measure! Ideally the CPU never waits for completed IO, it's busy building the next set of responses, or writing out the new state to the rest of the application. If you're ever "waiting", then you should probably be putting another/bigger nic in there or rethinking what your application actually needs to do: 10Gb ethernet is only about 10x slower than main memory, so 10 threads could be enough if you have a single memory fetch to service the response, and if you need two, then 20 threads. If you need more than six, then you should definitely rearchitect your application, since you're out of cores and time-slicing again.
  - cafxx 7y ago
    > 10Gb ethernet is only about 10x slower than main memory Server-class CPUs have much higher bandwidth nowadays: > The total memory bandwidth available to a single CPU clocks in at 170 GB/s. (i.e. ~136x slower than main memory) https://www.anandtech.com/show/11183/amd-prepares-32-core-naples-cpus-for-1p-and-2p-servers-coming-in-q2 https://www.anandtech.com/show/11183/amd-prepares-32-core-na...
    - geocar 7y ago
      The per-channel number[1] is much better (imo) for napkin spec'ing because it's what you'll get in a typical networked application that's already done and minimised stalls. But you're raising a good point: If you can arrange your data set across the physical DIMMS yes you can get much another 10x, and maybe the contortions you need to do to get it can be worth it (if you're actually memory bound). [1]: better link, since the one you posted is bust: https://developer.amd.com/wp-content/resources/56301_1.0.pdf https://developer.amd.com/wp-content/resources/56301_1.0.pdf
  - hinkley 7y ago
    Would you put a new ethernet card into a machine where the network is saturated 5% of the time? There's a huge space between 'never waiting' and 'always waiting'.
- Matthias247 7y ago
  That section describes the "polling". You are right that with normal IO, there doesn't need to be polling, since the completion will be signaled by interrupt. However as described in the document polling can be faster for some use-cases. And for those there needs to be one thread which performs the polling for completions. That one can be in the Kernel.
  - Const-me 7y ago
    Looked more closely, indeed you’re right. That section discusses an optional mechanism how the kernel gets notified about new IO requests submit by the application. It’s not for IO completion polling. For that use case, having a kernel thread per uring makes perfect sense.

How many rings should I create on a server with 100GB RAM, 64 cores, that handles 100k connections? Even if there's a definite answer to that, I still don't understand why threads are necessary? While CPU waits for completed IO, it doesn't poll anything. The CPU fires a DMA request at hardware, and forgets about it until it gets the response.

geocar 7y ago
It depends on your application, so measure! Ideally the CPU never waits for completed IO, it's busy building the next set of responses, or writing out the new state to the rest of the application. If you're ever "waiting", then you should probably be putting another/bigger nic in there or rethinking what your application actually needs to do: 10Gb ethernet is only about 10x slower than main memory, so 10 threads could be enough if you have a single memory fetch to service the response, and if you need two, then 20 threads. If you need more than six, then you should definitely rearchitect your application, since you're out of cores and time-slicing again.
- cafxx 7y ago
  > 10Gb ethernet is only about 10x slower than main memory Server-class CPUs have much higher bandwidth nowadays: > The total memory bandwidth available to a single CPU clocks in at 170 GB/s. (i.e. ~136x slower than main memory) https://www.anandtech.com/show/11183/amd-prepares-32-core-naples-cpus-for-1p-and-2p-servers-coming-in-q2 https://www.anandtech.com/show/11183/amd-prepares-32-core-na...
  - geocar 7y ago
    The per-channel number[1] is much better (imo) for napkin spec'ing because it's what you'll get in a typical networked application that's already done and minimised stalls. But you're raising a good point: If you can arrange your data set across the physical DIMMS yes you can get much another 10x, and maybe the contortions you need to do to get it can be worth it (if you're actually memory bound). [1]: better link, since the one you posted is bust: https://developer.amd.com/wp-content/resources/56301_1.0.pdf https://developer.amd.com/wp-content/resources/56301_1.0.pdf
- hinkley 7y ago
  Would you put a new ethernet card into a machine where the network is saturated 5% of the time? There's a huge space between 'never waiting' and 'always waiting'.
Matthias247 7y ago
That section describes the "polling". You are right that with normal IO, there doesn't need to be polling, since the completion will be signaled by interrupt. However as described in the document polling can be faster for some use-cases. And for those there needs to be one thread which performs the polling for completions. That one can be in the Kernel.
- Const-me 7y ago
  Looked more closely, indeed you’re right. That section discusses an optional mechanism how the kernel gets notified about new IO requests submit by the application. It’s not for IO completion polling. For that use case, having a kernel thread per uring makes perfect sense.