4 ms·

Sounds like servers handling lots of small UDP packets would be hit pretty hard.

revelation 9y ago
Applications like this where the syscall overhead (and latency) starts to be a significant factor in processing time and latency have moved to userland drivers anyway: DPDK for 10-100 Gbps networking: https://dpdk.org/ https://dpdk.org/ SPDK for NVMe storage: http://www.spdk.io/ http://www.spdk.io/ The queuing and balancing stuff the kernel does makes sense for spinning rust harddisks and residential networking, but when the underlying hardware is so fast that nothing is ever queued, really what are you doing. At 100 Gbps line speed, a 1518 byte packet takes all of ~ 120ns to transmit, or about 360 clock cycles for a 3 GHz processor.
- emj 9y ago
  Taking control over network cards in user space seems doable nowdays. There was a talk about doing such drivers with IOMMU/DMA at CCC: https://media.ccc.de/v/34c3-9159-demystifying_network_cards https://media.ccc.de/v/34c3-9159-demystifying_network_cards
  - arghwhat 9y ago
    User-space drivers have been doable for a while, and dpdk[1] is definitely worth a check. There's also some manufacturers[2] that only does user-space drivers for their high-performance cards (e.g. 4x10Gb/s, 2x40Gb/s, 2x100Gb/s cards). Being designed with this in mind helps performance a lot. 1: https://dpdk.org/ https://dpdk.org/ 2: http://www.napatech.com/ http://www.napatech.com/
- DSMan195276 9y ago
  > Applications like this where the syscall overhead (and latency) starts to be a significant factor in processing time and latency have moved to userland drivers anyway: I would personally think that is worse, though please correct me if I'm wrong. The userland driver will run with an isolated PT like any other userland process won't it? If so, it will suffer the same slowdown that every other process now has every-time it has to communicate with the kernel, which I would think would be a lot for a driver.
  - revelation 9y ago
    It's counter intuitive at first, but the key to understand how this works is that while you can use an MMU to assign chunks of physical memory to a process, you can of course also just use the MMU to assign the memory mapped IO registers of say a PCI express peripheral to a process. That is in a nutshell what a "userland driver" is. It's not too far removed from poking the parallel port at 0x378 on your DOS computer :)
  - arghwhat 9y ago
    A user-space driver doesn't communicate with the kernel. It is assigned DMA buffers, and communicates with the NIC solely through reading and writing to shared memory buffers. Even before this fix, the benefits were massive, as sending a buffer was just writing to some memory, rather than syscalls and copies galore.
zmatt 9y ago
Reduce number of syscalls by using sendmmsg()/recvmmsg() to batch together multiple packets per syscall?
jamiesonbecker 9y ago
Anyone run any DNS server benchmarks, esp BIND and PowerDNS?

Applications like this where the syscall overhead (and latency) starts to be a significant factor in processing time and latency have moved to userland drivers anyway: DPDK for 10-100 Gbps networking: https://dpdk.org/ https://dpdk.org/ SPDK for NVMe storage: http://www.spdk.io/ http://www.spdk.io/ The queuing and balancing stuff the kernel does makes sense for spinning rust harddisks and residential networking, but when the underlying hardware is so fast that nothing is ever queued, really what are you doing. At 100 Gbps line speed, a 1518 byte packet takes all of ~ 120ns to transmit, or about 360 clock cycles for a 3 GHz processor.

emj 9y ago
Taking control over network cards in user space seems doable nowdays. There was a talk about doing such drivers with IOMMU/DMA at CCC: https://media.ccc.de/v/34c3-9159-demystifying_network_cards https://media.ccc.de/v/34c3-9159-demystifying_network_cards
- arghwhat 9y ago
  User-space drivers have been doable for a while, and dpdk[1] is definitely worth a check. There's also some manufacturers[2] that only does user-space drivers for their high-performance cards (e.g. 4x10Gb/s, 2x40Gb/s, 2x100Gb/s cards). Being designed with this in mind helps performance a lot. 1: https://dpdk.org/ https://dpdk.org/ 2: http://www.napatech.com/ http://www.napatech.com/
DSMan195276 9y ago
> Applications like this where the syscall overhead (and latency) starts to be a significant factor in processing time and latency have moved to userland drivers anyway: I would personally think that is worse, though please correct me if I'm wrong. The userland driver will run with an isolated PT like any other userland process won't it? If so, it will suffer the same slowdown that every other process now has every-time it has to communicate with the kernel, which I would think would be a lot for a driver.
- revelation 9y ago
  It's counter intuitive at first, but the key to understand how this works is that while you can use an MMU to assign chunks of physical memory to a process, you can of course also just use the MMU to assign the memory mapped IO registers of say a PCI express peripheral to a process. That is in a nutshell what a "userland driver" is. It's not too far removed from poking the parallel port at 0x378 on your DOS computer :)
- arghwhat 9y ago
  A user-space driver doesn't communicate with the kernel. It is assigned DMA buffers, and communicates with the NIC solely through reading and writing to shared memory buffers. Even before this fix, the benefits were massive, as sending a buffer was just writing to some memory, rather than syscalls and copies galore.

zmatt 9y ago

Reduce number of syscalls by using sendmmsg()/recvmmsg() to batch together multiple packets per syscall?

jamiesonbecker 9y ago

Anyone run any DNS server benchmarks, esp BIND and PowerDNS?