3 ms·

There is nothing that forces at least a single copy. You could do some processing and write directly into a buffer registered to the kernel.

vlovich123 3y ago
I mean if we want to be pedantic, the buffer has to be copied from main memory into whatever the HW uses for staging to generate the PHY signal. But that’s typically an DMA operation not an CPU memcpy. Even with io_uring though, afaik there’s inevitably a memcpy to create the sk_buf to hand off to the network driver no? I’m more fuzzy on how that stuff works but I don’t think the sk_buf is implemented as a list of iovecs. You could go the DPDK route but that has downsides of your application needing exclusive access to the network interface. AFAIK, any network stack that supports multiple simultaneous applications typically involves at least 1 copy but I could be off.
- camel_gopher 3y ago
  “Even with io_uring though, afaik there’s inevitably a memcpy to create the sk_buf to hand off to the network driver no? I’m more fuzzy on how that stuff works but I don’t think the sk_buf is implemented as a list of iovecs.” Having worked with sk_bufs a bit I hope a copy isn’t needed for a network send. All the tooling I worked with around those was zero copy.
- Galanwe 3y ago
  > You could go the DPDK route but that has downsides of your application needing exclusive access to the network interface. Never did it, but I guess you could reroute traffic you don't care about to the Linux network stack for regular routing. https://doc.dpdk.org/guides-16.07/howto/flow_bifurcation.html https://doc.dpdk.org/guides-16.07/howto/flow_bifurcation.htm... Not sure that's a realistic scenario though. You rarely expose a DPDK app directly on a regular / open subnet. You often have a dedicated app, with a dedicated NIC, on a dedicated subnet, for a dedicated traffic.

vlovich123 3y ago

I mean if we want to be pedantic, the buffer has to be copied from main memory into whatever the HW uses for staging to generate the PHY signal. But that’s typically an DMA operation not an CPU memcpy. Even with io_uring though, afaik there’s inevitably a memcpy to create the sk_buf to hand off to the network driver no? I’m more fuzzy on how that stuff works but I don’t think the sk_buf is implemented as a list of iovecs. You could go the DPDK route but that has downsides of your application needing exclusive access to the network interface. AFAIK, any network stack that supports multiple simultaneous applications typically involves at least 1 copy but I could be off.

camel_gopher 3y ago
“Even with io_uring though, afaik there’s inevitably a memcpy to create the sk_buf to hand off to the network driver no? I’m more fuzzy on how that stuff works but I don’t think the sk_buf is implemented as a list of iovecs.” Having worked with sk_bufs a bit I hope a copy isn’t needed for a network send. All the tooling I worked with around those was zero copy.
Galanwe 3y ago
> You could go the DPDK route but that has downsides of your application needing exclusive access to the network interface. Never did it, but I guess you could reroute traffic you don't care about to the Linux network stack for regular routing. https://doc.dpdk.org/guides-16.07/howto/flow_bifurcation.html https://doc.dpdk.org/guides-16.07/howto/flow_bifurcation.htm... Not sure that's a realistic scenario though. You rarely expose a DPDK app directly on a regular / open subnet. You often have a dedicated app, with a dedicated NIC, on a dedicated subnet, for a dedicated traffic.