4 ms·

Only somewhat related, I've been wondering if you could use zero copy networking with io_uring to send the same bytes over and over. Say you have some data that

by foota 3y ago

Only somewhat related, I've been wondering if you could use zero copy networking with io_uring to send the same bytes over and over. Say you have some data that needs to be send to the different clients (imagine maybe it's a video game sending updates to players), from what I can tell you should be able to create a "fixed" buffer with this contents and re-use it between calls in the ring. Not only would this be zero copy, but you also wouldn't ever need to touch the data at all to send it after the first time. (normally, zero copy is more like 1 copy, since you need to copy it in to the kernel allocated buffer even with zero copy). My understanding is that there's some contention from the locking involved though which can make this not scale well, but maybe that could be avoided (e.g., maybe by trying to have per core buffers used by the kernel, although I'm not sure you'd have any control over the kernel threads).

rowanG077 3y ago
There is nothing that forces at least a single copy. You could do some processing and write directly into a buffer registered to the kernel.
- vlovich123 3y ago
  I mean if we want to be pedantic, the buffer has to be copied from main memory into whatever the HW uses for staging to generate the PHY signal. But that’s typically an DMA operation not an CPU memcpy. Even with io_uring though, afaik there’s inevitably a memcpy to create the sk_buf to hand off to the network driver no? I’m more fuzzy on how that stuff works but I don’t think the sk_buf is implemented as a list of iovecs. You could go the DPDK route but that has downsides of your application needing exclusive access to the network interface. AFAIK, any network stack that supports multiple simultaneous applications typically involves at least 1 copy but I could be off.
  - camel_gopher 3y ago
    “Even with io_uring though, afaik there’s inevitably a memcpy to create the sk_buf to hand off to the network driver no? I’m more fuzzy on how that stuff works but I don’t think the sk_buf is implemented as a list of iovecs.” Having worked with sk_bufs a bit I hope a copy isn’t needed for a network send. All the tooling I worked with around those was zero copy.
  - Galanwe 3y ago
    > You could go the DPDK route but that has downsides of your application needing exclusive access to the network interface. Never did it, but I guess you could reroute traffic you don't care about to the Linux network stack for regular routing. https://doc.dpdk.org/guides-16.07/howto/flow_bifurcation.html https://doc.dpdk.org/guides-16.07/howto/flow_bifurcation.htm... Not sure that's a realistic scenario though. You rarely expose a DPDK app directly on a regular / open subnet. You often have a dedicated app, with a dedicated NIC, on a dedicated subnet, for a dedicated traffic.
wmf 3y ago
This sounds pretty straightforward using sendfile(); you shouldn't even need io_uring.
- foota 3y ago
  I guess you would have some kind of in memory fd that you would write to and then use with sendfile calls? I'm not sure if you'd run into similar contention on the data structures involved there. I guess it would depend on specifically what mechanism you're using to store the data (something like tmpfs?), but probably doesn't have to involve contention? (as there wouldn't have to be any refcounting if you control the lifetime of the data). I was thinking of a situation where you want to also limit the system call overhead per send. (think maybe there's some number of 64k-ish chunks that you want to send to different clients, so each connection would involve multiple write per frame etc.,)
nelsondev 3y ago
Reminds me of a Kafka topic, where different Consumers each have their own commit that indicates how far along the stream they’ve caught up to.
chatmasta 3y ago
Interesting. But wouldn't this mean some clients would need to wait a long time for their download to start? (I'm imagining a multi-gigabyte game update - but maybe you meant a continuous stream of in-game positional updates of players.) They basically idle until the next "train" shows up at their station, and then they hop on, right? Also, if they have a network hiccup, they can't resume their download until one of the trains reaches the point where their download failed. (That is, if you get my weird train analogy, and if I'm understanding your idea correctly.)
- foota 3y ago
  I was thinking of a game like minecraft or similar (essentially, a ~large world with lots of entities updating in a grid of chunks). In that case, you generally have people fairly clustered in different grid tiles, and they're all likely to get similar information about each chunk, so it seems like re-using the buffers between calls would be beneficial. So instead of: for each player: for each block player needs: # Write block to player's connection You could do: for each player: for each block player needs: # Check if the block is already mapped in a kernel buffer. If it is, just append a write for it to the io ring. # Otherwise, copy the block into a new buffer and send it.
10000truths 3y ago
I imagine that wouldn't be very useful, since most production traffic is encrypted (e.g. TLS) and key negotiations necessitate a per-client cipher stream.
- toast0 3y ago
  Advanced NICs can do bulk TLS encryption... probably other encryption if you work with the manufacturer. See the work Netflix CDN has done, at high bitrates, memory bandwidth is a major bottleneck for them, eliminating the memory reads and writes to do TLS in software (user level or kernel level) allowed them to hit much higher speeds.
toast0 3y ago
> although I'm not sure you'd have any control over the kernel threads You don't have control from the application, but if you're really trying to get the most performance from your networking heavy application, you want to set up your server where the NIC rx and tx queues are cpu pinned, and the application thread that processes any given socket is pinned to the same cpu that the kernel is using for that socket. Then there's no cross cpu traffic for that socket (if all else goes well). Hopefully all the memory used is NUMA local for the CPU and the NIC as well. That gets trickier with multiple socket systems, although some NICs do have provisions to connect to two PCIe slots so they can be NUMA local to two sockets.
m463 3y ago
Multicast (and broadcast) was supposed to do things like this, but I have rarely seen it used for any sort of "visible" client application. (I mean visible like a videoconferencing application as opposed to something like invisible multicast dns)
- bpye 3y ago
  Multicast does get used for IPTV doesn’t it?
- Galanwe 3y ago
  > Multicast was supposed to do things like this, but I have rarely seen it used for any sort of "visible" client application. Well multicast is not possible on internet so I guess that's why you don't see it often. In private network centric protocols though (e.g. finance), multicast is ubiquitous.

by foota 3y ago

rowanG077 3y ago

There is nothing that forces at least a single copy. You could do some processing and write directly into a buffer registered to the kernel.

vlovich123 3y ago
I mean if we want to be pedantic, the buffer has to be copied from main memory into whatever the HW uses for staging to generate the PHY signal. But that’s typically an DMA operation not an CPU memcpy. Even with io_uring though, afaik there’s inevitably a memcpy to create the sk_buf to hand off to the network driver no? I’m more fuzzy on how that stuff works but I don’t think the sk_buf is implemented as a list of iovecs. You could go the DPDK route but that has downsides of your application needing exclusive access to the network interface. AFAIK, any network stack that supports multiple simultaneous applications typically involves at least 1 copy but I could be off.
- camel_gopher 3y ago
  “Even with io_uring though, afaik there’s inevitably a memcpy to create the sk_buf to hand off to the network driver no? I’m more fuzzy on how that stuff works but I don’t think the sk_buf is implemented as a list of iovecs.” Having worked with sk_bufs a bit I hope a copy isn’t needed for a network send. All the tooling I worked with around those was zero copy.
- Galanwe 3y ago
  > You could go the DPDK route but that has downsides of your application needing exclusive access to the network interface. Never did it, but I guess you could reroute traffic you don't care about to the Linux network stack for regular routing. https://doc.dpdk.org/guides-16.07/howto/flow_bifurcation.html https://doc.dpdk.org/guides-16.07/howto/flow_bifurcation.htm... Not sure that's a realistic scenario though. You rarely expose a DPDK app directly on a regular / open subnet. You often have a dedicated app, with a dedicated NIC, on a dedicated subnet, for a dedicated traffic.

wmf 3y ago

This sounds pretty straightforward using sendfile(); you shouldn't even need io_uring.

foota 3y ago
I guess you would have some kind of in memory fd that you would write to and then use with sendfile calls? I'm not sure if you'd run into similar contention on the data structures involved there. I guess it would depend on specifically what mechanism you're using to store the data (something like tmpfs?), but probably doesn't have to involve contention? (as there wouldn't have to be any refcounting if you control the lifetime of the data). I was thinking of a situation where you want to also limit the system call overhead per send. (think maybe there's some number of 64k-ish chunks that you want to send to different clients, so each connection would involve multiple write per frame etc.,)

nelsondev 3y ago

Reminds me of a Kafka topic, where different Consumers each have their own commit that indicates how far along the stream they’ve caught up to.

chatmasta 3y ago

Interesting. But wouldn't this mean some clients would need to wait a long time for their download to start? (I'm imagining a multi-gigabyte game update - but maybe you meant a continuous stream of in-game positional updates of players.) They basically idle until the next "train" shows up at their station, and then they hop on, right? Also, if they have a network hiccup, they can't resume their download until one of the trains reaches the point where their download failed. (That is, if you get my weird train analogy, and if I'm understanding your idea correctly.)

foota 3y ago
I was thinking of a game like minecraft or similar (essentially, a ~large world with lots of entities updating in a grid of chunks). In that case, you generally have people fairly clustered in different grid tiles, and they're all likely to get similar information about each chunk, so it seems like re-using the buffers between calls would be beneficial. So instead of: for each player: for each block player needs: # Write block to player's connection You could do: for each player: for each block player needs: # Check if the block is already mapped in a kernel buffer. If it is, just append a write for it to the io ring. # Otherwise, copy the block into a new buffer and send it.

10000truths 3y ago

I imagine that wouldn't be very useful, since most production traffic is encrypted (e.g. TLS) and key negotiations necessitate a per-client cipher stream.

toast0 3y ago
Advanced NICs can do bulk TLS encryption... probably other encryption if you work with the manufacturer. See the work Netflix CDN has done, at high bitrates, memory bandwidth is a major bottleneck for them, eliminating the memory reads and writes to do TLS in software (user level or kernel level) allowed them to hit much higher speeds.

toast0 3y ago

> although I'm not sure you'd have any control over the kernel threads You don't have control from the application, but if you're really trying to get the most performance from your networking heavy application, you want to set up your server where the NIC rx and tx queues are cpu pinned, and the application thread that processes any given socket is pinned to the same cpu that the kernel is using for that socket. Then there's no cross cpu traffic for that socket (if all else goes well). Hopefully all the memory used is NUMA local for the CPU and the NIC as well. That gets trickier with multiple socket systems, although some NICs do have provisions to connect to two PCIe slots so they can be NUMA local to two sockets.

m463 3y ago

Multicast (and broadcast) was supposed to do things like this, but I have rarely seen it used for any sort of "visible" client application. (I mean visible like a videoconferencing application as opposed to something like invisible multicast dns)

bpye 3y ago
Multicast does get used for IPTV doesn’t it?
Galanwe 3y ago
> Multicast was supposed to do things like this, but I have rarely seen it used for any sort of "visible" client application. Well multicast is not possible on internet so I guess that's why you don't see it often. In private network centric protocols though (e.g. finance), multicast is ubiquitous.