10 ms·

XDP for game programmers

109 points by gafferongames 3y ago

10000truths 3y ago
You don't need to create your own kernel modules to define custom packet processing logic. You can map an RX/TX ring into userspace memory by invoking setsockopt on your XDP socket with the XDP_RX_RING, XDP_TX_RING and XDP_UMEM_REG options. Your XDP BPF program can then choose to redirect an incoming packet to the RX ring (see BPF_MAP_TYPE_XSKMAP).
- tptacek 3y ago
  What I think you're describing here is AF_XDP (just for a simpler search term).
posix_monad 3y ago
I was hoping a few code snippets. So these mini kernel programs are written in a subset of C?
- gafferongames 3y ago
  All the code is here: https://github.com/mas-bandwidth/xdp https://github.com/mas-bandwidth/xdp
deleted 3y ago
[deleted]
spintin 3y ago
The real way to do this is to get user space networking. Also event-based protocols with deterministic physics. Last but not least, you need to use a language that can atomically share memory between threads; C (with Arrays of 64 byte Structs) or Java.
gafferongames 3y ago
[dead]
nicebyte 3y ago
I'm calling eBPF "kernel shaders" to confuse graphics, os and gpgpu people all at the same time.
- tgaj 3y ago
  This analogy works well when trying to describe how eBPF is used for network applications. The eBPF scripts are "packet shaders" - like a "pixel shaders" they are executed for every packet independently and can modify attributes and/or payload according to a certain algorithm.
- danwills 3y ago
  Seems to have worked on me! Well played! :)
- meindnoch 3y ago
  I'm stealing this.
- sakras 3y ago
  I've been exactly thinking about it this way for a long time. Actually once we're able to push computation down into our disk drives, I wouldn't be surprised if these "Disk Shaders" will be written in eBPF.
  - danwills 3y ago
    Totally sibling comment confirms it already exists! I hope that the 'shader' name sticks too! I find the idea of a shader has a very appropriate shape for tiny-program-embedded-in-specific-context so it seems perfect from a hacker POV! I have a VFX(Houdini now, RSL shaders etc earlier) and openCL-dabbling and demoscene-lurking background, based on which I think I prefer 'shader' to 'kernel', that's what OpenCL calls them.. but that conflicts with the name of like, 'the OS kernel' at least somewhat..
  - sirsinsalot 3y ago
    I read that as eBNF and was very confused
hobs 3y ago
> Within 10 years, everybody is going to have 10gbps internet LOL, no way in America is that going to be true.
- voidUpdate 3y ago
  I live on the outskirts of a city in the middle of England and I /very/ much doubt I'll see 10gig in 10 years
  - fragmede 3y ago
    What if that kind of bandwidth becomes available over cell networks?
    - voidUpdate 3y ago
      I've restricted my phone to only using 4G because I've had a lot of problems with 5G feeling a lot slower than 4G. I've not done any quantitative testing on it but it just generally feels a lot slower, even when I'm looking right at the 5G mast, and the 4G mast is behind a few buildings. If whatever problem it's having can be fixed, and it can be priced similarly to broadband, then maybe
- gafferongames 3y ago
  https://www.cablelabs.com/10g https://www.cablelabs.com/10g
  - hobs 3y ago
    Is this supposed to be a joke? I cant even reliably get 1G in most places.
    - hansvm 3y ago
      I can't get 0.05G reliably in most suburbs, even when I do the worst-case ping in a typical 10-minute window is 500+ms, and there are still many places in the US where the best internet I can find is under 0.001G.
Animats 3y ago
I've actually written a high-performance metaverse client, one that can usefully pull half a gigabit per second and more from the Internet. So I get to see this happening. I'm looking at a highly detailed area right now, from the air, and traffic peaked around 200Mb/s. This XDP thing seems to address the wrong problem. Actual traffic for a metaverse is mostly bulk content download. Highly interactive traffic over UDP is maybe 1MB/second, including voice. You're mostly sending positions and orientations for moving objects. Latency matters for that, but an extra few hundred microseconds won't hurt. The rest is large file transfers. Those may be from totally different servers than the ones that talk interactive UDP. There's probably a CDN involved, and you're talking to caches. Latency doesn't matter that much, but big-block bandwidth does. Practical problems include data caps. If you go driving around a big metaverse, you can easily pull 200GB/hour from the asset servers. Don't try this on "AT&T Unlimited Extra® EL". Check your data plan. The last thing you want is game-specific code in the kernel. That creates a whole new attack surface.
- lisplist 3y ago
  What is a metaverse client? Do you just mean a cross platform VR app?
  - sbierwagen 3y ago
    It's a client for https://secondlife.com/ https://secondlife.com/
- gafferongames 3y ago
  > Actual traffic for a metaverse is mostly bulk content download. Highly interactive traffic over UDP is maybe 1MB/second, including voice. Typical bandwidth for multiplayer games like FPS (Counterstrike, Apex Legends) are around 512kbps-1mbit per-second down per-client, and this is old information, newer games almost certainly use more. It's easy to see a more high fidelity gaming experience taking 10mbit - 100mbit traffic from server to client, just increase the size and fidelity of the world. next, increase player counts and you can easily fill 10gbit/sec for a future FPS/MMO hybrid. God save us from the egress BW costs though :)
Galanwe 3y ago
This article is plain ridiculous. > Why? Because otherwise, the overhead of processing each packet in the kernel and passing it down to user space and back up to the kernel and out to the NIC limits the throughput you can achieve. We're talking 10gbps and above here. _Throughpout_ is not problematic at all for the Linux network stack, even at 100gbps. What is problematic is >10gbps line rate. In other words, unless you're receiving 10gbps unshaped UDP datagrams with no payloads at line rate, the problem is non existant. Considering internet is 99% fat TCP packets, this sentence is completely absurd. > With other kernel bypass technologies like DPDK you needed to install a second NIC to run your program or basically implement (or license) an entire TCP/IP network stack to make sure that everything works correctly under the hood That is just wrong on so many levels. First, DPDK allows reinjecting packets in the Linux network stack. That is called queue splitting,is done by the NIC, and can be trivially achieved using e.g. the bifurcated driver. Second, there are plenty of available performant network stacks out there, especially considering high end NICs implement 80% of the performance sensitive parts of the stack on chip. Last, kernel bypassing is made on _trusted private networks_, you would have to be crazy or damn well know what you're doing to bypass on publicly addressable networks, otherwise you will have a bad reality check. There are decades of security checks and counter measures baked in the Linux network stack that a game would be irresponsible to ask his players to skip. I'm not even mentioning the ridiculous latency gains to be achieved here. Wire tapping the packet "NIC in" to userspace buffer should be in the ballpark of 3us. If you think you can do better and this latency is too much for your application, you're either day dreaming or you're not working in the video game industry.
- cornel_io 3y ago
  Are you under the impression that he was talking about game clients doing this? That would be absurd, since no gamers (within epsilon of zero) use Linux, and you'd need a completely different bag of tricks on Windows. He's fully focused on the backend.
- intelVISA 3y ago
  It's not ridiculous but it is quite dated - there's better approaches for packet processing at scale nowadays.
mondobe 3y ago
> Within 10 years, everybody is going to have 10gbps internet. How will this change how games are made? How are we going to use all this bandwidth? 5v5 shooters just aren't going to cut it anymore. I have no doubt that more bandwidth will change how games are made, but 5v5 shooters (and pretty much all existing multiplayer styles) are here to stay for a lot longer than that, in some form or another.
- deleted 3y ago
  [deleted]
jay_kyburz 3y ago
Hello from Canberra Glenn!
cookiengineer 3y ago
[flagged]
- gafferongames 3y ago
  The eBPF code runs on the game server not the client. Cool story tho
  - Galanwe 3y ago
    If you want to use kernel bypass on the server, not the client, then XDP is even more useless. Because on the server world, you have access to professional NICs that you cannot expect clients to have. It's been decades now that people routinely do kernel bypass on servers using e. g. Solarflare NICs with OnLoad. You just write your networking code normally using BSD sockets API, and LD_PRELOAD the OnLoad SDK which will intercept the socket API and perform a bypass. You get 90% of the latency benefits of kernel bypass without modifying your code, which still runs everywhere.
    - tptacek 3y ago
      I think you're a little bit on tilt here. "The server world" was what XDP was invented for, and it's used all over cloud providers. Watch the XDP mailing list.
    - mc10 3y ago
      Both Katran[1] by Facebook and Unimog[2] by Cloudflare are L4 load balancers written using XDP. It's factually incorrect to claim that XDP is useless on servers as many companies intentionally choose to design their systems around commodity hardware. [1]: https://engineering.fb.com/2018/05/22/open-source/open-sourcing-katran-a-scalable-network-load-balancer/ [2]: https://blog.cloudflare.com/unimog-cloudflares-edge-load-balancer
  - cookiengineer 3y ago
    Doesn't mean you should risk a takeover of all your game databases because you decided to bypass every single security measurement linux has in place already. I'd argue it's even worse, and network programming wise I'd argue that every custom implementation of TCP/UDP was broken in the past and lead to a lot of remote exploits, and eventually was abandoned due to maintenance costs. Just think about the shitshow of the source engine, for example. RCE'ing in the wild for what, 14+ years now? Took a long while and a lot of steam runtime sandboxing code to fix the decision to use a custom unsafe networking code.

109 points by gafferongames 3y ago

You don't need to create your own kernel modules to define custom packet processing logic. You can map an RX/TX ring into userspace memory by invoking setsockopt on your XDP socket with the XDP_RX_RING, XDP_TX_RING and XDP_UMEM_REG options. Your XDP BPF program can then choose to redirect an incoming packet to the RX ring (see BPF_MAP_TYPE_XSKMAP).

tptacek 3y ago
What I think you're describing here is AF_XDP (just for a simpler search term).

posix_monad 3y ago

I was hoping a few code snippets. So these mini kernel programs are written in a subset of C?

gafferongames 3y ago
All the code is here: https://github.com/mas-bandwidth/xdp https://github.com/mas-bandwidth/xdp

deleted 3y ago

[deleted]

spintin 3y ago

The real way to do this is to get user space networking. Also event-based protocols with deterministic physics. Last but not least, you need to use a language that can atomically share memory between threads; C (with Arrays of 64 byte Structs) or Java.

gafferongames 3y ago

[dead]

nicebyte 3y ago

I'm calling eBPF "kernel shaders" to confuse graphics, os and gpgpu people all at the same time.

tgaj 3y ago
This analogy works well when trying to describe how eBPF is used for network applications. The eBPF scripts are "packet shaders" - like a "pixel shaders" they are executed for every packet independently and can modify attributes and/or payload according to a certain algorithm.
danwills 3y ago
Seems to have worked on me! Well played! :)
meindnoch 3y ago
I'm stealing this.
sakras 3y ago
I've been exactly thinking about it this way for a long time. Actually once we're able to push computation down into our disk drives, I wouldn't be surprised if these "Disk Shaders" will be written in eBPF.
- danwills 3y ago
  Totally sibling comment confirms it already exists! I hope that the 'shader' name sticks too! I find the idea of a shader has a very appropriate shape for tiny-program-embedded-in-specific-context so it seems perfect from a hacker POV! I have a VFX(Houdini now, RSL shaders etc earlier) and openCL-dabbling and demoscene-lurking background, based on which I think I prefer 'shader' to 'kernel', that's what OpenCL calls them.. but that conflicts with the name of like, 'the OS kernel' at least somewhat..
- sirsinsalot 3y ago
  I read that as eBNF and was very confused

hobs 3y ago

> Within 10 years, everybody is going to have 10gbps internet LOL, no way in America is that going to be true.

voidUpdate 3y ago
I live on the outskirts of a city in the middle of England and I /very/ much doubt I'll see 10gig in 10 years
- fragmede 3y ago
  What if that kind of bandwidth becomes available over cell networks?
  - voidUpdate 3y ago
    I've restricted my phone to only using 4G because I've had a lot of problems with 5G feeling a lot slower than 4G. I've not done any quantitative testing on it but it just generally feels a lot slower, even when I'm looking right at the 5G mast, and the 4G mast is behind a few buildings. If whatever problem it's having can be fixed, and it can be priced similarly to broadband, then maybe
gafferongames 3y ago
https://www.cablelabs.com/10g https://www.cablelabs.com/10g
- hobs 3y ago
  Is this supposed to be a joke? I cant even reliably get 1G in most places.
  - hansvm 3y ago
    I can't get 0.05G reliably in most suburbs, even when I do the worst-case ping in a typical 10-minute window is 500+ms, and there are still many places in the US where the best internet I can find is under 0.001G.

Animats 3y ago

I've actually written a high-performance metaverse client, one that can usefully pull half a gigabit per second and more from the Internet. So I get to see this happening. I'm looking at a highly detailed area right now, from the air, and traffic peaked around 200Mb/s. This XDP thing seems to address the wrong problem. Actual traffic for a metaverse is mostly bulk content download. Highly interactive traffic over UDP is maybe 1MB/second, including voice. You're mostly sending positions and orientations for moving objects. Latency matters for that, but an extra few hundred microseconds won't hurt. The rest is large file transfers. Those may be from totally different servers than the ones that talk interactive UDP. There's probably a CDN involved, and you're talking to caches. Latency doesn't matter that much, but big-block bandwidth does. Practical problems include data caps. If you go driving around a big metaverse, you can easily pull 200GB/hour from the asset servers. Don't try this on "AT&T Unlimited Extra® EL". Check your data plan. The last thing you want is game-specific code in the kernel. That creates a whole new attack surface.

lisplist 3y ago
What is a metaverse client? Do you just mean a cross platform VR app?
- sbierwagen 3y ago
  It's a client for https://secondlife.com/ https://secondlife.com/
gafferongames 3y ago
> Actual traffic for a metaverse is mostly bulk content download. Highly interactive traffic over UDP is maybe 1MB/second, including voice. Typical bandwidth for multiplayer games like FPS (Counterstrike, Apex Legends) are around 512kbps-1mbit per-second down per-client, and this is old information, newer games almost certainly use more. It's easy to see a more high fidelity gaming experience taking 10mbit - 100mbit traffic from server to client, just increase the size and fidelity of the world. next, increase player counts and you can easily fill 10gbit/sec for a future FPS/MMO hybrid. God save us from the egress BW costs though :)

Galanwe 3y ago

This article is plain ridiculous. > Why? Because otherwise, the overhead of processing each packet in the kernel and passing it down to user space and back up to the kernel and out to the NIC limits the throughput you can achieve. We're talking 10gbps and above here. _Throughpout_ is not problematic at all for the Linux network stack, even at 100gbps. What is problematic is >10gbps line rate. In other words, unless you're receiving 10gbps unshaped UDP datagrams with no payloads at line rate, the problem is non existant. Considering internet is 99% fat TCP packets, this sentence is completely absurd. > With other kernel bypass technologies like DPDK you needed to install a second NIC to run your program or basically implement (or license) an entire TCP/IP network stack to make sure that everything works correctly under the hood That is just wrong on so many levels. First, DPDK allows reinjecting packets in the Linux network stack. That is called queue splitting,is done by the NIC, and can be trivially achieved using e.g. the bifurcated driver. Second, there are plenty of available performant network stacks out there, especially considering high end NICs implement 80% of the performance sensitive parts of the stack on chip. Last, kernel bypassing is made on _trusted private networks_, you would have to be crazy or damn well know what you're doing to bypass on publicly addressable networks, otherwise you will have a bad reality check. There are decades of security checks and counter measures baked in the Linux network stack that a game would be irresponsible to ask his players to skip. I'm not even mentioning the ridiculous latency gains to be achieved here. Wire tapping the packet "NIC in" to userspace buffer should be in the ballpark of 3us. If you think you can do better and this latency is too much for your application, you're either day dreaming or you're not working in the video game industry.

cornel_io 3y ago
Are you under the impression that he was talking about game clients doing this? That would be absurd, since no gamers (within epsilon of zero) use Linux, and you'd need a completely different bag of tricks on Windows. He's fully focused on the backend.
intelVISA 3y ago
It's not ridiculous but it is quite dated - there's better approaches for packet processing at scale nowadays.

mondobe 3y ago

> Within 10 years, everybody is going to have 10gbps internet. How will this change how games are made? How are we going to use all this bandwidth? 5v5 shooters just aren't going to cut it anymore. I have no doubt that more bandwidth will change how games are made, but 5v5 shooters (and pretty much all existing multiplayer styles) are here to stay for a lot longer than that, in some form or another.

deleted 3y ago
[deleted]

jay_kyburz 3y ago

Hello from Canberra Glenn!

cookiengineer 3y ago

[flagged]

gafferongames 3y ago
The eBPF code runs on the game server not the client. Cool story tho
- Galanwe 3y ago
  If you want to use kernel bypass on the server, not the client, then XDP is even more useless. Because on the server world, you have access to professional NICs that you cannot expect clients to have. It's been decades now that people routinely do kernel bypass on servers using e. g. Solarflare NICs with OnLoad. You just write your networking code normally using BSD sockets API, and LD_PRELOAD the OnLoad SDK which will intercept the socket API and perform a bypass. You get 90% of the latency benefits of kernel bypass without modifying your code, which still runs everywhere.
  - tptacek 3y ago
    I think you're a little bit on tilt here. "The server world" was what XDP was invented for, and it's used all over cloud providers. Watch the XDP mailing list.
  - mc10 3y ago
    Both Katran[1] by Facebook and Unimog[2] by Cloudflare are L4 load balancers written using XDP. It's factually incorrect to claim that XDP is useless on servers as many companies intentionally choose to design their systems around commodity hardware. [1]: https://engineering.fb.com/2018/05/22/open-source/open-sourcing-katran-a-scalable-network-load-balancer/ [2]: https://blog.cloudflare.com/unimog-cloudflares-edge-load-balancer
- cookiengineer 3y ago
  Doesn't mean you should risk a takeover of all your game databases because you decided to bypass every single security measurement linux has in place already. I'd argue it's even worse, and network programming wise I'd argue that every custom implementation of TCP/UDP was broken in the past and lead to a lot of remote exploits, and eventually was abandoned due to maintenance costs. Just think about the shitshow of the source engine, for example. RCE'ing in the wild for what, 14+ years now? Took a long while and a lot of steam runtime sandboxing code to fix the decision to use a custom unsafe networking code.