5 ms·
This article is plain ridiculous. > Why? Because otherwise, the overhead of processing each packet in the kernel and passing it down to user space and back up
by Galanwe 3y ago
This article is plain ridiculous.
> Why? Because otherwise, the overhead of processing each packet in the kernel and passing it down to user space and back up to the kernel and out to the NIC limits the throughput you can achieve. We're talking 10gbps and above here.
_Throughpout_ is not problematic at all for the Linux network stack, even at 100gbps. What is problematic is >10gbps line rate. In other words, unless you're receiving 10gbps unshaped UDP datagrams with no payloads at line rate, the problem is non existant. Considering internet is 99% fat TCP packets, this sentence is completely absurd.
> With other kernel bypass technologies like DPDK you needed to install a second NIC to run your program or basically implement (or license) an entire TCP/IP network stack to make sure that everything works correctly under the hood
That is just wrong on so many levels.
First, DPDK allows reinjecting packets in the Linux network stack. That is called queue splitting,is done by the NIC, and can be trivially achieved using e.g. the bifurcated driver.
Second, there are plenty of available performant network stacks out there, especially considering high end NICs implement 80% of the performance sensitive parts of the stack on chip.
Last, kernel bypassing is made on _trusted private networks_, you would have to be crazy or damn well know what you're doing to bypass on publicly addressable networks, otherwise you will have a bad reality check. There are decades of security checks and counter measures baked in the Linux network stack that a game would be irresponsible to ask his players to skip.
I'm not even mentioning the ridiculous latency gains to be achieved here. Wire tapping the packet "NIC in" to userspace buffer should be in the ballpark of 3us. If you think you can do better and this latency is too much for your application, you're either day dreaming or you're not working in the video game industry.
- cornel_io 3y agoAre you under the impression that he was talking about game clients doing this? That would be absurd, since no gamers (within epsilon of zero) use Linux, and you'd need a completely different bag of tricks on Windows. He's fully focused on the backend.
- intelVISA 3y agoIt's not ridiculous but it is quite dated - there's better approaches for packet processing at scale nowadays.
- gafferongames 3y ago> _Throughpout_ is not problematic at all for the Linux network stack, even at 100gbps. What is problematic is >10gbps line rate. In other words, unless you're receiving 10gbps unshaped UDP small datagrams at line rate, the problem is non existant. Considering internet is 99% fat TCP packets, this sentence is completely absurd. But games are not 99% fat TCP packets. Games are typically networked with UDP small datagrams sent at high rates for most recent state or inputs, with custom protocols built on top of UDP to avoid TCP head of line blocking. Packet send rates per-client can often exceed 60HZ, especially when games tie client packet send rate to the display frequency, eg. Valve and Apex Legends network models. Now imagine you have many thousands of players and you can see that the problem does indeed exist. If not for current games, for future games and metaverse applications when we start to scale up the player counts from the typical 16, 32 or 64 players per-server instance, and try to merge something like FPS techniques with the scale of MMOs, which is actively something I'm actually doing. XDP/eBPF is a useful set of technologies for people who develop multiplayer games with custom UDP based protocols. You'll see a lot more usage of this in the future moving forward, as player counts increase for multiplayer games, and other metaverse-like experiences. Best wishes - Glenn
- Galanwe 3y ago> Packet send rates per-client can often exceed 60HZ You do realize the most random Python program, written by the most random programmer, running on the most random computer, could easily read 10k TCP packets per second per core?
- gafferongames 3y agoI absolutely do. Now let's see that random Python program handle 1M connected players each sending and receiving 1384 byte UDP packets at 244HZ, while also running game logic and simulation in the background.
- gafferongames 3y ago> First, DPDK allows reinjecting packets in the Linux network stack. That is called queue splitting,is done by the NIC, and can be trivially achieved using e.g. the bifurcated driver. My apologies, the last time I looked at DPDK was in 2016, and I don't believe this was true back then. Either way, it seems that XDP/eBPF is much easier to use. YMMV!
- ncallaway 3y ago> In other words, unless you're receiving 10gbps unshaped UDP datagrams with no payloads at line rate, the problem is non existant. Considering internet is 99% fat TCP packets, this sentence is completely absurd. Uh, "the internet" traffic shape is a terrible model for low-latency multiplayer games traffic shape. Surely you don't think that 99% of the packets being exchanged during a session of Counter-Strike 2 are fat TCP packets, right?
- Galanwe 3y agoSurely not, but I don't expect a video game datagram size and rate to be anywhere near line rate. To put things in perspective, we are talking ~1M datagrams per seconds on a 1gbps link, 10M dgps on a 10gbps link. Hell, I don't even think a commodity gaming computer have enough cores to process line rate datagrams on a 1gbps link.
- gafferongames 3y agoThink of a game server with many thousands or millions of players connected at the same time, and these players are all sending 10-100mbps of UDP packets at very high rates (60pps bidirectionally, or higher). Kernel bypass is not going to be something that is necessary on the client (even though there is a trend towards symmetric 10G internet becoming available in the US today, and according to Neilsen's law, it should be widely available within 10 years).
- eqvinox 3y ago60kpps is a joke to the network stack, and by the time you'd reach 60Mpps, your problem is doing something useful with it, rather than getting it pulled off the NIC. If your game engine can handle even 10k players on one system, it'd be the gold plated unicorn with wings of game engines. Ask CCP (EVE online), they know a thing or two about this.
- deleted 3y ago[deleted]
- tptacek 3y agoI feel like I'm reasonably well read into low-level TCP/IP security (also: eBPF) and I'm not sure what you mean by "decades of security checks and countermeasures baked into the Linux network stack" than an XDP kernel bypass would skip. Can you say more?
- Galanwe 3y ago> I'm reasonably well read into low-level TCP/IP security > I'm not sure what you mean by "decades of security checks and countermeasures baked into the Linux network stack" than an XDP kernel bypass would skip. Can you say more? If you bypass the kernel stack on an publicly addressable network, then it is your responsibility to implement and calibrate backloging, handshake recycling, SYN cookies
- gafferongames 3y agoMost games use UDP. The article describes techniques for UDP. We already do all the stuff you talk about in custom UDP protocols.