31 ms·

> _Throughpout_ is not problematic at all for the Linux network stack, even at 100gbps. What is problematic is >10gbps line rate. In other words, unless you're

> _Throughpout_ is not problematic at all for the Linux network stack, even at 100gbps. What is problematic is >10gbps line rate. In other words, unless you're receiving 10gbps unshaped UDP small datagrams at line rate, the problem is non existant. Considering internet is 99% fat TCP packets, this sentence is completely absurd. But games are not 99% fat TCP packets. Games are typically networked with UDP small datagrams sent at high rates for most recent state or inputs, with custom protocols built on top of UDP to avoid TCP head of line blocking. Packet send rates per-client can often exceed 60HZ, especially when games tie client packet send rate to the display frequency, eg. Valve and Apex Legends network models. Now imagine you have many thousands of players and you can see that the problem does indeed exist. If not for current games, for future games and metaverse applications when we start to scale up the player counts from the typical 16, 32 or 64 players per-server instance, and try to merge something like FPS techniques with the scale of MMOs, which is actively something I'm actually doing. XDP/eBPF is a useful set of technologies for people who develop multiplayer games with custom UDP based protocols. You'll see a lot more usage of this in the future moving forward, as player counts increase for multiplayer games, and other metaverse-like experiences. Best wishes - Glenn

Galanwe 3y ago
> Packet send rates per-client can often exceed 60HZ You do realize the most random Python program, written by the most random programmer, running on the most random computer, could easily read 10k TCP packets per second per core?
- gafferongames 3y ago
  I absolutely do. Now let's see that random Python program handle 1M connected players each sending and receiving 1384 byte UDP packets at 244HZ, while also running game logic and simulation in the background.
gafferongames 3y ago
> First, DPDK allows reinjecting packets in the Linux network stack. That is called queue splitting,is done by the NIC, and can be trivially achieved using e.g. the bifurcated driver. My apologies, the last time I looked at DPDK was in 2016, and I don't believe this was true back then. Either way, it seems that XDP/eBPF is much easier to use. YMMV!
- Galanwe 3y ago
  No harm taken, kernel bypass technologies have moved fast in the last decade. As a side note though, I would encourage you to revisit DPDK. I've worked in the low latency space for a long time, and used pretty much every solution out there, open source or proprietary, and DPDK is one of my favorite. This is because DPDK is not _just_ poll mode drivers, it's a a full featured SDK for low latency packet processing. You get top notch thread safe memory pools to store packets, thread safe queues, hand written intrinsics optimized hashing and CRC algos, a cooperative scheduler in case you isolate your CPUs, etc, etc.
  - gafferongames 3y ago
    That sounds really good. To be honest, just not having to deal with the BPF verifier would be a plus.
deleted 3y ago
[deleted]
nsguy 3y ago
A modern server class machine can push "100Gbps" through the entire Linux stack just fine. TCP or UDP. With standard packet sizes (e.g. 1500 bytes.) We do this where I work. Yes a long time ago 1Gbps was hard, you needed jumbo frames, then 10Gbps was hard, then 100Gbps was hard. Right now where we are seems to be you don't need to do a kernel bypass unless you're running multiple 100Gbps NICs - that's an application where I've seen DPDK used in the wild. EDIT: Might be some benefits in terms of latency through the stack and resource usage on the machine... But I don't think 10Gbps is where the pain is for these either.
- gafferongames 3y ago
  > A modern server class machine can push "100Gbps" through the entire Linux stack just fine. Maybe we'd like to use some of that CPU to run the game instead of pushing packets. Maybe if we can push the packets more efficiently, we save $$$. Maybe game servers get DDoS'd and it's great to be able to quickly drop packets without any linux kernel overhead.
  - Galanwe 3y ago
    Maybe I would like my games to not access my kernel? Maybe I would like my games to not bypass my firewall and VPN? Also, eBPF is not a way to shave CPU cycles, you would use DPDK/Netmap/BSD BPF for that, it's a way to lower latency. And we're talking single digit microsecond latency, where a typical internet link has millisecond jitter.
    - mattnewport 3y ago
      The use case is for dedicated game servers controlled by the developer running in data centers, not clients running on player's home computers.
      - Galanwe 3y ago
        If we're talking server tech then why does the article claim that 10gbps is becoming retail commodity? 10,40 and 100gbps is already the baseline for servers since a decade, I have not seen a server with 1gbps NICs for anything else than remote management or off band communication in a decade. If it's for servers then the article still doesn't make any sense. See my other comment https://news.ycombinator.com/item?id=39939876 https://news.ycombinator.com/item?id=39939876
deleted 3y ago
[deleted]

Galanwe 3y ago

> Packet send rates per-client can often exceed 60HZ You do realize the most random Python program, written by the most random programmer, running on the most random computer, could easily read 10k TCP packets per second per core?

gafferongames 3y ago
I absolutely do. Now let's see that random Python program handle 1M connected players each sending and receiving 1384 byte UDP packets at 244HZ, while also running game logic and simulation in the background.

gafferongames 3y ago

> First, DPDK allows reinjecting packets in the Linux network stack. That is called queue splitting,is done by the NIC, and can be trivially achieved using e.g. the bifurcated driver. My apologies, the last time I looked at DPDK was in 2016, and I don't believe this was true back then. Either way, it seems that XDP/eBPF is much easier to use. YMMV!

Galanwe 3y ago
No harm taken, kernel bypass technologies have moved fast in the last decade. As a side note though, I would encourage you to revisit DPDK. I've worked in the low latency space for a long time, and used pretty much every solution out there, open source or proprietary, and DPDK is one of my favorite. This is because DPDK is not _just_ poll mode drivers, it's a a full featured SDK for low latency packet processing. You get top notch thread safe memory pools to store packets, thread safe queues, hand written intrinsics optimized hashing and CRC algos, a cooperative scheduler in case you isolate your CPUs, etc, etc.
- gafferongames 3y ago
  That sounds really good. To be honest, just not having to deal with the BPF verifier would be a plus.

deleted 3y ago

[deleted]

nsguy 3y ago

A modern server class machine can push "100Gbps" through the entire Linux stack just fine. TCP or UDP. With standard packet sizes (e.g. 1500 bytes.) We do this where I work. Yes a long time ago 1Gbps was hard, you needed jumbo frames, then 10Gbps was hard, then 100Gbps was hard. Right now where we are seems to be you don't need to do a kernel bypass unless you're running multiple 100Gbps NICs - that's an application where I've seen DPDK used in the wild. EDIT: Might be some benefits in terms of latency through the stack and resource usage on the machine... But I don't think 10Gbps is where the pain is for these either.

gafferongames 3y ago
> A modern server class machine can push "100Gbps" through the entire Linux stack just fine. Maybe we'd like to use some of that CPU to run the game instead of pushing packets. Maybe if we can push the packets more efficiently, we save $$$. Maybe game servers get DDoS'd and it's great to be able to quickly drop packets without any linux kernel overhead.
- Galanwe 3y ago
  Maybe I would like my games to not access my kernel? Maybe I would like my games to not bypass my firewall and VPN? Also, eBPF is not a way to shave CPU cycles, you would use DPDK/Netmap/BSD BPF for that, it's a way to lower latency. And we're talking single digit microsecond latency, where a typical internet link has millisecond jitter.
  - mattnewport 3y ago
    The use case is for dedicated game servers controlled by the developer running in data centers, not clients running on player's home computers.
    - Galanwe 3y ago
      If we're talking server tech then why does the article claim that 10gbps is becoming retail commodity? 10,40 and 100gbps is already the baseline for servers since a decade, I have not seen a server with 1gbps NICs for anything else than remote management or off band communication in a decade. If it's for servers then the article still doesn't make any sense. See my other comment https://news.ycombinator.com/item?id=39939876 https://news.ycombinator.com/item?id=39939876

deleted 3y ago

[deleted]