5 ms·
The big guys don't have the patience to wait for Linux kernel networking to be fast and scalable. They bypass the kernel and take over the hardware. https://bl
by jeffbee 3y ago
The big guys don't have the patience to wait for Linux kernel networking to be fast and scalable. They bypass the kernel and take over the hardware.
https://blog.acolyer.org/2019/11/11/snap-networking/ https://blog.acolyer.org/2019/11/11/snap-networking/
- corbet 3y agoThat's funny ... the "big guys" are some of the biggest contributors to the Linux network stack, almost as if they were actually using it and cared about how well it works.
- jeffbee 3y agoHistory has shown that tons of Linux networking scalability and performance contributions have been rejected by the gatekeepers/maintainers. The upstream kernel remains unsuitable for datacenter use, and all the major operators bypass or patch it.
- sophacles 3y agoAll the major operators sometimes bypass or patch it for some use cases. For others they use it as is. For other still they laugh at you for taking the type of drugs that makes one think any CPU is sufficient to handle networking in code. Networking isn't a one size fits all thing - different networks have different needs, and different systems in any network will have different needs. Userland networking is great until you start needing to deal with weird flows or unexpected traffic - then you end up either needing something a bit more robust and your performance starts dropping because you added a bunch of branches to your code or switched over to a kernel implementation that handles those cases. I've seen a few cases of userland networking being slower than just doing the kernel - and being kept because sometimes the what you care about is control over packet lifecycle more than raw throughput. Kernels prioritize robust network stacks that can handle a lot of cases good enough. Different implementations handle different scenarios better - there's plenty of very high performance networking done with vanilla linux and vanilla freebsd.
- eddtests 3y agoDo you have links on this? I’ve not heard anything about it
- tptacek 3y agoI believe they're paraphrasing the Snap paper, and also that they're extrapolating too far from it.
- suprjami 3y agoLook who you're arguing with.
- tptacek 3y agoOver the course of several years, the architecture underpinning Snap has been used in production for multiple networking applications, including network virtualization for cloud VMs [19], packet-processing for Internet peering [62], scalable load balancing [22], and Pony Express, a reliable transport and communications stack that is our focus for the remainder of this paper. This paper suggests, as I would have expected, that Google uses userland networking in strategic spots where low-level network development is important (SDNs and routing), and not for normal applications.
- jeffbee 3y ago"and Pony Express" is the operative phrase. As the paper states on page 1, "Snap is deployed to over half of our fleet of machines and supports the needs of numerous teams." According to the paper it is not niche.
- tptacek 3y agoIsn't Pony Express a ground-up replacement for all of TCP/IP? It doesn't even present a TCP/UDP socket interface.
- jeffbee 3y agoCorrect. That is my point. The sockets interface, and design choices within the Linux kernel, make ordinary TCP sockets too difficult to exploit in a datacenter environment. The general trend is away from TCP sockets. QUIC (HTTP/3) is a less extreme retreat from TCP, moving all the flow control, congestion, and retry logic out of the kernel and into the application. An example of how Linux TCP is unsuitable for datacenters is that the minimum RTO is hard-coded to 200ms, which is essentially forever. People have been trying to land better or at least more configurable parameters upstream for decades. I am hardly the first person to point out the deficiencies. Google presented tuning Linux for datacenter applications at LPC 2022, and their deck has barely changed in 15 years.
- tptacek 3y ago