6 ms·
There are a number of concrete problems: - syscall interfaces are a mess, the primitive APIs are too slow for regular sized packets (~1500 bytes), the overhead
by raggi 2y ago
There are a number of concrete problems:
- syscall interfaces are a mess, the primitive APIs are too slow for regular sized packets (~1500 bytes), the overhead is too high. GSO helps but it’s a horrible API, and it’s been buggy even lately due to complexity and poor code standards.
- the syscall costs got even higher with spectre mitigation - and this story likely isn’t over. We need a replacement for the BSD sockets / POSIX APIs they’re terrible this decade. Yes, uring is fancy, but there’s a tutorial level API middle ground possible that should be safe and 10x less overhead without resorting to uring level complexity.
- system udp buffers are far too small by default - they’re much much smaller than their tcp siblings, essentially no one but experts have been using them, and experts just retune stuff.
- udp stack optimizations are possible (such as possible route lookup reuse without connect(2)), gso demonstrates this, though as noted above gso is highly fallible, quite expensive itself, and the design is wholly unnecessarily intricate for what we need, particularly as we want to do this safely from unprivileged userspace.
- several optimizations currently available only work at low/mid-scale, such as connect binding to (potentially) avoid route lookups / GSO only being applicable on a socket without high peer-competition (competing peers result in short offload chains due to single-peer constraints, eroding the overhead wins).
Despite all this, you can implement GSO and get substantial performance improvements, we (tailscale) have on Linux. There will be a need at some point for platforms to increase platform side buffer sizes for lower end systems, high load/concurrency, bdp and so on, but buffers and congestion control are a high complex and sometimes quite sensitive topic - nonetheless, when you have many applications doing this (presumed future state), there will be a need.
- SomaticPirate 2y agoWhat is GSO?
- thorncorona 2y agopresumably generic segmentation offloading
- throwaway8481 2y agoGeneric Segmentation Offload https://www.kernel.org/doc/html/latest/networking/segmentation-offloads.html https://www.kernel.org/doc/html/latest/networking/segmentati...
- chaboud 2y agoLikely Generic Segmentation Offload (if memory serves), which is a generalization of TCP segmentation offload. Basically (hyper simple), the kernel can lump stuff together when working with the network interface, which cuts down on ultra slow hardware interactions.
- raggi 2y agoit was originally for the hardware, but it's also valuable on the software side as the cost of syscalls is far too high for packet sized transactions
- jesperwe 2y agoGeneric Segmentation Offload "GSO gains performance by enabling upper layer applications to process a smaller number of large packets (e.g. MTU size of 64KB), instead of processing higher numbers of small packets (e.g. MTU size of 1500B), thus reducing per-packet overhead."
- underdeserver 2y agoThis is more the result. Generally today an Ethernet frame, which is the basic atomic unit of information over the wire, is limited to 1500 bytes (the MTU, or Maximum Transmission Unit). If you want to send more - the IP layer allows for 64k bytes per IP packet - you need to split the IP packet into multiple (64k / 1500 plus some header overhead) frames. This is called segmentation. Before GSO the kernel would do that which takes buffering and CPU time to assemble the frame headers. GSO moves this to the ethernet hardware, which is essentially doing the same thing only hardware accelerated and without taking up a CPU core.
- wtarreau 2y agoWhat you're describing is for TCP. On TCP you can perform a write(64kB) and see the stack send it into 1460 segments. On UDP if you write(64kB) you'll get a single 64kB packet composed of 45 fragments. Needless to say, it suffices that any of them is lost in a buffer somewhere for the whole packet never being received and all of them having to be retransmitted by the application layer. GSO on UDP allows the application to send a large chunk of data, indicating the MTU to be applied, and lets the kernel pass it down the stack as-is, until the lowest layer that can split it (network stack, driver or hardware). In this case they will make packets, not fragments. On the wire there will really be independent datagrams with different IP IDs. In this case, if any of them is lost, the other ones are still received and the application can focus on retransmitting only the missing one(s). In terms of route lookups, it's as efficient as fragmentation (since there's a single lookup) but it will ensure that what is sent over the wire is usable all along the chain, at a much lower cost than it would be to make the application send all of them individually.
- USiBqidmOOkAqRb 2y agoShipping? Government services online? Piedmont airport? Alcoholics anonymous? Obviously not. Please introduce your initialisms, if it's not guaranteed that first result in a search will be correct.
- mh- 2y ago> first result in a search will be correct Searching for GSO network gives you the correct answer in the first result. I'd consider that condition met.
- cookiengineer 2y agoSay what you want but I bet we'll see lots of eBPF modules being loaded in the future for the very reason you're describing. An ebpf quic module? Why not! And that scares me, because there's not a single tool that has this on its radar for malware detection/prevention.
- raggi 2y agowe can consider ebpf "a solution" when there's even a remote chance you'll be able to do it from an unentitled ios app. somewhat hyperbole, but the point is, this problem is a problem for userspace client applications, and bpf isn't a particularly "good" solution for servers either, it's high cost of authorship for a problem that is easily solvable with a better API to the network stack.
- mgaunard 2y agoebpf is linux technology, you will never be able to do it from iOS.
- dan-robertson 2y agohttps://github.com/microsoft/ebpf-for-windows https://github.com/microsoft/ebpf-for-windows
- JoshTriplett 2y ago> Yes, uring is fancy, but there’s a tutorial level API middle ground possible that should be safe and 10x less overhead without resorting to uring level complexity. I don't think io_uring is as complex as its reputation suggests. I don't think we need a substantially simpler low-level API; I think we need more high-level APIs built on top of io_uring. (That will also help with portability: we need APIs that can be most efficiently implemented atop io_uring but that work on non-Linux systems.)
- raggi 2y ago> I don't think io_uring is as complex as its reputation suggests. uring is extremely problematic to integrate into many common application / language runtimes and it has been demonstrably difficult to integrate into linux safely and correctly as well, with a continual stream of bugs, security and policy control issues. in principle a shared memory queue is a reasonable basis for improving the IO cost between applications and IO stacks such as the network or filesystem stacks, but this isn't easy to do well, cf. uring bugs and binder bugs.
- JoshTriplett 2y ago> with a continual stream of bugs, security and policy control issues This has not been true for a long time. There was an early design mistake that made it quite prone to these, but that mistake has been fixed. Unfortunately, the reputational damage will stick around for a while.
- quotemstr 2y ago> Yes, uring is fancy, but there’s a tutorial level API middle ground possible that should be safe and 10x less overhead without resorting to uring level complexity. And the kernel has no business providing this middle-layer API. Why should it? Let people grab whatever they need from the ecosystem. Networking should be like Vulkan: it should have a high-performance, flexible API at the systems level with being "easy to use" a non-goal --- and higher-level facilities on top.
- astrange 2y agoThe kernel provides networking because it doesn't trust userspace to do it. If you provided a low level networking API you'd have to verify everything a client sends is not malicious or pretending to be from another process. And for the same reason, it'd only work for transmission, not receiving. That and nobody was able to get performant microkernels working at the time, so we ended up with everything in the monokernel. If you do trust the client processes then it could be better to just have them read/write IP packets though.
- namibj 2y agoAlso, it is really easy to do the normal IO "syscall wrappers" on top of io_uring instead, even easily exposing a very simple async/await variant of them that splits out the "block on completion (after which just like normal IO the data buffer has been copied into kernel space)" from the rest of the normal IO syscall, which allow pipelining & coalescing of requests.
- modeless 2y agoSeems to me that the real problem is the 1500 byte MTU that hasn't increased in practice in over 40 years.
- _cenw 2y agoThat's on the list that right after we all migrate to IPv6.
- p_l 2y agoFor all practical purposes, the internet MTU is lower than ethernet default MTU. Sometimes for ease of mind I end up clamping it to v6 minimum (1280) just in case .
- j16sdiz 2y agoThe real problem is some so called "sysadmin" drop all ICMP, breaking path mtu discovery.
- icedchai 2y agoThe most secure network is one that doesn't pass any traffic at all. ;)
- cryptonector 2y agoThat's why PMTUD (P for Passive) exists.
- throw0101c 2y ago> Seems to me that the real problem is the 1500 byte MTU that hasn't increased in practice in over 40 years. As per a sibling comment, 1500 is just for Ethernet (the default, jumbo frames being able to go to (at least) 9000). But the Internet is more than just Ethernet. If you're on DSL, then RFC 2516 states that PPPoE's MTU is 1492 (and you probably want an MSS of 1452). The PPP, L2TP, and ATM AAL5 standards all have 16-bit length fields allowing for packets up 64k in length. GPON ONT MTU is 2000. The default MTU for LTE is 1428. If you're on an HPC cluster, there's a good chance you're using Infiniband, which goes to 4096. What are size do you suggest everyone on the planet go to? Who exactly is going to get everyone to switch to the new value?
- Diggsey 2y agoHistorically there have been too many constraints on the Linux syscall interface: - Performance - Stability - Convenience - Security This differs from eg. Windows because on Windows the stable interface to the OS is in user-space, not tied to the syscall boundary. This has resulted in unfortunate compromises in the design of various pieces of OS functionality. Thankfully things like futex and io-uring have dropped the "convenience" constraint from the syscall itself and moved it into user-space. Convenience is still important, but it doesn't need to be a constraint at the lowest level, and shouldn't compromise the other ideals.
- amluto 2y agoHi, Tailscale person! If you want a fairly straightforward improvement you could make: Tailscale, by default uses SOCK_RAW. And having any raw socket listening at all hurts receive performance systemwide: https://lore.kernel.org/all/CALCETrVJqj1JJmHJhMoZ3Fuj685Unf=DYsiEY1uEfnJq3H+Tzg@mail.gmail.com/ https://lore.kernel.org/all/CALCETrVJqj1JJmHJhMoZ3Fuj685Unf=... It shouldn’t be particularly hard to port over the optimization that prevents this problem for SOCK_PACKET. I’ll get to it eventually (might be quite a while), but I only care about this because of Tailscale, and I don’t have a ton of bandwidth.
- raggi 2y agoVery interesting, thank you. We’ll take a look at this!
- bradfitz 2y agoBTW, that code changed just recently: https://github.com/tailscale/tailscale/commit/1c972bc7cbebfcc45f1421bc359909f54a394b3d https://github.com/tailscale/tailscale/commit/1c972bc7cbebfc... It's now a AF_PACKET/SOCK_DGRAM fd as it was originally meant to be.
- cryptonector 2y agoOf these the hardest one to deal with is route lookup caching and reuse w/o connect(2). Obviously the UDP connected TCB can cache that, but if you don't want a "connected" socket fd... then there's nowhere else to cache it except ancillary data, so ancillary data it would have to be. But getting return-to-sender ancillary data on every read (so as to be able to copy it to any sends back to the same peer) adds overhead, so that's not good. A system call to get that ancillary data adds overhead that can be amortized by having the application cache it, so that's probably the right design, and if it could be combined with sending (so a new flavor of sendto(2)) that would be even better, and it all has to be uring-friendly.
- nly 2y agoAnyone who cares about performance is already using NIC accelerated APIs like Onloads or VFI
- leshow 2y agowhich UDP settings do you usually tune?
- wtarreau 2y agoThe default UDP buffers of 212kB are indeed a big problem for every client at the moment. You can optimize your server as you want, all your clients will experience losses if they pause for half a millisecond to redraw a tab or update an image, just because the UDP buffers can only store so few packets. That's among the things that must urgently change if we want UDP so start to work well on end-user devices.