10 ms·
QUIC for the kernel
- WASDx 1y agoI recall this article on QUIC disadvantages: https://www.reddit.com/r/programming/comments/1g7vv66/quic_is_not_quick_enough_over_fast_internet/ https://www.reddit.com/r/programming/comments/1g7vv66/quic_i... Seems like this is a step in the right direction to resole some of those issues. I suppose nothing is preventing it from getting hardware support in future network cards as well.
- miohtama 1y agoQUIC does not work very well for use cases like machine-to-machine traffic. However most of traffic in Internet today is from mobile phones to servers and it is were QUIC and HTTP 3 shine. For other use cases we can keep using TCP.
- thickice 1y agoWhy doesn't QUIC work well for machine-to-machine traffic ? Is it due to the lack of offloads/optimizations for TCP and machine-to-machine traffic tend to me high volume/high rate ?
- yello_downunder 1y agoQUIC would work okay, but not really have many advantages for machine-to-machine traffic. Machine-to-machine you tend to have long-lived connections over a pretty good network. In this situation TCP already works well and is currently handled better in the kernel. Eventually QUIC will probably be just as good for TCP in this use case, but we're not there yet.
- jabart 1y agoYou still have latency, legacy window sizes, and packet schedulers to deal with.
- spwa4 1y agoBut that is the huge advantage of QUIC. It does NOT totally outcompete TCP traffic on links (we already have bittorrent over udp for that purpose). They redesigned the protocol 5 times or so to achieve that.
- m00x 1y agoIt's explained in the reddit thread. Most of it is because you have to handle a ton of what TCP does in userland.
- extropy 1y agoThe NAT firewalls do not like P2P UDP traffic. Majoritoy of the routers lack the smarts to passtrough QUIC correctly, they need to treat it the same as TCP essentially.
- beeflet 1y agoNAT is the devil. bring on the IPoc4lypse
- hdgvhicv 1y agoNat is massively useful for all sorts of reasons which has nothing to do with ip limitations.
- unethical_ban 1y agoRather, NAT is a bandage for all sorts of reasons besides IP exhaustion. Example: Janky way to get return routing for traffic when you don't control enterprise routes. Source: FW engineer
- hdgvhicv 1y agoSure. When I can bgp advertise my laptop with my phone provider and have it update is a second or so globally when I move from tethering to wifi, or one network to another. No doubt you think I should simply renumber all my VMs every time that happens, breaking internal connections. Or perhaps run a completely separate addrsssing in each vm in parallel and make sure each vm knows which connection to use. Perhaps the vms peer with my laptop and then the laptop decides what to push out which way via localprefs, as paths etc. that sounds so much simpler than a simple masquerade. What happens when I want vm1 out of connection A, vm 3 out of connection B, vm 4-7 out of connection C. Then I want to change them quickly and easily. I’m balancing outbound and inbound rules, reaching for communities, and causing bgp dampening all over the place. What when they aren’t VMs but instead physical devices. My $40 mifi is now processing the entire DFZ routing table? What happens when I want a single physical device like a tv to contact one service via connection 1 and another via connection 2 but the device doesn’t support multiple routing tables or selection of that. What if it does support it but I just want to be able to shift my ssh sessions to a low latency higher loss link but keep my streaming ups on the high latency no loss link. All this is trivial with nat. Now sure I can use NAT66, and do a 1:1 natting (no PAT here), but then I’m using nat and that breaks the ipv6 cult that believes translating network addresses is useless.
- dan-robertson 1y agoI think basically there is currently a lot of overhead and, when you control the network more and everything is more reliable, you can make tcp work better.
- exabrial 1y agoFor starters, why encrypt something literally in the same datacenter 6 feet away? Add significant latency and processing overhead.
- 20k 1y agoBecause the NSA actively intercepts that traffic. There's a reason why encryption is non optional
- Karrot_Kream 1y agoTo me this seems outlandish (e.g. if you're part of PRISM you know what's happening and you're forced to comply.) But to think through this threat model, you're worried that the NSA will tap intra-DC traffic but not that it will try to install software or hardware on your hosts to spy traffic at the NIC level? I guess it would be harder to intercept and untangle traffic at the NIC level than intra-DC, but I'm not sure?
- viraptor 1y ago> you're worried that the NSA will tap intra-DC traffic but not that it will try to install software or hardware on your hosts It doesn't have to be one or the other. We've known for over a decade that the traffic between DCs was tapped https://www.theguardian.com/technology/2013/oct/30/google-reports-nsa-secretly-intercepts-data-links https://www.theguardian.com/technology/2013/oct/30/google-re... Extending that to intra-DC wouldn't be surprising at all. Meanwhile backdoored chips and firmware attacks are a constant worry and shouldn't be discounted regardless of the first point.
- adgjlsfhk1 1y agoThe difference between tapping intra-DC and in computer spying is that in computer spying is much more likely to get caught and much less easily able to get data out. There's a pretty big difference between software/hardware weaknesses that require specific targeting to exploit and passive scooping everything up and scanning
- 1y ago
- kldx 1y agoLet me try providing a different perspective based on experience. QUIC works amazingly well for _some_ kinds of machine to machine traffic. ssh3, based on QUIC is quicker at dropping into a shell compared to ssh. The latency difference was clearly visible. QUIC with the unreliable dgram extension is also a great way to implement port forwarding over ssh. Tunneling one reliable transport over another hides the packer losses in the upper layer.
- exDM69 1y agoThe article that GP posted was specifically about throughput over a high speed connection inside a data center. It was not about latency. In my opinion, the lessons that one can draw from this article should not be applied for use cases that are not about maximum throughput inside a data center.
- mort96 1y agoI don't understand what you mean by "machine-to-machine" if a phone (a machine) talking to a server (a machine) is not machine-to-machine.
- szszrk 1y agoI hope you don't think that user-to-machine means that I have to stick my finger in a network switch? :) Machine-to-machine is usually meant as traffic where neither of the sides is the client device (desktop, mobile etc). Often not initiated by user, but that's debatable. I would say an server making a sync of database to passive node is machine-to-machine, while a user connection from his browser to webserver is not.
- deleted 1y ago[deleted]
- dahfizz 1y ago> QUIC is meant to be fast, but the benchmark results included with the patch series do not show the proposed in-kernel implementation living up to that. A comparison of in-kernel QUIC with in-kernel TLS shows the latter achieving nearly three times the throughput in some tests. A comparison between QUIC with encryption disabled and plain TCP is even worse, with TCP winning by more than a factor of four in some cases. Jesus, that's bad. Does anyone know if userspace QUIC implementations are also this slow?
- klabb3 1y agoYes, they are. Worse, I’ve seen them shrink down to nothing in the face of congestion with TCP traffic. If Quic is indeed the future protocol, it’s a good thing to move it into the kernel IMO. It’s just madness to provide these massive userspace impls everywhere, on a packet switched protocol nonetheless, and expect it to beat good old TCP. Wouldn’t surprise me if we need optimizations all the way down to the NIC layer, and maybe even middleboxes. Oh and I haven’t even mentioned the CPU cost of UDP. OTOH, TCP is like a quiet guy at the gym who always wears baggy clothes but does 4 plates on the bench when nobody is looking. Don't underestimate. I wasted months to learn that lesson.
- vladvasiliu 1y agoWhy is QUIC being pushed, then?
- favflam 1y agoI know in the p2p space, peers have to send lots of small pieces of data. QUIC stops stream blocking on a single packet delay.
- dan-robertson 1y agoThe problem it is trying to solve is not overhead of the Linux kernel on a big server in a datacenter
- toast0 1y agoIt has good properties compared to tcp-in-tcp (http/2), especially when connected to clients without access to modern congestion control on iffy networks. http/2 was perhaps adopted too broadly; binary protocol is useful, header compression is useful (but sometimes dangerous), but tcp multiplexing is bad, unless you have very low loss ... it's not ideal for phones with inconsistent networking.
- jeffbee 1y agoThis seems to be a categorical error, for reasons that are contained in the article itself. The whole appeal of QUIC is being immune to ossification, being free to change parameters of the protocol without having to beg Linux maintainers to agree.
- corbet 1y agoOssification does not come about from the decisions of "Linux maintainers". You need to look at the people who design, sell, and deploy middleboxes for that.
- jeffbee 1y agoI disagree. There is plenty of ossification coming from inside the house. Just some examples off the top of my head are the stuck-in-1974 minimum RTO and ack delay time parameters, and the unwillingness to land microsecond timestamps.
- otterley 1y agoNot a networking expert, but does TCP in IPv6 suffer the same maladies?
- pumplekin 1y agoYes. Layer4 TCP is pretty much just slapped on top of Layer3 IPv4 or IPv6 in exactly the same way for both of them. Outside of some little nitpicky things like details on how TCP MSS clamping works, it is basically the same.
- ComputerGuru 1y ago…which is basically how it’s supposed to work (or how we teach that it’s supposed to work). (Not that you said anything to the contrary!)
- 0xbadcafebee 1y ago
- darksaints 1y agoFor the love of god, can we please move to microkernel-based operating systems already? We're adding a million lines of code to the linux kernel every year. That's so much attack surface area. We're setting ourselves up for a kessler syndrome of sorts with every system that we add to the kernel.
- mdavid626 1y agoMost of that code is not loaded into the kernel, only when needed.
- darksaints 1y agoTrue, but the last time I checked (several years ago), the size of the portion of code that is not drivers or kernel modules was still 7 million lines of code, and the average system still has to load a few million more via kernel modules and drivers. That is still a phenomenally large attack surface. The SeL4 kernel is 10k lines of code. OKL4 is 13k. QNX is ~30k.
- deleted 1y ago[deleted]
- arp242 1y agoCan I run Firefox or PostgreSQL with reasonable performance on SeL4, OKL4, or QNX?
- doubled112 1y agoReasonable performance includes GPU acceleration for both rendering and decoding media, right?
- 0x457 1y agoyes
- 1y ago
- valorzard 1y agoWould this (eventually) include the unreliable datagram extension?
- wosined 1y agoDon't know if it could get faster than UDP if it is on top of it.
- valorzard 1y agoThe use case for this would be running a multiplayer game server over QUIC
- mudkipdev 1y agoOr a mix of both, like datagrams for voice chat / movement and reliable packets for important data
- 01HNNWZ0MV43FF 1y agoOther use cases include video / audio streaming, VPNs over QUIC, and QUIC-over-QUIC (you never know)
- wosined 1y agoThe general web is slowed down by bloated websites. But I guess this can make game latency lower.
- fmbb 1y agohttps://en.m.wikipedia.org/wiki/Jevons_paradox https://en.m.wikipedia.org/wiki/Jevons_paradox The Jevons Paradox is applicable in a lot of contexts. More efficient use of compute and communications resources will lead to higher demand. In games this is fine. We want more, prettier, smoother, pixels. In scientific computing this is fine. We need to know those simulation results. On the web this is not great. We don’t want more ads, tracking, JavaScript.
- 01HNNWZ0MV43FF 1y agoNo, the last 20 years of browser improvements has made my static site incredibly fast! I'm benefiting from WebP, JS JITs, Flexbox, zstd, Wasm, QUIC, etc, etc
- deleted 1y ago[deleted]
- Ericson2314 1y agoWhat will the socket API look like for multiple streams? I guess it is implied it is the same as multiple connections, with caching behind the scenes. I would hope for something more explicit, where you get a connection object and then open streams from it, but I guess that is fine for now. https://github.com/microsoft/msquic/discussions/4257 https://github.com/microsoft/msquic/discussions/4257 ah but look at this --- unless this is an extension, the server side can also create new streams, once a connection is established. The client creating new "connections" (actually streams) cannot abstract over this. Something fundamentally new is needed. My guess is recvmsg to get a new file descriptor for new stream.
- gte525u 1y agoI would look at SCTP socket API it supports multistreaming.
- Ericson2314 1y agoI checked that out and....yuck! - Send specifies which stream by ordinal number? (Can't have different parts of a concurrent app independently open new streams) - Receive doesn't specify which stream at all?!
- wahern 1y agoAPI RFC is https://datatracker.ietf.org/doc/html/draft-lxin-quic-socket-apis https://datatracker.ietf.org/doc/html/draft-lxin-quic-socket...
- Ericson2314 1y agoAh fuck, it still has a stream_id notion How are socket APIs always such garbage....
- wahern 1y agoAt least the SCTP API has sctp_peeloff, which gives you a new single-stream socket descriptor for the connection. Maybe QUIC will get something like that, eventually. Kind of a glaring omission, though, unless I'm misunderstanding.
- euphamism 1y ago[flagged]
- deleted 1y ago[deleted]
- Bender 1y agoI don't know about using it in the kernel but I would love to see OpenSSH support QUIC so that I get some of the benefits of Mosh [1] while still having all the features of OpenSSH including SFTP, SOCKS, port forwarding, less state table and keep alive issues, roaming support, etc... Could OpenSSH leverage the kernel support? [1] - https://mosh.org/ https://mosh.org/
- wmf 1y agoSSH would need a lot of work to replace its crypto and mux layers with QUIC. It's probably worth starting from scratch to create a QUIC login protocol. There are a bunch of different approaches to this in various states of prototyping out there.
- Bender 1y agoFair points. I suppose Mosh would be the proper starting point then. I'm just selfish and want the benefits of QUIC without losing all the really useful features of OpenSSH.
- bauruine 1y agoOpenSSH is an OpenBSD project therefore I guess a Linux api isn't that interesting but I could be wrong ofc.
- Bender 1y agoThat's a good point. At least it would not be an entirely new idea. [1] Curious what reactions he received. [1] - https://papers.freebsd.org/2022/eurobsdcon/jones-making_freebsd_quic/ https://papers.freebsd.org/2022/eurobsdcon/jones-making_free...
- skissane 1y agoOnce Linux implements it, I think odds are high that FreeBSD sooner or later does too. And maybe NetBSD and XNU/Darwin/macOS/iOS thereafter. And if they’ve all got it, that increases the odds that eventually OpenBSD also implements it. And if OpenBSD has the support in its kernel, then they might be willing to consider accepting code in OpenSSH which uses it. So OpenSSH supporting QUIC might eventually happen, but if it does, it is going to be some years away
- kibwen 1y agoI'm confused, I thought the revolution of the past decade or so was in moving network stacks to userspace for better performance.
- michaelsshaw 1y agoThe constant mode switching for hardware access is slow. TCP/IP remains in the kernel for windows and Linux.
- wmf 1y agoPerformance comes from dedicating core(s) to polling, not from userspace.
- shanemhansen 1y agoYou are right but it's confusing because there are two different approaches. I guess you could say both approaches improve performance by eliminating context switches and system calls. 1. Kernel bypass combined with DMA and techniques like dedicating a CPU to packet processing improve performance. 2. What I think of as "removing userspace from the data plane" improves performance for things like sendfile and ktls. To your point, Quic in the kernel seems to not have either advantage.
- FuriouslyAdrift 1y agoSo... RDMA?
- michaelsshaw 1y agoNo, the first technique describes the basic way they already operate, DMA, but giving access to userspace directly because it's a zerocopy buffer. This is handled by the OS. RDMA is directly from bus-to-bus, bypassing all the software.
- zamalek 1y agoWhat is done for that is userspace gets the network data directly without (I believe) involving syscalls. It's not something you'd do for end-user software, only the likes of MOFAANG need it. In theory the likes of io_uring would bring these benefits across the board, but we haven't seen that delivered (yet, I remain optimistic).
- deleted 1y ago[deleted]
- qwertox 1y agoI recently had to add `ssl_preread_server_name` to my NGINX configuration in order to `proxy_pass` requests for certain domains to another NGINX instance. In this setup, the first instance simply forwards the raw TLS stream (with `proxy_protocol` prepended), while the second instance handles the actual TLS termination. This approach works well when implementing a failover mechanism: if the default path to a server goes down, you can update DNS A records to point to a fallback machine running NGINX. That fallback instance can then route requests for specific domains to the original backend over an alternate path without needing to replicate the full TLS configuration locally. However, this method won't work with HTTP/3. Since HTTP/3 uses QUIC over UDP and encrypts the SNI during the handshake, `ssl_preread_server_name` can no longer be used to route based on domain name. What alternatives exist to support this kind of SNI-based routing with HTTP/3? Is the recommended solution to continue using HTTP/1.1 or HTTP/2 over TLS for setups requiring this behavior?
- jcgl 1y agoHm, that’s a good question. I suppose the same would apply to TCP+TLS with Encrypted Client Hello as well, right? Presumably the answer would be the same/similar between the two.
- MadnessASAP 1y agoUnfortunately I think that falls under the "Not a bug" category of bugs. Keeping the endpoint concealed all the way to the TLS endpoint is a feature* of HTTP/3. * I do actually consider it a feature, but do acknowledge https://xkcd.com/1172/ https://xkcd.com/1172/ PS. HAProxy can proxy raw TLS, but can't direct based on hostname. Cloudflare tunnel I think has some special sauce that can proxy on hostname without terminating TLS but requires using them as your DNS provider.
- dgl 1y agoUnless you're using ECH (encrypted client helo) the endpoint is obscured (known keys), not concealed. PS: HAProxy definitely can do this too, something using req.ssl_sni like this: frontend tcp-https-plain mode tcp tcp-request inspect-delay 10s bind [::]:443 v4v6 tfo acl clienthello req.ssl_hello_type 1 acl example.com req.ssl_sni,lower,word(-1,.,2) example.com tcp-request content accept if clienthello tcp-request content reject if !clienthello default_backend tcp-https-default-proxy use_backend tcp-https-example-proxy if example.com Then tcp-https-example-proxy is a backend which forwards to a server listening for HTTPS (and using send-proxy-v2, so the client IP is kept). Cloudflare really isn't doing anything special here; there are also other tools like sniproxy[1] which can intercept based on SNI (a common thing commerical proxies do for filtering reasons). [1]: https://github.com/ameshkov/sniproxy https://github.com/ameshkov/sniproxy
- jbritton 1y agoThe article didn’t discuss ACK. I have often wondered if it makes sense for the protocol to not have ACKs, and to leave that up to the application layer. I feel like the application layer has to ensure this anyway, so I don’t know how much benefit it is to additionally support this at a lower layer.
- xgpyc2qp 1y agoLooks good. Quick is a real game changer for many. Internet should be a little faster with it. Probably we will not care because of 5g, but still valuable. Wondering that there is a separate tow handshake, I was thinking that qick embeds tls, but seams like I am wrong.
- another_twist 1y agoI have a question - bottleneck for TCP is said to the handshake. But that can be solved by reusing connections and/or multiplexing. The current implementation is 3-4x slower than the Linux impl and performance gap is expected to close. If speed is touted as the advantage for QUIC and it is in fact slower, why bother with this protocol ? The author of the PR itself attributes some of the speed issues to the protocol design. Are there other problems in TCP that need fixing ?
- morning-coffee 1y agoThat's just one bottleneck. The other issue is head-of-line blocking. When there is packet loss on a TCP connection, nothing sent after that is delivered until the loss is repaired.
- anonymousiam 1y agoTCP windowing fixes the issue you are describing. Make the window big and TCP will keep sending when there is a packet loss. It will also retry and usually recover before the end of the window is reached. https://en.wikipedia.org/wiki/TCP_window_scale_option https://en.wikipedia.org/wiki/TCP_window_scale_option
- quietbritishjim 1y agoThe statement in the comment you're replying to is still true. While waiting for those missed packets, the later packets will not be dropped if you have a large window size. But they won't be delivered either. They'll be cached in the kennel, even though it may be that the application could make use of them before the earlier blocked packet.
- morning-coffee 1y agoThey are unrelated. Larger windows help achieve higher throughput over paths with high delay. You allude to selective acknowledgements as a way to repair loss before the window completely drains which is true, but my point is that no data can be delivered to the application until the loss is repaired (and that repair takes at least a round-trip time). (Then the follow-on effects from noticed loss on the congestion controller can limit subsequent in-flight data for a time, etc, etc.)
- gethly 1y agoI've been hearing about QUIC for ages, yet it is still an obscure tech and will likely end up like IPv6.
- gfody 1y agoisn't it just http3 now?
- tralarpa 1y agoYour browser is using it when you watch a video on youtube (HTTP/3).
- Jtsummers 1y agoQUIC is already out and in active use. Every major web browser supports it, and it's not like IPv6. There's no fundamental infrastructure change needed to support it since it's built on top of UDP. The end points obviously have to support it, but that's the same as any other protocol built on UDP or TCP (like HTTP, SNMP, etc.).
- rstuart4133 1y ago> yet it is still an obscure tech and will likely end up like IPv6. Probably. According to Google, IPv6 has a measly 46% of internet traffic now [0], and growing at about 5% per year. QUIC is 40% of Chrome traffic, and is growing at 5% every two years [1]. So yeah, their fates do look similar, which is to say both are headed for world domination in a couple of decades. [0] https://dnsmadeeasy.com/resources/the-state-of-ipv6-adoption-in-2025-progress-pitfalls-and-pathways-forward https://dnsmadeeasy.com/resources/the-state-of-ipv6-adoption... [1] https://www.cellstream.com/2025/02/14/an-update-on-quic-adoption-and-traffic-levels/ https://www.cellstream.com/2025/02/14/an-update-on-quic-adop...
- gethly 1y agoWhen you remove IoT, those numbers will look very differently.
- 1y ago
- snvzz 1y agoBrace for unauthenticated remote execution exploits on network stack.
- bjourne 1y agoWhat is the need for mashing more and more stuff into the kernel? I thought the job of the kernel was to manage memory, hardware, and tasks. Shouldn't protocols built on top of IP be handled by userland?
- leoh 1y agoMaybe. Getting stuff into the kernel means (in theory) it’s been hardened, it has a serious LTS, and benefits from… well, the performance of being part of the kernel.
- heavyset_go 1y agoHaving networking, routing, VPN etc all not leave kernel space can be a performance improvement for some use cases. Similarly, splitting the networking/etc stacks out from the kernel into userspace can also be a performance improvement for some use cases.
- stingraycharles 1y agoYup, context switches between kernelspace and userspace are very expensive in high-performance situations, which is why these types of offloads are used. At specific workloads (think: load balancers / proxy servers / etc), these things become extremely expensive.
- bjourne 1y agoCan't you say that about virtually anything? I'm sure having, say, MIDI synthesizers in the kernel would improve performance too, but not many think that is a good idea.
- heavyset_go 1y agoDepends on the workload and scale. There are cases where offloading everything to userspace in order to minimize context switches into kernel space would improve performance, as well.
- kortilla 1y ago
- codedokode 1y agoThat's so wrong, putting more and more stuff into the kernel and expanding attack surface. How long will it take before someone finds a vulnerability in QUIC handling? The kernel should be as minimal as possible and everything that can be moved to userspace should be moved there. If you are afraid of performance issues then maybe you should stop using legacy processors with slow context switch timing.
- NewJazz 1y agoUse a microkernel if this is your strong opinion. Linux is a monolithic kernel and includes a whole lot in kernel space for the sake of performance and (as mentioned in the article) hardware integration. A well designed microkernel may be able to provide similar performance with better security, but until people put serious work in, it won't be competitive with Linux.
- codedokode 1y agoSadly Linux distributions use large kernel and there is no simple way to get a working desktop system with a microkernel.
- surajrmal 1y agoUnfortunately the os community puts 99% of it'st collective energy into Linux. There is definitely pent up demand for a different architecture. China seems to be innovating here, but it's unclear if the west will get anything out of their designs.
- sevg 1y ago> If you are afraid of performance issues then maybe you should stop using legacy processors with slow context switch timing. By the same logic, we should never improve performance in software and just tell everyone to buy new hardware instead. A bit ridiculous.
- codedokode 1y agoWe should not compromise security for minor improvements in performance.
- EdSchouten 1y ago> Calls to bind(), connect(), listen(), and accept() can be used to initiate and accept connections in much the same way as with TCP, but then things diverge a bit. [...] The sendmsg() and recvmsg() system calls are used to carry out that setup I wish the article explained why this approach was chosen, as opposed to adding a dedicated system call API that matches the semantics of QUIC.
- deepsun 1y agoI thought nowadays every driver wants to be in userspace for exactly speed, especially network ones. So there's no overhead at kernel/userspace later. Strange this article claims the other way, I definitely miss something.