4 ms·
When “idle” isn't idle: how a Linux kernel optimization became a QUIC bug
- blahgeek 5mo agoThe more precise title should be: How we copied code from Linux kernel without fully understand it and missed its follow-up fixes, now it bites us
- embedding-shape 5mo agoAlso, not a single takeaway about how to prevent that very preventable issue in the first place, as you allude to. I wonder what happened with the very hardcore engineering that used to happen at Cloudflare and was published? Almost every blog post today seems to expose some weirdness at Cloudflare, rather than highlighting excellence in engineering, what changes? Been slowly changing over the years, did they change their hiring practices or something?
- rslashuser 5mo agoThe test is the hardcore engineering tell here. The test is dialed in on the key area, and when the graph wasn't coming out the right shape, they kept at it. Plus one from me!
- neuralkoi 5mo agoI can see why they rewrote QUIC in Rust and for use in userspace, though going the in-house approach would warrant keeping an eye on the relevant kernel commits like a hawk to avoid missing bug fixes like these. These in-house implementations tend to have less eyeballs than the kernel. I found it interesting that Cloudflare is not yet using BBR as the default in quiche. CUBIC's recovery in this day and age, and especially in datacenters with large pipes, seems so slooow to me. Almost two seconds with no loss whatsoever till achieving BDP again and then shooting itself in the foot every time it hits the ceiling. Each one of those losses a retransmission.
- masklinn 5mo ago> I can see why they rewrote QUIC in Rust and for use in userspace As far as I know, while they might have either way, they did not ("rewrite QUICK [...] for use in userspace"): the linux kernel implementation only landed late 2025. Quiche was started ca 2018 (that's when Cloudflare started beta-deploying QUIC, the first public alpha of quiche was january 2019). I don't know that there even was an in-kernel implementation of quic before msquic.sys which I believe first shipped in Server 2022 circa mid 2021 (and is used as the implementation backend by MsQuic on Server 2022 and W11).
- benmmurphy 5mo agoI think the original commenter confused taking the CUBIC implementation from the kernel and rewriting that in Rust for use in their QUIC implementation or they just jumbled their wording. It does make sense to use an existing battle tested implementation of a congestion algorithm because there are potential many real world failure modes that you might not anticipate if you try and write an implementation from scratch.
- neuralkoi 5mo agoYes, I meant CUBIC implementation! But I'm glad I made the mistake-I learned some interesting things from the responses above.
- vasilvv 5mo ago> though going the in-house approach would warrant keeping an eye on the relevant kernel commits like a hawk to avoid missing bug fixes like these. These in-house implementations tend to have less eyeballs than the kernel. This is somewhat funny to read because this specific issue in CUBIC (sudden CWND jump upon existing quiescence) was originally discovered in Google's QUIC library and then later reported to the team working on the TCP stack. I know this because I was the one who found that bug back in 2015. That said, congestion control algorithms are really prone to logic bugs, and very subtle changes in the algorithm can often lead to dramatically different outcomes. Because of that, there's a lot of value in running congestion control code that has been tested on a wide variety of real Internet traffic.
- extropy 5mo agoIs it just me, or the article structure and subtitles feel very AI?
- bonzini 5mo agoYes, and it becomes unbearable after a while.
- twoodfin 5mo agoI don’t get it. Unlike a lot of the technical article slop that is posted here, this obviously had a lot of human thought and effort put into the prompt. The LLM pass (unsurprisingly) made it worse. For example: The results were conclusive: 100% pass rate, showing Reno recovered cleanly after the loss phase, and revealing that this is a CUBIC-related bug. Look, I’m reading a description of a Linux kernel network congestion bug. I don’t need the hand-holding.
- bonzini 5mo agoYeah, you aren't selling anything. "Reno has a 100% pass rate for recovering cleanly after the loss phase, so the bug is almost certainly related to CUBIC" is a perfectly fine technical text.
- twoodfin 5mo agoAlso, the same event both “showing” and “revealing” two different things is just bad writing.
- phreack 5mo agoAnother LLM tell is that they penalize repetition so they'll use as many synonyms as possible. You may end up recognizing the same concept being rehashed with synonyms constantly. You can look up examples of thie as "elegant variation"
- echoangle 5mo agoLooking at the last plot, it seems like the backoff is roughly 1/5 of the total bandwith and it happens every 50 ms or so. Wouldn't it make sense to reduce the backoff and the growth speed if a backoff occurs repeatedly in rapid succession? We want to maximize the area under the curve (transmitted packages), right?
- neuralkoi 5mo agoAs per the article, CCAs aim to maximize data transfer by inferring the "available bandwidth" of the network. CUBIC relies primarily on packet loss as a congestion signal. For recovery, CUBIC's window size is a cubic function of time since the last congestion event. After the initial packet loss triggered purposefully the first two seconds in this experiment, the only thing which could cause loss is the network queue (i.e. a simple tail drop, fq-codel, etc) which cannot process packets faster than they can arrive. At this point the link is saturated. The loss becomes a signal for CUBIC to reduce its window. This causes the oscillations you pointed out. Unlike CUBIC, BBR [0] uses a model-based approach that estimates the available bandwidth and leaves some headroom kind of like you suggest to achieve higher throughput, and doesn't react as aggressively to loss as CUBIC. [0] https://datatracker.ietf.org/meeting/104/materials/slides-104-iccrg-an-update-on-bbr-00 https://datatracker.ietf.org/meeting/104/materials/slides-10...
- lproven 5mo agoThe article uses the term "CCAs" without ever defining it. I followed the links, and googled it, with no useful result. What is a CCA in this context?
- Rp8yXmdmr 5mo agoCongestion Control Algorithms
- einsteinx2 5mo agoAfter some searching apparently it means “congestion control algorithm”. Definitely should have been defined in the article, especially since they have a whole section dedicated to explaining what it is.
- gavinsyancey 5mo agoa Congestion Control Algorithm -- which uses various signals (mostly dropped packets) to try to estimate the available bandwidth and avoid network connection.
- lproven 5mo agoThanks! And to @einsteinx2 and @rp8yxmdmr too.
- yencabulator 5mo agoNot sure if it got edited, but the first use of the term is > Before we dive into the core problem, a quick refresher on Congestion Control Algorithms (CCAs) may help to set the stage.
- rslashuser 5mo agoWhat jumps out to me is that this is a success story of using a non-trivial test to illuminate an important but hard to observe bit of algorithm. I appreciate the engineering grit to put in a complex test like this, and follow it up when the graph does not have the expected shape. Imagine your team does not want to write a test because it's too much work or hard to model - this is a great example to bring up.
- Rpu-Micro 5mo ago[dead]
- OptionOfT 5mo agoI tried to read the article, and had to go back and forth between reading about terms and etc, because I'm not _that_ familiar in the space. But previously I could understand Cloudflare's blog posts. This one just feels... off. The buildup just doesn't feel right. The fact that there is an Em Dash (sorry...) in the code tells me that it's at least AI assisted, which explains the vibe the article emanates. And once I finally made it to the end I read the following: > If you're interested in congestion control, transport protocols, or contributing to open-source networking code, check out the quiche repository. We're always looking for talented engineers who love digging into problems like these, please explore our open positions. You don't add that to your blog-posts 5 days after laying off 20% of the company, regardless of whether they're sales people or engineers. If you want to add it, delay the post by 2 weeks. Equally, there is only 1 role open in Engineering, and it's an intern role, posted yesterday: https://www.cloudflare.com/careers/ https://www.cloudflare.com/careers/ (filter by Engineering). Did they lay off their PR team as well?
- xxs 5mo ago> it's at least AI assisted, which explains the vibe the article emanates You're being too kind -- "The [b]epoch[b] is the reference timestamp CUBIC". Weird style to have random bold words. Its a blog post for the sake of it - no real takeaway. Well, there is the takeaway section that's a special summary of the article, instead.
- hack1312 5mo agoThe vast majority of the marketing team was laid off. Prince said he only needs people to build and people to sell, fuck everyone else.
- johng 5mo agoI _believe_ i've been having some connectivity issues related to this. Where the site acts like it's down but you refresh the page and it works. The solution is to disable the HTTP-3/QUIC toggle in Cloudflare. Some discussion here: https://xenforo.com/community/threads/page-could-not-be-loaded.193001/page-2#post-1780168 https://xenforo.com/community/threads/page-could-not-be-load... May not be related, but it might. It only recently started happening to me/our sites in the past couple of days.
- insumanth 5mo agoThe line that stuck with me: "Recovery after congestion collapse is an uncommon regime, but it is exactly the regime a congestion controller exists to handle." This generalizes well beyond congestion control. Most control loops have the same property. The path the system follows 99% of the time gets well-exercised; the path it falls into when things go wrong is the path you actually need to be correct on. There's usually no way to discover the bug until you deliberately drive the system into the bad regime and watch it try to climb out.