4 ms·
The fact that such a simple endpoint-only control loop (mostly) just works is one of the most amazing parts of the Internet to me. Still, it has its limits – t
by lxgr 3y ago
The fact that such a simple endpoint-only control loop (mostly) just works is one of the most amazing parts of the Internet to me.
Still, it has its limits – the most obvious one to me is mentioned in the article: The optimal buffer size for each hop that TCP (or similarly flow-controlled traffic) traverses depends on the average delay and bandwidth, but these are per-flow quantities which an intermediate router can't properly estimate in principle.
I've played around with Linux's excellent fq_codel a bit with promising results in the past, but ultimately, that's not a solution in the spirit of the Internet (dumb routers, smart endpoints) since it depends on looking at individual flows, something routers aren't really supposed to have to do.
I really hope that something like Google's BBR will become the de-facto TCP congestion control algorithm in the end.
- sambazi 3y ago1. buffersize requirements are not that simple 2. fq_* is meant for end-/edge-routers where the number of flows is rather small https://web.stanford.edu/~sarslan/files/Updating_the_Theory_of_Buffer_Sizing.pdf https://web.stanford.edu/~sarslan/files/Updating_the_Theory_...
- dtaht 3y agofq+ we have been running at 25+gbit or more, for quite some time. There are typically 32k FQ_codels in a 10gbit system. I am pretty sure, that we would not have scaled past 10gbit as well as we did, without having made fq_codel and sch_fq (which does pacing), had we not. We run up to 10k instances of cake (4k queues each) in libreqos. Amazing results.
- sambazi 3y agoimpressive and interesting i assume performance optimizations for the fast-path were successful, or is this scale-up rooted in advances with multithreading and packetsteering?
- dtaht 3y agofq_codel was successfully offloaded in a few products, however, no offload here, just a xeon with a ton of cores and a big L3. On the libreqos case we leveraged some ebpf to do packet steering in what we call the heimdal bridge, and also kathie nichol´s wonderful passive ping concept: https://github.com/thebracket/cpumap-pping https://github.com/thebracket/cpumap-pping src here: https://github.com/LibreQoE/LibreQoS/ https://github.com/LibreQoE/LibreQoS/ You can get a transparent high speed bridge with shaping capability up pretty rapidly with supported hardware. I still long for an ethernet card that can do a trie lookup natively! The flamegraphs are mostly getting the right packet to the right cpu, still.
- dtaht 3y agoDumb enough to me, includes basic fq and aqm technology at the router. FIFOs are the worst form of queuing. FQ solves most of the problem thoroughly enabling delay based transports, Van and Kathie´s codel, most of the rest. Van was also on the team that produced BBR, which spends most of it´s time in delay based mode when fq_codel is on the path. I do wish more folk were actively applying cake nowadays. It works wonders. https://blog.cerowrt.org/post/juniper/ https://blog.cerowrt.org/post/juniper/ Libreqos is doing nice things with cake.
- Animats 3y ago> The fact that such a simple endpoint-only control loop (mostly) just works is one of the most amazing parts of the Internet to me. Me too. We really don't know what to do about congestion in the middle of a pure datagram network. What saved the Internet was cheap fiber backbones. If backbone bandwidth had remained expensive, this would never have worked. When I was working on congestion in the early 1980s, our goal was to run at about 70% long haul link utilization. We would back off quite a bit if things got anywhere near saturation. This was fine with the DoD people who funded us. They wanted reliability under stress, not maximum performance at minimum cost. Much of the work on congestion since then has involved trying to squeeze out that last 30%, at the cost of much-increased complexity. The "bufferbloat" mess is embarrassing. If you use FIFO queuing into a bandwidth choke point, latency is going to go way up. This is well known. It is still ignored in most cheap routers. Every home router feeding limited bandwidth should have outbound fair queuing. Looking at individual flows near the network edges is a big win.
- the8472 3y agoThere's fq_pie that's flow-agnostic.
- wmf 3y agoFlow Queuing is in the name so... no?
- sambazi 3y agopie is effectively (adaptive) red the fq part does most of the work and is intended for edge-routers as bufferspace requirements diminish with growing number of flows.
- dtaht 3y agoexactly right. https://blog.cerowrt.org/post/juniper/ https://blog.cerowrt.org/post/juniper/ We have now produced variants of the flow queuing algorithm for fq-codel, fq-pie, and the FQ in cake, which we are now using in libreqos. It is the best form of fq ever designed, IMHO, with really good min/max fairness properties, resistence to starvation, great rtt fairness and highly complementary simultaneously time based AQM technology easily applied. I would like to see more improvements on the AQM side in the future and for it to land in more hardware. Admittedly, I´m one of the authors... :)
- sambazi 3y agothanks a lot for your work! flow-queuing for the mainstream is a godsend in the information age.
- dtaht 3y agoThank you for leveraging it. but: Eric Dumazet came up with the flow queuing fast/slow queue concept in a single saturday afternoon in his hammock (after admittedly 9 months of hard work on every other fq and aqm algorithm), slammed it into the kernel the next day, and has been building ever more amazing things ever since. I try to make sure people understand that I am more the acolyte of kathie, van, and Eric, than inventor here. I was proud of this thing we´d developed called SFQred, and writing a paper on it... but then kathie and van came out with codel a month later... obsoleting that part... Eric and I implemented it over a week, saw that it was far better than RED... I went to sleep for a week, only to wake up to fq_codel, which solved everything. We have been just re-implementing it and improving the edge cases since. I am pretty happy with cake, now. The world evolves however, and there are new demands on queuing. I cover some of these here: https://docs.google.com/document/d/1tTYBPeaRdCO9AGTGQCpoiuLORQzN_bG3TAkEolJPh28/edit#heading=h.vbbnfu73wlpp https://docs.google.com/document/d/1tTYBPeaRdCO9AGTGQCpoiuLO...