4 ms·
Evaluating TCP BBRv2 on the Dropbox edge network
- danudey 6y agoFor anyone else who didn't know, BBR stands for "Bottleneck Bandwidth and RTT", and is a congestion control algorithm. More (but not much) information is available on Google's BBR repository: https://github.com/google/bbr https://github.com/google/bbr
- throwaway189262 6y agoBBR is great. It should be on by default in Linux kernel. Unfortunately its only for TCP, so usefulness is somewhat limited outside the datacentre. For us devs, it's sad how little software uses LEDBAT protocol for bulk downloads. All of you should be aware of the advantages and use it wherever possible. It allows bulk data transfer without slowing down priority data streams. And it works without any OS and router support. For those with slow home connections, OpenWrt with SQM CAKE enabled is incredible. If you turn on the options for per-ip flows, it almost perfectly eliminates bufferbloat. I run OpenWrt x64 edition on one of my old desktops. We have abysmally slow DSL at home and in office and it's a night and day difference. Multiple people torrenting, uploading, watching video and you can still browse the web
- Arie 6y agoSQM with Cake is a life saver in any kind of network with bufferbloat. The per-host/IP fairness is the cherry on top.
- the8472 6y ago> Unfortunately its only for TCP, so usefulness is somewhat limited outside the datacentre. Also works nicely for p2p protocols that have to connect to peers on other continents.
- luizfelberti 6y agoIs there any study comparing the impact of BBR vs CUBIC on a LEDBAT stream? I assume it to be equivalent because of LEDBAT's eager backoff, but it'd still be a cool thing to read
- throwaway189262 6y agoI haven't seen anything. They're both based primarily on RTT round trip time instead of packet loss %. So they should share roughly equally. The big innovation around these new schedulers is focusing on fairness instead of maximum throughout. For the longest time TCP flow control was judged only by how close it reached to theoretical maximum throughout. So we ended up with extremely aggressive schedulers. The challenge now is to make a scheduler that limits bufferbloat while not surrendering all it's throughout to old school bully schedulers. This is where BRR really innovates
- Diederich 6y agohttps://lwn.net/Articles/701165/ https://lwn.net/Articles/701165/ is a nice writeup.
- luizfelberti 6y agoHere's a few questions I have for any lurking network engineers "in the know", since there doesn't seem to be an official paper for BBRv2 yet (or at least I couldn't find it on https://research.google/pubs/ https://research.google/pubs/): - Is BBRv2 still modeled close to the PID equation, in a way that expanding the equation as shown by BBRv1x [0] is still possible, and we can tune for different target parameters? - How tunable is BBRv2? For example, say I'm doing some voice chat app, and in my use case an increased sensitivity to ECN is presumably better (I read somewhere that Facetime does this, but regardless of that, let's just assume it's true for a moment). Can I tune that? Or disable ECN-awareness entirely? Is there any good reference material on the parameters BBRv2 exposes? - I never saw any version of BBR be compared to TIMELY [1] which is also from Google. Even though I assume TIMELY would be worse given that it only takes RTT into consideration, it'd still be cool to see - BBR is great for clients, but ultimately dependent on trusting other clients in the network to be well behaved, else the game-theoretical fairness doesn't work out. What is the SOTA algorithm for AQM on middle-boxes nowadays, and preventing bad clients from hogging link capacity? Questions aside though, it's great to see more algorithms leveraging ECN as a signal. Good stuff, kudos to the researchers! [0] https://www.youtube.com/watch?v=PeYPqnLhUuc https://www.youtube.com/watch?v=PeYPqnLhUuc [1] https://research.google/pubs/pub43840/ https://research.google/pubs/pub43840/
- the8472 6y ago> What is the SOTA algorithm for AQM on middle-boxes nowadays, and preventing bad clients from hogging link capacity? fq_pie for constrained systems. fq_codel or cake if you have enough CPU cycles. They can be operated in ECN-aware mode.
- the8472 6y agoThe improved ss output mentioned in the paper has helped me immensely tracking down what was limiting the upload speed from a corporate net to AWS. Turned out no middleboxes were to blame, just a misconfiguration in nginx (receiving side) limiting the http2 send window (sender side). Yes, http2 has a separate send window on top of tcp. Yet another source of errors.
- luizfelberti 6y ago> Yes, http2 has a separate send window on top of tcp. Yet another source of errors Damn, I was not aware of this, but sure enough there it is [0] I suppose that's just one more thing to reaffirm my belief that SPDY is a horribly over-engineered protocol, that tried to shoehorn fixes for TCP's shortcomings on top of the wrong abstraction (which, ironically, is TCP itself). Had HTTP2 only introduced the binary frames encoding, instead of bolting this horrible new transport along with it, we wouldn't be so deep in this mess. QUIC can't come soon enough. [0] https://www.chromium.org/spdy/spdy-protocol/spdy-protocol-draft3-1#TOC-2.5-Data-flow https://www.chromium.org/spdy/spdy-protocol/spdy-protocol-dr...
- morei 6y agoISTR that SPDY et al was originally built on UDP, not TCP, so there was a need for a send window. But yes, it's now fighting with the TCP window. Ideally, the HTTP2 send window should be derived from the TCP window.
- Matthias247 6y agoMaybe the bad news for you: Quic is different, but it's not necessarily a lot easier. A lot of the challenges that are part of implementing HTTP/2 correctly are also present in Quic. You will still have to implement a flow control mechanism for individual Quic streams, apart from the being fair across streams, congestion control and pacing for Quic connections, and being fair across connections on an endpoint. Getting this all right is also far from an easy task - and resource wise it might never catch up with HTTP/1.1 + TLS due to missing hardware support. However I feel like at least in some areas the Quic specification became less complex. E.g. the use of absolute flow control offsets instead of relative increments makes things less error prone. And it's nice that now the defaults for the dynamic table are "off", which means one can skip implementing that part without the risk of not being interoperable.