7 ms·

BBR, the new kid on the TCP block

360 points by pjf 9y ago

huhtenberg 9y ago
Any data on BBR vs Reno and Vegas sharing? Link capacity estimation is easy. It's the co-existing gracefully with all other flow control options that's tricky.
- throwaway2048 9y ago
  There is also no mention of how it handles things like wireless links with high jitter and intermittant not related to flow rate packet loss.
  - tankenmate 9y ago
    Well it does mention that BBR uses "The RTT is the minimum of all RTT measurements over some time window that is described as “tens of seconds to minutes”." This is a much larger measuring window than previous RTT tracking TCP. At 10s of seconds it is also hopefully enough to tackle the most common forms of jitter and intermittent flows. At truly bad flow characteristics you need to start some truly weird line coding methods with stream data split over multiple packets with redundant coding to rebuild dropped packets etc, but this kind of thing just kills latency.
- baq 9y ago
  There's a 'Sharing' section near the end where two scenarios are compared. Doesn't look like an exhaustive test, rather the opposite.
  - huhtenberg 9y ago
    Sure, I read the article. That section covers just CUBIC.
    - dsr_ 9y ago
      It also doesn't discuss competition between BBR users. Playing nicely with your neighbors is important for the health of the Internet.
      - Arnt 9y ago
        It doesn't discuss competition with either many users, with things like webrtc, or with those evil dial-a-speed schemes. The internet is complex. However, since the dial-a-speed evildoers haven't caused much controversy in the grand scheme of things, I don't think BPR will do real harm in that case. I'm more concerned about unloaded parts of a circuit. Suppose some BPR streams fill one hop, and share other, higher-capacity, hops with Reno. Does Reno do as well then as it would if the BPR streams were Reno?
      - metafnord 9y ago
        Several BBR flows do actually converge quite nicely to a fair share of bandwidth. Take a look at the presentation the guys from google gave at the IETF 97 in Seoul: https://www.ietf.org/proceedings/97/slides/slides-97-iccrg-bbr-congestion-control-02.pdf https://www.ietf.org/proceedings/97/slides/slides-97-iccrg-b... I think there is also a recording of that session somewhere on youtube. Also, the complete paper can be downloaded for free at http://queue.acm.org/detail.cfm?id=3022184 http://queue.acm.org/detail.cfm?id=3022184
  - morecoffee 9y ago
    What about BBR sharing with BBR? Really, what is the reason this shouldn't be rolled out as the default everywhere?
deleted 9y ago
[deleted]
notacoward 9y ago
It's almost irresponsible to write an article on this topic in 2017 without explicitly mentioning bufferbloat or network-scheduling algorithms like CoDel designed to address it. If you really want to understand this article, read up on those first. https://en.wikipedia.org/wiki/CoDel https://en.wikipedia.org/wiki/CoDel
- throwasehasdwi 9y ago
  CoDel is different from the packet scheduling algorithm even though both fight bufferbloat in different ways. CoDel is a congestion control algorithm for controlling what happens when outgoing buffers start overflowing. This is on a lower level than TCP and happens to any type of packet. The scheduling algorithm, like VEGAS or BBR, controls the transmit rate of only the TCP protocol. When packets are being sent over the wires, the TCP scheduling algorithm (usually CUBIC, VEGAS, RENO, or now BBR) will send out packets until the parameters they monitor indicate the downstream device is about to overload. Then they will back off slightly to prevent packets from being lost. These TCP transmit strategies tend to either monitor packet loss rate or round trip time, sometimes both. What they do with these two parameters determines the biggest differences between the packet sending algorithms. CODEL comes into affect when the scheduling algorithm decides it can't send out packets quickly enough without losing them, and they build up on local buffers. This can happen with TCP but also other internet protocols. Something most people don't know is that without a scheduling algorithm like BBR,VEGAS, or RENO, you can send out packets at interface speed. In simpler protocols like UDP you need to do your own packet scheduling. Otherwise your machine will send out packets at interface speed until they are mostly dropped by the first slower link. This is why TCP has scheduling algorithms, they're all an attempt to monitoring the end to end link speed from A to B you can achieve without losing data. Edit: BBR is a new TCP scheduling algorithm to fight buffer-bloat at the TCP level. Since the majority of internet traffic is TCP, wide adoption would cause a big improvement. TCP scheduling only affects outgoing packets, so its important to get this into Windows and Linux so we can get the full benefit of having buffer-bloat reduction on both ends. I'm looking at MS here because they're the last major OS running an aggressive and buffer-bloat causing TCP algorithm.
  - 9y ago
brutuscat 9y ago
First saw it at the morning paper: https://blog.acolyer.org/2017/03/31/bbr-congestion-based-congestion-control/ https://blog.acolyer.org/2017/03/31/bbr-congestion-based-con... This is the story of how members of Google’s make-tcp-fast project developed and deployed a new congestion control algorithm for TCP called BBR (for Bandwidth Bottleneck and Round-trip propagation time), leading to 2-25x throughput improvement over the previous loss-based congestion control CUBIC algorithm.
kstenerud 9y ago
It seems like the best way to handle this situation is to assume that all other algorithms are hostile, and to seize as much bandwidth as you can without causing queue delay. That would reduce the problem set to a basic resource competition problem, which could then be solved with genetic algorithms.
abainbridge 9y ago
Not be confused with BBR enhancing the Mazda MX-5: https://www.pistonheads.com/news/ph-japanesecars/mazda-mx-5-bbr-stage-1-turbo-review/36187 https://www.pistonheads.com/news/ph-japanesecars/mazda-mx-5-... Also significantly reduces latency and increases throughput :-)
- TheSwordsman 9y ago
  It even has a flow-control mechanism (wastegate).
skyde 9y ago
How can we use it today! Is it in Linux code already and easy to enable ?
- metafnord 9y ago
  The code is already in net-next. This means you currently have to build your own kernel if you want to use it. This should be enough to get you going: https://github.com/google/bbr/blob/master/Documentation/bbr-quick-start.md https://github.com/google/bbr/blob/master/Documentation/bbr-...
- lossolo 9y ago
  It's in kernel 4.9
  - ymse 9y ago
    https://github.com/torvalds/linux/commit/0f8782ea14974ce992618b55f0c041ef43ed0b78 https://github.com/torvalds/linux/commit/0f8782ea14974ce9926...
- Scaevolus 9y ago
  It's available in Ubuntu 17.04. Add these lines to /etc/sysctl.conf: net.core.default_qdisc=fq net.ipv4.tcp_congestion_control=bbr I got 400Mbps (on a 1Gbps link) from Seattle to New York with a single TCP-BBR stream, vs ~50Mbps before :-).
  - newman314 9y ago
    How are you measuring this?
  - otterley 9y ago
    You could probably saturate that link with a single TCP stream -- even without BBR -- by simply choosing the correct send and receive buffer sizes in your test program. (I assume you're using iperf or a similar program to perform your tests.) BBR and other congestion-control algorithms come into play when the underlying IP network is congested. To test them, you have to have control of the congestion rate of the underlying circuit, which you might not have if you're testing this on an ordinary public IP network.
netheril96 9y ago
Network performance across national borders within China has been abysmal since the censorship got much more serious. BBR seems promising, so more and more people (that includes me) who bypass GFW with their own VPS has been deploying BBR, and seen marvelous results.
raldi 9y ago
For those dying to know what it stands for: Bottleneck Bandwidth and Round-trip time
gritzko 9y ago
Sounds like they adapted LEDBAT delay measuring tricks.
emmelaich 9y ago
This article is not only a great intro to BBR, but an excellent introduction the history of flow control. Congrats to Geoff and his team.
emmelaich 9y ago
> ... the startup procedure must rapidly converge to the available bandwidth irrespective of its capacity It seems to me that you'd be able to make a rough guesstimate by noting the ip address; whether it's on the same LAN, or continent/AS. It wouldn't matter if you got it very wrong as long as you converged quickly to a better one (as you have to do anyway)
skyde 9y ago
Would adding this only to the http reverse proxy machines provide most of the benefit without have to patch all servers. This seem to have the greatest effect over wan links.

360 points by pjf 9y ago

Any data on BBR vs Reno and Vegas sharing? Link capacity estimation is easy. It's the co-existing gracefully with all other flow control options that's tricky.

throwaway2048 9y ago
There is also no mention of how it handles things like wireless links with high jitter and intermittant not related to flow rate packet loss.
- tankenmate 9y ago
  Well it does mention that BBR uses "The RTT is the minimum of all RTT measurements over some time window that is described as “tens of seconds to minutes”." This is a much larger measuring window than previous RTT tracking TCP. At 10s of seconds it is also hopefully enough to tackle the most common forms of jitter and intermittent flows. At truly bad flow characteristics you need to start some truly weird line coding methods with stream data split over multiple packets with redundant coding to rebuild dropped packets etc, but this kind of thing just kills latency.
baq 9y ago
There's a 'Sharing' section near the end where two scenarios are compared. Doesn't look like an exhaustive test, rather the opposite.
- huhtenberg 9y ago
  Sure, I read the article. That section covers just CUBIC.
  - dsr_ 9y ago
    It also doesn't discuss competition between BBR users. Playing nicely with your neighbors is important for the health of the Internet.
    - Arnt 9y ago
      It doesn't discuss competition with either many users, with things like webrtc, or with those evil dial-a-speed schemes. The internet is complex. However, since the dial-a-speed evildoers haven't caused much controversy in the grand scheme of things, I don't think BPR will do real harm in that case. I'm more concerned about unloaded parts of a circuit. Suppose some BPR streams fill one hop, and share other, higher-capacity, hops with Reno. Does Reno do as well then as it would if the BPR streams were Reno?
    - metafnord 9y ago
      Several BBR flows do actually converge quite nicely to a fair share of bandwidth. Take a look at the presentation the guys from google gave at the IETF 97 in Seoul: https://www.ietf.org/proceedings/97/slides/slides-97-iccrg-bbr-congestion-control-02.pdf https://www.ietf.org/proceedings/97/slides/slides-97-iccrg-b... I think there is also a recording of that session somewhere on youtube. Also, the complete paper can be downloaded for free at http://queue.acm.org/detail.cfm?id=3022184 http://queue.acm.org/detail.cfm?id=3022184
- morecoffee 9y ago
  What about BBR sharing with BBR? Really, what is the reason this shouldn't be rolled out as the default everywhere?

deleted 9y ago

[deleted]

notacoward 9y ago

It's almost irresponsible to write an article on this topic in 2017 without explicitly mentioning bufferbloat or network-scheduling algorithms like CoDel designed to address it. If you really want to understand this article, read up on those first. https://en.wikipedia.org/wiki/CoDel https://en.wikipedia.org/wiki/CoDel

throwasehasdwi 9y ago
CoDel is different from the packet scheduling algorithm even though both fight bufferbloat in different ways. CoDel is a congestion control algorithm for controlling what happens when outgoing buffers start overflowing. This is on a lower level than TCP and happens to any type of packet. The scheduling algorithm, like VEGAS or BBR, controls the transmit rate of only the TCP protocol. When packets are being sent over the wires, the TCP scheduling algorithm (usually CUBIC, VEGAS, RENO, or now BBR) will send out packets until the parameters they monitor indicate the downstream device is about to overload. Then they will back off slightly to prevent packets from being lost. These TCP transmit strategies tend to either monitor packet loss rate or round trip time, sometimes both. What they do with these two parameters determines the biggest differences between the packet sending algorithms. CODEL comes into affect when the scheduling algorithm decides it can't send out packets quickly enough without losing them, and they build up on local buffers. This can happen with TCP but also other internet protocols. Something most people don't know is that without a scheduling algorithm like BBR,VEGAS, or RENO, you can send out packets at interface speed. In simpler protocols like UDP you need to do your own packet scheduling. Otherwise your machine will send out packets at interface speed until they are mostly dropped by the first slower link. This is why TCP has scheduling algorithms, they're all an attempt to monitoring the end to end link speed from A to B you can achieve without losing data. Edit: BBR is a new TCP scheduling algorithm to fight buffer-bloat at the TCP level. Since the majority of internet traffic is TCP, wide adoption would cause a big improvement. TCP scheduling only affects outgoing packets, so its important to get this into Windows and Linux so we can get the full benefit of having buffer-bloat reduction on both ends. I'm looking at MS here because they're the last major OS running an aggressive and buffer-bloat causing TCP algorithm.
- 9y ago

brutuscat 9y ago

First saw it at the morning paper: https://blog.acolyer.org/2017/03/31/bbr-congestion-based-congestion-control/ https://blog.acolyer.org/2017/03/31/bbr-congestion-based-con... This is the story of how members of Google’s make-tcp-fast project developed and deployed a new congestion control algorithm for TCP called BBR (for Bandwidth Bottleneck and Round-trip propagation time), leading to 2-25x throughput improvement over the previous loss-based congestion control CUBIC algorithm.

kstenerud 9y ago

It seems like the best way to handle this situation is to assume that all other algorithms are hostile, and to seize as much bandwidth as you can without causing queue delay. That would reduce the problem set to a basic resource competition problem, which could then be solved with genetic algorithms.

abainbridge 9y ago

Not be confused with BBR enhancing the Mazda MX-5: https://www.pistonheads.com/news/ph-japanesecars/mazda-mx-5-bbr-stage-1-turbo-review/36187 https://www.pistonheads.com/news/ph-japanesecars/mazda-mx-5-... Also significantly reduces latency and increases throughput :-)

TheSwordsman 9y ago
It even has a flow-control mechanism (wastegate).

skyde 9y ago

How can we use it today! Is it in Linux code already and easy to enable ?

metafnord 9y ago
The code is already in net-next. This means you currently have to build your own kernel if you want to use it. This should be enough to get you going: https://github.com/google/bbr/blob/master/Documentation/bbr-quick-start.md https://github.com/google/bbr/blob/master/Documentation/bbr-...
lossolo 9y ago
It's in kernel 4.9
- ymse 9y ago
  https://github.com/torvalds/linux/commit/0f8782ea14974ce992618b55f0c041ef43ed0b78 https://github.com/torvalds/linux/commit/0f8782ea14974ce9926...
Scaevolus 9y ago
It's available in Ubuntu 17.04. Add these lines to /etc/sysctl.conf: net.core.default_qdisc=fq net.ipv4.tcp_congestion_control=bbr I got 400Mbps (on a 1Gbps link) from Seattle to New York with a single TCP-BBR stream, vs ~50Mbps before :-).
- newman314 9y ago
  How are you measuring this?
- otterley 9y ago
  You could probably saturate that link with a single TCP stream -- even without BBR -- by simply choosing the correct send and receive buffer sizes in your test program. (I assume you're using iperf or a similar program to perform your tests.) BBR and other congestion-control algorithms come into play when the underlying IP network is congested. To test them, you have to have control of the congestion rate of the underlying circuit, which you might not have if you're testing this on an ordinary public IP network.

netheril96 9y ago

Network performance across national borders within China has been abysmal since the censorship got much more serious. BBR seems promising, so more and more people (that includes me) who bypass GFW with their own VPS has been deploying BBR, and seen marvelous results.

raldi 9y ago

For those dying to know what it stands for: Bottleneck Bandwidth and Round-trip time

gritzko 9y ago

Sounds like they adapted LEDBAT delay measuring tricks.

emmelaich 9y ago

This article is not only a great intro to BBR, but an excellent introduction the history of flow control. Congrats to Geoff and his team.

emmelaich 9y ago

> ... the startup procedure must rapidly converge to the available bandwidth irrespective of its capacity It seems to me that you'd be able to make a rough guesstimate by noting the ip address; whether it's on the same LAN, or continent/AS. It wouldn't matter if you got it very wrong as long as you converged quickly to a better one (as you have to do anyway)

skyde 9y ago

Would adding this only to the http reverse proxy machines provide most of the benefit without have to patch all servers. This seem to have the greatest effect over wan links.