4 ms·

Intra network, TCP slow start is often turned off to minimize latency on links especially ones that you have to respond very quickly to data initially or that i

Intra network, TCP slow start is often turned off to minimize latency on links especially ones that you have to respond very quickly to data initially or that is bursty. Google BBR seems to used the same exponential probing that slow start does, so I wonder how it will perform when you are staying in network and don't often have to worry about packet loss or congestion and want the link to start off at full throttle. Once BBR enters its steady state it intentionally cycles faster and slower, but this seems like it is creating additional latency when you don't want it. Think of a traffic burst that happens just as the link decides to cycle slower. It also seems like the protocol intentionally runs slower that possible as to not create buffer pressure on the receiving side, if I'm understanding this quick description properly: "then cruising at the estimated bandwidth to utilize the pipe without creating excess queue". The this line just scares me: "Occasionally, on an as-needed basis, it sends significantly slower to probe for RTT (PROBE_RTT mode). Google is going to make patches that work for them, but that doesn't always mean it will work for everybody else. This seems very close tailed to Google's traffic issues and serving HTTP over persistent connections, and not a general purpose feature, think of games, intranetwork low-latency applications, etc.

honkhonkpants 10y ago
"staying in network and don't often have to worry about packet loss or congestion." I don't think you would find many Googlers making this equivalence. If your network is highly utilized then it is characterized by both loss and congestion. The trick is optimizing throughput on a congested network without suffering from collapse. Whenever you see anyone talking about "traffic engineering" or "software-defined networking" you should just mentally substitute "packet loss". The only thing that an SDN management plane can do is instruct nodes on the network to drop. Then you can imagine that a highly utilized SDN-managed network has high loss. Edit: In Google's "B4" paper they show that their network operates at packet loss levels approaching 10%. See Figure 14 in http://cseweb.ucsd.edu/~vahdat/papers/b4-sigcomm13.pdf http://cseweb.ucsd.edu/~vahdat/papers/b4-sigcomm13.pdf
- jnordwick 10y ago
  In my work, I don't deal with Internet traffic too much, so internal packet loss is mostly a sign of poor provisioning or technology (if it is the network's fault) or poor programming (if it is the endpoint's fault). Both of those are fixable. It is very rare to have true congestion and sometimes impossible (if it is a dedicated subnet with only a couple hosts). For us latency is more important that throughput, so our needs don't match Google's. There are a lot of cases like this.
  - honkhonkpants 10y ago
    If your network is provisioned such that no congestion is possible, I submit that it might not present very interesting challenges in the domain of flow control :-)
    - jnordwick 10y ago
      I just saying flow control for some thing latency is important as well as throughput. We have to make sure we not packet storming the network down, but we also want data delivered as soon as possible. TCP is used in a lot of situations now from these dedicated intranets to wireless and from serving HTTP to low-latency games. I'm just hoping TCP isn't being pushed too far in one direction because a large institution lives on one side of the spectrum.
  - xorgar831 10y ago
    Some level of packet loss is not a sign of a problem, this is a common confusion which has lead to the buffer bloat issues, and many poor quality behaviours by apps within datacenters. It's such a point of confusion that cloud providers don't want to even share the network error rates with customers since they feel they will misinterpret them.
- solotronics 10y ago
  At my non SDN ISP we do not over provision any of the core connections in the network. We plan and implement link upgrades proactively to make sure congestion is never an issue in network. I have closely followed googles SDN papers and they have much lower costs in comparison. We actually use packet loss as an indicator that there is a hardware issue.
otoburb 10y ago
>>Google is going to make patches that work for them, but that doesn't always mean it will work for everybody else. That makes sense from a purely commercial standpoint, but I was particularly intrigued when I noticed Van Jacobson[1] listed as one of the authors. For those who have read TCP/IP Illustrated (Vols 1 & 2), the four fundamental algorithms (slow start, congestion avoidance, fast retransmit & fast recovery) were evidently designed by Van Jacobson as described in RFC2001.[2] If listing Van Jacobson as one of the patch authors was meant to lend credibility to the patch, then it certainly worked on me as an initial appeal to authority. Particularly interested to see how BBR will perform over wireless networks. [1] http://internethalloffame.org/blog/2012/05/25/van-jacobson-denies-averting-internet-meltdown-1980s http://internethalloffame.org/blog/2012/05/25/van-jacobson-d... [2] https://tools.ietf.org/html/rfc2001 https://tools.ietf.org/html/rfc2001
- jnordwick 10y ago
  Of course I've read the TCP Illustrated series, and of course I know who Van Jacobson is. I went to UC Berkeley.
  - otoburb 10y ago
    My apologies if my reply came across as a rebuttal to your own but I just found it hard to contain my excitement! I referred to Van Jacobson for others who may not know of his significant contributions and who were unaware that he signed off on this particular patch.
    - jnordwick 10y ago
      not at all. I didn't notice the Van Jacobson attribution until the second time I read the link actually.
convolvatron 10y ago
slow start is kind of orthogonal as you say, so lets forget about that. the real meat of this work is to try to address the fact that for whatever reason, buffers are being provisioned well in excess of something that might represent the fractional delay bandwidth product end to end. so this leaves traditional tcp filling up these huge buffers until it gets a loss. a decent red policy would take care of this. the key observation is that if you look at the end to end latency, once you start excess buffering, there is an inflection point. this is the point we are trying to find, where the send rate actually matches a fair share without just dumping packets up to be queued. if you plot latency vs issue rate you can see it very clearly. latency is flat until you start queueing, and then it goes up because you're waiting in line. it makes a lot of sense. particularly if everyone plays along. a really interesting question is what happens if this cc is a minority player in a mix of other adaptation machines. at the bottom end you're under the right set point. but thats a lot better than taking a classic exponential backoff on the eventual loss.
- jnordwick 10y ago
  I can definitely see its use over longish connections, and I can definitely see disabling the patch on a local network these shouldn't be issues (because you have control of the program endpoints). I wonder how it works for things like delivering gaming or real-time-ish feeds where dumping packets might not be such a bad thing (as in they can be written to handle data). Looks interesting though and definitely better for throughput over random networks.
  - an_account 10y ago
    Correct me if I'm wrong, but doesn't bufferbloat kill latency, which hurts gaming/real time more than normal web browsing? So wouldn't this be a major improvement for those type of applications?
    - Filligree 10y ago
      It does, and the usual solution is to not load your link heavily while gaming. If it's someone else's link that's loaded, then you're screwed. However, most games are deeply inflexible on how much bandwidth they use, and additionally tend to use UDP. Running out of bandwidth doesn't degrade the game; it makes them break entirely. As a result of both of these, with a few exceptions, this work won't be directly useful to games.
xorgar831 10y ago
You have to consider that there are other streams on the same network paths, so simply sending faster doesn't actually mean faster transfers. It's almost identical to exiting a building during a fire, everyone can rush or exit orderly. Where the bottleneck is the door, and if everyone tries to get through at once, no one will. Also what's interesting about not going off of packet loss, is that blips in the network are less likely to cause TCP connections to become synchronized and all backoff and re-probe at the same time.
wmf 10y ago
I suspect that Google uses DC-TCP and/or TIMELY inside their datacenters and BBR for WAN and Internet traffic.
toast0 10y ago
> It also seems like the protocol intentionally runs slower that possible as to not create buffer pressure on the receiving side, if I'm understanding this quick description properly: "then cruising at the estimated bandwidth to utilize the pipe without creating excess queue". > The this line just scares me: "Occasionally, on an as-needed basis, it sends significantly slower to probe for RTT (PROBE_RTT mode). I haven't read the proposal, but I think the reason for this is that they're comparing RTT during load with idle RTT to determine packet queuing, but the idle RTT may change over time. Depending on how accurate you want to be, it could be as simple as after some time or packet count of full data packets sent from socket buffer in response to ack moving the window, leave a small gap for the next packet, and then resume sending. If that packet is acked faster than the rest, the idle RTT is shorter than the under load RTT, which means you should slow down in general (to optimize latency). If the RTT is the same for the after gap packet, then the load RTT is close to idle, and you can keep going at the current rate. (I probably wouldn't implement it like I described it. With TCP timestamps, we have pretty continuous RTT measurements, some sort of last N packet min/max/average/stddev to drive the congestion window from all measurements, and a mechanism to add a small gap for the PROBE_RTT would make more sense: any low RTT response should inform the system, not just one that comes in response to a probe)
- jnordwick 10y ago
  I would just never want the situation to be I have open advertised window space and have never dropped a packet, but the other side throttles me to prevent and anticipated packet drop, especially if I'm advertising a very large receive buffer. Just send me the packet.
  - toast0 10y ago
    Even if the roundtrip delay has grown to several seconds? (I've seen pcaps of this kind of thing, something in the path had ridiculous buffers) I took a look at how the throttling is done in their patches, and I think it's probably too much (drops the congestion window to minimum for ~200ms every 10 seconds), I would just drop it a packet or two each interval.
YZF 10y ago
Right. I worked on congestion avoidance for a UDP based protocol and it's really difficult to get this right because you have to coexist with lots of other algorithms on the same network. If you are "fighting" a loss based algorithm TCP stream it's virtually impossible for you to get your fair share of bandwidth without getting packet loss. An algorithm for efficient local networking where everything is under you control is very different than something that runs of the Internet. Not even sure that this style of congestion avoidance is the best approach for a tightly controlled local network. Edit: and keep in mind that packet loss is correlated with queues filling up. So as long as there's lots of loss based algorithms in the wild it's difficult for someone to come in with a better solution that coexists with those other flavours (at least if you're not allowed to touch the "network" itself).
- deleted 10y ago
  [deleted]
- jamesblonde 10y ago
  We are also working on a congestion avoidance for a UDP based protocol - our goal is to share large volumes of data over the Internet but have high throughput for high latency links and support NAT traversal. So no UDT. We're using a varient of LEDBAT (utp/micro-torrent). What are your negative experiences based on?
  - YZF 10y ago
    It's not really a negative experience. It's just that it can be difficult. In general I think it's useful to think as the bandwidth of a TCP stream being a function of loss and latency (and connection time, at least initially, less of a factor for long lived active connections). The different variants of TCP have slightly different curves. You want your UDP algorithm to play nice with these otherwise you may use more than your fair share or you'll get hammered by competing TCP streams. Apologies if this is kind of basic :)
- StillBored 10y ago
  If you are "fighting" a loss based algorithm TCP stream it's virtually impossible for you to get your fair share of bandwidth without getting packet loss. While I agree with the general sentiment, a UDP algorithm matched with a block based FEC, can actually get far more than its "fair share" simply by ignoring the packet loss. Its bad enough, that if your routers aren't deprioritizing UDP traffic you can generally consume really close to 100% of the available bandwidth. Particularly if you pace your packets and pick an algorithm that can handle somewhere in the ballpark of 10-20% packet loss before backing down.

honkhonkpants 10y ago

"staying in network and don't often have to worry about packet loss or congestion." I don't think you would find many Googlers making this equivalence. If your network is highly utilized then it is characterized by both loss and congestion. The trick is optimizing throughput on a congested network without suffering from collapse. Whenever you see anyone talking about "traffic engineering" or "software-defined networking" you should just mentally substitute "packet loss". The only thing that an SDN management plane can do is instruct nodes on the network to drop. Then you can imagine that a highly utilized SDN-managed network has high loss. Edit: In Google's "B4" paper they show that their network operates at packet loss levels approaching 10%. See Figure 14 in http://cseweb.ucsd.edu/~vahdat/papers/b4-sigcomm13.pdf http://cseweb.ucsd.edu/~vahdat/papers/b4-sigcomm13.pdf

jnordwick 10y ago
In my work, I don't deal with Internet traffic too much, so internal packet loss is mostly a sign of poor provisioning or technology (if it is the network's fault) or poor programming (if it is the endpoint's fault). Both of those are fixable. It is very rare to have true congestion and sometimes impossible (if it is a dedicated subnet with only a couple hosts). For us latency is more important that throughput, so our needs don't match Google's. There are a lot of cases like this.
- honkhonkpants 10y ago
  If your network is provisioned such that no congestion is possible, I submit that it might not present very interesting challenges in the domain of flow control :-)
  - jnordwick 10y ago
    I just saying flow control for some thing latency is important as well as throughput. We have to make sure we not packet storming the network down, but we also want data delivered as soon as possible. TCP is used in a lot of situations now from these dedicated intranets to wireless and from serving HTTP to low-latency games. I'm just hoping TCP isn't being pushed too far in one direction because a large institution lives on one side of the spectrum.
- xorgar831 10y ago
  Some level of packet loss is not a sign of a problem, this is a common confusion which has lead to the buffer bloat issues, and many poor quality behaviours by apps within datacenters. It's such a point of confusion that cloud providers don't want to even share the network error rates with customers since they feel they will misinterpret them.
solotronics 10y ago
At my non SDN ISP we do not over provision any of the core connections in the network. We plan and implement link upgrades proactively to make sure congestion is never an issue in network. I have closely followed googles SDN papers and they have much lower costs in comparison. We actually use packet loss as an indicator that there is a hardware issue.

otoburb 10y ago

>>Google is going to make patches that work for them, but that doesn't always mean it will work for everybody else. That makes sense from a purely commercial standpoint, but I was particularly intrigued when I noticed Van Jacobson[1] listed as one of the authors. For those who have read TCP/IP Illustrated (Vols 1 & 2), the four fundamental algorithms (slow start, congestion avoidance, fast retransmit & fast recovery) were evidently designed by Van Jacobson as described in RFC2001.[2] If listing Van Jacobson as one of the patch authors was meant to lend credibility to the patch, then it certainly worked on me as an initial appeal to authority. Particularly interested to see how BBR will perform over wireless networks. [1] http://internethalloffame.org/blog/2012/05/25/van-jacobson-denies-averting-internet-meltdown-1980s http://internethalloffame.org/blog/2012/05/25/van-jacobson-d... [2] https://tools.ietf.org/html/rfc2001 https://tools.ietf.org/html/rfc2001

jnordwick 10y ago
Of course I've read the TCP Illustrated series, and of course I know who Van Jacobson is. I went to UC Berkeley.
- otoburb 10y ago
  My apologies if my reply came across as a rebuttal to your own but I just found it hard to contain my excitement! I referred to Van Jacobson for others who may not know of his significant contributions and who were unaware that he signed off on this particular patch.
  - jnordwick 10y ago
    not at all. I didn't notice the Van Jacobson attribution until the second time I read the link actually.

convolvatron 10y ago

slow start is kind of orthogonal as you say, so lets forget about that. the real meat of this work is to try to address the fact that for whatever reason, buffers are being provisioned well in excess of something that might represent the fractional delay bandwidth product end to end. so this leaves traditional tcp filling up these huge buffers until it gets a loss. a decent red policy would take care of this. the key observation is that if you look at the end to end latency, once you start excess buffering, there is an inflection point. this is the point we are trying to find, where the send rate actually matches a fair share without just dumping packets up to be queued. if you plot latency vs issue rate you can see it very clearly. latency is flat until you start queueing, and then it goes up because you're waiting in line. it makes a lot of sense. particularly if everyone plays along. a really interesting question is what happens if this cc is a minority player in a mix of other adaptation machines. at the bottom end you're under the right set point. but thats a lot better than taking a classic exponential backoff on the eventual loss.

jnordwick 10y ago
I can definitely see its use over longish connections, and I can definitely see disabling the patch on a local network these shouldn't be issues (because you have control of the program endpoints). I wonder how it works for things like delivering gaming or real-time-ish feeds where dumping packets might not be such a bad thing (as in they can be written to handle data). Looks interesting though and definitely better for throughput over random networks.
- an_account 10y ago
  Correct me if I'm wrong, but doesn't bufferbloat kill latency, which hurts gaming/real time more than normal web browsing? So wouldn't this be a major improvement for those type of applications?
  - Filligree 10y ago
    It does, and the usual solution is to not load your link heavily while gaming. If it's someone else's link that's loaded, then you're screwed. However, most games are deeply inflexible on how much bandwidth they use, and additionally tend to use UDP. Running out of bandwidth doesn't degrade the game; it makes them break entirely. As a result of both of these, with a few exceptions, this work won't be directly useful to games.

xorgar831 10y ago

You have to consider that there are other streams on the same network paths, so simply sending faster doesn't actually mean faster transfers. It's almost identical to exiting a building during a fire, everyone can rush or exit orderly. Where the bottleneck is the door, and if everyone tries to get through at once, no one will. Also what's interesting about not going off of packet loss, is that blips in the network are less likely to cause TCP connections to become synchronized and all backoff and re-probe at the same time.

wmf 10y ago

I suspect that Google uses DC-TCP and/or TIMELY inside their datacenters and BBR for WAN and Internet traffic.

toast0 10y ago

> It also seems like the protocol intentionally runs slower that possible as to not create buffer pressure on the receiving side, if I'm understanding this quick description properly: "then cruising at the estimated bandwidth to utilize the pipe without creating excess queue". > The this line just scares me: "Occasionally, on an as-needed basis, it sends significantly slower to probe for RTT (PROBE_RTT mode). I haven't read the proposal, but I think the reason for this is that they're comparing RTT during load with idle RTT to determine packet queuing, but the idle RTT may change over time. Depending on how accurate you want to be, it could be as simple as after some time or packet count of full data packets sent from socket buffer in response to ack moving the window, leave a small gap for the next packet, and then resume sending. If that packet is acked faster than the rest, the idle RTT is shorter than the under load RTT, which means you should slow down in general (to optimize latency). If the RTT is the same for the after gap packet, then the load RTT is close to idle, and you can keep going at the current rate. (I probably wouldn't implement it like I described it. With TCP timestamps, we have pretty continuous RTT measurements, some sort of last N packet min/max/average/stddev to drive the congestion window from all measurements, and a mechanism to add a small gap for the PROBE_RTT would make more sense: any low RTT response should inform the system, not just one that comes in response to a probe)

jnordwick 10y ago
I would just never want the situation to be I have open advertised window space and have never dropped a packet, but the other side throttles me to prevent and anticipated packet drop, especially if I'm advertising a very large receive buffer. Just send me the packet.
- toast0 10y ago
  Even if the roundtrip delay has grown to several seconds? (I've seen pcaps of this kind of thing, something in the path had ridiculous buffers) I took a look at how the throttling is done in their patches, and I think it's probably too much (drops the congestion window to minimum for ~200ms every 10 seconds), I would just drop it a packet or two each interval.

YZF 10y ago

Right. I worked on congestion avoidance for a UDP based protocol and it's really difficult to get this right because you have to coexist with lots of other algorithms on the same network. If you are "fighting" a loss based algorithm TCP stream it's virtually impossible for you to get your fair share of bandwidth without getting packet loss. An algorithm for efficient local networking where everything is under you control is very different than something that runs of the Internet. Not even sure that this style of congestion avoidance is the best approach for a tightly controlled local network. Edit: and keep in mind that packet loss is correlated with queues filling up. So as long as there's lots of loss based algorithms in the wild it's difficult for someone to come in with a better solution that coexists with those other flavours (at least if you're not allowed to touch the "network" itself).

deleted 10y ago
[deleted]
jamesblonde 10y ago
We are also working on a congestion avoidance for a UDP based protocol - our goal is to share large volumes of data over the Internet but have high throughput for high latency links and support NAT traversal. So no UDT. We're using a varient of LEDBAT (utp/micro-torrent). What are your negative experiences based on?
- YZF 10y ago
  It's not really a negative experience. It's just that it can be difficult. In general I think it's useful to think as the bandwidth of a TCP stream being a function of loss and latency (and connection time, at least initially, less of a factor for long lived active connections). The different variants of TCP have slightly different curves. You want your UDP algorithm to play nice with these otherwise you may use more than your fair share or you'll get hammered by competing TCP streams. Apologies if this is kind of basic :)
StillBored 10y ago
If you are "fighting" a loss based algorithm TCP stream it's virtually impossible for you to get your fair share of bandwidth without getting packet loss. While I agree with the general sentiment, a UDP algorithm matched with a block based FEC, can actually get far more than its "fair share" simply by ignoring the packet loss. Its bad enough, that if your routers aren't deprioritizing UDP traffic you can generally consume really close to 100% of the available bandwidth. Particularly if you pace your packets and pick an algorithm that can handle somewhere in the ballpark of 10-20% packet loss before backing down.