4 ms·

Why doesn't the congestion control part of TCP prevent buffer bloat[1]? Is it because ISP throttling of the internet connection doesn't touch the TCP packets th

Why doesn't the congestion control part of TCP prevent buffer bloat[1]? Is it because ISP throttling of the internet connection doesn't touch the TCP packets themselves? I recently started doing off-site backups, which requires my entire internet uplink to be used for uploading said backups for about a week at a time. The internet basically becomes unusable because all the packets end up in a buffer on the router and latency spikes to 5000ms. [1] https://www.bufferbloat.net/projects/bloat/wiki/What_can_I_do_about_Bufferbloat/ https://www.bufferbloat.net/projects/bloat/wiki/What_can_I_d...

jeffbee 6y ago
If there is a huge FIFO queue on your router, the rate-finding algorithms associated with TCP will be forced to conclude that the RTT to your site is enormous. They may try to open the window to compensate, but here's a fun fact: most operating system default settings are insufficient to utilize very high bandwidth-delay products. If you want to send a 1gbps flow across an 80ms distance on Linux, you'll need to change some parameters with sysctl before it will work. If your apparent RTT is 5000ms, the flow you can get will be reduced in proportion. In any case, the solution to bufferbloat is queue discipline, not congestion control.
- jfkebwjsbx 6y ago
  Up to what speeds/latencies are the default sysctl parameters alright? Is there any easy way to know whether you are getting hit by this? Nowadays many people is getting 1 Gbps links at home! What do you mean by queue discipline?
  - jeffbee 6y ago
    You know, the worst part is that Linux sets the maximum receive window size at boot time depending on how much memory the system contains, ensuring that it's never quite right. On this machine, with 32GB of main memory, it defaults to 6291456 bytes.
    - jfkebwjsbx 6y ago
      I see, thanks! What about the queue discipline?
      - jeffbee 6y ago
        If you face a choice of what frame to put on the wire at any moment, the queue discipline makes that choice. The easiest policy is to simply send the oldest frame, but this is also the worst policy.
        jfkebwjsbx 6y ago
        Ah, so the eviction/priority algorithm. Thanks!
toast0 6y ago
> Why doesn't the congestion control part of TCP prevent buffer bloat[1]? Is it because ISP throttling of the internet connection doesn't touch the TCP packets themselves? Most of the congestion control algorithms use packet loss as the only indicator of congestion. In a network with oversized buffers, congestion will result in delay and not packet loss. If the delay gets large enough, recieve and congestion windows will restrict the effective bandwidth, but the latency at that point is terrible. There are some alternate congestion control algorithms which do use latency as a signal, but they aren't universally available, and may not be a good fit for all flows. For your backup use case, probably the simplest thing is to reduce your sendbuffers for the backup sender process. Although allowing packets to drop instead of queue at your router/modem would really be best, often that's difficult to acheive.
- api 6y ago
  A major reason explicit congestion notification is not used is firewalls that block anything that isn't bog standard TCP or UDP. Some even ban odd combinations of flags. There are enough of these to make ECN useless.
  - toast0 6y ago
    A router that is willing to buffer 5 seconds worth of packets probably wasn't going to mark for congestion and drop either. Note also, Apple is using MP-TCP and ECN in iOS, and the world didn't stop. It might not work everywhere, and I don't praise Apple lightly, but there's a pretty clear path to using things like this. Send a syn with it enabled, wait a bit, and send one with it disabled. Keep track of networks where it doesn't work and stop trying it there. If you have leverage, yell at people to not do dumb things, otherwise, let them figure out why expensive things work better on their competetors' networks. You can't rely on being able to use these things, but you can use them for progressive enhancement.
kqr 6y ago
Big buffers that can be filled fast trick congestion control algorithms into thinking your wire is really fast. The point of the buffer is to be transparent to the transmitting ends, so they see the packets going out at lightning speed and assume it's because they're actually going that fast, and not just piled into a buffer that fast.
milesvp 6y ago
This is a fundamental problem on the internet. Ram is so cheap that every device has too big buffers that don’t allow for proper TCP back pressure. Eric Raymond gave a talk on this a few years ago. He was going to distribute a lot of small embedded devices around the world to measure this to try to address it. I’m curious what happened to that effort.
the8472 6y ago
> Why doesn't the congestion control part of TCP prevent buffer bloat[1]? It can. Enable BBR + fq/fq_codel on the box in question and CAKE on your router.
vasilvv 6y ago
Most of the common TCP congestion control algorithms (Reno, Cubic) are loss-based: they try to send more and more data until the link no longer can buffer all of the packets, and drops some of them. Naturally, this approach requires the buffer to fill up, causing the latency to spike. There are algorithms that try to use increased delay as a signal that the link is full. This approach has multiple problems, one of which is that delay can be really noisy on wireless networks; another is that if you have a loss-based and a delay-based connection sharing the same link, the delay-based one will get much less than a fair share of its bandwidth. People have been trying to make an algorithm that both coexists with Reno/CUBIC and does not induce bufferbloat for the last 25 years or so, and there's been some progress, but none of it has reached the point where it could be used as a default congestion control for all operating systems. The problem of "I have files to transfer in background, but I want my connection to yield to more important traffic" can actually solved using a special congestion control algorithm called LEDBAT [1]; it's used by Apple for things like software updates, and BitTorrent uses it too. Unfortunately, I think only Apple implements it in its TCP stack, so anyone who wants to do that would have to roll their own thing using UDP. [1] https://en.wikipedia.org/wiki/LEDBAT https://en.wikipedia.org/wiki/LEDBAT
richbhanover 6y ago
Your router is holding ("buffering") packets in the hopes that they can be sent soon. Your measurements indicate that the router is "bloated", holding about five seconds (5,000 ms) worth of data. This gives the sending TCP algorithm the wrong impression. It's waiting to hear about a dropped packet to indicate that there's congestion. When your router holds on to those packets (instead of dropping them), the TCP algorithm doesn't get any feedback, so it keeps shoveling data into the connection. This leads to the bad state you're seeing. And that's where the advice on "What can I do about Bufferbloat?" comes in. There's no benefit to having more than one packet buffered by the router. (Hanging on to more than one packet per connection only causes the latency/lag you're seeing.) There are routers that actually check the time the router has held packets. If packets have been queued for "too long", the router discards them immediately, giving the vital feedback to the sending TCP. Those routers use the technique known as SQM (Smart Queue Management) and the fq_codel, cake, PIE algorithms to keep the queues within the router short - typically less than 5 msec. To solve your problem, investigate getting a router that implements one of those SQM algorithms. They're listed on the "What can I do..." page. I am a fan of OpenWrt (use it at home), but have installed a bunch of IQrouters and Ubuquiti devices for friends.
- JoeAltmaier 6y ago
  TCP has its own buffers too. In a media application I had to use UDP because I could know how deep the local transmit buffering went. TCP just swallows the packets and maybe sends them, maybe buffers them. Adding to the problem.

jeffbee 6y ago

If there is a huge FIFO queue on your router, the rate-finding algorithms associated with TCP will be forced to conclude that the RTT to your site is enormous. They may try to open the window to compensate, but here's a fun fact: most operating system default settings are insufficient to utilize very high bandwidth-delay products. If you want to send a 1gbps flow across an 80ms distance on Linux, you'll need to change some parameters with sysctl before it will work. If your apparent RTT is 5000ms, the flow you can get will be reduced in proportion. In any case, the solution to bufferbloat is queue discipline, not congestion control.

jfkebwjsbx 6y ago
Up to what speeds/latencies are the default sysctl parameters alright? Is there any easy way to know whether you are getting hit by this? Nowadays many people is getting 1 Gbps links at home! What do you mean by queue discipline?
- jeffbee 6y ago
  You know, the worst part is that Linux sets the maximum receive window size at boot time depending on how much memory the system contains, ensuring that it's never quite right. On this machine, with 32GB of main memory, it defaults to 6291456 bytes.
  - jfkebwjsbx 6y ago
    I see, thanks! What about the queue discipline?
    - jeffbee 6y ago
      If you face a choice of what frame to put on the wire at any moment, the queue discipline makes that choice. The easiest policy is to simply send the oldest frame, but this is also the worst policy.
      - jfkebwjsbx 6y ago
        Ah, so the eviction/priority algorithm. Thanks!

toast0 6y ago

> Why doesn't the congestion control part of TCP prevent buffer bloat[1]? Is it because ISP throttling of the internet connection doesn't touch the TCP packets themselves? Most of the congestion control algorithms use packet loss as the only indicator of congestion. In a network with oversized buffers, congestion will result in delay and not packet loss. If the delay gets large enough, recieve and congestion windows will restrict the effective bandwidth, but the latency at that point is terrible. There are some alternate congestion control algorithms which do use latency as a signal, but they aren't universally available, and may not be a good fit for all flows. For your backup use case, probably the simplest thing is to reduce your sendbuffers for the backup sender process. Although allowing packets to drop instead of queue at your router/modem would really be best, often that's difficult to acheive.

api 6y ago
A major reason explicit congestion notification is not used is firewalls that block anything that isn't bog standard TCP or UDP. Some even ban odd combinations of flags. There are enough of these to make ECN useless.
- toast0 6y ago
  A router that is willing to buffer 5 seconds worth of packets probably wasn't going to mark for congestion and drop either. Note also, Apple is using MP-TCP and ECN in iOS, and the world didn't stop. It might not work everywhere, and I don't praise Apple lightly, but there's a pretty clear path to using things like this. Send a syn with it enabled, wait a bit, and send one with it disabled. Keep track of networks where it doesn't work and stop trying it there. If you have leverage, yell at people to not do dumb things, otherwise, let them figure out why expensive things work better on their competetors' networks. You can't rely on being able to use these things, but you can use them for progressive enhancement.

kqr 6y ago

Big buffers that can be filled fast trick congestion control algorithms into thinking your wire is really fast. The point of the buffer is to be transparent to the transmitting ends, so they see the packets going out at lightning speed and assume it's because they're actually going that fast, and not just piled into a buffer that fast.

milesvp 6y ago

This is a fundamental problem on the internet. Ram is so cheap that every device has too big buffers that don’t allow for proper TCP back pressure. Eric Raymond gave a talk on this a few years ago. He was going to distribute a lot of small embedded devices around the world to measure this to try to address it. I’m curious what happened to that effort.

the8472 6y ago

> Why doesn't the congestion control part of TCP prevent buffer bloat[1]? It can. Enable BBR + fq/fq_codel on the box in question and CAKE on your router.

vasilvv 6y ago

Most of the common TCP congestion control algorithms (Reno, Cubic) are loss-based: they try to send more and more data until the link no longer can buffer all of the packets, and drops some of them. Naturally, this approach requires the buffer to fill up, causing the latency to spike. There are algorithms that try to use increased delay as a signal that the link is full. This approach has multiple problems, one of which is that delay can be really noisy on wireless networks; another is that if you have a loss-based and a delay-based connection sharing the same link, the delay-based one will get much less than a fair share of its bandwidth. People have been trying to make an algorithm that both coexists with Reno/CUBIC and does not induce bufferbloat for the last 25 years or so, and there's been some progress, but none of it has reached the point where it could be used as a default congestion control for all operating systems. The problem of "I have files to transfer in background, but I want my connection to yield to more important traffic" can actually solved using a special congestion control algorithm called LEDBAT [1]; it's used by Apple for things like software updates, and BitTorrent uses it too. Unfortunately, I think only Apple implements it in its TCP stack, so anyone who wants to do that would have to roll their own thing using UDP. [1] https://en.wikipedia.org/wiki/LEDBAT https://en.wikipedia.org/wiki/LEDBAT

richbhanover 6y ago

Your router is holding ("buffering") packets in the hopes that they can be sent soon. Your measurements indicate that the router is "bloated", holding about five seconds (5,000 ms) worth of data. This gives the sending TCP algorithm the wrong impression. It's waiting to hear about a dropped packet to indicate that there's congestion. When your router holds on to those packets (instead of dropping them), the TCP algorithm doesn't get any feedback, so it keeps shoveling data into the connection. This leads to the bad state you're seeing. And that's where the advice on "What can I do about Bufferbloat?" comes in. There's no benefit to having more than one packet buffered by the router. (Hanging on to more than one packet per connection only causes the latency/lag you're seeing.) There are routers that actually check the time the router has held packets. If packets have been queued for "too long", the router discards them immediately, giving the vital feedback to the sending TCP. Those routers use the technique known as SQM (Smart Queue Management) and the fq_codel, cake, PIE algorithms to keep the queues within the router short - typically less than 5 msec. To solve your problem, investigate getting a router that implements one of those SQM algorithms. They're listed on the "What can I do..." page. I am a fan of OpenWrt (use it at home), but have installed a bunch of IQrouters and Ubuquiti devices for friends.

JoeAltmaier 6y ago
TCP has its own buffers too. In a media application I had to use UDP because I could know how deep the local transmit buffering went. TCP just swallows the packets and maybe sends them, maybe buffers them. Adding to the problem.