3 ms·

> Kernel TCP has been inappropriate for datacenter networking for decades I'm not sure your conclusion can be drawn from the cited paper alone, which is now 11

> Kernel TCP has been inappropriate for datacenter networking for decades I'm not sure your conclusion can be drawn from the cited paper alone, which is now 11 years old. The paper describes a phenomenon that occurs when switch hardware buffers overflow on high-throughput networks -- a problem that has since been resolved in many datacenters that use software-defined networking and more modern hardware.

jeffbee 7y ago
All of the people I've met in my career who believed their frames were not being dropped in practice were people who simply hadn't bothered measuring it. Perhaps my experience was warped by working at Google where they build the dumbest, cheapest switches they can imagine, where frame drops even for the highest priority flows are rampant, but after Google I've seen the same thing in multiple production networks so I'm pretty sure at this point that frame drops are endemic. Keep in mind that there are only two outcomes to networking: the frame is either forwarded or dropped. Whenever you encounter software-defined networks you need to ask which frames the network is dropping, because that's the only action an SDN can take. Incast is a stochastic phenomenon and the point of the paper is there is some probability, which is not zero, that too many frames will arrive at the same point in the network at the same time, forcing some to be dropped. This probability cannot be zero regardless of the buffer depth. Eric Dumazet has written a lot about how adding entropy in packet scheduling at the host level helps avoid frame drops. The fact is the problem cannot be solved by the network, it has to be solved by the hosts at the edge (and this is basic nethead-vs-bellhead philosophy going back 40 years).
- otterley 7y ago
  So now the question is, in the presence of dropped frames/packets, is this a problem that needs a wholesale L4 protocol replacement (a la SCTP or QUIC), or is this something that only requires some improvements to TCP or maybe tuning some knobs? I've been hearing arguments that "TCP sucks and is irredeemable" for decades, too, yet the world still turns and our devices that use it continue to work reasonably well at increasing bandwidth (2.4kbps to 40Gbps+) - at least as far as the public is concerned.
  - jeffbee 7y ago
    A number of the knobs are cast in stone in the RFCs and kernel maintainers have been resistant to exposing them. These slides mention several of the TCP parameters that are off by 2-4 orders of magnitude and some ways to remediate this. Note the last slide: Google has been patching around Linux TCP brain damage for 15+ years. https://www.ietf.org/proceedings/97/slides/slides-97-tcpm-tcp-options-for-low-latency-00.pdf https://www.ietf.org/proceedings/97/slides/slides-97-tcpm-tc...
- toast0 7y ago
  I no longer have access to stats, but at my last job, we did measurements of packetloss on the backend tcp connections between our hosts. The vast majority of the time, there was either no loss, or some very small amount of loss (maybe < 10 packets lost in a minute). Generally, when we saw losses, we were able to work with the datacenter to track down bad cables, or overloaded paths, or overloaded fabrics (which was usually in their 'legacy' datacenter). I expect the datacenter networking was overprovisioned (separate backend network, dual NICs on hosts, multiple uplinks on switches, etc), but it was darn reliable. Of course, it would have been nice if the people who owned the datacenter would have been monitoring all the stuff their customer (us) was able to find was broken. And, some sort of multistream connectivity would have helped avoid the paths with loss; ocassionaly the loss would be bad enough to impact service, and we'd need to force reconnects until the tcp stream went on a different path.
  - otterley 7y ago
    I can't speak for the parent, but I think what he's saying is that these are phenomena that start to become apparent at extremely high aggregate throughput rates with extremely high device density - i.e., Google scale. By saying it's "inappropriate for datacenters," I think he painted too broad a brush - obviously not all are the same, and not all run at Google scale. Depends on the datacenter and the workload.
    - jeffbee 7y ago
      Sort of. These things are relevant whenever you have congestion. If you have no congestion then the congestion-control features of your protocol aren't worth discussing. On a perfect network any protocol will do the job. All real network are imperfect, which is why the congestion control features of TCP and QUIC are worth discussing.

jeffbee 7y ago

All of the people I've met in my career who believed their frames were not being dropped in practice were people who simply hadn't bothered measuring it. Perhaps my experience was warped by working at Google where they build the dumbest, cheapest switches they can imagine, where frame drops even for the highest priority flows are rampant, but after Google I've seen the same thing in multiple production networks so I'm pretty sure at this point that frame drops are endemic. Keep in mind that there are only two outcomes to networking: the frame is either forwarded or dropped. Whenever you encounter software-defined networks you need to ask which frames the network is dropping, because that's the only action an SDN can take. Incast is a stochastic phenomenon and the point of the paper is there is some probability, which is not zero, that too many frames will arrive at the same point in the network at the same time, forcing some to be dropped. This probability cannot be zero regardless of the buffer depth. Eric Dumazet has written a lot about how adding entropy in packet scheduling at the host level helps avoid frame drops. The fact is the problem cannot be solved by the network, it has to be solved by the hosts at the edge (and this is basic nethead-vs-bellhead philosophy going back 40 years).

otterley 7y ago
So now the question is, in the presence of dropped frames/packets, is this a problem that needs a wholesale L4 protocol replacement (a la SCTP or QUIC), or is this something that only requires some improvements to TCP or maybe tuning some knobs? I've been hearing arguments that "TCP sucks and is irredeemable" for decades, too, yet the world still turns and our devices that use it continue to work reasonably well at increasing bandwidth (2.4kbps to 40Gbps+) - at least as far as the public is concerned.
- jeffbee 7y ago
  A number of the knobs are cast in stone in the RFCs and kernel maintainers have been resistant to exposing them. These slides mention several of the TCP parameters that are off by 2-4 orders of magnitude and some ways to remediate this. Note the last slide: Google has been patching around Linux TCP brain damage for 15+ years. https://www.ietf.org/proceedings/97/slides/slides-97-tcpm-tcp-options-for-low-latency-00.pdf https://www.ietf.org/proceedings/97/slides/slides-97-tcpm-tc...
toast0 7y ago
I no longer have access to stats, but at my last job, we did measurements of packetloss on the backend tcp connections between our hosts. The vast majority of the time, there was either no loss, or some very small amount of loss (maybe < 10 packets lost in a minute). Generally, when we saw losses, we were able to work with the datacenter to track down bad cables, or overloaded paths, or overloaded fabrics (which was usually in their 'legacy' datacenter). I expect the datacenter networking was overprovisioned (separate backend network, dual NICs on hosts, multiple uplinks on switches, etc), but it was darn reliable. Of course, it would have been nice if the people who owned the datacenter would have been monitoring all the stuff their customer (us) was able to find was broken. And, some sort of multistream connectivity would have helped avoid the paths with loss; ocassionaly the loss would be bad enough to impact service, and we'd need to force reconnects until the tcp stream went on a different path.
- otterley 7y ago
  I can't speak for the parent, but I think what he's saying is that these are phenomena that start to become apparent at extremely high aggregate throughput rates with extremely high device density - i.e., Google scale. By saying it's "inappropriate for datacenters," I think he painted too broad a brush - obviously not all are the same, and not all run at Google scale. Depends on the datacenter and the workload.
  - jeffbee 7y ago
    Sort of. These things are relevant whenever you have congestion. If you have no congestion then the congestion-control features of your protocol aren't worth discussing. On a perfect network any protocol will do the job. All real network are imperfect, which is why the congestion control features of TCP and QUIC are worth discussing.