4 ms·

No, the point of QUIC is to shave a few % of traffic off of the loads the likes of FAANG have, saving millions, while having a marginal benefit to pretty much e

No, the point of QUIC is to shave a few % of traffic off of the loads the likes of FAANG have, saving millions, while having a marginal benefit to pretty much everyone else. QUIC can be 'a bit better' in 'some scenarios', but most of the time it is totally irrelevant, except for the fact that it requires TLS 1.3 which is a good requirement overall.

jeffbee 7y ago
QUIC in the datacenter is hugely superior to Linux kernel TCP. Kernel TCP has been inappropriate for datacenter networking for decades and this has been a known issue mentioned many times in the literature (start with [1] if this is new to you). Datacenter operators have been forced to either patch the kernel or bypass it to get decent TCP performance. QUIC is a rational response to this situation. Instead of waiting a quarter century or more for Linux to get its TCP stack in order, switch to UDP and innovate in user space. 1: https://www.cs.cmu.edu/~dga/papers/incast-sigcomm2009.pdf https://www.cs.cmu.edu/~dga/papers/incast-sigcomm2009.pdf
- derefr 7y ago
  Are you talking about TCP over datacenter hardware Ethernet, or TCP over a software-defined network (potentially terminated on the same host)?
  - jeffbee 7y ago
    Not sure these are opposing topics. My understanding of SDN is simply that a program may decide to drop certain frames, and this can run on top of ordinary Ethernet.
    - oneplane 7y ago
      In theory, if SDN is on top of it, or below it (or both) it would be better to use QUIC than IP-in-TCP or EoIP (or GRE, PPP etc.) because the connections within the SDN would already assume the medium might be lossy and having stacked retransmission will really blow performance (hence VXLAN and the likes are used more than the classical encapsulation AFAIK).
- oneplane 7y ago
  So again: it's for when you do large scale data center communication. Perhaps not just FAANG level, but say, having a few private cages before it starts delivering much of an improvement?
  - jeffbee 7y ago
    I don't think you need large scale to benefit. Any time you are latency-sensitive, have several machines that are microseconds apart, and there is a chance of dropping frames, you will suffer from Linux TCP bogosity. It can't comprehend microsecond RTTs. If you drop a frame it will be a minimum of 15ms before Linux notices (with TLP, 200ms otherwise). That's orders of magnitude worse than it needs to be.
    - oneplane 7y ago
      So again, large scale or real time, which is not something most setups do. Say you process video and audio streams, that makes sense, or if retransmissions are costly at scale. But does it matter for anything else? I'm trying to find the actual general benefit, but so far I'm not finding it. There is a technical benefit, but not on the scale of say h2 multiplexing or binary vs. plain-text protocols.
      - jeffbee 7y ago
        Sure, if you're not trying to serve with low latency then this doesn't matter to you. But if you're happy with hundreds of milliseconds of tail latency you also don't need SSDs, 64-core CPUs, 100g ethernet, or anything else newer than ten years old. It's only interesting to evaluate the software on the frontier of performance.
- otterley 7y ago
  > Kernel TCP has been inappropriate for datacenter networking for decades I'm not sure your conclusion can be drawn from the cited paper alone, which is now 11 years old. The paper describes a phenomenon that occurs when switch hardware buffers overflow on high-throughput networks -- a problem that has since been resolved in many datacenters that use software-defined networking and more modern hardware.
  - jeffbee 7y ago
    All of the people I've met in my career who believed their frames were not being dropped in practice were people who simply hadn't bothered measuring it. Perhaps my experience was warped by working at Google where they build the dumbest, cheapest switches they can imagine, where frame drops even for the highest priority flows are rampant, but after Google I've seen the same thing in multiple production networks so I'm pretty sure at this point that frame drops are endemic. Keep in mind that there are only two outcomes to networking: the frame is either forwarded or dropped. Whenever you encounter software-defined networks you need to ask which frames the network is dropping, because that's the only action an SDN can take. Incast is a stochastic phenomenon and the point of the paper is there is some probability, which is not zero, that too many frames will arrive at the same point in the network at the same time, forcing some to be dropped. This probability cannot be zero regardless of the buffer depth. Eric Dumazet has written a lot about how adding entropy in packet scheduling at the host level helps avoid frame drops. The fact is the problem cannot be solved by the network, it has to be solved by the hosts at the edge (and this is basic nethead-vs-bellhead philosophy going back 40 years).
    - otterley 7y ago
      So now the question is, in the presence of dropped frames/packets, is this a problem that needs a wholesale L4 protocol replacement (a la SCTP or QUIC), or is this something that only requires some improvements to TCP or maybe tuning some knobs? I've been hearing arguments that "TCP sucks and is irredeemable" for decades, too, yet the world still turns and our devices that use it continue to work reasonably well at increasing bandwidth (2.4kbps to 40Gbps+) - at least as far as the public is concerned.

jeffbee 7y ago

QUIC in the datacenter is hugely superior to Linux kernel TCP. Kernel TCP has been inappropriate for datacenter networking for decades and this has been a known issue mentioned many times in the literature (start with [1] if this is new to you). Datacenter operators have been forced to either patch the kernel or bypass it to get decent TCP performance. QUIC is a rational response to this situation. Instead of waiting a quarter century or more for Linux to get its TCP stack in order, switch to UDP and innovate in user space. 1: https://www.cs.cmu.edu/~dga/papers/incast-sigcomm2009.pdf https://www.cs.cmu.edu/~dga/papers/incast-sigcomm2009.pdf

derefr 7y ago
Are you talking about TCP over datacenter hardware Ethernet, or TCP over a software-defined network (potentially terminated on the same host)?
- jeffbee 7y ago
  Not sure these are opposing topics. My understanding of SDN is simply that a program may decide to drop certain frames, and this can run on top of ordinary Ethernet.
  - oneplane 7y ago
    In theory, if SDN is on top of it, or below it (or both) it would be better to use QUIC than IP-in-TCP or EoIP (or GRE, PPP etc.) because the connections within the SDN would already assume the medium might be lossy and having stacked retransmission will really blow performance (hence VXLAN and the likes are used more than the classical encapsulation AFAIK).
oneplane 7y ago
So again: it's for when you do large scale data center communication. Perhaps not just FAANG level, but say, having a few private cages before it starts delivering much of an improvement?
- jeffbee 7y ago
  I don't think you need large scale to benefit. Any time you are latency-sensitive, have several machines that are microseconds apart, and there is a chance of dropping frames, you will suffer from Linux TCP bogosity. It can't comprehend microsecond RTTs. If you drop a frame it will be a minimum of 15ms before Linux notices (with TLP, 200ms otherwise). That's orders of magnitude worse than it needs to be.
  - oneplane 7y ago
    So again, large scale or real time, which is not something most setups do. Say you process video and audio streams, that makes sense, or if retransmissions are costly at scale. But does it matter for anything else? I'm trying to find the actual general benefit, but so far I'm not finding it. There is a technical benefit, but not on the scale of say h2 multiplexing or binary vs. plain-text protocols.
    - jeffbee 7y ago
      Sure, if you're not trying to serve with low latency then this doesn't matter to you. But if you're happy with hundreds of milliseconds of tail latency you also don't need SSDs, 64-core CPUs, 100g ethernet, or anything else newer than ten years old. It's only interesting to evaluate the software on the frontier of performance.
otterley 7y ago
> Kernel TCP has been inappropriate for datacenter networking for decades I'm not sure your conclusion can be drawn from the cited paper alone, which is now 11 years old. The paper describes a phenomenon that occurs when switch hardware buffers overflow on high-throughput networks -- a problem that has since been resolved in many datacenters that use software-defined networking and more modern hardware.
- jeffbee 7y ago
  All of the people I've met in my career who believed their frames were not being dropped in practice were people who simply hadn't bothered measuring it. Perhaps my experience was warped by working at Google where they build the dumbest, cheapest switches they can imagine, where frame drops even for the highest priority flows are rampant, but after Google I've seen the same thing in multiple production networks so I'm pretty sure at this point that frame drops are endemic. Keep in mind that there are only two outcomes to networking: the frame is either forwarded or dropped. Whenever you encounter software-defined networks you need to ask which frames the network is dropping, because that's the only action an SDN can take. Incast is a stochastic phenomenon and the point of the paper is there is some probability, which is not zero, that too many frames will arrive at the same point in the network at the same time, forcing some to be dropped. This probability cannot be zero regardless of the buffer depth. Eric Dumazet has written a lot about how adding entropy in packet scheduling at the host level helps avoid frame drops. The fact is the problem cannot be solved by the network, it has to be solved by the hosts at the edge (and this is basic nethead-vs-bellhead philosophy going back 40 years).
  - otterley 7y ago
    So now the question is, in the presence of dropped frames/packets, is this a problem that needs a wholesale L4 protocol replacement (a la SCTP or QUIC), or is this something that only requires some improvements to TCP or maybe tuning some knobs? I've been hearing arguments that "TCP sucks and is irredeemable" for decades, too, yet the world still turns and our devices that use it continue to work reasonably well at increasing bandwidth (2.4kbps to 40Gbps+) - at least as far as the public is concerned.