4 ms·

If I understand correctly the whole point of QUIC is to not be in hardware level. That's why QUIC exists, because TCP can't be changed because it is baked in at

If I understand correctly the whole point of QUIC is to not be in hardware level. That's why QUIC exists, because TCP can't be changed because it is baked in at such low, fundamental levels.

oneplane 6y ago
No, the point of QUIC is to shave a few % of traffic off of the loads the likes of FAANG have, saving millions, while having a marginal benefit to pretty much everyone else. QUIC can be 'a bit better' in 'some scenarios', but most of the time it is totally irrelevant, except for the fact that it requires TLS 1.3 which is a good requirement overall.
- jeffbee 6y ago
  QUIC in the datacenter is hugely superior to Linux kernel TCP. Kernel TCP has been inappropriate for datacenter networking for decades and this has been a known issue mentioned many times in the literature (start with [1] if this is new to you). Datacenter operators have been forced to either patch the kernel or bypass it to get decent TCP performance. QUIC is a rational response to this situation. Instead of waiting a quarter century or more for Linux to get its TCP stack in order, switch to UDP and innovate in user space. 1: https://www.cs.cmu.edu/~dga/papers/incast-sigcomm2009.pdf https://www.cs.cmu.edu/~dga/papers/incast-sigcomm2009.pdf
  - derefr 6y ago
    Are you talking about TCP over datacenter hardware Ethernet, or TCP over a software-defined network (potentially terminated on the same host)?
    - jeffbee 6y ago
      Not sure these are opposing topics. My understanding of SDN is simply that a program may decide to drop certain frames, and this can run on top of ordinary Ethernet.
      - oneplane 6y ago
        In theory, if SDN is on top of it, or below it (or both) it would be better to use QUIC than IP-in-TCP or EoIP (or GRE, PPP etc.) because the connections within the SDN would already assume the medium might be lossy and having stacked retransmission will really blow performance (hence VXLAN and the likes are used more than the classical encapsulation AFAIK).
  - oneplane 6y ago
    So again: it's for when you do large scale data center communication. Perhaps not just FAANG level, but say, having a few private cages before it starts delivering much of an improvement?
    - jeffbee 6y ago
      I don't think you need large scale to benefit. Any time you are latency-sensitive, have several machines that are microseconds apart, and there is a chance of dropping frames, you will suffer from Linux TCP bogosity. It can't comprehend microsecond RTTs. If you drop a frame it will be a minimum of 15ms before Linux notices (with TLP, 200ms otherwise). That's orders of magnitude worse than it needs to be.
      - oneplane 6y ago
        So again, large scale or real time, which is not something most setups do. Say you process video and audio streams, that makes sense, or if retransmissions are costly at scale. But does it matter for anything else? I'm trying to find the actual general benefit, but so far I'm not finding it. There is a technical benefit, but not on the scale of say h2 multiplexing or binary vs. plain-text protocols.
        jeffbee 6y ago
        Sure, if you're not trying to serve with low latency then this doesn't matter to you. But if you're happy with hundreds of milliseconds of tail latency you also don't need SSDs, 64-core CPUs, 100g ethernet, or anything else newer than ten years old. It's only interesting to evaluate the software on the frontier of performance.
  - otterley 6y ago
    > Kernel TCP has been inappropriate for datacenter networking for decades I'm not sure your conclusion can be drawn from the cited paper alone, which is now 11 years old. The paper describes a phenomenon that occurs when switch hardware buffers overflow on high-throughput networks -- a problem that has since been resolved in many datacenters that use software-defined networking and more modern hardware.
    - jeffbee 6y ago
      All of the people I've met in my career who believed their frames were not being dropped in practice were people who simply hadn't bothered measuring it. Perhaps my experience was warped by working at Google where they build the dumbest, cheapest switches they can imagine, where frame drops even for the highest priority flows are rampant, but after Google I've seen the same thing in multiple production networks so I'm pretty sure at this point that frame drops are endemic. Keep in mind that there are only two outcomes to networking: the frame is either forwarded or dropped. Whenever you encounter software-defined networks you need to ask which frames the network is dropping, because that's the only action an SDN can take. Incast is a stochastic phenomenon and the point of the paper is there is some probability, which is not zero, that too many frames will arrive at the same point in the network at the same time, forcing some to be dropped. This probability cannot be zero regardless of the buffer depth. Eric Dumazet has written a lot about how adding entropy in packet scheduling at the host level helps avoid frame drops. The fact is the problem cannot be solved by the network, it has to be solved by the hosts at the edge (and this is basic nethead-vs-bellhead philosophy going back 40 years).
      - otterley 6y ago
        So now the question is, in the presence of dropped frames/packets, is this a problem that needs a wholesale L4 protocol replacement (a la SCTP or QUIC), or is this something that only requires some improvements to TCP or maybe tuning some knobs? I've been hearing arguments that "TCP sucks and is irredeemable" for decades, too, yet the world still turns and our devices that use it continue to work reasonably well at increasing bandwidth (2.4kbps to 40Gbps+) - at least as far as the public is concerned.
thwarted 6y ago
It's not that TCP can't be changed, it's that it's the only option that works. TCP is baked into firmware on edge routers and firewalls and wifi devices that are beyond the period of support and the companies that created them aren't interested in supporting them or have gone out of business. There are other protocols in layer 4 that could have been used that are better on some workloads than TCP, but TCP and UDP were the only things configured/supported; many firewall default configurations filter out things they don't recognize so anything new is a non-starter. And because of the perception of responsibility when it comes to networks, if I support a new protocol and you can't connect to it because there's older devices between you and me, it becomes my problem.
- JoeAltmaier 6y ago
  So much this. Its ossified so far that if your DHCP packet isn't padded in just the right way, some firewalls will block it. Somebody mis-read the RFC and put in some rule, and now many Enterprise firewalls do this. Sigh. Anyway, anothing not 'normal' is being filtered out somewhere. We even (at one startup) considered tunneling RTP through HTTP! My god.
- PudgePacket 6y ago
  QUIC is built on UDP, not sure if you were aware based on your 3rd paragraph? > One concern about the move from TCP to UDP is that TCP is widely adopted and many of the "middle-boxes" in the internet infrastructure are tuned for TCP and rate-limit or even block UDP. Google carried out a number of exploratory experiments to characterize this and found that only a small number of connections were blocked in this manner.[3] This led to the use of a rapid fallback-to-TCP system; Chromium's network stack opens both a QUIC and traditional TCP connection at the same time, which allows it to fallback with zero latency.[17] https://en.wikipedia.org/wiki/QUIC#Characteristics https://en.wikipedia.org/wiki/QUIC#Characteristics
jayd16 6y ago
The whole point is to improve network traffic. The only avenue to do that is in user space until new strategies make their way down to the OS and hardware. If QUIC is good enough eventually someone will make hardware for it.

If I understand correctly the whole point of QUIC is to not be in hardware level. That's why QUIC exists, because TCP can't be changed because it is baked in at such low, fundamental levels.

oneplane 6y ago

No, the point of QUIC is to shave a few % of traffic off of the loads the likes of FAANG have, saving millions, while having a marginal benefit to pretty much everyone else. QUIC can be 'a bit better' in 'some scenarios', but most of the time it is totally irrelevant, except for the fact that it requires TLS 1.3 which is a good requirement overall.

jeffbee 6y ago
QUIC in the datacenter is hugely superior to Linux kernel TCP. Kernel TCP has been inappropriate for datacenter networking for decades and this has been a known issue mentioned many times in the literature (start with [1] if this is new to you). Datacenter operators have been forced to either patch the kernel or bypass it to get decent TCP performance. QUIC is a rational response to this situation. Instead of waiting a quarter century or more for Linux to get its TCP stack in order, switch to UDP and innovate in user space. 1: https://www.cs.cmu.edu/~dga/papers/incast-sigcomm2009.pdf https://www.cs.cmu.edu/~dga/papers/incast-sigcomm2009.pdf
- derefr 6y ago
  Are you talking about TCP over datacenter hardware Ethernet, or TCP over a software-defined network (potentially terminated on the same host)?
  - jeffbee 6y ago
    Not sure these are opposing topics. My understanding of SDN is simply that a program may decide to drop certain frames, and this can run on top of ordinary Ethernet.
    - oneplane 6y ago
      In theory, if SDN is on top of it, or below it (or both) it would be better to use QUIC than IP-in-TCP or EoIP (or GRE, PPP etc.) because the connections within the SDN would already assume the medium might be lossy and having stacked retransmission will really blow performance (hence VXLAN and the likes are used more than the classical encapsulation AFAIK).
- oneplane 6y ago
  So again: it's for when you do large scale data center communication. Perhaps not just FAANG level, but say, having a few private cages before it starts delivering much of an improvement?
  - jeffbee 6y ago
    I don't think you need large scale to benefit. Any time you are latency-sensitive, have several machines that are microseconds apart, and there is a chance of dropping frames, you will suffer from Linux TCP bogosity. It can't comprehend microsecond RTTs. If you drop a frame it will be a minimum of 15ms before Linux notices (with TLP, 200ms otherwise). That's orders of magnitude worse than it needs to be.
    - oneplane 6y ago
      So again, large scale or real time, which is not something most setups do. Say you process video and audio streams, that makes sense, or if retransmissions are costly at scale. But does it matter for anything else? I'm trying to find the actual general benefit, but so far I'm not finding it. There is a technical benefit, but not on the scale of say h2 multiplexing or binary vs. plain-text protocols.
      - jeffbee 6y ago
        Sure, if you're not trying to serve with low latency then this doesn't matter to you. But if you're happy with hundreds of milliseconds of tail latency you also don't need SSDs, 64-core CPUs, 100g ethernet, or anything else newer than ten years old. It's only interesting to evaluate the software on the frontier of performance.
- otterley 6y ago
  > Kernel TCP has been inappropriate for datacenter networking for decades I'm not sure your conclusion can be drawn from the cited paper alone, which is now 11 years old. The paper describes a phenomenon that occurs when switch hardware buffers overflow on high-throughput networks -- a problem that has since been resolved in many datacenters that use software-defined networking and more modern hardware.
  - jeffbee 6y ago
    All of the people I've met in my career who believed their frames were not being dropped in practice were people who simply hadn't bothered measuring it. Perhaps my experience was warped by working at Google where they build the dumbest, cheapest switches they can imagine, where frame drops even for the highest priority flows are rampant, but after Google I've seen the same thing in multiple production networks so I'm pretty sure at this point that frame drops are endemic. Keep in mind that there are only two outcomes to networking: the frame is either forwarded or dropped. Whenever you encounter software-defined networks you need to ask which frames the network is dropping, because that's the only action an SDN can take. Incast is a stochastic phenomenon and the point of the paper is there is some probability, which is not zero, that too many frames will arrive at the same point in the network at the same time, forcing some to be dropped. This probability cannot be zero regardless of the buffer depth. Eric Dumazet has written a lot about how adding entropy in packet scheduling at the host level helps avoid frame drops. The fact is the problem cannot be solved by the network, it has to be solved by the hosts at the edge (and this is basic nethead-vs-bellhead philosophy going back 40 years).
    - otterley 6y ago
      So now the question is, in the presence of dropped frames/packets, is this a problem that needs a wholesale L4 protocol replacement (a la SCTP or QUIC), or is this something that only requires some improvements to TCP or maybe tuning some knobs? I've been hearing arguments that "TCP sucks and is irredeemable" for decades, too, yet the world still turns and our devices that use it continue to work reasonably well at increasing bandwidth (2.4kbps to 40Gbps+) - at least as far as the public is concerned.

thwarted 6y ago

It's not that TCP can't be changed, it's that it's the only option that works. TCP is baked into firmware on edge routers and firewalls and wifi devices that are beyond the period of support and the companies that created them aren't interested in supporting them or have gone out of business. There are other protocols in layer 4 that could have been used that are better on some workloads than TCP, but TCP and UDP were the only things configured/supported; many firewall default configurations filter out things they don't recognize so anything new is a non-starter. And because of the perception of responsibility when it comes to networks, if I support a new protocol and you can't connect to it because there's older devices between you and me, it becomes my problem.

JoeAltmaier 6y ago
So much this. Its ossified so far that if your DHCP packet isn't padded in just the right way, some firewalls will block it. Somebody mis-read the RFC and put in some rule, and now many Enterprise firewalls do this. Sigh. Anyway, anothing not 'normal' is being filtered out somewhere. We even (at one startup) considered tunneling RTP through HTTP! My god.
PudgePacket 6y ago
QUIC is built on UDP, not sure if you were aware based on your 3rd paragraph? > One concern about the move from TCP to UDP is that TCP is widely adopted and many of the "middle-boxes" in the internet infrastructure are tuned for TCP and rate-limit or even block UDP. Google carried out a number of exploratory experiments to characterize this and found that only a small number of connections were blocked in this manner.[3] This led to the use of a rapid fallback-to-TCP system; Chromium's network stack opens both a QUIC and traditional TCP connection at the same time, which allows it to fallback with zero latency.[17] https://en.wikipedia.org/wiki/QUIC#Characteristics https://en.wikipedia.org/wiki/QUIC#Characteristics

jayd16 6y ago

The whole point is to improve network traffic. The only avenue to do that is in user space until new strategies make their way down to the OS and hardware. If QUIC is good enough eventually someone will make hardware for it.