8 ms·

Ok, so there have been literally decades spent in managing and optimising TCP to behave in various network conditions, from modem lines to multi-gigabit fiber,

by ivoras 8y ago

Ok, so there have been literally decades spent in managing and optimising TCP to behave in various network conditions, from modem lines to multi-gigabit fiber, in all kinds of congestion conditions, lossy-ness, and other hairy edge cases of real physical networks. So, what's the point of ditching all that and reinventing HTTP over a thin wrapper (UDP) over (lossy) IP? Has there been a huge theoretical breakthrough which simply can't be applied to TCP?

dagenix 8y ago
One issue is head of line blocking in TCP when multiplexing requests over a single connection. That problem can't be solved without something like QUIC.
- billyhoffman 8y ago
  To be clear, this became a bigger issue because HTTP/2 moved to “1 TCP connection per origin with all requests multiplexed on it”. There has always been head of line blocking in tcp, however H2 compounded it since it now impacts all requests to the origin instead of just a few requests in one of four or one of six tcp connections to origin. H2 effectively moved the problem to TCP (really just made underlyubg problem more impactful). Quic is an attempt to solve the TCP problem
  - ivoras 8y ago
    But moving the solution away from TCP into something new (in this case userland UDP-based QUIC) would make sense if (and probably only if) the overhead of QUIC connections + request bookkeeping is lower in memory and CPU usage (ideally both) than just establishing multiple TCP connections. Right?
    - jayd16 8y ago
      My understanding is multiplexing over an existing connection also provides time savings as new data does not need the full handshake and ramp up a completely new connection would.
    - dagenix 8y ago
      TCP flow control is generally applied per-connection. So, a web browser that creates 6 connections to load a website is going to get a greater share of available bandwidth than an application that just uses one. And what will get greater bandwidth still is if the website owner can trick the browser into opening 12 or 18 or more connections by sharding their site over multiple domains. "Just open more connections" has issues. QUIC making it less necessary to do that has value even if it didn't save memory or CPU usage.
      - ivoras 8y ago
        The need for maintaining flow control still exists, both in QUIC and TCP, it doesn't just vanish. Also, big(ish) routers are aware of TCP and can adapt as the load changes. They literally won't know what to do with a huge increase in UDP traffic. I'd make an educated guess that mass deployment of QUIC will prove to be pretty awful until it's hacked to look more or less exactly like TCP with multiple connections, just with more overhead.
        anticensor 8y ago
        > They literally won't know what to do with a huge increase in UDP traffic. Randomly drop UDP packets.
        dagenix 8y ago
        Afaik, the bigger the router, the more likely it is to just route ip packets. I'll make the educated guess that deployment of QUIC will be just fine as routers already know how to handle UDP and IP just fine.
  - josteink 8y ago
    So HTTP/1.1 worked fine for around 2+ decades. Google(tm) HTTP/2.0 was unasked for, tried to reinvent TCP, found buggy and fundamentally broken after less than a year. So now Google is pushing HTTP/3 to solve the bugs they made with HTTP/2, by reinventing even more of the IP-stack. What could possibly go wrong? Can someone please tell Google that protocols are not Chrome-releases? They need to work and stay working and supported for decades. You can’t ship a new revision every 6 month. Edit: Correct HTTP/1.1 longdetivity.
    - manigandham 8y ago
      3 decades? 1.1 was released in 1997. Networks change as the world evolves and supports more people and applications, so new protocols are expected. HTTP/2 was great progress at the application layer, and QUIC is now about improving the transport layer. Can you clearly list the bugs you claim HTTP/2 has?
      - josteink 8y ago
        > 3 decades? 1.1 was released in 1997. Fair enough. Corrected. > Can you clearly list the bugs you claim HTTP/2 has? They seemed fairly well outlined in the post I was replying to, no?
        profmonocle 8y ago
        The only issue raised in the post you replied to was that TCP head-of-line blocking can be a bigger issue in HTTP/2. Claiming this makes the protocol "fundamentally broken" is a bit of a stretch. Plus, this wasn't "found after less than a year". It was a known drawback from the very beginning, when the protocol was still SPDY, and deemed an acceptable tradeoff.
        manigandham 8y ago
        That is not a bug with HTTP/2. It is designed to be more efficient by using a single TCP connection, thereby magnifying any efficiency issues that TCP had all along. These issues are not only about HTTP but include the massively increased amount of devices, network types, security features, and general throughput over the years. Perhaps a new version of TCP could've been made, and that's a valid concern, but it's a slow moving standard that takes decades to alter. QUIC evolved faster on UDP so that's what we have. What strategy do you suggest for improving TCP faster than the current process?
  - deathanatos 8y ago
    > There has always been head of line blocking in tcp, however H2 compounded it since it now impacts all requests to the origin instead of just a few requests in one of four or one of six tcp connections to origin. I feel like you're conflating two very different things into "head of line blockings". Head of line blocking in HTTP/1.1 was that HTTP/1.1 cannot transmit responses out of order, so a slow response "blocks" any responses behind it. HTTP/2 solves this by being multiplexed (effectively, it tags the requests & responses s.t. you can match out of order responses back to the appropriate request). This isn't necessarily that the network can't handle it; a request that might be easy for the server to handle (e.g., a static asset) can get caught behind one that is not (e.g., one that entails a slow DB query). The other "head of line blocking" that I think you're getting at is just network loss or saturation. If the network can't deliver packets, I don't see how having 6 connections is going to help: you're going to have to wait for six different state machines to determine that the link is saturated, back off, etc. Regardless, my understanding is that QUIC / HTTP/3 is still a multiplexed protocol; I would think it would behave very similarly. Nonetheless, Google's announcement of it backs up your claim[1]: > Like HTTP/2, QUIC multiplexes multiple streams into one connection, so that a connection can serve several HTTP requests simultaneously. But HTTP/2 uses TCP as its transport, so all of its streams can be blocked when a single TCP packet is lost—a problem called head-of-line blocking. QUIC is different: Loss of a UDP packet within a QUIC connection only affects the streams contained within that packet. In other words, QUIC won’t let a problem with one request slow the others down, even on an unreliable connection. But what here makes QUIC think that those other requests have any hope where the first didn't? To me, this approach seems like you're just spamming the link with packets and hoping for the best; those packets could just end up dropped like the other ones were. What makes QUIC think this approach will have any benefit? (It could, perhaps, if the link is just going to drop one packet and we can get right back to sending. I'm not sure that still conveys any benefit over TCP. Not saying QUIC is wrong, just that I don't see a decent reason to buy into it.) (This whole thing makes me think there are two problems here: there's the actual network connection between two endpoints, and what its state is, how lossy it is, its PMTU, how much data we should allow in-flight between those two endpoints, etc. But then there is also logical streams, such as HTTP request/response pairs. It almost seems like to me any data about the link, such as PMTU/lossiness/window, should be shared between all TCP connections, and the individual logical streams should be otherwise cheap/easy to make.) [1]: https://cloudplatform.googleblog.com/2018/06/Introducing-QUIC-support-for-HTTPS-load-balancing.html https://cloudplatform.googleblog.com/2018/06/Introducing-QUI...
- deleted 8y ago
  [deleted]
cpeterso 8y ago
Fixing bugs and experimenting with new protocols will be easier in user space than waiting for operating systems and routers to get fixed and their users to deploy them. Look at all the problems TLS 1.3 has faced with busted routers and enterprise MITM boxes.
tomohawk 8y ago
TCP is a general stream oriented protocol that must cover a variety of use cases and situations. That's why there are so many OS knobs to tweak it if you have a special use. Since HTTP is the use case here, it is possible to create a protocol that is optimized for it, whereas TCP cannot be optimized specifically for HTTP. Source: I've implemented a reliable, stream oriented protocol over UDP which has been in use for many years.
devwastaken 8y ago
Udp is preferred for content that can be lossy. The overhead of tcp for streamable content, especially over WiFi, is too high. I ran into this issue by trying to have a simple video server through nginx on a DigitalOcean droplet. Wouldn't work well over WiFi at range, but YouTube would work easy on chrome.
- colde 8y ago
  I think that has a lot more to do with YouTube having an a very optimized video player, and using ABR streams. That works fine over WiFi in general. If you are doing MP4 pseudo-streaming, thats really a completely different beast. You don't really want to drop data for streaming video either, sure, in some cases it might not make a huge difference, but if you drop i-frames, you are going to have quality issues.
  - devwastaken 8y ago
    Problem is on wifi that the whole syn/ack pattern gets lost and has to retry. I could have probably optimized this by having a range, but afaik if any in the range fail it has to still re-send that packet and get an ack. I'm not sure if HTTP/3 will have a way to do selective re-sending, but that is how very versatile lossy video streams do it. I-frames can get re-sent while normal frames can be lost.
ploxiln 8y ago
UDP is the new IP. You can't practically have your users install an upgraded OS kernel with TCP rate control improvements, you can't fix queuing in their NAT gateway, and you can't get SCTP working through their ISP. So, you do it all in UDP. (Your servers can handle the streams in kernel or even accelerated by the NIC chip, or in a userspace IP stack e.g. DPDK.)
- fulafel 8y ago
  OS modifications are not strictly required for new L3 protocols, though some new APIs would be nice to handle it more ergonomically. You can receive/send any IP packets with standard socket APIs. ref: https://docs.microsoft.com/en-us/windows/desktop/WinSock/tcp-ip-raw-sockets-2 https://docs.microsoft.com/en-us/windows/desktop/WinSock/tcp... https://linux.die.net/man/7/raw https://linux.die.net/man/7/raw
toast0 8y ago
TCP has some real issues, MTU discovery is either slow or not enabled or not very smart on a lot of clients. Google, for some reason, doesn't want to fix tcp in Android, instead they went with implementing tcp inside of tcp (http/2), and since that ran into the obvious problems, they've realized that tcp over UDP is a better choice. As far as I can tell, they still haven't enabled TCP MTU discovery in Android though. Or build apis to allow applications to influence packetization and congestion control -- which is what is probably needed. The java apis for TCP are fairly inflexible too, to set most socket options you have to call into c code, and hunt around to get the FD.
- est 8y ago
  h2 is not tcp on tcp. h2 failed because it's mux everything into 1 single tcp connection, and it only takes 1 failed tcp to block everything in line. h2 really need to do demux. But h2 actually can technically do M:N connections, but browsers do only 1:N. Also h2 is a failed promise. Most sites deliver web assets and page and js in different host, also ws/wss in a different host. Thus renders h2 advantages complete useless.
  - bcherny 8y ago
    > “h2 failed” Just curious, what do you mean? Quickly Googling this, it looks like 1/3 websites use HTTP2 today, which is darn good for a relatively new protocol. https://w3techs.com/technologies/details/ce-http2/all/all https://w3techs.com/technologies/details/ce-http2/all/all
    - Vektorweg 8y ago
      Can't always correlate usage share with performance, especially on things that are extremely hyped.
    - provolone 8y ago
      Googlebot never adopted HTTP2, only SPDY
    - est 8y ago
nly 8y ago
Biggest problem with multiplexing over TCP is mixing latency sensitive flows (like getting HTML to render) with high throughput flows (like downloading a YouTube video). You can try and diddle with chunk size and message prioritization in your application to reduce HOL blocking of low latency messages behind chunks of high throughput data, but your OS will buffer everything in one big queue anyway (because TCP is a single stream and flow control applies to that entire stream as a whole). How do you fix that? Well you can reduce socket buffer sizes on your TCP connection, and that will fix latency... but small buffer sizes mean your total throughput is limited by round trip time (because of the way the TCP receive windows and positive acknowledgment work). If you're working over a LAN, things will be fine, but over the Internet you're hosed. Of course, the solution really is to have separate long-lived TCP connections for different kinds of flows, so you can set buffer sizes appropriately... but that's impossible for your browser (which is responsible for opening connections). It has no way to know whether a GET request triggered by an <img> tag is going to yield a 100 byte response or a 100MB response.
- bcherny 8y ago
  Does HTTP2 server push help solve this? Is there a way to send headers for things you’re about to push before pushing them, then letting the browser switch some subset of your resource loads over to another TCP thread without interrupting things?
  - bastawhiz 8y ago
    It doesn't, and there is not.
romed 8y ago
TCP has major issues in low-latency environments. It has too many timers and they are too coarse for 50us RTTs and 100gbps bandwidth. QUIC as a transport for gRPC can solve a lot of this. What we know for sure today is that Linux distributions, out of the box, are garbage for TCP in datacenters. You can tune some of it away, if you happen to know where the knobs are, but some things like microsecond packet time stamps you have to patch. Switching to a UDP transport means you don’t have to argue with Linux kernel maintainers in order to make progress.
- ivoras 8y ago
  First actually useful and technical answer I've seen in this thread. But... which use cases of HTTP need us timers? It's not like gmail or other bloated web apps will load faster on my phone because of QUIC?
  - romed 8y ago
    The timers are in TCP not HTTP. TCP only ACKs every other packet. If the stream pauses after an odd number of packets, the sender has to just wait for the receiver to perform a delayed ACK, for which there is a timer on both sides. The sender infers that the last packet has been lost if the ACK doesn't arrive after a while. The sender has a thing called the retransmission timeout (RTO) and the receiver's delayed ACK timeout (ATO) sets a lower bound on the RTO. Modern TCP stacks have a thing called the tail loss probe that just sends a retransmit early (before the RTO) if the stream has stopped at an odd boundary. This is a bit of a hack. All of these things affect client-observed latency since obviously the client can't proceed until it gets the last packet. On mobile this is pretty important because mobile networks have massive packet loss. Anyway, that's mobile. In the datacenter all of these timers are way too high, by several orders of magnitude. 1ms is an outrageously long RTO in a datacenter fabric but 200ms is the minimum RTO allowed by the TCP RFCs.
rstuart4133 8y ago
QUIC does things TCP will never do. Firstly QUIC is a combination of SSL + TCP. This allows it put the SSL handshake into the TCP handshake - thus reducing the overhead of starting an SSL connection. It also leverages SSL session reuse to avoid the TCP 3 package handshake completely. As others have mentioned, QUIC is optimised for something that HTTP/2 does often that TCP does not support very well - multiple child streams over the one connection. In general, it is true that TCP is highly optimised, but it is only optimised for one use case: a long lived single flow over a low error rate connection to a single destination. That was the predominate type of connection back in the FTP and early web days. But is hasn't been that way for a while now. HTTP is short lived, video goes to multi destinations, HTTP/2 and multi media in general has multiple flows. Wireless only appears to be low error rate because the physical layer had to cover up it's shortcomings otherwise TCP's performance became abysmal. TCP hasn't remained king of the mountain because its fantastic protocol that can't be improved. On the contrary - looks to be the simplest thing it's inventors could come up with. It even lacked round trip time measurement - which is something I suspect that would have killed it if an extension hadn't saved it. TCP has remained king of the mountain purely because it appears to be near impossible to replace. Now that Google has done it, I expect we will see a lot more experimentation in this area.
- ktpsns 8y ago
  Nice review. Just a theoretical question: Wasn't IPsec supposed to do the encryption right in the IP layer? I wonder how this compares with the handshake roundtrip time needed (for IPsec, for TCP).
  - pas 8y ago
    Well, the problem is complex. For security you need integrity, authenticity and anonimity/secrecy. Crypto of course is the way we implement all that, via keys and other kind of information theoretic tokens that somehow build upon a pre-shared secret. You need an anchor, a mutual trust foundation to build your secure channel upon. If you want to put security on/into IP, you suddenly need the whole key management thing, and if you want it to be automatic, you need to somehow cobble together the assignment, provisioning, allocation, publication, revocation of keys with IP addresses (or hostnames, domain names, and you just put DNS into IP). Eventually it means either you need manual key management (thus IPsec becames a hard to manage old rusty and poorly supported concept from an old era) or you just descended into the recursive pit of all hells of circular dependencies. Sure, it can be done, we can put keys into DNS, and we can use that, but then we're back to QUIC or DTLS basically.

by ivoras 8y ago

dagenix 8y ago

One issue is head of line blocking in TCP when multiplexing requests over a single connection. That problem can't be solved without something like QUIC.

billyhoffman 8y ago
To be clear, this became a bigger issue because HTTP/2 moved to “1 TCP connection per origin with all requests multiplexed on it”. There has always been head of line blocking in tcp, however H2 compounded it since it now impacts all requests to the origin instead of just a few requests in one of four or one of six tcp connections to origin. H2 effectively moved the problem to TCP (really just made underlyubg problem more impactful). Quic is an attempt to solve the TCP problem
- ivoras 8y ago
  But moving the solution away from TCP into something new (in this case userland UDP-based QUIC) would make sense if (and probably only if) the overhead of QUIC connections + request bookkeeping is lower in memory and CPU usage (ideally both) than just establishing multiple TCP connections. Right?
  - jayd16 8y ago
    My understanding is multiplexing over an existing connection also provides time savings as new data does not need the full handshake and ramp up a completely new connection would.
  - dagenix 8y ago
    TCP flow control is generally applied per-connection. So, a web browser that creates 6 connections to load a website is going to get a greater share of available bandwidth than an application that just uses one. And what will get greater bandwidth still is if the website owner can trick the browser into opening 12 or 18 or more connections by sharding their site over multiple domains. "Just open more connections" has issues. QUIC making it less necessary to do that has value even if it didn't save memory or CPU usage.
    - ivoras 8y ago
      The need for maintaining flow control still exists, both in QUIC and TCP, it doesn't just vanish. Also, big(ish) routers are aware of TCP and can adapt as the load changes. They literally won't know what to do with a huge increase in UDP traffic. I'd make an educated guess that mass deployment of QUIC will prove to be pretty awful until it's hacked to look more or less exactly like TCP with multiple connections, just with more overhead.
      - anticensor 8y ago
        > They literally won't know what to do with a huge increase in UDP traffic. Randomly drop UDP packets.
      - dagenix 8y ago
        Afaik, the bigger the router, the more likely it is to just route ip packets. I'll make the educated guess that deployment of QUIC will be just fine as routers already know how to handle UDP and IP just fine.
- josteink 8y ago
  So HTTP/1.1 worked fine for around 2+ decades. Google(tm) HTTP/2.0 was unasked for, tried to reinvent TCP, found buggy and fundamentally broken after less than a year. So now Google is pushing HTTP/3 to solve the bugs they made with HTTP/2, by reinventing even more of the IP-stack. What could possibly go wrong? Can someone please tell Google that protocols are not Chrome-releases? They need to work and stay working and supported for decades. You can’t ship a new revision every 6 month. Edit: Correct HTTP/1.1 longdetivity.
  - manigandham 8y ago
    3 decades? 1.1 was released in 1997. Networks change as the world evolves and supports more people and applications, so new protocols are expected. HTTP/2 was great progress at the application layer, and QUIC is now about improving the transport layer. Can you clearly list the bugs you claim HTTP/2 has?
    - josteink 8y ago
      > 3 decades? 1.1 was released in 1997. Fair enough. Corrected. > Can you clearly list the bugs you claim HTTP/2 has? They seemed fairly well outlined in the post I was replying to, no?
      - profmonocle 8y ago
        The only issue raised in the post you replied to was that TCP head-of-line blocking can be a bigger issue in HTTP/2. Claiming this makes the protocol "fundamentally broken" is a bit of a stretch. Plus, this wasn't "found after less than a year". It was a known drawback from the very beginning, when the protocol was still SPDY, and deemed an acceptable tradeoff.
      - manigandham 8y ago
        That is not a bug with HTTP/2. It is designed to be more efficient by using a single TCP connection, thereby magnifying any efficiency issues that TCP had all along. These issues are not only about HTTP but include the massively increased amount of devices, network types, security features, and general throughput over the years. Perhaps a new version of TCP could've been made, and that's a valid concern, but it's a slow moving standard that takes decades to alter. QUIC evolved faster on UDP so that's what we have. What strategy do you suggest for improving TCP faster than the current process?
- deathanatos 8y ago
  > There has always been head of line blocking in tcp, however H2 compounded it since it now impacts all requests to the origin instead of just a few requests in one of four or one of six tcp connections to origin. I feel like you're conflating two very different things into "head of line blockings". Head of line blocking in HTTP/1.1 was that HTTP/1.1 cannot transmit responses out of order, so a slow response "blocks" any responses behind it. HTTP/2 solves this by being multiplexed (effectively, it tags the requests & responses s.t. you can match out of order responses back to the appropriate request). This isn't necessarily that the network can't handle it; a request that might be easy for the server to handle (e.g., a static asset) can get caught behind one that is not (e.g., one that entails a slow DB query). The other "head of line blocking" that I think you're getting at is just network loss or saturation. If the network can't deliver packets, I don't see how having 6 connections is going to help: you're going to have to wait for six different state machines to determine that the link is saturated, back off, etc. Regardless, my understanding is that QUIC / HTTP/3 is still a multiplexed protocol; I would think it would behave very similarly. Nonetheless, Google's announcement of it backs up your claim[1]: > Like HTTP/2, QUIC multiplexes multiple streams into one connection, so that a connection can serve several HTTP requests simultaneously. But HTTP/2 uses TCP as its transport, so all of its streams can be blocked when a single TCP packet is lost—a problem called head-of-line blocking. QUIC is different: Loss of a UDP packet within a QUIC connection only affects the streams contained within that packet. In other words, QUIC won’t let a problem with one request slow the others down, even on an unreliable connection. But what here makes QUIC think that those other requests have any hope where the first didn't? To me, this approach seems like you're just spamming the link with packets and hoping for the best; those packets could just end up dropped like the other ones were. What makes QUIC think this approach will have any benefit? (It could, perhaps, if the link is just going to drop one packet and we can get right back to sending. I'm not sure that still conveys any benefit over TCP. Not saying QUIC is wrong, just that I don't see a decent reason to buy into it.) (This whole thing makes me think there are two problems here: there's the actual network connection between two endpoints, and what its state is, how lossy it is, its PMTU, how much data we should allow in-flight between those two endpoints, etc. But then there is also logical streams, such as HTTP request/response pairs. It almost seems like to me any data about the link, such as PMTU/lossiness/window, should be shared between all TCP connections, and the individual logical streams should be otherwise cheap/easy to make.) [1]: https://cloudplatform.googleblog.com/2018/06/Introducing-QUIC-support-for-HTTPS-load-balancing.html https://cloudplatform.googleblog.com/2018/06/Introducing-QUI...
deleted 8y ago
[deleted]

cpeterso 8y ago

Fixing bugs and experimenting with new protocols will be easier in user space than waiting for operating systems and routers to get fixed and their users to deploy them. Look at all the problems TLS 1.3 has faced with busted routers and enterprise MITM boxes.

tomohawk 8y ago

TCP is a general stream oriented protocol that must cover a variety of use cases and situations. That's why there are so many OS knobs to tweak it if you have a special use. Since HTTP is the use case here, it is possible to create a protocol that is optimized for it, whereas TCP cannot be optimized specifically for HTTP. Source: I've implemented a reliable, stream oriented protocol over UDP which has been in use for many years.

devwastaken 8y ago

Udp is preferred for content that can be lossy. The overhead of tcp for streamable content, especially over WiFi, is too high. I ran into this issue by trying to have a simple video server through nginx on a DigitalOcean droplet. Wouldn't work well over WiFi at range, but YouTube would work easy on chrome.

colde 8y ago
I think that has a lot more to do with YouTube having an a very optimized video player, and using ABR streams. That works fine over WiFi in general. If you are doing MP4 pseudo-streaming, thats really a completely different beast. You don't really want to drop data for streaming video either, sure, in some cases it might not make a huge difference, but if you drop i-frames, you are going to have quality issues.
- devwastaken 8y ago
  Problem is on wifi that the whole syn/ack pattern gets lost and has to retry. I could have probably optimized this by having a range, but afaik if any in the range fail it has to still re-send that packet and get an ack. I'm not sure if HTTP/3 will have a way to do selective re-sending, but that is how very versatile lossy video streams do it. I-frames can get re-sent while normal frames can be lost.

ploxiln 8y ago

UDP is the new IP. You can't practically have your users install an upgraded OS kernel with TCP rate control improvements, you can't fix queuing in their NAT gateway, and you can't get SCTP working through their ISP. So, you do it all in UDP. (Your servers can handle the streams in kernel or even accelerated by the NIC chip, or in a userspace IP stack e.g. DPDK.)

fulafel 8y ago
OS modifications are not strictly required for new L3 protocols, though some new APIs would be nice to handle it more ergonomically. You can receive/send any IP packets with standard socket APIs. ref: https://docs.microsoft.com/en-us/windows/desktop/WinSock/tcp-ip-raw-sockets-2 https://docs.microsoft.com/en-us/windows/desktop/WinSock/tcp... https://linux.die.net/man/7/raw https://linux.die.net/man/7/raw

toast0 8y ago

TCP has some real issues, MTU discovery is either slow or not enabled or not very smart on a lot of clients. Google, for some reason, doesn't want to fix tcp in Android, instead they went with implementing tcp inside of tcp (http/2), and since that ran into the obvious problems, they've realized that tcp over UDP is a better choice. As far as I can tell, they still haven't enabled TCP MTU discovery in Android though. Or build apis to allow applications to influence packetization and congestion control -- which is what is probably needed. The java apis for TCP are fairly inflexible too, to set most socket options you have to call into c code, and hunt around to get the FD.

est 8y ago
h2 is not tcp on tcp. h2 failed because it's mux everything into 1 single tcp connection, and it only takes 1 failed tcp to block everything in line. h2 really need to do demux. But h2 actually can technically do M:N connections, but browsers do only 1:N. Also h2 is a failed promise. Most sites deliver web assets and page and js in different host, also ws/wss in a different host. Thus renders h2 advantages complete useless.
- bcherny 8y ago
  > “h2 failed” Just curious, what do you mean? Quickly Googling this, it looks like 1/3 websites use HTTP2 today, which is darn good for a relatively new protocol. https://w3techs.com/technologies/details/ce-http2/all/all https://w3techs.com/technologies/details/ce-http2/all/all
  - Vektorweg 8y ago
    Can't always correlate usage share with performance, especially on things that are extremely hyped.
  - provolone 8y ago
    Googlebot never adopted HTTP2, only SPDY
  - est 8y ago

nly 8y ago

Biggest problem with multiplexing over TCP is mixing latency sensitive flows (like getting HTML to render) with high throughput flows (like downloading a YouTube video). You can try and diddle with chunk size and message prioritization in your application to reduce HOL blocking of low latency messages behind chunks of high throughput data, but your OS will buffer everything in one big queue anyway (because TCP is a single stream and flow control applies to that entire stream as a whole). How do you fix that? Well you can reduce socket buffer sizes on your TCP connection, and that will fix latency... but small buffer sizes mean your total throughput is limited by round trip time (because of the way the TCP receive windows and positive acknowledgment work). If you're working over a LAN, things will be fine, but over the Internet you're hosed. Of course, the solution really is to have separate long-lived TCP connections for different kinds of flows, so you can set buffer sizes appropriately... but that's impossible for your browser (which is responsible for opening connections). It has no way to know whether a GET request triggered by an <img> tag is going to yield a 100 byte response or a 100MB response.

bcherny 8y ago
Does HTTP2 server push help solve this? Is there a way to send headers for things you’re about to push before pushing them, then letting the browser switch some subset of your resource loads over to another TCP thread without interrupting things?
- bastawhiz 8y ago
  It doesn't, and there is not.

romed 8y ago

TCP has major issues in low-latency environments. It has too many timers and they are too coarse for 50us RTTs and 100gbps bandwidth. QUIC as a transport for gRPC can solve a lot of this. What we know for sure today is that Linux distributions, out of the box, are garbage for TCP in datacenters. You can tune some of it away, if you happen to know where the knobs are, but some things like microsecond packet time stamps you have to patch. Switching to a UDP transport means you don’t have to argue with Linux kernel maintainers in order to make progress.

ivoras 8y ago
First actually useful and technical answer I've seen in this thread. But... which use cases of HTTP need us timers? It's not like gmail or other bloated web apps will load faster on my phone because of QUIC?
- romed 8y ago
  The timers are in TCP not HTTP. TCP only ACKs every other packet. If the stream pauses after an odd number of packets, the sender has to just wait for the receiver to perform a delayed ACK, for which there is a timer on both sides. The sender infers that the last packet has been lost if the ACK doesn't arrive after a while. The sender has a thing called the retransmission timeout (RTO) and the receiver's delayed ACK timeout (ATO) sets a lower bound on the RTO. Modern TCP stacks have a thing called the tail loss probe that just sends a retransmit early (before the RTO) if the stream has stopped at an odd boundary. This is a bit of a hack. All of these things affect client-observed latency since obviously the client can't proceed until it gets the last packet. On mobile this is pretty important because mobile networks have massive packet loss. Anyway, that's mobile. In the datacenter all of these timers are way too high, by several orders of magnitude. 1ms is an outrageously long RTO in a datacenter fabric but 200ms is the minimum RTO allowed by the TCP RFCs.

rstuart4133 8y ago

QUIC does things TCP will never do. Firstly QUIC is a combination of SSL + TCP. This allows it put the SSL handshake into the TCP handshake - thus reducing the overhead of starting an SSL connection. It also leverages SSL session reuse to avoid the TCP 3 package handshake completely. As others have mentioned, QUIC is optimised for something that HTTP/2 does often that TCP does not support very well - multiple child streams over the one connection. In general, it is true that TCP is highly optimised, but it is only optimised for one use case: a long lived single flow over a low error rate connection to a single destination. That was the predominate type of connection back in the FTP and early web days. But is hasn't been that way for a while now. HTTP is short lived, video goes to multi destinations, HTTP/2 and multi media in general has multiple flows. Wireless only appears to be low error rate because the physical layer had to cover up it's shortcomings otherwise TCP's performance became abysmal. TCP hasn't remained king of the mountain because its fantastic protocol that can't be improved. On the contrary - looks to be the simplest thing it's inventors could come up with. It even lacked round trip time measurement - which is something I suspect that would have killed it if an extension hadn't saved it. TCP has remained king of the mountain purely because it appears to be near impossible to replace. Now that Google has done it, I expect we will see a lot more experimentation in this area.

ktpsns 8y ago
Nice review. Just a theoretical question: Wasn't IPsec supposed to do the encryption right in the IP layer? I wonder how this compares with the handshake roundtrip time needed (for IPsec, for TCP).
- pas 8y ago
  Well, the problem is complex. For security you need integrity, authenticity and anonimity/secrecy. Crypto of course is the way we implement all that, via keys and other kind of information theoretic tokens that somehow build upon a pre-shared secret. You need an anchor, a mutual trust foundation to build your secure channel upon. If you want to put security on/into IP, you suddenly need the whole key management thing, and if you want it to be automatic, you need to somehow cobble together the assignment, provisioning, allocation, publication, revocation of keys with IP addresses (or hostnames, domain names, and you just put DNS into IP). Eventually it means either you need manual key management (thus IPsec becames a hard to manage old rusty and poorly supported concept from an old era) or you just descended into the recursive pit of all hells of circular dependencies. Sure, it can be done, we can put keys into DNS, and we can use that, but then we're back to QUIC or DTLS basically.