3 ms·

The improved ss output mentioned in the paper has helped me immensely tracking down what was limiting the upload speed from a corporate net to AWS. Turned out n

The improved ss output mentioned in the paper has helped me immensely tracking down what was limiting the upload speed from a corporate net to AWS. Turned out no middleboxes were to blame, just a misconfiguration in nginx (receiving side) limiting the http2 send window (sender side). Yes, http2 has a separate send window on top of tcp. Yet another source of errors.

luizfelberti 6y ago
> Yes, http2 has a separate send window on top of tcp. Yet another source of errors Damn, I was not aware of this, but sure enough there it is [0] I suppose that's just one more thing to reaffirm my belief that SPDY is a horribly over-engineered protocol, that tried to shoehorn fixes for TCP's shortcomings on top of the wrong abstraction (which, ironically, is TCP itself). Had HTTP2 only introduced the binary frames encoding, instead of bolting this horrible new transport along with it, we wouldn't be so deep in this mess. QUIC can't come soon enough. [0] https://www.chromium.org/spdy/spdy-protocol/spdy-protocol-draft3-1#TOC-2.5-Data-flow https://www.chromium.org/spdy/spdy-protocol/spdy-protocol-dr...
- morei 6y ago
  ISTR that SPDY et al was originally built on UDP, not TCP, so there was a need for a send window. But yes, it's now fighting with the TCP window. Ideally, the HTTP2 send window should be derived from the TCP window.
- Matthias247 6y ago
  Maybe the bad news for you: Quic is different, but it's not necessarily a lot easier. A lot of the challenges that are part of implementing HTTP/2 correctly are also present in Quic. You will still have to implement a flow control mechanism for individual Quic streams, apart from the being fair across streams, congestion control and pacing for Quic connections, and being fair across connections on an endpoint. Getting this all right is also far from an easy task - and resource wise it might never catch up with HTTP/1.1 + TLS due to missing hardware support. However I feel like at least in some areas the Quic specification became less complex. E.g. the use of absolute flow control offsets instead of relative increments makes things less error prone. And it's nice that now the defaults for the dynamic table are "off", which means one can skip implementing that part without the risk of not being interoperable.
  - luizfelberti 6y ago
    I'm well aware of these things actually lol, it's pretty obvious that if the protocol is built on top of UDP it'd need to implement some FSM to handle retransmission and flow control > A lot of the challenges that are part of implementing HTTP/2 correctly are also present in QUIC I'd dispute this because it doesn't have nested flight windows, and doesn't have to multiplex and interleave streams over the same tunnel (it does that over the same socket/port, but it's way easier because UDP is a stateless packet machine-gun, so you don't need to keep both TCP and SPDY's concept of a stream at the same time). It has about the same challenges of implementing TCP correctly (which is what you'd expect really), because it needs to implement congestion control and etc pretty much in all the same places, the only "extra" it shares with SPDY is stream multiplexing (which is still much simpler because it doesn't need to worry about TCP's semantics on top of it). > resource wise, it might never catch up with HTTP/1.1 + TLS (I think you meant TCP here) due to missing hardware support I have two comments about this: 1 - This is actually more of a software problem than a hardware problem. Hardware acceleration for QUIC depends mostly on evolving the tooling we have for "talking" to our network cards across different platforms than anything else really. For a while now, most NICs (especially those you'd see in data-center gear) use a Motorola 68k based chip, or something else equally programmable, and offloading arbitrary programs to those is already something that happens quite often. For example, Linux XDP programs are automatically offloaded to the NIC if it supports that. This is also not a problem for middle-boxes, since they just treat UDP blindly as UDP and will just do AQM on top of it like they always did, no need to implement flow control. This might be a problem for legacy consumer hardware, but then again those are already inefficient in all kinds of ways, and for these cases being able to run the entire networking stack from userland is actually a benefit: being able to fallback on full-software implementations means we can make Windows XP support QUIC if we wanted to. Can we even measure how much shit we'd break if we tried to "upgrade" TCP? 2 - This might be underestimating the amortized gains from switching to a more efficient protocol: having to retransmit less data, 0-rtt handshakes, and such things add up. Specifically for HTTP1.1, being a text based protocol doesn't help either: how many CPU cycles do we waste with parsing CRLFs, lowercasing headers, and crap like that? At least HTTP2 doesn't suffer from these problems, but it still screwed up transport big time. > I feel like at least in some areas the QUIC specification became less complex This I agree with, wholeheartedly. So far, the only thing I've found absolutely detestable about it is mandated encryption, as that prohibits me from outsourcing some L7 intelligence to a proxy/sidecar/service-mesh/whatever, and having to shoehorn some bullshit certificate between my application and the proxy so that it can have some visibility into traffic will be error prone and annoying. Maybe we'll make a variant of QUIC with TLS disabled for gRPC and other API shennanigans? We'll see when the time comes...
Matthias247 6y ago
Yes, it has. And it's one of the prime reasons HTTP/2 is slower for lots of applications than pure HTTP/1.1. The tricky part of HTTP/2 is that you now have 2 flow control algorithms running on top of each other. And they are rather good at harming the other one instead of cooperating. E.g. a too small HTTP window can lead to poor performance as you observed. However a too big HTTP window can lead to excessive buffering in the application for HTTP/2 streams. It's a rather hard challenge to get to a good compromise solution here (and not a lot of people have arrived there yet). In doubt the rule of thumb is to use HTTP/2 for small requests where the benefit of multiplexing is biggest. And stick to HTTP/1.1 (a non multiplexed TCP connection) for high throughput.
- mobilio 6y ago
  This is fixed on HTTP/3
  - Matthias247 6y ago
    How do you define "fixed"? Quic still defines per stream plus per connection flow control windows, on top of the lower level congestion control algorithm.
    - xxpor 6y ago
      Does it not use UDP?
      - mobilio 6y ago
        It uses UDP because simply can't change all TCP infrastructure worldwide. Example - i'm still using on some place router Linksys WRT54GL with kernel 2.4 that is almost 15 years old. Because device still works and there was low network requirements.
        xxpor 6y ago
        Sure, I totally get that. But my point is UDP doesn't have any underlying congestion control you have to fight.

luizfelberti 6y ago

> Yes, http2 has a separate send window on top of tcp. Yet another source of errors Damn, I was not aware of this, but sure enough there it is [0] I suppose that's just one more thing to reaffirm my belief that SPDY is a horribly over-engineered protocol, that tried to shoehorn fixes for TCP's shortcomings on top of the wrong abstraction (which, ironically, is TCP itself). Had HTTP2 only introduced the binary frames encoding, instead of bolting this horrible new transport along with it, we wouldn't be so deep in this mess. QUIC can't come soon enough. [0] https://www.chromium.org/spdy/spdy-protocol/spdy-protocol-draft3-1#TOC-2.5-Data-flow https://www.chromium.org/spdy/spdy-protocol/spdy-protocol-dr...

morei 6y ago
ISTR that SPDY et al was originally built on UDP, not TCP, so there was a need for a send window. But yes, it's now fighting with the TCP window. Ideally, the HTTP2 send window should be derived from the TCP window.
Matthias247 6y ago
Maybe the bad news for you: Quic is different, but it's not necessarily a lot easier. A lot of the challenges that are part of implementing HTTP/2 correctly are also present in Quic. You will still have to implement a flow control mechanism for individual Quic streams, apart from the being fair across streams, congestion control and pacing for Quic connections, and being fair across connections on an endpoint. Getting this all right is also far from an easy task - and resource wise it might never catch up with HTTP/1.1 + TLS due to missing hardware support. However I feel like at least in some areas the Quic specification became less complex. E.g. the use of absolute flow control offsets instead of relative increments makes things less error prone. And it's nice that now the defaults for the dynamic table are "off", which means one can skip implementing that part without the risk of not being interoperable.
- luizfelberti 6y ago
  I'm well aware of these things actually lol, it's pretty obvious that if the protocol is built on top of UDP it'd need to implement some FSM to handle retransmission and flow control > A lot of the challenges that are part of implementing HTTP/2 correctly are also present in QUIC I'd dispute this because it doesn't have nested flight windows, and doesn't have to multiplex and interleave streams over the same tunnel (it does that over the same socket/port, but it's way easier because UDP is a stateless packet machine-gun, so you don't need to keep both TCP and SPDY's concept of a stream at the same time). It has about the same challenges of implementing TCP correctly (which is what you'd expect really), because it needs to implement congestion control and etc pretty much in all the same places, the only "extra" it shares with SPDY is stream multiplexing (which is still much simpler because it doesn't need to worry about TCP's semantics on top of it). > resource wise, it might never catch up with HTTP/1.1 + TLS (I think you meant TCP here) due to missing hardware support I have two comments about this: 1 - This is actually more of a software problem than a hardware problem. Hardware acceleration for QUIC depends mostly on evolving the tooling we have for "talking" to our network cards across different platforms than anything else really. For a while now, most NICs (especially those you'd see in data-center gear) use a Motorola 68k based chip, or something else equally programmable, and offloading arbitrary programs to those is already something that happens quite often. For example, Linux XDP programs are automatically offloaded to the NIC if it supports that. This is also not a problem for middle-boxes, since they just treat UDP blindly as UDP and will just do AQM on top of it like they always did, no need to implement flow control. This might be a problem for legacy consumer hardware, but then again those are already inefficient in all kinds of ways, and for these cases being able to run the entire networking stack from userland is actually a benefit: being able to fallback on full-software implementations means we can make Windows XP support QUIC if we wanted to. Can we even measure how much shit we'd break if we tried to "upgrade" TCP? 2 - This might be underestimating the amortized gains from switching to a more efficient protocol: having to retransmit less data, 0-rtt handshakes, and such things add up. Specifically for HTTP1.1, being a text based protocol doesn't help either: how many CPU cycles do we waste with parsing CRLFs, lowercasing headers, and crap like that? At least HTTP2 doesn't suffer from these problems, but it still screwed up transport big time. > I feel like at least in some areas the QUIC specification became less complex This I agree with, wholeheartedly. So far, the only thing I've found absolutely detestable about it is mandated encryption, as that prohibits me from outsourcing some L7 intelligence to a proxy/sidecar/service-mesh/whatever, and having to shoehorn some bullshit certificate between my application and the proxy so that it can have some visibility into traffic will be error prone and annoying. Maybe we'll make a variant of QUIC with TLS disabled for gRPC and other API shennanigans? We'll see when the time comes...

Matthias247 6y ago

Yes, it has. And it's one of the prime reasons HTTP/2 is slower for lots of applications than pure HTTP/1.1. The tricky part of HTTP/2 is that you now have 2 flow control algorithms running on top of each other. And they are rather good at harming the other one instead of cooperating. E.g. a too small HTTP window can lead to poor performance as you observed. However a too big HTTP window can lead to excessive buffering in the application for HTTP/2 streams. It's a rather hard challenge to get to a good compromise solution here (and not a lot of people have arrived there yet). In doubt the rule of thumb is to use HTTP/2 for small requests where the benefit of multiplexing is biggest. And stick to HTTP/1.1 (a non multiplexed TCP connection) for high throughput.

mobilio 6y ago
This is fixed on HTTP/3
- Matthias247 6y ago
  How do you define "fixed"? Quic still defines per stream plus per connection flow control windows, on top of the lower level congestion control algorithm.
  - xxpor 6y ago
    Does it not use UDP?
    - mobilio 6y ago
      It uses UDP because simply can't change all TCP infrastructure worldwide. Example - i'm still using on some place router Linksys WRT54GL with kernel 2.4 that is almost 15 years old. Because device still works and there was low network requirements.
      - xxpor 6y ago
        Sure, I totally get that. But my point is UDP doesn't have any underlying congestion control you have to fight.