8 ms·

QUIC is a really nice protocol as well, I find. It basically gives you an end-to-end encrypted & authenticated channel over which you can transport multiple str

QUIC is a really nice protocol as well, I find. It basically gives you an end-to-end encrypted & authenticated channel over which you can transport multiple streams in parallel, as well as datagrams. A lost packet in a given stream won't block other streams, and the overhead is quite low. Both ends of the connection can open streams as well, so it's really easy to build bidirectional communication over QUIC. You can also do things like building a VPN tunnel using the datagram mechanism, there are protocols like MASQUE that aim to standardize this. Apple is using a custom MASQUE implementation for their private relay, for example. HTTP/3 is a protocol on top of QUIC that adds a few more really interesting things, like qpack header compression. If you e.g. send a "Content-type: text/html" header it will compress to 2 bytes as the protocol has a Huffman table with the most commonly used header values. I found that quite confusing when testing connections as I thought "It's impossible that I only get 2 bytes, I sent a long header string..." until I found out about this.

promiseofbeans 3y ago
Woah that's actually pretty neat about header compression. Thanks for sharing!
- ithkuil 3y ago
  it builds on the experience gathered with the HTTP/2 HPACK header compression.
culebron21 3y ago
That makes sense -- if a packet is lost, and it affected just one asset, but you're on TCP, then everything has to wait till the packet is re-requested and resent.
- afiori 3y ago
  I wonder why the OS does not let the application peek at upcoming incomplete data.
  - londons_explore 3y ago
    There is an API for this on Linux. It's used to checkpoint the state of a TCP connection, for example to move a live TCP connection to another machine incase you want to do zero downtime hardware replacement while keeping all TCP sockets open.
- ReactiveJelly 3y ago
  For the curious, this problem is called head-of-line blocking https://en.wikipedia.org/wiki/Head-of-line_blocking https://en.wikipedia.org/wiki/Head-of-line_blocking HTTP2 allowed multiple streams over one TCP stream, but that kinda made HoL blocking worse, because in the same scenario HTTP 1.1 would have just opened multiple TCP streams. QUIC, as GP said, basically gives you a VPN connection to the server. Open 100 streams for reliable downloads and uploads, send datagrams for a Quake match, all over one UDP 'connection' that can also reconnect even if you change IPs.
binwiederhier 3y ago
I dabbled with QUIC a few years ago and I couldn't agree more. It was pleasant to work with, and because it's UDP based, suddenly you can do NAT hole punching more easily. Funny that you mentioned a VPN, because I made a little experimental project back then to hole-punch between two behind-the-NAT machines and deliver traffic between them over QUIC. I was able to make my own L2 and L3 bridges across the WAN, or just port forward from one natted endpoint to an endpoint behind a different NAT. At one point I used it to L2-bridge my company's network (10.x) to my home network (192.168.x), and I was able to ping my home server from the bridging host, even though it was different networks, because it was essentially just connecting a cable between the networks. It was quite fun. Here's the project if anyone is interested: https://github.com/binwiederhier/natter https://github.com/binwiederhier/natter -- it's probably defunct, but it was only experimental anyway.
- abhishekjha 3y ago
  How mcuh config did it requir on your home local network and office network to do the ping on 192.168.x.x?
  - binwiederhier 3y ago
    I only tested one hop, e.g. A(10.x)---->B(192.x). All I had to do there was to adapt the routing tables: On A, route 192.x traffic to the "natter" tap/tun interface (I always forget which is L2), and on B, route traffic to 10.x accordingly. That's all. For it to be routable in the entire network, you'd need to obviously mess with a lot more :-D
- heyoni 3y ago
  THAT was your first project in golang? I’m impressed. it looks like it might it could replace ngrok? I’ll try it later.
  - binwiederhier 3y ago
    Well it's abandoned and experimental, and there are better ways to hole punch than what I did, e.g. using STUN and TURN. But yeah it could replace one ngrok use case, though I think ngrok does not do L2/3 bridges. Also: I think technically this was my first Go program (https://github.com/binwiederhier/re https://github.com/binwiederhier/re), but that was so tiny that it doesn't really count. ;-)
d-z-m 3y ago
I believe that qpack(like hpack) still has some sharp edges. As in...low entropy headers are still vulnerable to leakage via a CRIME[0] style attack on the HTTP/3 header compression. In practice, high entropy headers aren't vulnerable, as an attacker has to match the entire header:value line in order to see a difference in the compression ratio[1]. [0]: https://en.wikipedia.org/wiki/CRIME https://en.wikipedia.org/wiki/CRIME [1]: https://www.ietf.org/archive/id/draft-ietf-quic-qpack-20.html#section-7 https://www.ietf.org/archive/id/draft-ietf-quic-qpack-20.htm...
- RenThraysk 3y ago
  Given that Qpack has a dedicated stream for dynamic table updates, can atleast limit which header:value combinations are inserted. And there are bit flags in the encoder instructions that carry header values that signal they should never be inserted into a dynamic table.
  - d-z-m 3y ago
    > And there are bit flags in the encoder instructions that carry header values that signal they should never be inserted into a dynamic table. My understanding is that these header values have to be known to the Qpack implementation beforehand. As a result, it isn't possible for an HTTP/3 client to signal to Qpack to treat a particular custom header as a never-indexed literal.
nly 3y ago
The problem is there's no standard equivalent of the BSD sockets API for writing programs that communicate over QUIC. It'll always be niche outside the browser until this exists.
- jeffbee 3y ago
  Not having a system API is the entire point of QUIC. The only reason QUIC needs to exist is because the sockets API and the system TCP stacks are too ossified to be improved. If you move that boundary then QUIC will inevitably suffer from the same ossification that TCP displays today.
  - tsimionescu 3y ago
    No, the reason QUIC exists is that TCP is ossified at the level of middle boxes on the internet. If it had been possible to modify TCP with just some changes in the Linux, BSD and Windows kernels, it would have been done.
    - gjvc 3y ago
      worth defining "middlebox" """ A middlebox is a computer networking device that transforms, inspects, filters, and manipulates traffic for purposes other than packet forwarding. Examples of middleboxes include firewalls, network address translators (NATs), load balancers, and deep packet inspection (DPI) devices. """ https://en.wikipedia.org/wiki/Middlebox https://en.wikipedia.org/wiki/Middlebox
    - Dylan16807 3y ago
      I don't know about that. Without middlebox problems we might have used SCTP as a basis and upgraded it. But it's so different from TCP that I doubt we would have done it as a modification of TCP.
    - lima 3y ago
      Both, really. The OS-level ossification isn't quite as bad as the middleboxes, but is still glacially slow and bad enough to justify QUIC on its own. Case in point: the whole TCP Fast Open drama.
weird-eye-issue 3y ago
> If you e.g. send a "Content-type: text/html" header it will compress to 2 bytes as the protocol has a Huffman table with the most commonly used header values Reminds me of tokenization for LLMs. 48 dashes ("------------------------------------------------") is only a single token for GPT-3.5 / GPT-4 (they use the cl100k_base encoding). I suppose since that is used in Markdown. Also "professional illustration" is only two tokens despite being a long string. Whereas if you convert that to a language like Thai it is 17 tokens which sucks in some cases but I suppose tradeoffs had to be made
Borg3 3y ago
QUIC is junk.. You people all care about raw throughput but not about multiuser friendless. Selfish. Problem is, QUIC is UDP and so its hard to police/shape. I really want to play some FPS game while someone is watching/browsing web. Also, I really want my corpo VPN have a bit of priority over web, but no, now I cannot police it easly. TCP is good for best effort traffic, and thats where I classify web browsing, downloading, VoD streaming. UDP is good for gaming, voice/video conferencing, VPNs (because they are encapsulate stuff you put another layer somewhere else).
- KMag 3y ago
  QUIC has its own flow control. It's not raw UDP. Now, I wish ToS/QoS were more broadly usable for traffic prioritization. It sounds like you're using UDP vs. TCP as a proxy for ToS/QoS. At a minimum, you're still going to have a bad time with TCP streams getting encapsulated in UDP WireGuard VPN connections.
- vlovich123 3y ago
  I feel like that’s an ungenerous characterization. First, QUIC should contain some minimal connection info unencrypted that can be middleware to do some basic traffic shaping. It’s also intentionally very careful to avoid showing too much to avoid “smart” middleware that permanently ossifies the standard as has happened to TCP. Finally, traffic shaping on a single machine is pretty easy and most routers will prefer TCP traffic to UDP. Finally, the correct response to overwhelm is to drop packets. This is true for TCP and UDP to trigger congestion control. Middleware has gotten way too clever by half and we have bufferbloat. To drop packets you don’t need knowledge of streams - just that you have a non-skewed distribution you apply to dropping the packets so that proportionally all traffic overwhelming you from a source gets equally likely to be dropped. This ironically improves performance and latency because well behaving protocols like TCP and QUIC will throttle back their connections and UDP protocols without throttling will just deal with elevated error rates.
  - Borg3 3y ago
    So what? You dropping packets, and they are still coming, eating BW and buckets. Because traditionally UDP did not have any flow control, you just treat is as kinda CBR traffic and so you just want to leave it queues as fast as it can. If there was a lot of TCP traffic around, you just drop packets there and vioala, congestion kick sin and you have more room for importand UDP traffic. Now, if you start to drop UDP packets your UX drops.. packet loss in FPS games is terrible, even worse than a bit of jitter. Thank you.
ComputerGuru 3y ago
I’ve been trying to wrap my mind around whether (and how much) QUIC would be better than TCP for video streams that rely on order-sensitive delivery of frames, especially for frames that have to be split into multiple packets (and losing one packet or receiving it out of order would lose the entire frame and any dependent frames). We used to use UDP for mpegts packets but found TCP with a reset when buffers are backlogged after switching to h264 to be a much better option over lossy WAN uplinks, (scenario is even worse when you have b or p frames present). The problem would be a lot easier if there were a feedback loop to the compressor where you can dynamically reduce quality/bandwidth as the connection quality deteriorates but currently using the stock raspivid (or v4l2) interface makes that a bit difficult unless you’re willing to explicitly stop and start the encoding all over again, which breaks the stream anyway.
- londons_explore 3y ago
  > stop and start the encoding all over again, which breaks the stream anyway. It's a common thing I wish encoders could do - if I try to compress a frame and the result is too large to fit in my packet/window, I wish I could 'undo' and retry compression of the same frame with different options. Sadly all hardware video compressors mutate internal state when something is compressed, so there is no way to undo.
- mleo 3y ago
  QUIC handles the packet ordering and retransmission of lost packets before it is handed to the upper layer of the application. However, if you are just sending one channel of video packets through the pipe, it probably won’t buy you much more. Where QUIC additionally excels is being able to send and/or receive multiple streams of data across the “single” connection where each is effectively independent and does not suffer head of line blocking across all streams.
Karrot_Kream 3y ago
I can't wait to start implementing RTP atop QUIC so we can stop having to deal with the highly stateful SIP stack and open a media connection the same way we open any other Application layer connection.

promiseofbeans 3y ago

Woah that's actually pretty neat about header compression. Thanks for sharing!

ithkuil 3y ago
it builds on the experience gathered with the HTTP/2 HPACK header compression.

culebron21 3y ago

That makes sense -- if a packet is lost, and it affected just one asset, but you're on TCP, then everything has to wait till the packet is re-requested and resent.

afiori 3y ago
I wonder why the OS does not let the application peek at upcoming incomplete data.
- londons_explore 3y ago
  There is an API for this on Linux. It's used to checkpoint the state of a TCP connection, for example to move a live TCP connection to another machine incase you want to do zero downtime hardware replacement while keeping all TCP sockets open.
ReactiveJelly 3y ago
For the curious, this problem is called head-of-line blocking https://en.wikipedia.org/wiki/Head-of-line_blocking https://en.wikipedia.org/wiki/Head-of-line_blocking HTTP2 allowed multiple streams over one TCP stream, but that kinda made HoL blocking worse, because in the same scenario HTTP 1.1 would have just opened multiple TCP streams. QUIC, as GP said, basically gives you a VPN connection to the server. Open 100 streams for reliable downloads and uploads, send datagrams for a Quake match, all over one UDP 'connection' that can also reconnect even if you change IPs.

binwiederhier 3y ago

I dabbled with QUIC a few years ago and I couldn't agree more. It was pleasant to work with, and because it's UDP based, suddenly you can do NAT hole punching more easily. Funny that you mentioned a VPN, because I made a little experimental project back then to hole-punch between two behind-the-NAT machines and deliver traffic between them over QUIC. I was able to make my own L2 and L3 bridges across the WAN, or just port forward from one natted endpoint to an endpoint behind a different NAT. At one point I used it to L2-bridge my company's network (10.x) to my home network (192.168.x), and I was able to ping my home server from the bridging host, even though it was different networks, because it was essentially just connecting a cable between the networks. It was quite fun. Here's the project if anyone is interested: https://github.com/binwiederhier/natter https://github.com/binwiederhier/natter -- it's probably defunct, but it was only experimental anyway.

abhishekjha 3y ago
How mcuh config did it requir on your home local network and office network to do the ping on 192.168.x.x?
- binwiederhier 3y ago
  I only tested one hop, e.g. A(10.x)---->B(192.x). All I had to do there was to adapt the routing tables: On A, route 192.x traffic to the "natter" tap/tun interface (I always forget which is L2), and on B, route traffic to 10.x accordingly. That's all. For it to be routable in the entire network, you'd need to obviously mess with a lot more :-D
heyoni 3y ago
THAT was your first project in golang? I’m impressed. it looks like it might it could replace ngrok? I’ll try it later.
- binwiederhier 3y ago
  Well it's abandoned and experimental, and there are better ways to hole punch than what I did, e.g. using STUN and TURN. But yeah it could replace one ngrok use case, though I think ngrok does not do L2/3 bridges. Also: I think technically this was my first Go program (https://github.com/binwiederhier/re https://github.com/binwiederhier/re), but that was so tiny that it doesn't really count. ;-)

d-z-m 3y ago

I believe that qpack(like hpack) still has some sharp edges. As in...low entropy headers are still vulnerable to leakage via a CRIME[0] style attack on the HTTP/3 header compression. In practice, high entropy headers aren't vulnerable, as an attacker has to match the entire header:value line in order to see a difference in the compression ratio[1]. [0]: https://en.wikipedia.org/wiki/CRIME https://en.wikipedia.org/wiki/CRIME [1]: https://www.ietf.org/archive/id/draft-ietf-quic-qpack-20.html#section-7 https://www.ietf.org/archive/id/draft-ietf-quic-qpack-20.htm...

RenThraysk 3y ago
Given that Qpack has a dedicated stream for dynamic table updates, can atleast limit which header:value combinations are inserted. And there are bit flags in the encoder instructions that carry header values that signal they should never be inserted into a dynamic table.
- d-z-m 3y ago
  > And there are bit flags in the encoder instructions that carry header values that signal they should never be inserted into a dynamic table. My understanding is that these header values have to be known to the Qpack implementation beforehand. As a result, it isn't possible for an HTTP/3 client to signal to Qpack to treat a particular custom header as a never-indexed literal.

nly 3y ago

The problem is there's no standard equivalent of the BSD sockets API for writing programs that communicate over QUIC. It'll always be niche outside the browser until this exists.

jeffbee 3y ago
Not having a system API is the entire point of QUIC. The only reason QUIC needs to exist is because the sockets API and the system TCP stacks are too ossified to be improved. If you move that boundary then QUIC will inevitably suffer from the same ossification that TCP displays today.
- tsimionescu 3y ago
  No, the reason QUIC exists is that TCP is ossified at the level of middle boxes on the internet. If it had been possible to modify TCP with just some changes in the Linux, BSD and Windows kernels, it would have been done.
  - gjvc 3y ago
    worth defining "middlebox" """ A middlebox is a computer networking device that transforms, inspects, filters, and manipulates traffic for purposes other than packet forwarding. Examples of middleboxes include firewalls, network address translators (NATs), load balancers, and deep packet inspection (DPI) devices. """ https://en.wikipedia.org/wiki/Middlebox https://en.wikipedia.org/wiki/Middlebox
  - Dylan16807 3y ago
    I don't know about that. Without middlebox problems we might have used SCTP as a basis and upgraded it. But it's so different from TCP that I doubt we would have done it as a modification of TCP.
  - lima 3y ago
    Both, really. The OS-level ossification isn't quite as bad as the middleboxes, but is still glacially slow and bad enough to justify QUIC on its own. Case in point: the whole TCP Fast Open drama.

weird-eye-issue 3y ago

> If you e.g. send a "Content-type: text/html" header it will compress to 2 bytes as the protocol has a Huffman table with the most commonly used header values Reminds me of tokenization for LLMs. 48 dashes ("------------------------------------------------") is only a single token for GPT-3.5 / GPT-4 (they use the cl100k_base encoding). I suppose since that is used in Markdown. Also "professional illustration" is only two tokens despite being a long string. Whereas if you convert that to a language like Thai it is 17 tokens which sucks in some cases but I suppose tradeoffs had to be made

Borg3 3y ago

QUIC is junk.. You people all care about raw throughput but not about multiuser friendless. Selfish. Problem is, QUIC is UDP and so its hard to police/shape. I really want to play some FPS game while someone is watching/browsing web. Also, I really want my corpo VPN have a bit of priority over web, but no, now I cannot police it easly. TCP is good for best effort traffic, and thats where I classify web browsing, downloading, VoD streaming. UDP is good for gaming, voice/video conferencing, VPNs (because they are encapsulate stuff you put another layer somewhere else).

KMag 3y ago
QUIC has its own flow control. It's not raw UDP. Now, I wish ToS/QoS were more broadly usable for traffic prioritization. It sounds like you're using UDP vs. TCP as a proxy for ToS/QoS. At a minimum, you're still going to have a bad time with TCP streams getting encapsulated in UDP WireGuard VPN connections.
vlovich123 3y ago
I feel like that’s an ungenerous characterization. First, QUIC should contain some minimal connection info unencrypted that can be middleware to do some basic traffic shaping. It’s also intentionally very careful to avoid showing too much to avoid “smart” middleware that permanently ossifies the standard as has happened to TCP. Finally, traffic shaping on a single machine is pretty easy and most routers will prefer TCP traffic to UDP. Finally, the correct response to overwhelm is to drop packets. This is true for TCP and UDP to trigger congestion control. Middleware has gotten way too clever by half and we have bufferbloat. To drop packets you don’t need knowledge of streams - just that you have a non-skewed distribution you apply to dropping the packets so that proportionally all traffic overwhelming you from a source gets equally likely to be dropped. This ironically improves performance and latency because well behaving protocols like TCP and QUIC will throttle back their connections and UDP protocols without throttling will just deal with elevated error rates.
- Borg3 3y ago
  So what? You dropping packets, and they are still coming, eating BW and buckets. Because traditionally UDP did not have any flow control, you just treat is as kinda CBR traffic and so you just want to leave it queues as fast as it can. If there was a lot of TCP traffic around, you just drop packets there and vioala, congestion kick sin and you have more room for importand UDP traffic. Now, if you start to drop UDP packets your UX drops.. packet loss in FPS games is terrible, even worse than a bit of jitter. Thank you.

ComputerGuru 3y ago

I’ve been trying to wrap my mind around whether (and how much) QUIC would be better than TCP for video streams that rely on order-sensitive delivery of frames, especially for frames that have to be split into multiple packets (and losing one packet or receiving it out of order would lose the entire frame and any dependent frames). We used to use UDP for mpegts packets but found TCP with a reset when buffers are backlogged after switching to h264 to be a much better option over lossy WAN uplinks, (scenario is even worse when you have b or p frames present). The problem would be a lot easier if there were a feedback loop to the compressor where you can dynamically reduce quality/bandwidth as the connection quality deteriorates but currently using the stock raspivid (or v4l2) interface makes that a bit difficult unless you’re willing to explicitly stop and start the encoding all over again, which breaks the stream anyway.

londons_explore 3y ago
> stop and start the encoding all over again, which breaks the stream anyway. It's a common thing I wish encoders could do - if I try to compress a frame and the result is too large to fit in my packet/window, I wish I could 'undo' and retry compression of the same frame with different options. Sadly all hardware video compressors mutate internal state when something is compressed, so there is no way to undo.
mleo 3y ago
QUIC handles the packet ordering and retransmission of lost packets before it is handed to the upper layer of the application. However, if you are just sending one channel of video packets through the pipe, it probably won’t buy you much more. Where QUIC additionally excels is being able to send and/or receive multiple streams of data across the “single” connection where each is effectively independent and does not suffer head of line blocking across all streams.

Karrot_Kream 3y ago

I can't wait to start implementing RTP atop QUIC so we can stop having to deal with the highly stateful SIP stack and open a media connection the same way we open any other Application layer connection.