6 ms·

QUIC, a multiplexed stream transport over UDP

44 points by joubert 11y ago

aidenn0 11y ago
I see FEC is mentioned; isn't most error-correction over noisy channels implemented at the link-layer? Even over e.g. wifi or cellular radios, my understanding is you typically either get the packet or not, with the internet layer rarely seeing corrupted packets. [edit] I read deeper in, and apparently, QUIC optionally allows sending packets in groups, with an additional packet that is the XOR of the other packets in the groups; this means that you can fill-in any single dropped packet by sending an extra packet ahead of time.
stormbrew 11y ago
Does anyone have a clear explanation for why QUIC exists when SCTP (which can be run over UDP) existed? What does QUIC bring over it?
- wmf 11y ago
  QUIC is supposed to be quicker than SCTP+TLS or SCTP+IPSec by using some "rampant layering violations" to combine security and transport into one layer and eliminate a lot of round trips.
- waqf 11y ago
  Your question is answered in the design doc at https://docs.google.com/document/d/1RNHkx_VvKWyWg6Lr8SZ-saqsQx7rFV-ev2jRFUoVD34/edit?pli=1#heading=h.nvk2biis6bpk https://docs.google.com/document/d/1RNHkx_VvKWyWg6Lr8SZ-saqs... tl;dr minimizing latency, particularly in establishing a connection, is an important design objective.
- dragontamer 11y ago
  SCTP isn't implemented on most routers, firewalls. UDP is.
  - imglorp 11y ago
    SCTP is in Cisco IOS[1], but regardless, it sits at the IP layer so all routers need to do is pass it as IP, unless you want to do more. It's also in plenty of OS kernels already, such as solaris and linux, and can be written as userspace libraries. 1: http://www.cisco.com/c/en/us/td/docs/ios/12_4t/12_4t15/htsctp.html http://www.cisco.com/c/en/us/td/docs/ios/12_4t/12_4t15/htsct...
    - dragontamer 11y ago
      I'm talking about your typical Netgear router / firewall application. > it sits at the IP layer so all routers need to do is pass it as IP Erm... no. The vast majority of consumer electronics are built up from NAT. You are NOT getting past your Verizon/Comcast router with IP alone. It's getting interpreted as TCP, and then getting NATted over to something else. As far as i can tell, UDP has a similar process. But its a major problem with SCTP and consumer routers. Have you ever tried to send an SCTP connection across your typical consumer Verizon router?
      - wtallis 11y ago
        > You are NOT getting past your Verizon/Comcast router with IP alone. ... on IPv4. On IPv6 you don't have to worry about NAT, and IPv6 usage is growing: the percentage of Google traffic conducted over IPv6 has doubled in the past year and is significantly higher on weekends, which means consumers are driving a lot of that traffic. It's now over 7% on weekends and growing by about one percentage point every 3-4 months. That seems to me like a sufficiently large pool of potential testers. The NAT problem is really finally going away. What we need to be worrying about is dumb firewalls, but I'd bet that there will be just as many consumer routers that fail to make any of their firewall rules apply to IPv6 as there will be routers that can pass TCP over IPv6 but not arbitrary protocols.
    - acdha 11y ago
      The catch is that most NAT devices don't support it, which is a deal-breaker for something to be deployed on the public internet, and many firewalls either don't support it or would require firmware updates / policy changes to support it. There's a wicked chicken-and-egg problem deploying SCTP: a popular service would be a good reason to push vendors, enterprise firewall admins, etc. to support SCTP but it'd be financial suicide to build a general service which depends on it now.
- pthatcherg 11y ago
  I know that one big deal is fewer round trips. DTLS+SCTP requires several round trips to establish a secure connection (around 4 I think). DTLS 1.3 should improve that on the DTLS side of things, but QUIC is designed from the beginning to have 0 round trips for many cases, and 1 round trip in the worst case. For high RTT scenarios, that can make a big difference.
Scramblejams 11y ago
I can think of a lot of things this could be good for that aren't web servers. I didn't see it clearly treated in the documents at the linked site: is out of order packet delivery supported? I see they they use multiple streams to avoid head of line blocking, but within a stream, can packets be delivered out of order?
- aidenn0 11y ago
  This looks like purely a stream protocol, so at the transport layer, there are no "packets" to be delivered, just a stream of bytes. [edit] See SCTP for reliable multiplexed out-of-order datagrams.
  - Scramblejams 11y ago
    SCTP isn't friendly with cheap consumer routers, which makes it a non-starter for me, unfortunately.
    - aidenn0 11y ago
      https://tools.ietf.org/html/rfc6951 https://tools.ietf.org/html/rfc6951
- chetanahuja 11y ago
  What would be your use-case for reliable out-of-order packet delivery? Streaming/VOIP are convincing use-cases for "slightly unreliable" in-order delivery (that is, sacrifice reliability but not order). I'd love to hear your case for the revers (sacrifice order but not reliability).
boardwaalk 11y ago
I've played with QUIC [1] and SCTP [2]. QUIC does not have good documentation and SCTP's userlevel library feels like it's intended on being used a certain way (one-to-one sockets) and is easy to use incorrectly, which makes me uneasy about the quality of its code. I really wish there was a solid "TCP replacement" with optional reliability, optional ordering and multiplexing that I can run over real networks as user level code, among other features. But the view from the ground from what I've seen is that people either use TCP or build their own protocol that runs over UDP (I'm thinking about games specifically here). Am I missing an option here? Maybe I should dig into libquic and really figure out how to use it and if it can do what I want. [1] https://github.com/devsisters/libquic https://github.com/devsisters/libquic [2] https://code.google.com/p/sctp-refimpl/ https://code.google.com/p/sctp-refimpl/
- Scramblejams 11y ago
  Yeah, I have the same wish. JIC you haven't seen it, it covers some of the bases: http://enet.bespin.org/ http://enet.bespin.org/ IIRC I got frustrated with its (at least at the time) inability to send a packet while waiting on a packet to be received so I gave up and started writing my own spin on RFC 908.
  - corysama 11y ago
    IIRC, packets on different "channels" in enet don't block each other. So, a troublesome reliable packet that needs retransmitting unusually many times won't hold up packets on other channels -reliable or not. Edit: from the docs > To combat this latency and reduce the ordering restrictions on packets, ENet provides multiple channels of communication over a given connection. Each channel is independently sequenced, and so the delivery status of a packet in one channel will not stall the delivery of other packets in another channel.
    - Scramblejams 11y ago
      I was referring to their non-thread-safe (within a single ENet instance) architecture combined with the need to block if you want to pick up a packet the moment it arrives. I have to block to receive that packet, but while I'm blocking, I can't send a packet (because ENet isn't thread safe). Now, it will let you set a receive timeout, but then you just spend a lot of time spinning.
simfoo 11y ago
I've said it before and I'll say it again. If Google wants others to use QUIC they will have to stop pointing at their Chromium source tree and say "there it is". Provide a standalone library that implements QUIC for the major platforms and compiles without having to install custom build environments and people/projects will start using it.
- apendleton 11y ago
  I don't think they really want other people to use it, at least right now. I think they want the latitude to be able to make backwards-incompatible changes and only have to worry about roll-out on their own servers and to their own client software.
  - billyhoffman 11y ago
    Yes. This. It's the same model as SPDY. See a problem (HTTP1/x utilizing the network poorly). Implement a proprietary protocol from just your clients talking with your web properties. Work out kinks. Standardize as HTTP/2. I wonder what the HTTP/2 equivalent of QUIC will be. Interesting aside. As I typed that summary above. I couldn't help but think of Microsoft and how SPDY might still exists as a private "market leading feature" if Google sold server software...
- convivialdingo 11y ago
  And more to the point, it's a massive pain to track down all the outside header references to build a stand-along library. In all honesty, I gave up and iterated my own secure UDP protocol based on the excellent Micro Transport Protocol.
- DonPellegrino 11y ago
  This, I'd love to give it a try, but I'm not going to touch it until there's a Makefile and a minimal documentation. Until then, it feels like an engineering experiment, not a major company-backed project.
- lttlrck 11y ago
  There is a standalone test client & server [0] not quite a library but it looks manageable. And there is libquicc [1], though yes, it would be nice if this came from google. [0]http://src.chromium.org/viewvc/chrome/trunk/src/net/tools/quic/ http://src.chromium.org/viewvc/chrome/trunk/src/net/tools/qu... [1]https://github.com/devsisters/libquic https://github.com/devsisters/libquic
s800 11y ago
Mixing (doubling up) network layers like this is a bad plan. Yes, it would be a new protocol on ip, and would be less backward compatible. But if it's that important to do, then let's do it right. As long as we're probing to see if it's supported, then why not?
- wtallis 11y ago
  Especially since building atop IP instead of UDP will mean that you won't have to give up TCP's support for ECN.
- aidenn0 11y ago
  NAT traversal [edit] And implementation in userspace.
fredliu 11y ago
QUIC has come a (relative) long way: http://blog.chromium.org/2015/04/a-quic-update-on-googles-experimental.html http://blog.chromium.org/2015/04/a-quic-update-on-googles-ex...
- vvanders 11y ago
  That's a pretty awesome overview, I had no idea of the scale they are using it in the wild.
seoguru 11y ago
reminds me of this comment: "Of these, quic is my favorite option for the future. It is a truly phenmenal transport, with tons of things done very, very well. However, though the dragon is ready to hatch from its egg, it must still be unearthed from the enormous Code Caverns, of the Chromium Mountains, a quest not to be taken lightly." https://github.com/ipfs/go-ipfs/issues/57#issuecomment-70162015 https://github.com/ipfs/go-ipfs/issues/57#issuecomment-70162...

44 points by joubert 11y ago

I see FEC is mentioned; isn't most error-correction over noisy channels implemented at the link-layer? Even over e.g. wifi or cellular radios, my understanding is you typically either get the packet or not, with the internet layer rarely seeing corrupted packets. [edit] I read deeper in, and apparently, QUIC optionally allows sending packets in groups, with an additional packet that is the XOR of the other packets in the groups; this means that you can fill-in any single dropped packet by sending an extra packet ahead of time.

stormbrew 11y ago

Does anyone have a clear explanation for why QUIC exists when SCTP (which can be run over UDP) existed? What does QUIC bring over it?

wmf 11y ago
QUIC is supposed to be quicker than SCTP+TLS or SCTP+IPSec by using some "rampant layering violations" to combine security and transport into one layer and eliminate a lot of round trips.
waqf 11y ago
Your question is answered in the design doc at https://docs.google.com/document/d/1RNHkx_VvKWyWg6Lr8SZ-saqsQx7rFV-ev2jRFUoVD34/edit?pli=1#heading=h.nvk2biis6bpk https://docs.google.com/document/d/1RNHkx_VvKWyWg6Lr8SZ-saqs... tl;dr minimizing latency, particularly in establishing a connection, is an important design objective.
dragontamer 11y ago
SCTP isn't implemented on most routers, firewalls. UDP is.
- imglorp 11y ago
  SCTP is in Cisco IOS[1], but regardless, it sits at the IP layer so all routers need to do is pass it as IP, unless you want to do more. It's also in plenty of OS kernels already, such as solaris and linux, and can be written as userspace libraries. 1: http://www.cisco.com/c/en/us/td/docs/ios/12_4t/12_4t15/htsctp.html http://www.cisco.com/c/en/us/td/docs/ios/12_4t/12_4t15/htsct...
  - dragontamer 11y ago
    I'm talking about your typical Netgear router / firewall application. > it sits at the IP layer so all routers need to do is pass it as IP Erm... no. The vast majority of consumer electronics are built up from NAT. You are NOT getting past your Verizon/Comcast router with IP alone. It's getting interpreted as TCP, and then getting NATted over to something else. As far as i can tell, UDP has a similar process. But its a major problem with SCTP and consumer routers. Have you ever tried to send an SCTP connection across your typical consumer Verizon router?
    - wtallis 11y ago
      > You are NOT getting past your Verizon/Comcast router with IP alone. ... on IPv4. On IPv6 you don't have to worry about NAT, and IPv6 usage is growing: the percentage of Google traffic conducted over IPv6 has doubled in the past year and is significantly higher on weekends, which means consumers are driving a lot of that traffic. It's now over 7% on weekends and growing by about one percentage point every 3-4 months. That seems to me like a sufficiently large pool of potential testers. The NAT problem is really finally going away. What we need to be worrying about is dumb firewalls, but I'd bet that there will be just as many consumer routers that fail to make any of their firewall rules apply to IPv6 as there will be routers that can pass TCP over IPv6 but not arbitrary protocols.
  - acdha 11y ago
    The catch is that most NAT devices don't support it, which is a deal-breaker for something to be deployed on the public internet, and many firewalls either don't support it or would require firmware updates / policy changes to support it. There's a wicked chicken-and-egg problem deploying SCTP: a popular service would be a good reason to push vendors, enterprise firewall admins, etc. to support SCTP but it'd be financial suicide to build a general service which depends on it now.
pthatcherg 11y ago
I know that one big deal is fewer round trips. DTLS+SCTP requires several round trips to establish a secure connection (around 4 I think). DTLS 1.3 should improve that on the DTLS side of things, but QUIC is designed from the beginning to have 0 round trips for many cases, and 1 round trip in the worst case. For high RTT scenarios, that can make a big difference.

Scramblejams 11y ago

I can think of a lot of things this could be good for that aren't web servers. I didn't see it clearly treated in the documents at the linked site: is out of order packet delivery supported? I see they they use multiple streams to avoid head of line blocking, but within a stream, can packets be delivered out of order?

aidenn0 11y ago
This looks like purely a stream protocol, so at the transport layer, there are no "packets" to be delivered, just a stream of bytes. [edit] See SCTP for reliable multiplexed out-of-order datagrams.
- Scramblejams 11y ago
  SCTP isn't friendly with cheap consumer routers, which makes it a non-starter for me, unfortunately.
  - aidenn0 11y ago
    https://tools.ietf.org/html/rfc6951 https://tools.ietf.org/html/rfc6951
chetanahuja 11y ago
What would be your use-case for reliable out-of-order packet delivery? Streaming/VOIP are convincing use-cases for "slightly unreliable" in-order delivery (that is, sacrifice reliability but not order). I'd love to hear your case for the revers (sacrifice order but not reliability).

boardwaalk 11y ago

I've played with QUIC [1] and SCTP [2]. QUIC does not have good documentation and SCTP's userlevel library feels like it's intended on being used a certain way (one-to-one sockets) and is easy to use incorrectly, which makes me uneasy about the quality of its code. I really wish there was a solid "TCP replacement" with optional reliability, optional ordering and multiplexing that I can run over real networks as user level code, among other features. But the view from the ground from what I've seen is that people either use TCP or build their own protocol that runs over UDP (I'm thinking about games specifically here). Am I missing an option here? Maybe I should dig into libquic and really figure out how to use it and if it can do what I want. [1] https://github.com/devsisters/libquic https://github.com/devsisters/libquic [2] https://code.google.com/p/sctp-refimpl/ https://code.google.com/p/sctp-refimpl/

Scramblejams 11y ago
Yeah, I have the same wish. JIC you haven't seen it, it covers some of the bases: http://enet.bespin.org/ http://enet.bespin.org/ IIRC I got frustrated with its (at least at the time) inability to send a packet while waiting on a packet to be received so I gave up and started writing my own spin on RFC 908.
- corysama 11y ago
  IIRC, packets on different "channels" in enet don't block each other. So, a troublesome reliable packet that needs retransmitting unusually many times won't hold up packets on other channels -reliable or not. Edit: from the docs > To combat this latency and reduce the ordering restrictions on packets, ENet provides multiple channels of communication over a given connection. Each channel is independently sequenced, and so the delivery status of a packet in one channel will not stall the delivery of other packets in another channel.
  - Scramblejams 11y ago
    I was referring to their non-thread-safe (within a single ENet instance) architecture combined with the need to block if you want to pick up a packet the moment it arrives. I have to block to receive that packet, but while I'm blocking, I can't send a packet (because ENet isn't thread safe). Now, it will let you set a receive timeout, but then you just spend a lot of time spinning.

simfoo 11y ago

I've said it before and I'll say it again. If Google wants others to use QUIC they will have to stop pointing at their Chromium source tree and say "there it is". Provide a standalone library that implements QUIC for the major platforms and compiles without having to install custom build environments and people/projects will start using it.

apendleton 11y ago
I don't think they really want other people to use it, at least right now. I think they want the latitude to be able to make backwards-incompatible changes and only have to worry about roll-out on their own servers and to their own client software.
- billyhoffman 11y ago
  Yes. This. It's the same model as SPDY. See a problem (HTTP1/x utilizing the network poorly). Implement a proprietary protocol from just your clients talking with your web properties. Work out kinks. Standardize as HTTP/2. I wonder what the HTTP/2 equivalent of QUIC will be. Interesting aside. As I typed that summary above. I couldn't help but think of Microsoft and how SPDY might still exists as a private "market leading feature" if Google sold server software...
convivialdingo 11y ago
And more to the point, it's a massive pain to track down all the outside header references to build a stand-along library. In all honesty, I gave up and iterated my own secure UDP protocol based on the excellent Micro Transport Protocol.
DonPellegrino 11y ago
This, I'd love to give it a try, but I'm not going to touch it until there's a Makefile and a minimal documentation. Until then, it feels like an engineering experiment, not a major company-backed project.
lttlrck 11y ago
There is a standalone test client & server [0] not quite a library but it looks manageable. And there is libquicc [1], though yes, it would be nice if this came from google. [0]http://src.chromium.org/viewvc/chrome/trunk/src/net/tools/quic/ http://src.chromium.org/viewvc/chrome/trunk/src/net/tools/qu... [1]https://github.com/devsisters/libquic https://github.com/devsisters/libquic

s800 11y ago

Mixing (doubling up) network layers like this is a bad plan. Yes, it would be a new protocol on ip, and would be less backward compatible. But if it's that important to do, then let's do it right. As long as we're probing to see if it's supported, then why not?

wtallis 11y ago
Especially since building atop IP instead of UDP will mean that you won't have to give up TCP's support for ECN.
aidenn0 11y ago
NAT traversal [edit] And implementation in userspace.

fredliu 11y ago

QUIC has come a (relative) long way: http://blog.chromium.org/2015/04/a-quic-update-on-googles-experimental.html http://blog.chromium.org/2015/04/a-quic-update-on-googles-ex...

vvanders 11y ago
That's a pretty awesome overview, I had no idea of the scale they are using it in the wild.

seoguru 11y ago

reminds me of this comment: "Of these, quic is my favorite option for the future. It is a truly phenmenal transport, with tons of things done very, very well. However, though the dragon is ready to hatch from its egg, it must still be unearthed from the enormous Code Caverns, of the Chromium Mountains, a quest not to be taken lightly." https://github.com/ipfs/go-ipfs/issues/57#issuecomment-70162015 https://github.com/ipfs/go-ipfs/issues/57#issuecomment-70162...