8 ms·

Tcpls: Modern Transport Services with TCP and TLS

101 points by enz 4y ago

simmervigor 4y ago
Some related discussion in this Twitter thread: https://twitter.com/alagoutte/status/1532013841718120449?t=W0Z7TeyPltRNM3p2XpkXMQ&s=19 https://twitter.com/alagoutte/status/1532013841718120449?t=W...
bediger4000 4y ago
Lots of dupes: https://news.ycombinator.com/item?id=31589575 https://news.ycombinator.com/item?id=31589575 https://news.ycombinator.com/item?id=31569066 https://news.ycombinator.com/item?id=31569066
- deleted 4y ago
  [deleted]
- pvg 4y ago
  They need to have some discussion to count as HN-dupes and those don't.
  - bediger4000 4y ago
    That seems a little wrong - I can just spam uninteresting, yet topical things to HN, and no negative consequences accrue? People will see the headline at least, maybe click on the article. I get the clicks every time, and I get to maybe repeat a Big Lie over and over, which will help my goal of framing the discussion my way. But thanks for the correction! I appreciate it.
    - pvg 4y ago
      There are other mechanisms that handle things like spamming and even copycat links.
scarmig 4y ago
https://dl.acm.org/doi/10.1145/3485983.3494865 https://dl.acm.org/doi/10.1145/3485983.3494865 seems like a more understandable source.
- metadat 4y ago
  This link was much easier for me to read and parse, thank you scarming.
throwaway787544 4y ago
I would appreciate it if the industry would stop trying to shove everything into user space. Imagine it's 40 years in the future, and we still have to kludge new protocols over an 80 year old TCP/IP stack because nobody wants to add a new protocol to the standard. Every application ships with its own unique protocol stack implementation. Duplicate code abounds. Bugs are fixed haphazardly. Quality and stability varies by application. RCEs are more common because an increase in implementations increases the probability of security bugs. Compare that to finalizing a new protocol, putting it in the kernel, and letting all applications just connect() and send() and recv(). We lose the ability to constantly mutate the protocol whenever FAANG feels like it, but we also gain the ability for every application to use the new protocol just by changing a flag passed to a syscall.
- xxpor 4y ago
  Pushing it to the kernel means you have to upgrade your kernel to use it. Instead, you can just include a library that exposes a send() and recv() style API to the main app code, while the library just throws everything into a UDP socket. Seems better to me, IMO. The real issue is that we can't get away from UDP/TCP because of middleboxes (firewalls, load balancers, etc) and the fact that switches only know how to hash on 5 tuples of protocols they know about. If you're lucky, your router will be able to look inside a GRE header or something, but that's about it. When was the last time you used SCTP, for example? (If you've used it recently, I bet it was tunneled over UDP)
  - kazen44 4y ago
    technically speaking, switches do not hash TCP information but hash based on data in the Ethernet frame. (this is mainly used for correctly balancing load across links in an lacp bundle). High end routers are able to do pretty conplex lookups into their asics, but this usually only extends to specific use cases. (as far as I am aware for instance, an juniper MX series is able to do a lookup of both vlan tags in a single asic pipeline step when using q-in-q. the same also applies to mpls stacked lable lookups on provider edge routers).
    - 4y ago
a-dub 4y ago
interesting debate whether encryption should occur at the network vs. transport layer. on the one hand, encryption tends to have session state, so the transport layer makes sense but on the other i often feel i'd like to see a new version of IP where encryption is baked in as a primary design principle thereby eliminating all notions of leaks.
- zamadatix 4y ago
  IPsec tunnel mode delivers that function https://datatracker.ietf.org/doc/html/rfc4303 https://datatracker.ietf.org/doc/html/rfc4303 This contrasts to IPsec transport mode which only encrypts the payload not the IP info https://upload.wikimedia.org/wikipedia/commons/6/64/Ipsec-esp-tunnel-and-transport.svg https://upload.wikimedia.org/wikipedia/commons/6/64/Ipsec-es... Generally there is encryption available at each layer in the network stack but middleware (operating systems, firewalls, NAT) tend to get in the way at lower layers. E.g. IPsec is often now run with NAT traversal which means the ESP header is stuck under a UDP header so it can get through to end devices even though IPsec itself doesn't require this. This, and a few other things, have left IPsec as a thing most people only know for corporate VPNs.
  - a-dub 4y ago
    i'm envisioning more of a sort of ipvx, 64 bit addresses, encryption is non-optional and dos resistance is built-in.
    - zamadatix 4y ago
      It's mandatory to support in IPv6 so the only thing lacking in the general population's mind is a convincing enough reason for people to force it for every connection which will cause a lot of breakage due to the aforementioned middleboxes. But if you can convince that there's nothing about existing IPv6 implementation that would prevent it from working as described today. Switching to IPvX would be the same problem as convincing people to always encrypt with IPv6 just amplified as there is 0 hardware with existing support for such a protocol and IPv6 has had a 20 years head start on rollout that would be wiped away. It'd have the added "bonus" of being even harder to pitch adoption for since "NAT on IPv4 is working fine" and you'd still have to wait for those people THEN the IPv6 people to migrate. Meanwhile things like QUIC, HTTP/3, and TCPLS give all the same advantages as migrating to the hypothetical IPvX without breaking when they run into middleboxes. Just think of it as IPvX but with a wider header field than you were planning. After all you'll still need some header that tells the network where to deliver the packet then the encrypted payload it's just that that dumb destination header uses more bits than it needs to if it were designed fresh.
_8j50 4y ago
TCP is terrible at long distance data transfer. I am sure some people will still prefer it since it can manage congestion but shouldn't newer things use UDP and handle delivery guarantees at the application layer?
- gsliepen 4y ago
  There is nothing inherently wrong with modern TCP for long distance data transfer. You do need to use the right congestion algorithm and some extensions enabled to handle large window sizes to get high speeds at long distances. But UDP does not magically solve anything; if you want reliable data transfer over UDP you have to basically implement almost everything TCP does on top of it. The main benefit is that you are no longer relying on the operating system's implementation of TCP, the drawback is that you won't get any help from the OS or even hardware.
  - _8j50 4y ago
    I thought you can't keep increasing window sizes, it is a physics limit? The mere fact that you need to wait for an ACK is the problem? https://en.m.wikipedia.org/wiki/Bandwidth-delay_product https://en.m.wikipedia.org/wiki/Bandwidth-delay_product
    - zinekeller 4y ago
      As GP said: > if you want reliable data transfer over UDP you have to basically implement almost everything TCP does on top of it. Even QUIC has an ACK-like signal so you will never avoid it as otherwise you will deal with the two generals' problem (https://en.wikipedia.org/wiki/Two_Generals%27_Problem https://en.wikipedia.org/wiki/Two_Generals%27_Problem). The reason QUIC is more efficient than TCP is simply due to the fact that TCP can't evolve further due to middleboxes (called ossification, https://en.wikipedia.org/wiki/Protocol_ossification https://en.wikipedia.org/wiki/Protocol_ossification), not that TCP is inherently bad.
      - _8j50 4y ago
gsliepen 4y ago
This is quite interesting. The only issue I see is when you want to stream real-time audio and video over TCPLS. That is when you want packets that are dropped, late or reordered to not interfere with the delivery of other packets. Using TCP as the underlying transport makes that problematic, and even having multiple TCP connections cannot help much with that.
ThePhysicist 4y ago
QUIC's connection migration feature is really amazing for applications where you switch IPs, e.g. when your phone goes from mobile data to Wifi. Another nice thing is that you can cache sessions and resume them later, so connections can even survive reboots and server failures, or you can move them between different machines. TCPLS seems like a good proposal but I won't hold my breath for it as the big players like Cloudflare, Google, Akamai, Fastly are already committed to QUIC. There's still significant churn in the proposals and some features like MASQUE are still being finalized, but overall QUIC works really well.
- stormbrew 4y ago
  I'm not a huge fan of how spdy and then quic were kinda ramrodded into the HTTP standards process, but realistically I think QUIC is basically the future of connection transport wherever tcp or udp alone are failing at this point. I recently discovered the nQuick proposal to do noise-based connection initiation for peer-to-peer connections and I love it[1]. It's an incredibly versatile protocol. It seems likely that a lot of the issues raised in the OP will be somewhat forced to be resolved due to the aforementioned ramrodding of QUIC into the HTTP3 path, but maybe in the long run the two ideas will co-evolve to the same solution. [1] https://eprint.iacr.org/2019/028.pdf https://eprint.iacr.org/2019/028.pdf
  - unmole 4y ago
    > quic were kinda ramrodded into the HTTP standards process QUIC took 4 years and 34 drafts to get standardized. The final IETF version is significantly different from Google initially came up with. So, no it most definitely was not ramrodded through.
    - stormbrew 4y ago
      I didn't say quic itself was -- if quic had been introduced completely independently of http then sure, that's fine. But it was introduced intermingled with making yet another drastic change to a backbone protocol of the internet. Considering HTTP/1.1 stood for over 15 years before http/2 came along (and was a pretty modest improvement over http/1.0 compared to what we're talking about now, so I'm being generous by not saying over 20 years). http/2 had only been promoted to a proposed standard for 3 before http/3 was proposed. That, to me, is a very aggressive process. But again, it's the relationship to http that bothers me, not the standardization process for quic itself.
Matthias247 4y ago
Some thoughts from someone working on QUIC for a couple of years: There's some good ideas in this one! The idea of just keeping a TCP stream per Tcpls stream while sharing some crypto state means it can be really efficient, since all optimizations that exist for TCP (including hardware offloads) will continue to work. That is also visible in the efficiency benchmarks [1] on the site. One might even argue that with such a design where the most expensive parts of TLS (handshakes) become no-ops an introduce no additional latency, techniques like HTTP/2 become unnecessary - one could keep just having a single TCP stream per request and enjoy better flow control, fairness and observability than with anything application managed. However there are some challenges that need to be resolved for running protocols on "real-world infrastructure and datacenters", which change the picture a bit. One is that a single IP address announced by large-scale webservices doesn't necessarily point to a single server, but to e.g. a rack of them or even bigger units. This means that a second TCP connection might (or even should be) established to a different server - which would not be aware about the TLS session state. If the state is kept on a different server, the original one would need to forward all packets either via IP-in-IP encapsulation or by creating a separate upstream TCP connection and proxying all data bidirectionally. All those approaches would be fairly expensive - every time a TLS session would be joined the amount of packets handled by a server would be double of the ideal version. While a similar challenge exists for QUIC, it's actually cheaper on average: Since multiple streams use the same QUIC connection and usually also the same source/destination IP and port pairs, traffic would usually be directed to the same host by common infrastructure and require no special handling. It would only require effort to "fix" misdirected QUIC connections in the event of connection migrations - which are rather rare. If packet forwarding is required, it can be achieved rather cheap and easily by interpreting the QUIC connection ID fields in all UDP packets and doing raw UDP packet forwarding. Therefore as far as I understand the proposals so far, it might not be necessarily clear-cut on whether running QUIC or TCPLS would be cheaper and an ingress or load-balancing layer - which are the layers where both protocols have the biggest benefits due to the highest amount of concurrent connections and higher client-facing RTT. But maybe the challenge for TCPLS could be somehow solved by having a shared place for the TLS state between hosts and making sure streams are operating fully independently - but it requires a bit more thoughts. I'm wondering how close such a proposal is then to the existing TLS session resumption mechanisms - just extended by allowing more than 1 follow-up session. [1] Note that the diagrams are really about efficiency and not performance. They tell nothing about what throughput might be available for end-users that are e.g. 100ms away from the server. For local tests the delay and packet introduced by large-scale internet infrastructure doesn't play a role.

101 points by enz 4y ago

Some related discussion in this Twitter thread: https://twitter.com/alagoutte/status/1532013841718120449?t=W0Z7TeyPltRNM3p2XpkXMQ&s=19 https://twitter.com/alagoutte/status/1532013841718120449?t=W...

bediger4000 4y ago

Lots of dupes: https://news.ycombinator.com/item?id=31589575 https://news.ycombinator.com/item?id=31589575 https://news.ycombinator.com/item?id=31569066 https://news.ycombinator.com/item?id=31569066

deleted 4y ago
[deleted]
pvg 4y ago
They need to have some discussion to count as HN-dupes and those don't.
- bediger4000 4y ago
  That seems a little wrong - I can just spam uninteresting, yet topical things to HN, and no negative consequences accrue? People will see the headline at least, maybe click on the article. I get the clicks every time, and I get to maybe repeat a Big Lie over and over, which will help my goal of framing the discussion my way. But thanks for the correction! I appreciate it.
  - pvg 4y ago
    There are other mechanisms that handle things like spamming and even copycat links.

scarmig 4y ago

https://dl.acm.org/doi/10.1145/3485983.3494865 https://dl.acm.org/doi/10.1145/3485983.3494865 seems like a more understandable source.

metadat 4y ago
This link was much easier for me to read and parse, thank you scarming.

throwaway787544 4y ago

I would appreciate it if the industry would stop trying to shove everything into user space. Imagine it's 40 years in the future, and we still have to kludge new protocols over an 80 year old TCP/IP stack because nobody wants to add a new protocol to the standard. Every application ships with its own unique protocol stack implementation. Duplicate code abounds. Bugs are fixed haphazardly. Quality and stability varies by application. RCEs are more common because an increase in implementations increases the probability of security bugs. Compare that to finalizing a new protocol, putting it in the kernel, and letting all applications just connect() and send() and recv(). We lose the ability to constantly mutate the protocol whenever FAANG feels like it, but we also gain the ability for every application to use the new protocol just by changing a flag passed to a syscall.

xxpor 4y ago
Pushing it to the kernel means you have to upgrade your kernel to use it. Instead, you can just include a library that exposes a send() and recv() style API to the main app code, while the library just throws everything into a UDP socket. Seems better to me, IMO. The real issue is that we can't get away from UDP/TCP because of middleboxes (firewalls, load balancers, etc) and the fact that switches only know how to hash on 5 tuples of protocols they know about. If you're lucky, your router will be able to look inside a GRE header or something, but that's about it. When was the last time you used SCTP, for example? (If you've used it recently, I bet it was tunneled over UDP)
- kazen44 4y ago
  technically speaking, switches do not hash TCP information but hash based on data in the Ethernet frame. (this is mainly used for correctly balancing load across links in an lacp bundle). High end routers are able to do pretty conplex lookups into their asics, but this usually only extends to specific use cases. (as far as I am aware for instance, an juniper MX series is able to do a lookup of both vlan tags in a single asic pipeline step when using q-in-q. the same also applies to mpls stacked lable lookups on provider edge routers).
  - 4y ago

a-dub 4y ago

interesting debate whether encryption should occur at the network vs. transport layer. on the one hand, encryption tends to have session state, so the transport layer makes sense but on the other i often feel i'd like to see a new version of IP where encryption is baked in as a primary design principle thereby eliminating all notions of leaks.

zamadatix 4y ago
IPsec tunnel mode delivers that function https://datatracker.ietf.org/doc/html/rfc4303 https://datatracker.ietf.org/doc/html/rfc4303 This contrasts to IPsec transport mode which only encrypts the payload not the IP info https://upload.wikimedia.org/wikipedia/commons/6/64/Ipsec-esp-tunnel-and-transport.svg https://upload.wikimedia.org/wikipedia/commons/6/64/Ipsec-es... Generally there is encryption available at each layer in the network stack but middleware (operating systems, firewalls, NAT) tend to get in the way at lower layers. E.g. IPsec is often now run with NAT traversal which means the ESP header is stuck under a UDP header so it can get through to end devices even though IPsec itself doesn't require this. This, and a few other things, have left IPsec as a thing most people only know for corporate VPNs.
- a-dub 4y ago
  i'm envisioning more of a sort of ipvx, 64 bit addresses, encryption is non-optional and dos resistance is built-in.
  - zamadatix 4y ago
    It's mandatory to support in IPv6 so the only thing lacking in the general population's mind is a convincing enough reason for people to force it for every connection which will cause a lot of breakage due to the aforementioned middleboxes. But if you can convince that there's nothing about existing IPv6 implementation that would prevent it from working as described today. Switching to IPvX would be the same problem as convincing people to always encrypt with IPv6 just amplified as there is 0 hardware with existing support for such a protocol and IPv6 has had a 20 years head start on rollout that would be wiped away. It'd have the added "bonus" of being even harder to pitch adoption for since "NAT on IPv4 is working fine" and you'd still have to wait for those people THEN the IPv6 people to migrate. Meanwhile things like QUIC, HTTP/3, and TCPLS give all the same advantages as migrating to the hypothetical IPvX without breaking when they run into middleboxes. Just think of it as IPvX but with a wider header field than you were planning. After all you'll still need some header that tells the network where to deliver the packet then the encrypted payload it's just that that dumb destination header uses more bits than it needs to if it were designed fresh.

_8j50 4y ago

TCP is terrible at long distance data transfer. I am sure some people will still prefer it since it can manage congestion but shouldn't newer things use UDP and handle delivery guarantees at the application layer?

gsliepen 4y ago
There is nothing inherently wrong with modern TCP for long distance data transfer. You do need to use the right congestion algorithm and some extensions enabled to handle large window sizes to get high speeds at long distances. But UDP does not magically solve anything; if you want reliable data transfer over UDP you have to basically implement almost everything TCP does on top of it. The main benefit is that you are no longer relying on the operating system's implementation of TCP, the drawback is that you won't get any help from the OS or even hardware.
- _8j50 4y ago
  I thought you can't keep increasing window sizes, it is a physics limit? The mere fact that you need to wait for an ACK is the problem? https://en.m.wikipedia.org/wiki/Bandwidth-delay_product https://en.m.wikipedia.org/wiki/Bandwidth-delay_product
  - zinekeller 4y ago
    As GP said: > if you want reliable data transfer over UDP you have to basically implement almost everything TCP does on top of it. Even QUIC has an ACK-like signal so you will never avoid it as otherwise you will deal with the two generals' problem (https://en.wikipedia.org/wiki/Two_Generals%27_Problem https://en.wikipedia.org/wiki/Two_Generals%27_Problem). The reason QUIC is more efficient than TCP is simply due to the fact that TCP can't evolve further due to middleboxes (called ossification, https://en.wikipedia.org/wiki/Protocol_ossification https://en.wikipedia.org/wiki/Protocol_ossification), not that TCP is inherently bad.
    - _8j50 4y ago

gsliepen 4y ago

This is quite interesting. The only issue I see is when you want to stream real-time audio and video over TCPLS. That is when you want packets that are dropped, late or reordered to not interfere with the delivery of other packets. Using TCP as the underlying transport makes that problematic, and even having multiple TCP connections cannot help much with that.

ThePhysicist 4y ago

QUIC's connection migration feature is really amazing for applications where you switch IPs, e.g. when your phone goes from mobile data to Wifi. Another nice thing is that you can cache sessions and resume them later, so connections can even survive reboots and server failures, or you can move them between different machines. TCPLS seems like a good proposal but I won't hold my breath for it as the big players like Cloudflare, Google, Akamai, Fastly are already committed to QUIC. There's still significant churn in the proposals and some features like MASQUE are still being finalized, but overall QUIC works really well.

stormbrew 4y ago
I'm not a huge fan of how spdy and then quic were kinda ramrodded into the HTTP standards process, but realistically I think QUIC is basically the future of connection transport wherever tcp or udp alone are failing at this point. I recently discovered the nQuick proposal to do noise-based connection initiation for peer-to-peer connections and I love it[1]. It's an incredibly versatile protocol. It seems likely that a lot of the issues raised in the OP will be somewhat forced to be resolved due to the aforementioned ramrodding of QUIC into the HTTP3 path, but maybe in the long run the two ideas will co-evolve to the same solution. [1] https://eprint.iacr.org/2019/028.pdf https://eprint.iacr.org/2019/028.pdf
- unmole 4y ago
  > quic were kinda ramrodded into the HTTP standards process QUIC took 4 years and 34 drafts to get standardized. The final IETF version is significantly different from Google initially came up with. So, no it most definitely was not ramrodded through.
  - stormbrew 4y ago
    I didn't say quic itself was -- if quic had been introduced completely independently of http then sure, that's fine. But it was introduced intermingled with making yet another drastic change to a backbone protocol of the internet. Considering HTTP/1.1 stood for over 15 years before http/2 came along (and was a pretty modest improvement over http/1.0 compared to what we're talking about now, so I'm being generous by not saying over 20 years). http/2 had only been promoted to a proposed standard for 3 before http/3 was proposed. That, to me, is a very aggressive process. But again, it's the relationship to http that bothers me, not the standardization process for quic itself.

Matthias247 4y ago

Some thoughts from someone working on QUIC for a couple of years: There's some good ideas in this one! The idea of just keeping a TCP stream per Tcpls stream while sharing some crypto state means it can be really efficient, since all optimizations that exist for TCP (including hardware offloads) will continue to work. That is also visible in the efficiency benchmarks [1] on the site. One might even argue that with such a design where the most expensive parts of TLS (handshakes) become no-ops an introduce no additional latency, techniques like HTTP/2 become unnecessary - one could keep just having a single TCP stream per request and enjoy better flow control, fairness and observability than with anything application managed. However there are some challenges that need to be resolved for running protocols on "real-world infrastructure and datacenters", which change the picture a bit. One is that a single IP address announced by large-scale webservices doesn't necessarily point to a single server, but to e.g. a rack of them or even bigger units. This means that a second TCP connection might (or even should be) established to a different server - which would not be aware about the TLS session state. If the state is kept on a different server, the original one would need to forward all packets either via IP-in-IP encapsulation or by creating a separate upstream TCP connection and proxying all data bidirectionally. All those approaches would be fairly expensive - every time a TLS session would be joined the amount of packets handled by a server would be double of the ideal version. While a similar challenge exists for QUIC, it's actually cheaper on average: Since multiple streams use the same QUIC connection and usually also the same source/destination IP and port pairs, traffic would usually be directed to the same host by common infrastructure and require no special handling. It would only require effort to "fix" misdirected QUIC connections in the event of connection migrations - which are rather rare. If packet forwarding is required, it can be achieved rather cheap and easily by interpreting the QUIC connection ID fields in all UDP packets and doing raw UDP packet forwarding. Therefore as far as I understand the proposals so far, it might not be necessarily clear-cut on whether running QUIC or TCPLS would be cheaper and an ingress or load-balancing layer - which are the layers where both protocols have the biggest benefits due to the highest amount of concurrent connections and higher client-facing RTT. But maybe the challenge for TCPLS could be somehow solved by having a shared place for the TLS state between hosts and making sure streams are operating fully independently - but it requires a bit more thoughts. I'm wondering how close such a proposal is then to the existing TLS session resumption mechanisms - just extended by allowing more than 1 follow-up session. [1] Note that the diagrams are really about efficiency and not performance. They tell nothing about what throughput might be available for end-users that are e.g. 100ms away from the server. For local tests the delay and packet introduced by large-scale internet infrastructure doesn't play a role.