5 ms·

Going multipath without Multipath TCP

114 points by FiloSottile 5y ago

est 5y ago
This is the future of censorship-resistant network. Middleman only happens because every info is dumped inside one series of packets transferred as a single src-dst ip:port connection. Crack one stream and you get all. If we somehow split bytes and bits and shuffled them into multiple ip:port connections, span on multiple ISPs, then middleboxes are rendered useless. Targeting one TCP stream is easy, but correctly select M from N connections is nearly impossible.
- eru 5y ago
  Most real world censorship doesn't seem to be happen at the level you are describing as a major weakness here, does it?
  - est 5y ago
    It's the link layer. You can't defeat censorship without the physical ground work
brenns10 5y ago
I worked with the MPTCP Linux implementation before it was upstreamed. One of the exciting things about it was that it was the default: IPPROTO_TCP would attempt an MPTCP handshake. This meant that any unmodified application that had used TCP could inherit the magic of connection bonding and bandwidth aggregation, when used with an MPTCP capable server. Browser downloads, SSH connections, iperf, you name it! It seems that one of the concessions necessary to get MPTCP upstreamed was to make it opt-in. Even if the distribution enables MPTCP in its config, and if the sysctl parameter disabling it is enabled, the applications must STILL request it via IPPROTO_MPTCP. It was probably the correct choice, but to me it destroyed a major benefit of MPTCP. If it can't (easily) be used to enable unmodified applications to reap its benefits, then why was it written as a standards-based, backward compatible extension to TCP? There were so many extra hurdles to overcome by adopting those constraints. Modern protocol development, especially at the "transport" layer (though usually based on UDP), is easiest done in userspace these days. This enables easier experimentation, iteration, and delivering updates to your users. See Chrome and SPDY/QUIC. (Incidentally, I recall some discussion on "multipath QUIC", wonder where that led to). To bring this comment back to the article, I love it. It sidesteps the protocol development hurdles and just says "if we're going to require modified clients and servers anyway, let's shove it into the application layer in a library". I'm sure approaches like this could handle all the same use cases: - maximizing throughput of multiple interfaces, as the article does - connection handover (by making a new connection over a new IP/interface before breaking the old one) - reducing latency by racing the same data across all interfaces - using metered (e.g. LTE) connections as backup for poorly performing default connections Of course, applications would need to be modified to use such a library, but that's what the community seems to have decided is the way forward regardless :/
- grayfaced 5y ago
  I didn't realize it was such a drop-in replacement. Couldn't you then use a LD_PRELOAD to translate TCP into MPTCP? Then applications wouldn't need to be modified.
  - brenns10 5y ago
    It's probably possible to do that, assuming you have a kernel properly configured and the sysctls and privileges established. I had no problems using the out-of-tree MPTCP kernel for a variety of TCP based tasks, so most applications had no issue with it. Of course then the next requirement is a server which supports MPTCP as well... which means you need at least two hosts with very specific kernels, and maybe LD_PRELOAD setups :/ And to be honest, there are good reasons not to enable MPTCP by default. For one, there's security consequences. "Hi Bob, my name is Alice, I have a totally different IP address but would like to join into this conversation as well..." MPTCP has protections for this based on session keys that are exchanged. But for many people, it's still a concern. Another good reason is that most applications don't benefit from multiple paths, so it's a lot of extra complexity and packets being sent for no benefit. And finally, there are many possible use cases for multipath reliable sockets, as I detailed above. Each requires different policy (to determine when to make new connections across different IP addresses, or how to schedule data to transmit across each subflow). It's unlikely that an unmodified application would get the kind of behavior that the user desires. How do you decide whether you should treat one subflow as a metered "backup" connection, or whether you should be blasting data across all subflows? This could all be configured by user who has dutifully read enough manual pages and implemented enough netlink applications (or BPF scripts?) to control the kernel's behavior, I'm sure. So, the moral of the story to me is that, drop-in replacement was an exciting goal, and something that felt important. Losing it definitely meant that MPTCP was doomed to not reach critical mass, and probably won't see widespread deployment. But losing it also means that we can see applications fine-tune a solution like this one for their use case, which feels like a win.
    - brenns10 5y ago
      To expand on this - if you are going to mess with LD_PRELOAD, why not stick this whole library behind LD_PRELOAD? You get similar levels of compatibility (except for the odd app that shirks libc and hand-codes their system calls...). But you also get the ability to do userspace protocol development, and you have an easier time of configuring the policy for how you handle new connections and subflows.
ec109685 5y ago
Great post! One quibble is that it says this: > This extension is currently used sparsely, with the only two commonly deployed uses being the OpenMTCPRouter Project and Apple’s Siri Siri was the first application supported, but now it’s available more broadly on iOS: https://developer.apple.com/documentation/foundation/urlsessionconfiguration/improving_network_reliability_using_multipath_tcp https://developer.apple.com/documentation/foundation/urlsess...
- pouulet 5y ago
  There is much more than "two commonly deployed used" of MPTCP, to name a few : OpenMPTCPRouter is an improved fork from OVH Overthebox project ( https://www.ovhtelecom.fr/overthebox/ https://www.ovhtelecom.fr/overthebox/ ), Tessares is a company that provide key in hand MPTCP solutions for many ISPs ( https://www.tessares.net/customers https://www.tessares.net/customers ). MPTCP was designed to be transparent for the end user, so you might be using it without knowing it, from your phone, your computer or your internet box
justinsaccount 5y ago
This is interesting. I was thinking of doing a similar thing but running it on top of quic instead of tcp. Furthest I got was having a somewhat working `quicat`.

114 points by FiloSottile 5y ago

est 5y ago

This is the future of censorship-resistant network. Middleman only happens because every info is dumped inside one series of packets transferred as a single src-dst ip:port connection. Crack one stream and you get all. If we somehow split bytes and bits and shuffled them into multiple ip:port connections, span on multiple ISPs, then middleboxes are rendered useless. Targeting one TCP stream is easy, but correctly select M from N connections is nearly impossible.

eru 5y ago
Most real world censorship doesn't seem to be happen at the level you are describing as a major weakness here, does it?
- est 5y ago
  It's the link layer. You can't defeat censorship without the physical ground work

brenns10 5y ago

I worked with the MPTCP Linux implementation before it was upstreamed. One of the exciting things about it was that it was the default: IPPROTO_TCP would attempt an MPTCP handshake. This meant that any unmodified application that had used TCP could inherit the magic of connection bonding and bandwidth aggregation, when used with an MPTCP capable server. Browser downloads, SSH connections, iperf, you name it! It seems that one of the concessions necessary to get MPTCP upstreamed was to make it opt-in. Even if the distribution enables MPTCP in its config, and if the sysctl parameter disabling it is enabled, the applications must STILL request it via IPPROTO_MPTCP. It was probably the correct choice, but to me it destroyed a major benefit of MPTCP. If it can't (easily) be used to enable unmodified applications to reap its benefits, then why was it written as a standards-based, backward compatible extension to TCP? There were so many extra hurdles to overcome by adopting those constraints. Modern protocol development, especially at the "transport" layer (though usually based on UDP), is easiest done in userspace these days. This enables easier experimentation, iteration, and delivering updates to your users. See Chrome and SPDY/QUIC. (Incidentally, I recall some discussion on "multipath QUIC", wonder where that led to). To bring this comment back to the article, I love it. It sidesteps the protocol development hurdles and just says "if we're going to require modified clients and servers anyway, let's shove it into the application layer in a library". I'm sure approaches like this could handle all the same use cases: - maximizing throughput of multiple interfaces, as the article does - connection handover (by making a new connection over a new IP/interface before breaking the old one) - reducing latency by racing the same data across all interfaces - using metered (e.g. LTE) connections as backup for poorly performing default connections Of course, applications would need to be modified to use such a library, but that's what the community seems to have decided is the way forward regardless :/

grayfaced 5y ago
I didn't realize it was such a drop-in replacement. Couldn't you then use a LD_PRELOAD to translate TCP into MPTCP? Then applications wouldn't need to be modified.
- brenns10 5y ago
  It's probably possible to do that, assuming you have a kernel properly configured and the sysctls and privileges established. I had no problems using the out-of-tree MPTCP kernel for a variety of TCP based tasks, so most applications had no issue with it. Of course then the next requirement is a server which supports MPTCP as well... which means you need at least two hosts with very specific kernels, and maybe LD_PRELOAD setups :/ And to be honest, there are good reasons not to enable MPTCP by default. For one, there's security consequences. "Hi Bob, my name is Alice, I have a totally different IP address but would like to join into this conversation as well..." MPTCP has protections for this based on session keys that are exchanged. But for many people, it's still a concern. Another good reason is that most applications don't benefit from multiple paths, so it's a lot of extra complexity and packets being sent for no benefit. And finally, there are many possible use cases for multipath reliable sockets, as I detailed above. Each requires different policy (to determine when to make new connections across different IP addresses, or how to schedule data to transmit across each subflow). It's unlikely that an unmodified application would get the kind of behavior that the user desires. How do you decide whether you should treat one subflow as a metered "backup" connection, or whether you should be blasting data across all subflows? This could all be configured by user who has dutifully read enough manual pages and implemented enough netlink applications (or BPF scripts?) to control the kernel's behavior, I'm sure. So, the moral of the story to me is that, drop-in replacement was an exciting goal, and something that felt important. Losing it definitely meant that MPTCP was doomed to not reach critical mass, and probably won't see widespread deployment. But losing it also means that we can see applications fine-tune a solution like this one for their use case, which feels like a win.
  - brenns10 5y ago
    To expand on this - if you are going to mess with LD_PRELOAD, why not stick this whole library behind LD_PRELOAD? You get similar levels of compatibility (except for the odd app that shirks libc and hand-codes their system calls...). But you also get the ability to do userspace protocol development, and you have an easier time of configuring the policy for how you handle new connections and subflows.

ec109685 5y ago

Great post! One quibble is that it says this: > This extension is currently used sparsely, with the only two commonly deployed uses being the OpenMTCPRouter Project and Apple’s Siri Siri was the first application supported, but now it’s available more broadly on iOS: https://developer.apple.com/documentation/foundation/urlsessionconfiguration/improving_network_reliability_using_multipath_tcp https://developer.apple.com/documentation/foundation/urlsess...

pouulet 5y ago
There is much more than "two commonly deployed used" of MPTCP, to name a few : OpenMPTCPRouter is an improved fork from OVH Overthebox project ( https://www.ovhtelecom.fr/overthebox/ https://www.ovhtelecom.fr/overthebox/ ), Tessares is a company that provide key in hand MPTCP solutions for many ISPs ( https://www.tessares.net/customers https://www.tessares.net/customers ). MPTCP was designed to be transparent for the end user, so you might be using it without knowing it, from your phone, your computer or your internet box

justinsaccount 5y ago

This is interesting. I was thinking of doing a similar thing but running it on top of quic instead of tcp. Furthest I got was having a somewhat working `quicat`.