5 ms·

OpenMPTCProuter: Aggregate and encrypt multiple internet connections using MPTCP

170 points by rzk 2y ago

fefferkorn 2y ago
Love it, but aggregating different connections (latency, LTE vs Adsl, fiber) is hard. Tried different algos, but always had network hogs or even higher ping that slowest connection. What made it work best (and rock stable) was using LTE only, or ADSL only connections having same ping to aggregator (VPS) and have the VPS as near as possible. (latency wise) I did not had the time to set up multiple redundant aggregators, so my SPOF was the VPS some times. Maybe there is a solution out there. So far my experience a year ago. Our provider then offered a way cheaper managed solution, thats why we stopped using it.
elnappo 2y ago
TCP Over TCP Is A Bad Idea https://web.archive.org/web/20230228035749/http://sites.inka.de/%7EW1011/devel/tcp-tcp.html https://web.archive.org/web/20230228035749/http://sites.inka...
- chgs 2y ago
  It’s odd > If MPTCP is not supported, OpenMPTCProuter can also use Multi-link VPN (MLVPN) or Glorytun UDP with multipath support. It’s unclear to me from a brief look why they use mptcp as a backend for bonding
- riobard 2y ago
  MPTCP isn't TCP over TCP.
- ranger_danger 2y ago
  I don't doubt that it can be a problem in some cases with bad connections but personally I have never had any issues using TCP-based (TLS) VPNs in the 15 years I have been using them, even with MPTCP, which I routinely use to bond several different connections together.
kodama-lens 2y ago
In my last year of university (5 years ago) I took a networking seminar. Each student took a look at a different technology to utilize multiple links for internet data transfers. Initially I was amazed by MPTCP and wondered why it had so little adoption. As I looked into the papers I slowly figured out why. With different links (WLAN, LAN, LTE) their real world characteristics are too different for efficient aggregation. It is the head of line blocking problem times ten. It might be fine as a back up link, but there are other problems like the limit to TCP and middelboxes dropping unknowns packets. The challenges outnumber the benefits for consumers and in data centers there are other technologies to aggregate links that operate on a level below TCP.
- riobard 2y ago
  I hope QUIC with its many advanced features gets better adoption to provide many of the benefits so we can just stop messing with TCP for it's completely screwed up by middleboxes.
  - chgs 2y ago
    Middle boxes simply block QUIC so that won’t help
    - vlovich123 2y ago
      No they do not unless they’ve been specifically updated to do so since QUIC is just UDP. From Google’s experiments very few middleware had problems such that they made QUIC impossible. That’s why Chrome has been using QUiC by default to Google services for a decade or maybe even slightly more. And given that it’s the next evolution of TCP and a requirement for HTTP/3 it seems like the opposite direction of better QUIC compatibility is likely.
      - chgs 2y ago
        If you have a middle box in the way fiddling with your traffic then bypassing it is just a policy matter. Companies don’t put these things inline for fun, they do it because they want to block traffic. Allowing bypassing it breaks their policy. If you own your own network then you don’t need to worry about middle boxes interfering with quic or http.
Y_Y 2y ago
What's a Prouter?
- riobard 2y ago
  It's Open-MPTCP-router.
- sans_souse 2y ago
  Rasprouter the bruter! I musta got lost
ajb 2y ago
Cool, but needs a VPS. A simpler approach is to load balance/ fail over individual TCP/UDP flows, eg using mwan3 I'm hoping that with QUIC, there will be a way to use it's migration feature to load balance more accurately (no need to wait for new flows to start). But, right now there is no way for a middlebox to tell if the server end of an individual flow supports migration, as this is only visible to the client.
steelbrain 2y ago
I used this when I was in Tallinn. Place I was in didnt have gigabit fiber (probably the only place on the whole street). It worked well for me. I was using copper internet from local ISP paired with cellular and starlink. Starlink went out for 30 seconds every few minutes but when it worked, it was the fastest of the bunch. I rented a cheap VPS in the city to use as the other end of this router. The setup worked well overall, I was getting work done along with downloading games with sizes above 100G without worrying too much
ivan4th 2y ago
I used OpenMPTCPRouter to aggregate 3 LTE connections (via routers connected to directional antenna, with SIM cards from different operators) when I was living in a house in the woods before the war has started I had to leave Russia. Worked like a charm, giving me up to 180 Mbps or so. May not be that good for aggregating different types of links together, but for using multiple cellular connections it's nearly a perfect solution. BTW 5G 3GPP specs include MPTCP support, IIRC for aggregating connections going via different gNodeBs (base stations)
edude03 2y ago
I used this product when I lived in a building where I could get a 500mb and a 100mbit connection but not a gigabit connection. I had the server side setup on digital ocean and it "worked" in so far as I could pull 600mbit/s but was impractical since 1) I would get the latency of the worse (and in my case more variable) connection and 2) since it was essentially a VPN to a cloud provider many sites blocked me as a suspected bot.
andrewmackrodt 2y ago
I've been using this for around 6 months now to aggregate a 70 mbit FTTC connection and 150-450 mbit 5G connection; overall it works pretty well. The FTTC connection is the "master" link and it seems is preferred for the first several bytes of the connection, so the latency is better than using the 5G connection directly. This provides a nice balance for general web browsing, as loading web pages is still quite quick and overall download speed is more than either connection alone. In my setup I'd say it's about 80% efficient in terms of aggregating the total download. There are a lot of configuration options and the stability of them can be an issue. I've found using XRay VLESS for the Proxy and Glorytun TCP for VPN to offer the best overall speed and reliability. (Edit: I've disabled SQM too). It's possible to mostly bypass the VPN by enabling proxy UDP over XRay but I found that breaks various bypass rules, e.g. so that Netflix connections always go over 5G, as some content has restriction from being accessed by my VPS data center IP. Port forwarding is also a bit hit and miss; I have configured my 2 WAN routers to use the OMR router as a DMZ and then if I want to play a game, or enable remote access, I will use the bypass feature so that the device's MAC goes through my FTTC connection. UPnP works correctly in this scenario which is handy for consoles.
- dfc 2y ago
  I've never heard of XRay or V2Ray. It seems like a niche thing?
  - andrewmackrodt 2y ago
    Unfortunately I don't really understand the various protocols OMR supports, so my experience comes from measuring aggregated speed/latency and stability. XRay worked best for me and also supported QUIC if enabling proxy UDP over XRay/V2Ray. However, due to the omr-bypass issue, I've disabled that option and QUIC (which is the default setting).
  - ranger_danger 2y ago
    They are extremely popular in the "niche" of censorship evasion especially in countries like China and Russia. There are many more such protocols as well like OBFS/Shadowsocks/Snowflake/Meek etc.
NelsonMinar 2y ago
When Starlink was new I really wanted channel bonding to take advantage of its speed an work around the early beta unreliability. I ended up using Speedify which has a really nice desktop client implementation. But it only works for one computer, it's more like a VPN client. This system was the best option for a full network for a router. Fortunately Starlink got more reliable so I stopped needing it. Bonding disparate network connections has a lot of really funky behaviors. In practice the biggest problem is the Speedify VPNs kept getting flagged as spammy: running your own OpenMPTCrouter endpoint fixes that.
S3raph 2y ago
I've been using it for a few years, and it's an awesome solution if you have slow or flaky network connections. The project is great, however, it takes some time to find the best configuration. I'm not sure about the latest version, but I didn’t have a great experience with versions above 0.60 and still stick to 0.59. I also recommend saving your working configuration once you have it, as a few changes can mess up the system—probably due to a bug.
ggm 2y ago
I got hit by MPTCP shifting gigs of data from hetzner Germany to Australia. The solution was to intrude a relay host on the west coast. It's great tech which can reach out and bite you in the bum at random. In my case it was a 10x drop in throughput by out of order packet arrival and effects on window and flow control.
jedisct1 2y ago
With support for Glorytun. Very cool!
loongloong 2y ago
Would it be more practical to consider MPTCP use in a smaller network such as across servers and racks in a data center /room while using QUIC (and MP-QUIC) over WAN / Internet?

170 points by rzk 2y ago

Love it, but aggregating different connections (latency, LTE vs Adsl, fiber) is hard. Tried different algos, but always had network hogs or even higher ping that slowest connection. What made it work best (and rock stable) was using LTE only, or ADSL only connections having same ping to aggregator (VPS) and have the VPS as near as possible. (latency wise) I did not had the time to set up multiple redundant aggregators, so my SPOF was the VPS some times. Maybe there is a solution out there. So far my experience a year ago. Our provider then offered a way cheaper managed solution, thats why we stopped using it.

elnappo 2y ago

TCP Over TCP Is A Bad Idea https://web.archive.org/web/20230228035749/http://sites.inka.de/%7EW1011/devel/tcp-tcp.html https://web.archive.org/web/20230228035749/http://sites.inka...

chgs 2y ago
It’s odd > If MPTCP is not supported, OpenMPTCProuter can also use Multi-link VPN (MLVPN) or Glorytun UDP with multipath support. It’s unclear to me from a brief look why they use mptcp as a backend for bonding
riobard 2y ago
MPTCP isn't TCP over TCP.
ranger_danger 2y ago
I don't doubt that it can be a problem in some cases with bad connections but personally I have never had any issues using TCP-based (TLS) VPNs in the 15 years I have been using them, even with MPTCP, which I routinely use to bond several different connections together.

kodama-lens 2y ago

In my last year of university (5 years ago) I took a networking seminar. Each student took a look at a different technology to utilize multiple links for internet data transfers. Initially I was amazed by MPTCP and wondered why it had so little adoption. As I looked into the papers I slowly figured out why. With different links (WLAN, LAN, LTE) their real world characteristics are too different for efficient aggregation. It is the head of line blocking problem times ten. It might be fine as a back up link, but there are other problems like the limit to TCP and middelboxes dropping unknowns packets. The challenges outnumber the benefits for consumers and in data centers there are other technologies to aggregate links that operate on a level below TCP.

riobard 2y ago
I hope QUIC with its many advanced features gets better adoption to provide many of the benefits so we can just stop messing with TCP for it's completely screwed up by middleboxes.
- chgs 2y ago
  Middle boxes simply block QUIC so that won’t help
  - vlovich123 2y ago
    No they do not unless they’ve been specifically updated to do so since QUIC is just UDP. From Google’s experiments very few middleware had problems such that they made QUIC impossible. That’s why Chrome has been using QUiC by default to Google services for a decade or maybe even slightly more. And given that it’s the next evolution of TCP and a requirement for HTTP/3 it seems like the opposite direction of better QUIC compatibility is likely.
    - chgs 2y ago
      If you have a middle box in the way fiddling with your traffic then bypassing it is just a policy matter. Companies don’t put these things inline for fun, they do it because they want to block traffic. Allowing bypassing it breaks their policy. If you own your own network then you don’t need to worry about middle boxes interfering with quic or http.

Y_Y 2y ago

What's a Prouter?

riobard 2y ago
It's Open-MPTCP-router.
sans_souse 2y ago
Rasprouter the bruter! I musta got lost

ajb 2y ago

Cool, but needs a VPS. A simpler approach is to load balance/ fail over individual TCP/UDP flows, eg using mwan3 I'm hoping that with QUIC, there will be a way to use it's migration feature to load balance more accurately (no need to wait for new flows to start). But, right now there is no way for a middlebox to tell if the server end of an individual flow supports migration, as this is only visible to the client.

steelbrain 2y ago

I used this when I was in Tallinn. Place I was in didnt have gigabit fiber (probably the only place on the whole street). It worked well for me. I was using copper internet from local ISP paired with cellular and starlink. Starlink went out for 30 seconds every few minutes but when it worked, it was the fastest of the bunch. I rented a cheap VPS in the city to use as the other end of this router. The setup worked well overall, I was getting work done along with downloading games with sizes above 100G without worrying too much

ivan4th 2y ago

I used OpenMPTCPRouter to aggregate 3 LTE connections (via routers connected to directional antenna, with SIM cards from different operators) when I was living in a house in the woods before the war has started I had to leave Russia. Worked like a charm, giving me up to 180 Mbps or so. May not be that good for aggregating different types of links together, but for using multiple cellular connections it's nearly a perfect solution. BTW 5G 3GPP specs include MPTCP support, IIRC for aggregating connections going via different gNodeBs (base stations)

edude03 2y ago

I used this product when I lived in a building where I could get a 500mb and a 100mbit connection but not a gigabit connection. I had the server side setup on digital ocean and it "worked" in so far as I could pull 600mbit/s but was impractical since 1) I would get the latency of the worse (and in my case more variable) connection and 2) since it was essentially a VPN to a cloud provider many sites blocked me as a suspected bot.

andrewmackrodt 2y ago

I've been using this for around 6 months now to aggregate a 70 mbit FTTC connection and 150-450 mbit 5G connection; overall it works pretty well. The FTTC connection is the "master" link and it seems is preferred for the first several bytes of the connection, so the latency is better than using the 5G connection directly. This provides a nice balance for general web browsing, as loading web pages is still quite quick and overall download speed is more than either connection alone. In my setup I'd say it's about 80% efficient in terms of aggregating the total download. There are a lot of configuration options and the stability of them can be an issue. I've found using XRay VLESS for the Proxy and Glorytun TCP for VPN to offer the best overall speed and reliability. (Edit: I've disabled SQM too). It's possible to mostly bypass the VPN by enabling proxy UDP over XRay but I found that breaks various bypass rules, e.g. so that Netflix connections always go over 5G, as some content has restriction from being accessed by my VPS data center IP. Port forwarding is also a bit hit and miss; I have configured my 2 WAN routers to use the OMR router as a DMZ and then if I want to play a game, or enable remote access, I will use the bypass feature so that the device's MAC goes through my FTTC connection. UPnP works correctly in this scenario which is handy for consoles.

dfc 2y ago
I've never heard of XRay or V2Ray. It seems like a niche thing?
- andrewmackrodt 2y ago
  Unfortunately I don't really understand the various protocols OMR supports, so my experience comes from measuring aggregated speed/latency and stability. XRay worked best for me and also supported QUIC if enabling proxy UDP over XRay/V2Ray. However, due to the omr-bypass issue, I've disabled that option and QUIC (which is the default setting).
- ranger_danger 2y ago
  They are extremely popular in the "niche" of censorship evasion especially in countries like China and Russia. There are many more such protocols as well like OBFS/Shadowsocks/Snowflake/Meek etc.

NelsonMinar 2y ago

When Starlink was new I really wanted channel bonding to take advantage of its speed an work around the early beta unreliability. I ended up using Speedify which has a really nice desktop client implementation. But it only works for one computer, it's more like a VPN client. This system was the best option for a full network for a router. Fortunately Starlink got more reliable so I stopped needing it. Bonding disparate network connections has a lot of really funky behaviors. In practice the biggest problem is the Speedify VPNs kept getting flagged as spammy: running your own OpenMPTCrouter endpoint fixes that.

S3raph 2y ago

I've been using it for a few years, and it's an awesome solution if you have slow or flaky network connections. The project is great, however, it takes some time to find the best configuration. I'm not sure about the latest version, but I didn’t have a great experience with versions above 0.60 and still stick to 0.59. I also recommend saving your working configuration once you have it, as a few changes can mess up the system—probably due to a bug.

ggm 2y ago

I got hit by MPTCP shifting gigs of data from hetzner Germany to Australia. The solution was to intrude a relay host on the west coast. It's great tech which can reach out and bite you in the bum at random. In my case it was a 10x drop in throughput by out of order packet arrival and effects on window and flow control.

jedisct1 2y ago

With support for Glorytun. Very cool!

loongloong 2y ago

Would it be more practical to consider MPTCP use in a smaller network such as across servers and racks in a data center /room while using QUIC (and MP-QUIC) over WAN / Internet?