4 ms·

Tailscale is P2P which is nicer than a VPS as a hub and spoke approach. But one thing that Tailscale didn't do well (at least early on) is performance. It's us

by jamra 3y ago

Tailscale is P2P which is nicer than a VPS as a hub and spoke approach. But one thing that Tailscale didn't do well (at least early on) is performance. It's user space Go, which seemed to cap the data transfers when I tested it out. I would prefer a really fast data transfer P2P so I could use Tailscale in between my web server and DB.

klabb3 3y ago
> It's user space Go, which seemed to cap the data transfers when I tested it out. Compared to another VPN? I’d be curious to know whether the kernel mode byte shuffling solves that problem. But even so, a kernel module is a pretty big ask only for connectivity. In my experience, UDP in general isn’t as performant in practice as one would think. Not saying you can’t push the limits, and even outperform TCP, but to do so with a reliable cross platform way isn’t exactly trivial today. All my benchmarks (albeit user space) have shown that pushing bytes over udp has a higher CPU overhead, and that’s even if you omit retransmission, congestion control, etc etc (ie just push garbage bytes). And even if your cpu can handle the throughput, the congestion control can still bite you for god knows what reason. When I ran quic benchmarks they got deprioritized in the presence of tcp traffic. Don’t know all the reasons why (sorry, just didn’t have the time) but at least to me TCP wins the bang-for-the-buck-throughput-on-commodity-hardware category, hands down. Maybe this changes with platform-specific optimized vectored IO, but that alone would be a huge effort. No, the more time I spent on it, the more I appreciated all the things TCP gives me for free, and it’s remarkable resilience in complex conditions. I am also happy I don’t have to worry about bulky 3p libraries. This is what the OS is supposed to do, imo. So this UDP-hype-renaissance we’ve seen over the last years is a bit premature or at least not as obvious as people hoped (including myself). Another fun fact (for anyone who read this far): contrary to popular belief p2p TCP isn’t harder to do than UDP, not really.
- Tor3 3y ago
  The problem with TCP compared to UDP is when you do VPN over links where there's some amount of round trip time. I routinely run VPN over a 300+ ms ping times, and any TCP-based VPN suffers dramatically when doing a TCP connection through that TCP-based VPN. Switch to a UDP-based VPN and the problem disappears (easy to test by using OpenVPN and switch the configuration between TCP and UDP. But I've tested with other types of VPN as well). When you're closer to home the problem disappears.
  - klabb3 3y ago
    Thanks, yes this makes a ton of sense. Another thing I wondered about is how much CPU overhead VPNs add, and how it performs when maximizing throughput. (Not Tailscale but a “regular” one with kernel packet switching). Do you have any experience with that?
    - Tor3 3y ago
      I only use OpenVPN regularly. My internet connection is either 100Mbit or 1Gb both up and down, but depending on where I am the actual external bandwidth varies - mostly it's around 50Mbit end-to-end, subjectively. If I send data at max speed through the network then I observe that OpenVPN may use quite a bit of CPU (maybe up to around 40% of one core (i7-7500U), but it doesn't limit the transfer speed I get compared to when I do a direct transfer without VPN (interestingly, on long latency lines (when I go OpenVPN from Japan to Europe) I often get better and more consistent performance when going through OpenVPN (configured to use UDP).
- benmmurphy 3y ago
  usually VPNs in linux push IP (TUN) or ethernet (TAP) frames between the nodes so you really need to be using UDP or else you are going to have problems with running TCP over TCP and the congestion algorithms conflicting with each other. openvpn which supports TCP refers to this problem as TCP meltdown and advise using UDP where possible (https://openvpn.net/faq/what-is-tcp-meltdown/ https://openvpn.net/faq/what-is-tcp-meltdown/) VPNs that use TCP as the transport layer could try and special case TCP handling and treat them as a flow and just transport the TCP data streams instead of the IP packets but you would still be left with issues when you are transferring UDP across TCP which is not ideal.
  - klabb3 3y ago
    Thanks! I don’t have a lot of experience with VPNs but for sure UDP is much better suited for packets which is the abstraction layer that VPNs operate on. > VPNs that use TCP as the transport layer could try and special case TCP handling and treat them as a flow and just transport the TCP data streams instead Yes! An interesting observation is that TCP composes really well, ie relaying works excellent. However, for VPNs it’d be nesting TCP which melts down quickly.
5e92cb50239222b 3y ago
netbird is quite similar and uses kernel WireGuard if one of the peers has a publicly accessible IP, or both are on the same subnet.
- jamra 3y ago
  I wonder how hard it would be to create a STUN server for netbird.
nekonek2 3y ago
>> But one thing that Tailscale didn't do well (at least early on) is performance. AFAIK currently user space wireguard-go is faster then kernel implementation due to improvements[1] that landed there such as Generic receive offload (GRO) and TCP Segmentation Offload (TSO). [1] https://tailscale.com/blog/throughput-improvements/ https://tailscale.com/blog/throughput-improvements/

by jamra 3y ago

klabb3 3y ago

> It's user space Go, which seemed to cap the data transfers when I tested it out. Compared to another VPN? I’d be curious to know whether the kernel mode byte shuffling solves that problem. But even so, a kernel module is a pretty big ask only for connectivity. In my experience, UDP in general isn’t as performant in practice as one would think. Not saying you can’t push the limits, and even outperform TCP, but to do so with a reliable cross platform way isn’t exactly trivial today. All my benchmarks (albeit user space) have shown that pushing bytes over udp has a higher CPU overhead, and that’s even if you omit retransmission, congestion control, etc etc (ie just push garbage bytes). And even if your cpu can handle the throughput, the congestion control can still bite you for god knows what reason. When I ran quic benchmarks they got deprioritized in the presence of tcp traffic. Don’t know all the reasons why (sorry, just didn’t have the time) but at least to me TCP wins the bang-for-the-buck-throughput-on-commodity-hardware category, hands down. Maybe this changes with platform-specific optimized vectored IO, but that alone would be a huge effort. No, the more time I spent on it, the more I appreciated all the things TCP gives me for free, and it’s remarkable resilience in complex conditions. I am also happy I don’t have to worry about bulky 3p libraries. This is what the OS is supposed to do, imo. So this UDP-hype-renaissance we’ve seen over the last years is a bit premature or at least not as obvious as people hoped (including myself). Another fun fact (for anyone who read this far): contrary to popular belief p2p TCP isn’t harder to do than UDP, not really.

Tor3 3y ago
The problem with TCP compared to UDP is when you do VPN over links where there's some amount of round trip time. I routinely run VPN over a 300+ ms ping times, and any TCP-based VPN suffers dramatically when doing a TCP connection through that TCP-based VPN. Switch to a UDP-based VPN and the problem disappears (easy to test by using OpenVPN and switch the configuration between TCP and UDP. But I've tested with other types of VPN as well). When you're closer to home the problem disappears.
- klabb3 3y ago
  Thanks, yes this makes a ton of sense. Another thing I wondered about is how much CPU overhead VPNs add, and how it performs when maximizing throughput. (Not Tailscale but a “regular” one with kernel packet switching). Do you have any experience with that?
  - Tor3 3y ago
    I only use OpenVPN regularly. My internet connection is either 100Mbit or 1Gb both up and down, but depending on where I am the actual external bandwidth varies - mostly it's around 50Mbit end-to-end, subjectively. If I send data at max speed through the network then I observe that OpenVPN may use quite a bit of CPU (maybe up to around 40% of one core (i7-7500U), but it doesn't limit the transfer speed I get compared to when I do a direct transfer without VPN (interestingly, on long latency lines (when I go OpenVPN from Japan to Europe) I often get better and more consistent performance when going through OpenVPN (configured to use UDP).
benmmurphy 3y ago
usually VPNs in linux push IP (TUN) or ethernet (TAP) frames between the nodes so you really need to be using UDP or else you are going to have problems with running TCP over TCP and the congestion algorithms conflicting with each other. openvpn which supports TCP refers to this problem as TCP meltdown and advise using UDP where possible (https://openvpn.net/faq/what-is-tcp-meltdown/ https://openvpn.net/faq/what-is-tcp-meltdown/) VPNs that use TCP as the transport layer could try and special case TCP handling and treat them as a flow and just transport the TCP data streams instead of the IP packets but you would still be left with issues when you are transferring UDP across TCP which is not ideal.
- klabb3 3y ago
  Thanks! I don’t have a lot of experience with VPNs but for sure UDP is much better suited for packets which is the abstraction layer that VPNs operate on. > VPNs that use TCP as the transport layer could try and special case TCP handling and treat them as a flow and just transport the TCP data streams instead Yes! An interesting observation is that TCP composes really well, ie relaying works excellent. However, for VPNs it’d be nesting TCP which melts down quickly.

5e92cb50239222b 3y ago

netbird is quite similar and uses kernel WireGuard if one of the peers has a publicly accessible IP, or both are on the same subnet.

jamra 3y ago
I wonder how hard it would be to create a STUN server for netbird.

nekonek2 3y ago

>> But one thing that Tailscale didn't do well (at least early on) is performance. AFAIK currently user space wireguard-go is faster then kernel implementation due to improvements[1] that landed there such as Generic receive offload (GRO) and TCP Segmentation Offload (TSO). [1] https://tailscale.com/blog/throughput-improvements/ https://tailscale.com/blog/throughput-improvements/