8 ms·

Easily accessing all your stuff with a zero-trust mesh VPN

177 points by JNRowe 4y ago

AHOHA 4y ago
Nice list, zerotier does offer sharing as long as you add it to the network, or even having it public, but keep in mind it will share everything as if you’re normally connected with other members in the network, or create a separate network for the sharing purposes.
- tiernano 4y ago
  Their rules option allow you to lock down access
viraptor 4y ago
> I wouldn’t want to have a lot of programs broadcasting on a slow link. While in theory this could let you run Netware or DECNet across Zerotier There are less obscure uses for broadcast in those networks. For example mDNS "just works". You may not have an automated DNS at the service level, but you can use avahi to discover your other services or use .local without any extra config.
goodpoint 4y ago
What about legal liability? Forwarding somebody else's traffic can be very risky. Other than that, yggdrasil and tinc sound so much better than the alternatives.
chaxor 4y ago
Headscale seems like the best answer in this space
- linsomniac 4y ago
  Headscale is very good, but a few caveats: There's really no redundancy. If your headscale node goes down it can take your tailnet with it. In particular, if you drop in some ACLs that have issues, your entire tailnet can drop until you get it fixed. The recent (0.21) "configtest" can help there, but it still feels a bit brittle. Headscale can use a lot of system resources. ~100 nodes can saturate a t3a.small instance in cpu time and disc access. Reducing the update frequency can help, but there are hard limits here. I'm imagining much of this is database updates to sqlite, but I haven't tried switching to an external postgres server yet to see how much of the load is database related.
alchemist1e9 4y ago
tinc performance is limited by the encryption performance on single core. I suspect for Linux network engineers it could be a better choice than author suggests. tinc is very mature and battle tested.
- password4321 4y ago
  I don't know much about tinc for mobile devices, are there decent clients?
  - alchemist1e9 4y ago
    I don’t use it on mobile devices so not sure. Instead I use a cloud server which joins the tinc mesh and and also runs wireguard service which then acts as a router.
  - wkat4242 4y ago
    There's a great client for Android yes. https://tincapp.pacien.org/ https://tincapp.pacien.org/ For iOS there isn't. Last time I checked there's only one without a GUI that needs root. PS tinc is not fully zero trust. Every node can connect to every other one. This includes vps nodes you'll probably use for firewall traversal. Other systems have a 'lighthouse' concept where the vps just coordinates traffic but isn't able to actually read it.
    - progval 4y ago
      > Every node can connect to every other one. At the Tinc level, they can't connect to you unless you have their public key configured locally. At the IP level, set 'StrictSubnets = yes' in the main config file to prevent nodes not explicitly configured locally to send packets to you.
plaguuuuuu 4y ago
semi-related - does anyone know of any software frameworks/libraries for writing IoT event based systems stuff on this sort of mesh network? I have a side project in mind where IoT devices (dropping in and out of 4G and/or ad-hoc wifi or bluetooth) need to communicate and send/receive events, reconciling with event sourcing or whatever The tough parts of the architecture for this seem pretty generic so I'm hopeful that something already exists!
- codegangsta 4y ago
  Have you looked at nats.io? High performance messaging with a flexible persistence layer. Servers can be meshed via leaf node connections. Perfect for IoT and edge
  - piceas 4y ago
    Works nicely with zerotier to simplify testing it out.
- jacquesm 4y ago
  Mail me?
- somat 4y ago
  It's not lightweight and it's not exactly a mesh network, that is, it's not a level 2 mesh it is level 5 mesh so you still need to get an ip somehow. but ipfs exposes it's dht for general purpose use via it's pubsub sytem. I had a lot of fun abusing this to make a serverless[1] video streaming platform. The video streaming ended up being shit with fundamental issues I don't think I could solve. but it was fun to put together the proof of concept. 1. actually serverless. not "the server is in the cloud" bullshit people usually mean when they say serverless.
  - bboygravity 4y ago
    Sounds like https://lbry.com/ https://lbry.com/ ?? How did they solve or work around what you couldn't solve? Curious due to some deeply engrained dislike of anything centralized :p
jrm4 4y ago
Long-time tinc user here. My gut is that tinc is going to be as good as anything else here -- specifically what this means is that his "for your friends" idea doesn't hold much water for me. I.e. -- either you have friends techy enough such that you could get tinc going, or you don't, and at this level ALL of these solutions are going to be too annoying and complicated for them? For the non-techy types, I don't think any VPN like thing has been built yet? Would have to be like "syncthing" level easy.
- remram 4y ago
  hamachi was pretty close, we used it back in the days to play LAN games without having to enter IP addresses or mess with routers/modems.
- imiric 4y ago
  I'm a big fan of tinc. It's fully private and doesn't rely on public servers, like some of the mentioned solutions. Adding new peers is a minor chore, but if you have a network that doesn't change frequently, this is not an issue. Also unlike most of the alternatives, it runs absolutely everywhere. I had it setup on my pfSense router, and on my Android phone, and it worked great. Not sure why the article says that mobile support is iffy. I also used it professionally to setup a small Docker Swarm network over the internet. It also worked without issues, and was relatively simple to setup and maintain. Performance could be an issue, as the article mentions, but unless you're transferring large files, in practice it's good enough. I'm currently looking into setting this up again for personal use, and was hoping that there would be native mesh support within Wireguard, which doesn't seem to be the case.
  - jrm4 4y ago
    Right? Honestly, I'm no expert, but I coded up a little crappy "good enough" onboarding shell script for myself (for adding new clients). I don't know if something similar exists already, but it seems like it could be done.
HereBeBeasties 4y ago
Twingate is missing from this list. It has nice DNS-based routing and access control.
jis 4y ago
Another tool worth looking at is vpncloud (https://github.com/dswd/vpncloud https://github.com/dswd/vpncloud). I used to use tinc, but switched to vpncloud 2 years ago. In my use case, I have a modest number of nodes. Although nodes learn of other nodes from each other, I use ansible to keep each node's config updated. I use vpncloud (and previously, tinc) between docker hosts. So, you have to be careful about interface MTU's inside of docker, particularly if you use containers based on Alpine.
aborsy 4y ago
What’s the problem with all devices connecting to a vpn server running somewhere? I mean, the usual access VPN solution. It’s not peer to peer, but you can set up or rent a server near clients, like in the same city.
- wmf 4y ago
  A hub-and-spoke VPN is slower and it can be more expensive. It's easier if you're configuring each tunnel by hand, but now that mesh VPNs are easy to use you might as well use mesh.
  - SOLAR_FIELDS 4y ago
    It’s also very much a reduction of what some of these technologies are capable of. For instance, I’ve used Netmaker somewhat extensively as a VPN solution, and I mostly agree with the sentiments that the original article expresses about Netmaker (young, unstable). However, that is not the most interesting use case that Netmaker offers at all. Rather, a much more interesting use case is something like having an easy and straightforward way for Kube pods in AWS and GCP spanning across multiple regions to talk to each other. Multi-cloud architectures are greatly simplified by these kinds of technologies, especially Netmaker, partly because of their abilities to punch through NATs. Also, Netmaker is kernel level WireGuard which is much more performant. If you’re just using Tailscale to talk to clusters from your MacBook, for instance, userland WireGuard is probably going to be more than enough. When you need servers to talk to each other, though, you usually need the extra perf
  - linsomniac 4y ago
    It's not necessarily slower. It really depends on the connectivity of the VPN hub. At my previous company our VPN server was on a VERY good network, including InterNAP traffic optimizers. One of our employees ran some testing of it and found that in most cases going over the VPN was faster than natively using his Comcast connection. Comcast would tend to route the traffic over their own network links as much as possible. In the case of the VPN server, it was well enough connected that Comcast could get to it quickly, and the good connectivity it had (largely thanks to Level-3, but the InterNAP optimizer would help it pick great connections in any case), would get it to the final destination faster than Comcast's network could.
deleted 4y ago
[deleted]
nbk_2000 4y ago
Privacy VPN providers are in a unique position to offer this as an add-on. NordVPN recently started offering Meshnet which effectively accomplishes the same thing as Tailscale, Zerotier etc. I vaguely recall another privacy VPN provider (maybe PIA) used to offer this as well. Seems like a good value add-on for services that already require setting up encrypted tunnels.
- e12e 4y ago
  Interesting - i wasn't aware of that: https://nordvpn.com/blog/nordlynx-protocol-wireguard/ https://nordvpn.com/blog/nordlynx-protocol-wireguard/
  - nbk_2000 4y ago
    Here's a more direct link to what I was referring to: https://nordvpn.com/meshnet/ https://nordvpn.com/meshnet/
mjg59 4y ago
The point of zero-trust is that you don't implicitly trust something, but instead re-evaluate its trustworthiness every time you have to make an access control decision. Trusting a device simply because it's on your VPN is equivalent to trusting a device simply because it's on your LAN, and this is absolutely not a zero-trust architecture.
- barnabee 4y ago
  It's zero trust networking, unlike the internet or a traditional VPN.
  - mjg59 4y ago
    And every definition of "zero trust networking" I can find still states that the point is that trust is something that's continually evaluated, which isn't the case if you're trusting someone simply because they're on your VPN.
    - barnabee 4y ago
      Tailscale describe zero trust networking more eloquently than I can, here: https://tailscale.com/kb/1123/zero-trust/ https://tailscale.com/kb/1123/zero-trust/ The gist being that you are no longer making unverifiable trust assumptions about the identity of nodes connected to your network, as you have encrypted links between all nodes (not just between external nodes and a VPN gaetway to a non-zero-trust internal LAN) based on their verified identity and allowed access. A node's identity can be verified by its public key, so you no longer implicitly need to trust that only known and expected nodes are plugged into the switch in the office (for example). I do not have to care about whether the physical network infrastructure can be trusted (zero trust) because I can verify the identity of all nodes. This of course does not mean your entire architecture is zero trust. There are pretty much always trust decisions and assumptions somewhere, but that is why it's described as zero trust [mesh] networking, not zero trust complete end to end systems. edit/aside: some people prefer the term "trust minimising" rather than zero trust to emphasise that there's always trust somewhere and you are just pushing it outside of some component or layer, and I agree this is probably a better/more accurate term.
voytec 4y ago
"Zero trust" is such a bad, repulsive name.
Trias11 4y ago
Gl-Inet routers with embedded support for VPN's + ZeroTier. You may even install ZeroTier directly in these routers to [re-]configure router from anywhere.

177 points by JNRowe 4y ago

Nice list, zerotier does offer sharing as long as you add it to the network, or even having it public, but keep in mind it will share everything as if you’re normally connected with other members in the network, or create a separate network for the sharing purposes.

tiernano 4y ago
Their rules option allow you to lock down access

viraptor 4y ago

> I wouldn’t want to have a lot of programs broadcasting on a slow link. While in theory this could let you run Netware or DECNet across Zerotier There are less obscure uses for broadcast in those networks. For example mDNS "just works". You may not have an automated DNS at the service level, but you can use avahi to discover your other services or use .local without any extra config.

goodpoint 4y ago

What about legal liability? Forwarding somebody else's traffic can be very risky. Other than that, yggdrasil and tinc sound so much better than the alternatives.

chaxor 4y ago

Headscale seems like the best answer in this space

linsomniac 4y ago
Headscale is very good, but a few caveats: There's really no redundancy. If your headscale node goes down it can take your tailnet with it. In particular, if you drop in some ACLs that have issues, your entire tailnet can drop until you get it fixed. The recent (0.21) "configtest" can help there, but it still feels a bit brittle. Headscale can use a lot of system resources. ~100 nodes can saturate a t3a.small instance in cpu time and disc access. Reducing the update frequency can help, but there are hard limits here. I'm imagining much of this is database updates to sqlite, but I haven't tried switching to an external postgres server yet to see how much of the load is database related.

alchemist1e9 4y ago

tinc performance is limited by the encryption performance on single core. I suspect for Linux network engineers it could be a better choice than author suggests. tinc is very mature and battle tested.

password4321 4y ago
I don't know much about tinc for mobile devices, are there decent clients?
- alchemist1e9 4y ago
  I don’t use it on mobile devices so not sure. Instead I use a cloud server which joins the tinc mesh and and also runs wireguard service which then acts as a router.
- wkat4242 4y ago
  There's a great client for Android yes. https://tincapp.pacien.org/ https://tincapp.pacien.org/ For iOS there isn't. Last time I checked there's only one without a GUI that needs root. PS tinc is not fully zero trust. Every node can connect to every other one. This includes vps nodes you'll probably use for firewall traversal. Other systems have a 'lighthouse' concept where the vps just coordinates traffic but isn't able to actually read it.
  - progval 4y ago
    > Every node can connect to every other one. At the Tinc level, they can't connect to you unless you have their public key configured locally. At the IP level, set 'StrictSubnets = yes' in the main config file to prevent nodes not explicitly configured locally to send packets to you.

plaguuuuuu 4y ago

semi-related - does anyone know of any software frameworks/libraries for writing IoT event based systems stuff on this sort of mesh network? I have a side project in mind where IoT devices (dropping in and out of 4G and/or ad-hoc wifi or bluetooth) need to communicate and send/receive events, reconciling with event sourcing or whatever The tough parts of the architecture for this seem pretty generic so I'm hopeful that something already exists!

codegangsta 4y ago
Have you looked at nats.io? High performance messaging with a flexible persistence layer. Servers can be meshed via leaf node connections. Perfect for IoT and edge
- piceas 4y ago
  Works nicely with zerotier to simplify testing it out.
jacquesm 4y ago
Mail me?
somat 4y ago
It's not lightweight and it's not exactly a mesh network, that is, it's not a level 2 mesh it is level 5 mesh so you still need to get an ip somehow. but ipfs exposes it's dht for general purpose use via it's pubsub sytem. I had a lot of fun abusing this to make a serverless[1] video streaming platform. The video streaming ended up being shit with fundamental issues I don't think I could solve. but it was fun to put together the proof of concept. 1. actually serverless. not "the server is in the cloud" bullshit people usually mean when they say serverless.
- bboygravity 4y ago
  Sounds like https://lbry.com/ https://lbry.com/ ?? How did they solve or work around what you couldn't solve? Curious due to some deeply engrained dislike of anything centralized :p

jrm4 4y ago

Long-time tinc user here. My gut is that tinc is going to be as good as anything else here -- specifically what this means is that his "for your friends" idea doesn't hold much water for me. I.e. -- either you have friends techy enough such that you could get tinc going, or you don't, and at this level ALL of these solutions are going to be too annoying and complicated for them? For the non-techy types, I don't think any VPN like thing has been built yet? Would have to be like "syncthing" level easy.

remram 4y ago
hamachi was pretty close, we used it back in the days to play LAN games without having to enter IP addresses or mess with routers/modems.
imiric 4y ago
I'm a big fan of tinc. It's fully private and doesn't rely on public servers, like some of the mentioned solutions. Adding new peers is a minor chore, but if you have a network that doesn't change frequently, this is not an issue. Also unlike most of the alternatives, it runs absolutely everywhere. I had it setup on my pfSense router, and on my Android phone, and it worked great. Not sure why the article says that mobile support is iffy. I also used it professionally to setup a small Docker Swarm network over the internet. It also worked without issues, and was relatively simple to setup and maintain. Performance could be an issue, as the article mentions, but unless you're transferring large files, in practice it's good enough. I'm currently looking into setting this up again for personal use, and was hoping that there would be native mesh support within Wireguard, which doesn't seem to be the case.
- jrm4 4y ago
  Right? Honestly, I'm no expert, but I coded up a little crappy "good enough" onboarding shell script for myself (for adding new clients). I don't know if something similar exists already, but it seems like it could be done.

HereBeBeasties 4y ago

Twingate is missing from this list. It has nice DNS-based routing and access control.

jis 4y ago

Another tool worth looking at is vpncloud (https://github.com/dswd/vpncloud https://github.com/dswd/vpncloud). I used to use tinc, but switched to vpncloud 2 years ago. In my use case, I have a modest number of nodes. Although nodes learn of other nodes from each other, I use ansible to keep each node's config updated. I use vpncloud (and previously, tinc) between docker hosts. So, you have to be careful about interface MTU's inside of docker, particularly if you use containers based on Alpine.

aborsy 4y ago

What’s the problem with all devices connecting to a vpn server running somewhere? I mean, the usual access VPN solution. It’s not peer to peer, but you can set up or rent a server near clients, like in the same city.

wmf 4y ago
A hub-and-spoke VPN is slower and it can be more expensive. It's easier if you're configuring each tunnel by hand, but now that mesh VPNs are easy to use you might as well use mesh.
- SOLAR_FIELDS 4y ago
  It’s also very much a reduction of what some of these technologies are capable of. For instance, I’ve used Netmaker somewhat extensively as a VPN solution, and I mostly agree with the sentiments that the original article expresses about Netmaker (young, unstable). However, that is not the most interesting use case that Netmaker offers at all. Rather, a much more interesting use case is something like having an easy and straightforward way for Kube pods in AWS and GCP spanning across multiple regions to talk to each other. Multi-cloud architectures are greatly simplified by these kinds of technologies, especially Netmaker, partly because of their abilities to punch through NATs. Also, Netmaker is kernel level WireGuard which is much more performant. If you’re just using Tailscale to talk to clusters from your MacBook, for instance, userland WireGuard is probably going to be more than enough. When you need servers to talk to each other, though, you usually need the extra perf
- linsomniac 4y ago
  It's not necessarily slower. It really depends on the connectivity of the VPN hub. At my previous company our VPN server was on a VERY good network, including InterNAP traffic optimizers. One of our employees ran some testing of it and found that in most cases going over the VPN was faster than natively using his Comcast connection. Comcast would tend to route the traffic over their own network links as much as possible. In the case of the VPN server, it was well enough connected that Comcast could get to it quickly, and the good connectivity it had (largely thanks to Level-3, but the InterNAP optimizer would help it pick great connections in any case), would get it to the final destination faster than Comcast's network could.

deleted 4y ago

[deleted]

nbk_2000 4y ago

Privacy VPN providers are in a unique position to offer this as an add-on. NordVPN recently started offering Meshnet which effectively accomplishes the same thing as Tailscale, Zerotier etc. I vaguely recall another privacy VPN provider (maybe PIA) used to offer this as well. Seems like a good value add-on for services that already require setting up encrypted tunnels.

e12e 4y ago
Interesting - i wasn't aware of that: https://nordvpn.com/blog/nordlynx-protocol-wireguard/ https://nordvpn.com/blog/nordlynx-protocol-wireguard/
- nbk_2000 4y ago
  Here's a more direct link to what I was referring to: https://nordvpn.com/meshnet/ https://nordvpn.com/meshnet/

mjg59 4y ago

The point of zero-trust is that you don't implicitly trust something, but instead re-evaluate its trustworthiness every time you have to make an access control decision. Trusting a device simply because it's on your VPN is equivalent to trusting a device simply because it's on your LAN, and this is absolutely not a zero-trust architecture.

barnabee 4y ago
It's zero trust networking, unlike the internet or a traditional VPN.
- mjg59 4y ago
  And every definition of "zero trust networking" I can find still states that the point is that trust is something that's continually evaluated, which isn't the case if you're trusting someone simply because they're on your VPN.
  - barnabee 4y ago
    Tailscale describe zero trust networking more eloquently than I can, here: https://tailscale.com/kb/1123/zero-trust/ https://tailscale.com/kb/1123/zero-trust/ The gist being that you are no longer making unverifiable trust assumptions about the identity of nodes connected to your network, as you have encrypted links between all nodes (not just between external nodes and a VPN gaetway to a non-zero-trust internal LAN) based on their verified identity and allowed access. A node's identity can be verified by its public key, so you no longer implicitly need to trust that only known and expected nodes are plugged into the switch in the office (for example). I do not have to care about whether the physical network infrastructure can be trusted (zero trust) because I can verify the identity of all nodes. This of course does not mean your entire architecture is zero trust. There are pretty much always trust decisions and assumptions somewhere, but that is why it's described as zero trust [mesh] networking, not zero trust complete end to end systems. edit/aside: some people prefer the term "trust minimising" rather than zero trust to emphasise that there's always trust somewhere and you are just pushing it outside of some component or layer, and I agree this is probably a better/more accurate term.

voytec 4y ago

"Zero trust" is such a bad, repulsive name.

Trias11 4y ago

Gl-Inet routers with embedded support for VPN's + ZeroTier. You may even install ZeroTier directly in these routers to [re-]configure router from anywhere.