7 ms·

WireGuard on K8s: road-warrior-style VPN server

165 points by sclevine 6y ago

vpner 6y ago
GitHub has several projects that automate setting up a wireguard VPN on various cloud VMs without K8s: https://github.com/topics/wireguard https://github.com/topics/wireguard. There's also this tutorial that sets up a VPN along with proper DNS configuration so that DNS doesn't leak: https://www.ckn.io/blog/2017/11/14/wireguard-vpn-typical-setup/ https://www.ckn.io/blog/2017/11/14/wireguard-vpn-typical-set....
rektide 6y ago
Worth metnioning Kilo, which is an enhancement or a CNI (container network interface) provider that does Wireguard for Kubernetes. https://github.com/squat/kilo https://github.com/squat/kilo
- jzelinskie 6y ago
  Yes! When you think of Wireguard and Kubernetes, you should think of Lucas! He spends a lot of his free time experimenting with the combination of these two technologies. At KubeCon EU Barcelona, he gave a talk about cross-cluster networking using Wireguard: https://www.youtube.com/watch?v=iPz_DAOOCKA https://www.youtube.com/watch?v=iPz_DAOOCKA
wferrell 6y ago
Worth taking look at http://tailscale.com http://tailscale.com - Their tag line: Private networks made easy. No affiliation -- just like their product.
- 0az 6y ago
  My main annoyance with Tailscale is the reliance on Google. I need to refresh my memory, but I think this makes a VLAN shared with other people impossible. This is why I'm still using https://zerotier.com https://zerotier.com -- also no affiliation.
  - pot8n 6y ago
    Honestly that's the least of all problems and catastrophes of Tailscale. You must have 1000% of confidence in their own servers security, if the published public keys hosted on their servers have been tampered then the entire network is compromised. Also, if their service is down, you will be unable to connect to your network even if it is completely fine and working.
    - lima 6y ago
      Tailscale is open source, it should be possible to set up your own server. The hosted Tailscale product is meant for GSuite customers who want an peer-to-peer VPN with corporate SSO. Yes, you have to trust them - SSO login is inherently centralized. My company uses it, it works great.
      - pot8n 6y ago
        I am not really sure you understand how it works. There is no hosted/not hosted versions of it. You must connect your "opensource" client/agent through their coordination servers hosted by them to host and publish the public key to the other devices in your network and you can not skip their service. So Tailscale is effectively as opensource as any commercial opensource VPN client. It's entirely useless when not used with their commercial service and users have zero control over the software unless when used with their servers. The "open source" thing is great from a marketing and business perspective because you basically benefit from the open source marketing and the community thing from the unsuspecting users and enthusiasts pros without giving away literally anything.
        lima 6y ago
        The backend (minus the web UI?) is open source as well.
      - deleted 6y ago
        [deleted]
    - j88439h84 6y ago
- smartbit 6y ago
  A quick search for overlay mesh network wireguard resulted in https://github.com/costela/wesher https://github.com/costela/wesher. Anyone experience with wesher or the like?
microcolonel 6y ago
Well, there's road-warrior, and then there's road-warrior. I've been trying out Glorytun, it does multi-path VPN with a relatively similar wire format to WireGuard. Being mostly indoors, due to the microbial boogaloo, I've not been trying it with the most interesting applications.
- jcims 6y ago
  Would like to use something like this to aggregate a few DSL connections. Any idea how well it works for that use case?
  - microcolonel 6y ago
    It seems to work well when the connections are of roughly equal speed and stability, so that sounds like a rather ideal use case. :+ ) I think it'll need work for connections with varied performance.
    - jcims 6y ago
      Nice! Thanks for the reply, May have to give this a go.
  - tsukurimashou 6y ago
    I looked into aggregating DSL connections in the past, a few years ago I think you had to get your own router for that as well as a VPS. OVH launched a service "over the box" that goes just that, they provide a router and a VPS where a VPN runs. They claimed you'd get a total bandwidth equals to the sum of all connections and I think it's not required for the connections to be similar in bandwidth.
Legogris 6y ago
Is there any reason why OP installs iproute2 and iptables not in the builder together with the wireguard package but in the final container image?
- simcop2387 6y ago
  The packages installed in the builders are essentially never used since it never runs. The builder makes the files to install in the final container during the build phase, and then gets thrown away.
  - Legogris 6y ago
    Oh, of course. Brainfart there.
stzup7 6y ago
I'm not sure to understand the use case. Is the goal to replace things like flannel? Or route all your traffic from a single gateway?
BillinghamJ 6y ago
You can install the WireGuard tools only, without the kernel extensions etc, with: apt-get install -y --no-install-recommends wireguard-tools This is all you need with the server flavour of 20.04. For the minimal one, you need a couple more. So no need to use a builder image
tombh 6y ago
I think we all understand the usefulness of a road-warrior-style VPN. But it doesn't seem so clear what k8s is adding here? Anyway, on the topic of scalable UDP services, does anyone have any experience of load balancing a UDP service? Because UDP is connectionless there's no obvious way to make UDP packets "sticky". Are there any established practices that could help scale this k8s Wireguard service to 2 or more containers?
- sclevine 6y ago
  I'm just using K8s (specifically: K3s) for configuration management in this case. This post hits the nail on the head: https://news.ycombinator.com/item?id=23006114 https://news.ycombinator.com/item?id=23006114 That said, NGINX can do UDP load balancing and WireGuard is stateless, so it should be possible to use this with a Service + NGINX ingress controller at scale: https://kubernetes.github.io/ingress-nginx/user-guide/exposing-tcp-udp-services/ https://kubernetes.github.io/ingress-nginx/user-guide/exposi... I have not tried it though.
- KenanSulayman 6y ago
  Well if you make it a DaemonSet you could technically use the container as the network interface of other containers throughout the whole cluster. That said, I'm very happy that his example k8s deployment uses secrets. I didn't know Ubuntu 20.04 back ported WG into its 5.4 kernel. I spent a few hours yesterday fixing a node after breaking ZFS because I upgraded to 5.6 for WG support. I feel rather silly now.. edit: rektide mentioned 'kilo' which actually does exactly what I said (https://github.com/squat/kilo https://github.com/squat/kilo).
  - tombh 6y ago
    That's an interesting idea about using a unified network interface. Do you know how you might then get the right packets to the right containers/processes? Does that even matter with Wireguard?
    - KenanSulayman 6y ago
      You can use a different container as network in docker: 'container:<name>' would route the container traffic through specified container. Example vpn container: docker run --name foo --cap-add=NET_ADMIN ... Other container: docker run --net=container:foo ... Now you'd need to specify the respective routing rules [1] in the container. [1] i.e. https://github.com/bubuntux/nordvpn/blob/master/start_vpn.sh#L72-L79 https://github.com/bubuntux/nordvpn/blob/master/start_vpn.sh...
sandGorgon 6y ago
this example uses k3s - which is the k8s distribution by the Rancher guys. Really cool distro - simple UX. Runs equally good on a raspberry pi or the cloud
aequitas 6y ago
A few people seem to be confused why K8s is needed when you can just run this on the OS itself. I think they miss the point that this is not a guide to setup Wireguard using K8s but setup Wireguard if you only have/want a K8s environment. As the author notes: "you can run a road-warrior-style Wireguard server in K8s without making changes to the node." Which makes this guide ideal for me. I run a lightweight K8s flavor (K3s, https://k3s.io/ https://k3s.io/) as "configuration management" on my home server and home automation Raspberry Pi's because I don't want to mess with OS/userland configuration or the associated tools (Puppet, Ansible, hacked together scripts, etc) or want to maintain any OS state manually. For my setup I just flash K3s to disk or SD card and let it join the cluster. Everything else is configured in Kubernetes and stored nicely as configuration files on my laptop so I have an overview of everthing and can modify/rebuild whenever I want.
- sclevine 6y ago
  This was indeed the motivation for my write-up :)
  - aequitas 6y ago
    Thanks, I was meaning to look into this but your post will save me some research work.
- darkwater 6y ago
  You say you don't want to use Puppet or Ansible but you are basically using kubernetes manifests for the same exact reason: configuration management. I know it can be funny and I totally support it but I thought it should be pointed out anyway.
  - aequitas 6y ago
    The problem I have with traditional configuration management is that in the end, even if it's declerative, you are stil modifying a imperative OS/userland. So it will collect state at some point. Things like undoing changes with those tools is not that trivial. You have to actively reverse them in your configuration. Which turns nice CM code into mess. Want to try out something quick? Better not be afraid it messes up your OS/userland as there is no simple undo. So since I'm doing isolation in containers/Docker already it's a small step to a lightweight Kubernetes. What Kubernetes gives me on top of that is that I can consider everything below the application layer as a declarative API.
godelmachine 6y ago
While we are discussing this, I see NordVPN has also released support for Wireguard and named it NordLynx

165 points by sclevine 6y ago

GitHub has several projects that automate setting up a wireguard VPN on various cloud VMs without K8s: https://github.com/topics/wireguard https://github.com/topics/wireguard. There's also this tutorial that sets up a VPN along with proper DNS configuration so that DNS doesn't leak: https://www.ckn.io/blog/2017/11/14/wireguard-vpn-typical-setup/ https://www.ckn.io/blog/2017/11/14/wireguard-vpn-typical-set....

rektide 6y ago

Worth metnioning Kilo, which is an enhancement or a CNI (container network interface) provider that does Wireguard for Kubernetes. https://github.com/squat/kilo https://github.com/squat/kilo

jzelinskie 6y ago
Yes! When you think of Wireguard and Kubernetes, you should think of Lucas! He spends a lot of his free time experimenting with the combination of these two technologies. At KubeCon EU Barcelona, he gave a talk about cross-cluster networking using Wireguard: https://www.youtube.com/watch?v=iPz_DAOOCKA https://www.youtube.com/watch?v=iPz_DAOOCKA

wferrell 6y ago

Worth taking look at http://tailscale.com http://tailscale.com - Their tag line: Private networks made easy. No affiliation -- just like their product.

0az 6y ago
My main annoyance with Tailscale is the reliance on Google. I need to refresh my memory, but I think this makes a VLAN shared with other people impossible. This is why I'm still using https://zerotier.com https://zerotier.com -- also no affiliation.
- pot8n 6y ago
  Honestly that's the least of all problems and catastrophes of Tailscale. You must have 1000% of confidence in their own servers security, if the published public keys hosted on their servers have been tampered then the entire network is compromised. Also, if their service is down, you will be unable to connect to your network even if it is completely fine and working.
  - lima 6y ago
    Tailscale is open source, it should be possible to set up your own server. The hosted Tailscale product is meant for GSuite customers who want an peer-to-peer VPN with corporate SSO. Yes, you have to trust them - SSO login is inherently centralized. My company uses it, it works great.
    - pot8n 6y ago
      I am not really sure you understand how it works. There is no hosted/not hosted versions of it. You must connect your "opensource" client/agent through their coordination servers hosted by them to host and publish the public key to the other devices in your network and you can not skip their service. So Tailscale is effectively as opensource as any commercial opensource VPN client. It's entirely useless when not used with their commercial service and users have zero control over the software unless when used with their servers. The "open source" thing is great from a marketing and business perspective because you basically benefit from the open source marketing and the community thing from the unsuspecting users and enthusiasts pros without giving away literally anything.
      - lima 6y ago
        The backend (minus the web UI?) is open source as well.
    - deleted 6y ago
      [deleted]
  - j88439h84 6y ago
smartbit 6y ago
A quick search for overlay mesh network wireguard resulted in https://github.com/costela/wesher https://github.com/costela/wesher. Anyone experience with wesher or the like?

microcolonel 6y ago

Well, there's road-warrior, and then there's road-warrior. I've been trying out Glorytun, it does multi-path VPN with a relatively similar wire format to WireGuard. Being mostly indoors, due to the microbial boogaloo, I've not been trying it with the most interesting applications.

jcims 6y ago
Would like to use something like this to aggregate a few DSL connections. Any idea how well it works for that use case?
- microcolonel 6y ago
  It seems to work well when the connections are of roughly equal speed and stability, so that sounds like a rather ideal use case. :+ ) I think it'll need work for connections with varied performance.
  - jcims 6y ago
    Nice! Thanks for the reply, May have to give this a go.
- tsukurimashou 6y ago
  I looked into aggregating DSL connections in the past, a few years ago I think you had to get your own router for that as well as a VPS. OVH launched a service "over the box" that goes just that, they provide a router and a VPS where a VPN runs. They claimed you'd get a total bandwidth equals to the sum of all connections and I think it's not required for the connections to be similar in bandwidth.

Legogris 6y ago

Is there any reason why OP installs iproute2 and iptables not in the builder together with the wireguard package but in the final container image?

simcop2387 6y ago
The packages installed in the builders are essentially never used since it never runs. The builder makes the files to install in the final container during the build phase, and then gets thrown away.
- Legogris 6y ago
  Oh, of course. Brainfart there.

stzup7 6y ago

I'm not sure to understand the use case. Is the goal to replace things like flannel? Or route all your traffic from a single gateway?

BillinghamJ 6y ago

You can install the WireGuard tools only, without the kernel extensions etc, with: apt-get install -y --no-install-recommends wireguard-tools This is all you need with the server flavour of 20.04. For the minimal one, you need a couple more. So no need to use a builder image

tombh 6y ago

I think we all understand the usefulness of a road-warrior-style VPN. But it doesn't seem so clear what k8s is adding here? Anyway, on the topic of scalable UDP services, does anyone have any experience of load balancing a UDP service? Because UDP is connectionless there's no obvious way to make UDP packets "sticky". Are there any established practices that could help scale this k8s Wireguard service to 2 or more containers?

sclevine 6y ago
I'm just using K8s (specifically: K3s) for configuration management in this case. This post hits the nail on the head: https://news.ycombinator.com/item?id=23006114 https://news.ycombinator.com/item?id=23006114 That said, NGINX can do UDP load balancing and WireGuard is stateless, so it should be possible to use this with a Service + NGINX ingress controller at scale: https://kubernetes.github.io/ingress-nginx/user-guide/exposing-tcp-udp-services/ https://kubernetes.github.io/ingress-nginx/user-guide/exposi... I have not tried it though.
KenanSulayman 6y ago
Well if you make it a DaemonSet you could technically use the container as the network interface of other containers throughout the whole cluster. That said, I'm very happy that his example k8s deployment uses secrets. I didn't know Ubuntu 20.04 back ported WG into its 5.4 kernel. I spent a few hours yesterday fixing a node after breaking ZFS because I upgraded to 5.6 for WG support. I feel rather silly now.. edit: rektide mentioned 'kilo' which actually does exactly what I said (https://github.com/squat/kilo https://github.com/squat/kilo).
- tombh 6y ago
  That's an interesting idea about using a unified network interface. Do you know how you might then get the right packets to the right containers/processes? Does that even matter with Wireguard?
  - KenanSulayman 6y ago
    You can use a different container as network in docker: 'container:<name>' would route the container traffic through specified container. Example vpn container: docker run --name foo --cap-add=NET_ADMIN ... Other container: docker run --net=container:foo ... Now you'd need to specify the respective routing rules [1] in the container. [1] i.e. https://github.com/bubuntux/nordvpn/blob/master/start_vpn.sh#L72-L79 https://github.com/bubuntux/nordvpn/blob/master/start_vpn.sh...

sandGorgon 6y ago

this example uses k3s - which is the k8s distribution by the Rancher guys. Really cool distro - simple UX. Runs equally good on a raspberry pi or the cloud

aequitas 6y ago

A few people seem to be confused why K8s is needed when you can just run this on the OS itself. I think they miss the point that this is not a guide to setup Wireguard using K8s but setup Wireguard if you only have/want a K8s environment. As the author notes: "you can run a road-warrior-style Wireguard server in K8s without making changes to the node." Which makes this guide ideal for me. I run a lightweight K8s flavor (K3s, https://k3s.io/ https://k3s.io/) as "configuration management" on my home server and home automation Raspberry Pi's because I don't want to mess with OS/userland configuration or the associated tools (Puppet, Ansible, hacked together scripts, etc) or want to maintain any OS state manually. For my setup I just flash K3s to disk or SD card and let it join the cluster. Everything else is configured in Kubernetes and stored nicely as configuration files on my laptop so I have an overview of everthing and can modify/rebuild whenever I want.

sclevine 6y ago
This was indeed the motivation for my write-up :)
- aequitas 6y ago
  Thanks, I was meaning to look into this but your post will save me some research work.
darkwater 6y ago
You say you don't want to use Puppet or Ansible but you are basically using kubernetes manifests for the same exact reason: configuration management. I know it can be funny and I totally support it but I thought it should be pointed out anyway.
- aequitas 6y ago
  The problem I have with traditional configuration management is that in the end, even if it's declerative, you are stil modifying a imperative OS/userland. So it will collect state at some point. Things like undoing changes with those tools is not that trivial. You have to actively reverse them in your configuration. Which turns nice CM code into mess. Want to try out something quick? Better not be afraid it messes up your OS/userland as there is no simple undo. So since I'm doing isolation in containers/Docker already it's a small step to a lightweight Kubernetes. What Kubernetes gives me on top of that is that I can consider everything below the application layer as a declarative API.

godelmachine 6y ago

While we are discussing this, I see NordVPN has also released support for Wireguard and named it NordLynx