26 ms·

I'm curious to hear well-informed reasons from this crowd for why we can trust Tailscale given the non-self-hosted part of the architecture? Does it come down t

I'm curious to hear well-informed reasons from this crowd for why we can trust Tailscale given the non-self-hosted part of the architecture? Does it come down to Tailnet locks [1], not worrying that Tailscale will be compromised, not worrying that your home network is worth compromising, or something else? [1]: https://tailscale.com/kb/1226/tailnet-lock https://tailscale.com/kb/1226/tailnet-lock

sgc 2y ago
Use headscale: https://headscale.net/stable/ https://headscale.net/stable/
- haswell 2y ago
  What are the primary downsides of self-hosting this? The top issues that come to mind: 1. Maintaining high availability 2. Dealing with patches/upgrades But I'm also really curious how likely a self-hosted instance is to be an attack vector potentially more dangerous than using something like Tailscale.
  - huslage 2y ago
    You own your attack surface at that point. Tailscale/Headscale is a matchmaker and key broker for the most part, the clients almost always (barring NAT issues) connect directly to one another. The normal security considerations apply as with running any service.
  - eddieroger 2y ago
    > 1. Maintaining high availability In my experience as a poor sysadmin (as in, bad), you don't /need/ HA for Headscale because the clients are pretty resilient. I've had my instance go down for a little bit and it's fine. Stale and new connections aren't, obviously, but it will work well enough that you won't realize Headscale itself has gone down until a while after it did.
    - PLG88 2y ago
      Probably fine for a home lab, I don't think its fine for a production organisation running critical services across the overlay.
    - linsomniac 2y ago
      My experience has been the opposite: I have to restart headscale at night because a significant %age of the time when I do, the tailnet goes down. I'd say maybe 30% of the time, maybe more. I'm talking about when I update ACLs and OS updates. I run a single instance, and I will say it's been reliable over ~2 years. For the record, I just rebooted that node for OS updates and the tailnet stayed up.
stego-tech 2y ago
Call me Cappy Paranoid, but I fall into the camp of "You should never trust a service provider, ever," and build infrastructure accordingly; I believe this falls into an extreme interpretation of "zero trust". So while also implementing Tailnet locks and other security measures to constrict traffic flow, I'd also consider going a step further by only permitting server or resource access based on client certificate validation (in other words, a client that's missing a trusted certificate is rejected from even attempting to initiate AuthN); that way even if your Tailscale network is compromised somehow, untrusted clients and endpoints can't make inroads into your infrastructure as easily. Is that a gigantic PITA to implement? Oh heck, you betcha it is, and I doubt 99% of folks need to go that far with their homelabs or home services. Still, that'd be my approach to zero trust - trusting Tailscale only so far as enabling virtual networking, but not blindly trusting traffic coming over that network at any point.
- oDot 2y ago
  Yes. The best way to avoid trouble is build redundancies to it, rather than refine the troublesome part to no end
- tjoff 2y ago
  Then why go with tailscale in the first place? There is slacks nebula and other options that are completely self-hosted from the start. Feels like such a weird hype around tailscale.
  - stego-tech 2y ago
    I feel like a lot of hype around Tailscale is because it vastly simplifies VPNs and their associated networking, especially for businesses, startups, or homelabs where the focus might be elsewhere or specific talent is unavailable. The problem arises when folks don't quite understand why specific decisions are being made, or use the product in nonstandard (or even negative) ways. I've seen stories of folks deploying Tailscale on every machine in their LAN, thinking that secures their traffic; using it to cross boundaries in the firewall or router between secure and insecure VLANs; and using it to connect to servers in lieu of a proper router or firewall with appropriate ACLs. Tailscale is an excellent piece of software, provided it's implemented in a way to emphasize security, and not weaken it. In OPs case, being used as an accessibility aide to a system that couldn't be secured any other way while preserving external access (in their case due to CGNAT) was an excellent use of Tailscale.
porridgeraisin 2y ago
If you really want to host the control server yourself, you can use this open source implementation of it at https://github.com/juanfont/headscale https://github.com/juanfont/headscale
some-guy 2y ago
I use Tailscale a lot. I don't fully trust anybody but I trust them more than I trust myself to set it up properly.
phito 2y ago
Yeah, I don't understand how it is so prevalent in the self-hosted community. I would never install this on my server, just use wireguard/openvpn ... edit: okay, CGNAT
- gabeio 2y ago
  > Yeah, I don't understand how it is so prevalent in the self-hosted community. Not just CGNAT but not having _any_ external ports open can be a beautiful thing. I used to have an ssh port (not on the standard 22) and the amount of auth attempts back then was insane. I now have a full firewall zero open ports but, thanks to tailscale, I can still safely access my machines while not being at home with zero unauthorized attempts. And since I am a security person, I use the tailscale lock feature so not even tailscale themselves can add nodes to my network. Even if they had a breach. I am a very happy customer.
  - Carrok 2y ago
    If you're using only key-auth and have password auth disabled, I'm not sure why unauthorized attempts are a problem.
    - gabeio 2y ago
      > If you're using only key-auth and have password auth disabled, I'm not sure why unauthorized attempts are a problem. See xz vulnerability for more details. It’s about not trusting people with any of my ports/software (directly).
      - papichulo2023 2y ago
        Did you vet all Tailgate infra? Because now your attack surface is way higher. Wouldnt surprise the xz is somewhere there as well.
        gabeio 2y ago
        > Did you vet all Tailgate infra? > I use the tailscale lock feature so not even tailscale themselves can add nodes to my network. https://tailscale.com/blog/tailnet-lock https://tailscale.com/blog/tailnet-lock https://tailscale.com/kb/1226/tailnet-lock https://tailscale.com/kb/1226/tailnet-lock
GauntletWizard 2y ago
I trust Tailscale with my network traffic. I also trust a $50 cheap chinese 10G switch that I bought off amazon with a terrible and surely insecure management interface. Which is to say - I don't, but I don't need to trust it far. I do have enough trust in their client that's installed on my machine to believe that it's not actively malicious. I do trust that I can find my other devices, and trust tailscale to keep a list of them, and not randomly add other devices that I don't know, but I don't have perfect trust of that. All my internal services are still E2E encrypted over the Wireguard link; They run HTTPS with an internal cert authority. There's not ports open on them that shouldn't be, and while it's possible that one of them still gets popped, it's much less likely.
mbesto 2y ago
The data sent from one node to another doesn't pass through TS's infra. I basically just see Tailscale as an auth paradigm for managing wireguard keys.
- sureglymop 2y ago
  I don't really understand this though.. The key exchange is perhaps the most important aspect. Just hypothetically, what if an intelligence service records your encrypted traffic and also happened to get AWS to mitm your communication with the tailscale key distribution server? Doesn't really matter if most of your traffic doesn't use their infrastructure if the most important parts of it do.
  - showerst 2y ago
    If your threat model includes intelligence services and mitming AWS you should not be using tailscale, and you would hopefully already know that.
  - mbesto 2y ago
    > Using Tailscale introduces a dependency on Tailscale’s security. Using WireGuard directly does not. It is important to note that a device’s private key never leaves the device and thus Tailscale cannot decrypt network traffic. Our client code is open source, so you can confirm that yourself. https://tailscale.com/compare/wireguard https://tailscale.com/compare/wireguard My understanding is that (in theory) the only way this is possible is if the attacker introduces a new node and then connected to other nodes that are in the tailnet. What you're suggesting is that a single node that is connected to the other nodes gets compromised, but this isn't possible without already being able to compromise that specific node. Alternatively, if someone hacks Tailscale itself, the only way they could get access to any nodes would be to add their own node, but if you have alerting set up you would know and you could shut down the attacker.
harrall 2y ago
For me, Tailscale is worth the trouble of not maintaining my own Wireguard setup. Everything on my home network is set up as if it were public-facing.
- selfhoster 2y ago
  > Everything on my home network is set up as if it were public-facing. That's Wireguard, I have the same, just Wireguard + VPS, everything I want available that is. I don't put every PC on my home network on the VPN, I could though, pretty easily.
- felbane 2y ago
  This baffles me. What's to maintain? I've been running wireguard for years and never had to do anything except scan a QR code when I get a new phone. By "as if it were public facing" I assume you mean locked down as much as possible using either router or host-based firewall rules?
  - harrall 2y ago
    By locked down I mean everything requires authentication (and authorization), everything is containerized, and I have fairly strict firewall defaults. Let me explain what I mean by low maintenance... I was a very early containerization adopter and set up a company and also my home network using Docker around 10 years ago. I chose Docker because I thought it was reasonably polished and was the future of deployment. Even though the landscape keeps moving with changes in Kubernetes, Helm, Rancher and stuff like that, the actual Docker part hasn't changed in 10+ years so I haven't had to change my setup for a decade. Low maintenance for me is software that can be left mostly untouched (except for minor updates) for a long time and I judge that based on the project's future, which for me is partly judged from a project's polish. Every time I tried WireGuard in the past, it didn't seem so polished. I don't want to waste time learning something that could go away. On the other hand, not only did Tailscale look pretty well set up, it was pretty much click and run which means that even if it were to fail, I would have not lost any time learning much about it. So low maintenance for me is "get the most out of as little work as possible" and choosing Tailscale was the decision to achieve that. So given that I've been using Tailscale for 1.5 years with near 0 amount of configuration and so far, no real downtime, it is adequately low maintenance.
idatum 2y ago
I still find SSH adequate for connecting to a home server remotely. I don't have the CGNAT terrible problem but I also don't do any port forwarding on my home router. Instead, I have a VM running on a cloud provider that I SSH to from an OpenBSD box inside my home network. The SSH connection establishes a reverse SSH tunnel. This opens a port on the cloud VM to tunnel to my OpenBSD sshd port. With the reverse proxy to my home OpenBSD box established, I can use the SSH jump box option, -J. I connect to the cloud VM and "jump" through the tunnel to the OpenBSD box at home. You can even specify multiple jumps if I need to connect to another machine in my home. I can also set up a local tunnel through that jump for things like connecting to my Home Assistant server from my remote laptop or phone. I only have to trust my cloud provider.
- sfink 2y ago
  If I'm understanding correctly, this will break whenever the IP address of your tunnel changes. You'll have to reestablish all of your connections. My use case for tailscale: have an SSH (or other) connection to my home server while working from home. Drive to a coffee shop, register on their network, and continue using the same connection. (Or hotspot, if I'm somewhere without Wifi.) The IP address of my server does not change. When at home, the packets do not leave my home network. When out and about, they do. It's magic to me. I set up a sophisticated (read: overkill) SSH tunneling setup previously, using Match rules in .ssh/config to autodetect the network I was on so that `ssh myserver` would always go via the correct route. But my connections were still interrupted broke when I switched, and I'm not good enough at networking to do any better. (I guess this is what Wireguard is for? I could access my server via a fixed IP address on my machine that goes to a tun device, and that would send the packets to the actual server if nearby otherwise hand off to the carrier pigeons? Is that what the tailnet is doing? I don't understand how packets get intercepted by tailscaled, though I do see a tailscale0 device. Is that just a vanity license plate version of tun0? Why does `ip route show` give me only routes through my actual devices, then? Never mind, this isn't a helpdesk. I'm just getting old and stupid, I think.)
  - idatum 2y ago
    > If I'm understanding correctly, this will break whenever the IP address of your tunnel changes. You'll have to reestablish all of your connections. The tunnel is on localhost only. The VM has a static IPv4/IPv6 with DNS. Connecting the SSH tunnel from my home is stable as well as connecting to the VM remotely. I do appreciate Tailscale and Wireguard. I was more responding to the fact that I don't have to trust any provider here, other than the one keeping my VM running. Also, there's tmux for preserving sessions.
goodpoint 2y ago
We should not.

sgc 2y ago

Use headscale: https://headscale.net/stable/ https://headscale.net/stable/

haswell 2y ago
What are the primary downsides of self-hosting this? The top issues that come to mind: 1. Maintaining high availability 2. Dealing with patches/upgrades But I'm also really curious how likely a self-hosted instance is to be an attack vector potentially more dangerous than using something like Tailscale.
- huslage 2y ago
  You own your attack surface at that point. Tailscale/Headscale is a matchmaker and key broker for the most part, the clients almost always (barring NAT issues) connect directly to one another. The normal security considerations apply as with running any service.
- eddieroger 2y ago
  > 1. Maintaining high availability In my experience as a poor sysadmin (as in, bad), you don't /need/ HA for Headscale because the clients are pretty resilient. I've had my instance go down for a little bit and it's fine. Stale and new connections aren't, obviously, but it will work well enough that you won't realize Headscale itself has gone down until a while after it did.
  - PLG88 2y ago
    Probably fine for a home lab, I don't think its fine for a production organisation running critical services across the overlay.
  - linsomniac 2y ago
    My experience has been the opposite: I have to restart headscale at night because a significant %age of the time when I do, the tailnet goes down. I'd say maybe 30% of the time, maybe more. I'm talking about when I update ACLs and OS updates. I run a single instance, and I will say it's been reliable over ~2 years. For the record, I just rebooted that node for OS updates and the tailnet stayed up.

stego-tech 2y ago

Call me Cappy Paranoid, but I fall into the camp of "You should never trust a service provider, ever," and build infrastructure accordingly; I believe this falls into an extreme interpretation of "zero trust". So while also implementing Tailnet locks and other security measures to constrict traffic flow, I'd also consider going a step further by only permitting server or resource access based on client certificate validation (in other words, a client that's missing a trusted certificate is rejected from even attempting to initiate AuthN); that way even if your Tailscale network is compromised somehow, untrusted clients and endpoints can't make inroads into your infrastructure as easily. Is that a gigantic PITA to implement? Oh heck, you betcha it is, and I doubt 99% of folks need to go that far with their homelabs or home services. Still, that'd be my approach to zero trust - trusting Tailscale only so far as enabling virtual networking, but not blindly trusting traffic coming over that network at any point.

oDot 2y ago
Yes. The best way to avoid trouble is build redundancies to it, rather than refine the troublesome part to no end
tjoff 2y ago
Then why go with tailscale in the first place? There is slacks nebula and other options that are completely self-hosted from the start. Feels like such a weird hype around tailscale.
- stego-tech 2y ago
  I feel like a lot of hype around Tailscale is because it vastly simplifies VPNs and their associated networking, especially for businesses, startups, or homelabs where the focus might be elsewhere or specific talent is unavailable. The problem arises when folks don't quite understand why specific decisions are being made, or use the product in nonstandard (or even negative) ways. I've seen stories of folks deploying Tailscale on every machine in their LAN, thinking that secures their traffic; using it to cross boundaries in the firewall or router between secure and insecure VLANs; and using it to connect to servers in lieu of a proper router or firewall with appropriate ACLs. Tailscale is an excellent piece of software, provided it's implemented in a way to emphasize security, and not weaken it. In OPs case, being used as an accessibility aide to a system that couldn't be secured any other way while preserving external access (in their case due to CGNAT) was an excellent use of Tailscale.

porridgeraisin 2y ago

If you really want to host the control server yourself, you can use this open source implementation of it at https://github.com/juanfont/headscale https://github.com/juanfont/headscale

some-guy 2y ago

I use Tailscale a lot. I don't fully trust anybody but I trust them more than I trust myself to set it up properly.

phito 2y ago

Yeah, I don't understand how it is so prevalent in the self-hosted community. I would never install this on my server, just use wireguard/openvpn ... edit: okay, CGNAT

gabeio 2y ago
> Yeah, I don't understand how it is so prevalent in the self-hosted community. Not just CGNAT but not having _any_ external ports open can be a beautiful thing. I used to have an ssh port (not on the standard 22) and the amount of auth attempts back then was insane. I now have a full firewall zero open ports but, thanks to tailscale, I can still safely access my machines while not being at home with zero unauthorized attempts. And since I am a security person, I use the tailscale lock feature so not even tailscale themselves can add nodes to my network. Even if they had a breach. I am a very happy customer.
- Carrok 2y ago
  If you're using only key-auth and have password auth disabled, I'm not sure why unauthorized attempts are a problem.
  - gabeio 2y ago
    > If you're using only key-auth and have password auth disabled, I'm not sure why unauthorized attempts are a problem. See xz vulnerability for more details. It’s about not trusting people with any of my ports/software (directly).
    - papichulo2023 2y ago
      Did you vet all Tailgate infra? Because now your attack surface is way higher. Wouldnt surprise the xz is somewhere there as well.
      - gabeio 2y ago
        > Did you vet all Tailgate infra? > I use the tailscale lock feature so not even tailscale themselves can add nodes to my network. https://tailscale.com/blog/tailnet-lock https://tailscale.com/blog/tailnet-lock https://tailscale.com/kb/1226/tailnet-lock https://tailscale.com/kb/1226/tailnet-lock

GauntletWizard 2y ago

I trust Tailscale with my network traffic. I also trust a $50 cheap chinese 10G switch that I bought off amazon with a terrible and surely insecure management interface. Which is to say - I don't, but I don't need to trust it far. I do have enough trust in their client that's installed on my machine to believe that it's not actively malicious. I do trust that I can find my other devices, and trust tailscale to keep a list of them, and not randomly add other devices that I don't know, but I don't have perfect trust of that. All my internal services are still E2E encrypted over the Wireguard link; They run HTTPS with an internal cert authority. There's not ports open on them that shouldn't be, and while it's possible that one of them still gets popped, it's much less likely.

mbesto 2y ago

The data sent from one node to another doesn't pass through TS's infra. I basically just see Tailscale as an auth paradigm for managing wireguard keys.

sureglymop 2y ago
I don't really understand this though.. The key exchange is perhaps the most important aspect. Just hypothetically, what if an intelligence service records your encrypted traffic and also happened to get AWS to mitm your communication with the tailscale key distribution server? Doesn't really matter if most of your traffic doesn't use their infrastructure if the most important parts of it do.
- showerst 2y ago
  If your threat model includes intelligence services and mitming AWS you should not be using tailscale, and you would hopefully already know that.
- mbesto 2y ago
  > Using Tailscale introduces a dependency on Tailscale’s security. Using WireGuard directly does not. It is important to note that a device’s private key never leaves the device and thus Tailscale cannot decrypt network traffic. Our client code is open source, so you can confirm that yourself. https://tailscale.com/compare/wireguard https://tailscale.com/compare/wireguard My understanding is that (in theory) the only way this is possible is if the attacker introduces a new node and then connected to other nodes that are in the tailnet. What you're suggesting is that a single node that is connected to the other nodes gets compromised, but this isn't possible without already being able to compromise that specific node. Alternatively, if someone hacks Tailscale itself, the only way they could get access to any nodes would be to add their own node, but if you have alerting set up you would know and you could shut down the attacker.

harrall 2y ago

For me, Tailscale is worth the trouble of not maintaining my own Wireguard setup. Everything on my home network is set up as if it were public-facing.

selfhoster 2y ago
> Everything on my home network is set up as if it were public-facing. That's Wireguard, I have the same, just Wireguard + VPS, everything I want available that is. I don't put every PC on my home network on the VPN, I could though, pretty easily.
felbane 2y ago
This baffles me. What's to maintain? I've been running wireguard for years and never had to do anything except scan a QR code when I get a new phone. By "as if it were public facing" I assume you mean locked down as much as possible using either router or host-based firewall rules?
- harrall 2y ago
  By locked down I mean everything requires authentication (and authorization), everything is containerized, and I have fairly strict firewall defaults. Let me explain what I mean by low maintenance... I was a very early containerization adopter and set up a company and also my home network using Docker around 10 years ago. I chose Docker because I thought it was reasonably polished and was the future of deployment. Even though the landscape keeps moving with changes in Kubernetes, Helm, Rancher and stuff like that, the actual Docker part hasn't changed in 10+ years so I haven't had to change my setup for a decade. Low maintenance for me is software that can be left mostly untouched (except for minor updates) for a long time and I judge that based on the project's future, which for me is partly judged from a project's polish. Every time I tried WireGuard in the past, it didn't seem so polished. I don't want to waste time learning something that could go away. On the other hand, not only did Tailscale look pretty well set up, it was pretty much click and run which means that even if it were to fail, I would have not lost any time learning much about it. So low maintenance for me is "get the most out of as little work as possible" and choosing Tailscale was the decision to achieve that. So given that I've been using Tailscale for 1.5 years with near 0 amount of configuration and so far, no real downtime, it is adequately low maintenance.

idatum 2y ago

I still find SSH adequate for connecting to a home server remotely. I don't have the CGNAT terrible problem but I also don't do any port forwarding on my home router. Instead, I have a VM running on a cloud provider that I SSH to from an OpenBSD box inside my home network. The SSH connection establishes a reverse SSH tunnel. This opens a port on the cloud VM to tunnel to my OpenBSD sshd port. With the reverse proxy to my home OpenBSD box established, I can use the SSH jump box option, -J. I connect to the cloud VM and "jump" through the tunnel to the OpenBSD box at home. You can even specify multiple jumps if I need to connect to another machine in my home. I can also set up a local tunnel through that jump for things like connecting to my Home Assistant server from my remote laptop or phone. I only have to trust my cloud provider.

sfink 2y ago
If I'm understanding correctly, this will break whenever the IP address of your tunnel changes. You'll have to reestablish all of your connections. My use case for tailscale: have an SSH (or other) connection to my home server while working from home. Drive to a coffee shop, register on their network, and continue using the same connection. (Or hotspot, if I'm somewhere without Wifi.) The IP address of my server does not change. When at home, the packets do not leave my home network. When out and about, they do. It's magic to me. I set up a sophisticated (read: overkill) SSH tunneling setup previously, using Match rules in .ssh/config to autodetect the network I was on so that `ssh myserver` would always go via the correct route. But my connections were still interrupted broke when I switched, and I'm not good enough at networking to do any better. (I guess this is what Wireguard is for? I could access my server via a fixed IP address on my machine that goes to a tun device, and that would send the packets to the actual server if nearby otherwise hand off to the carrier pigeons? Is that what the tailnet is doing? I don't understand how packets get intercepted by tailscaled, though I do see a tailscale0 device. Is that just a vanity license plate version of tun0? Why does `ip route show` give me only routes through my actual devices, then? Never mind, this isn't a helpdesk. I'm just getting old and stupid, I think.)
- idatum 2y ago
  > If I'm understanding correctly, this will break whenever the IP address of your tunnel changes. You'll have to reestablish all of your connections. The tunnel is on localhost only. The VM has a static IPv4/IPv6 with DNS. Connecting the SSH tunnel from my home is stable as well as connecting to the VM remotely. I do appreciate Tailscale and Wireguard. I was more responding to the fact that I don't have to trust any provider here, other than the one keeping my VM running. Also, there's tmux for preserving sessions.

goodpoint 2y ago

We should not.