4 ms·

Call me Cappy Paranoid, but I fall into the camp of "You should never trust a service provider, ever," and build infrastructure accordingly; I believe this fall

Call me Cappy Paranoid, but I fall into the camp of "You should never trust a service provider, ever," and build infrastructure accordingly; I believe this falls into an extreme interpretation of "zero trust". So while also implementing Tailnet locks and other security measures to constrict traffic flow, I'd also consider going a step further by only permitting server or resource access based on client certificate validation (in other words, a client that's missing a trusted certificate is rejected from even attempting to initiate AuthN); that way even if your Tailscale network is compromised somehow, untrusted clients and endpoints can't make inroads into your infrastructure as easily. Is that a gigantic PITA to implement? Oh heck, you betcha it is, and I doubt 99% of folks need to go that far with their homelabs or home services. Still, that'd be my approach to zero trust - trusting Tailscale only so far as enabling virtual networking, but not blindly trusting traffic coming over that network at any point.

oDot 2y ago
Yes. The best way to avoid trouble is build redundancies to it, rather than refine the troublesome part to no end
tjoff 2y ago
Then why go with tailscale in the first place? There is slacks nebula and other options that are completely self-hosted from the start. Feels like such a weird hype around tailscale.
- stego-tech 2y ago
  I feel like a lot of hype around Tailscale is because it vastly simplifies VPNs and their associated networking, especially for businesses, startups, or homelabs where the focus might be elsewhere or specific talent is unavailable. The problem arises when folks don't quite understand why specific decisions are being made, or use the product in nonstandard (or even negative) ways. I've seen stories of folks deploying Tailscale on every machine in their LAN, thinking that secures their traffic; using it to cross boundaries in the firewall or router between secure and insecure VLANs; and using it to connect to servers in lieu of a proper router or firewall with appropriate ACLs. Tailscale is an excellent piece of software, provided it's implemented in a way to emphasize security, and not weaken it. In OPs case, being used as an accessibility aide to a system that couldn't be secured any other way while preserving external access (in their case due to CGNAT) was an excellent use of Tailscale.
  - haswell 2y ago
    Yeah, I mentally sum this up as the "Just Works" factor. As a happy Tailscale user, it's easy to see why it's so popular. I do think this simplicity is exactly what contributes to those weird and non-standard configurations.
    - stego-tech 2y ago
      > I do think this simplicity is exactly what contributes to those weird and non-standard configurations. This is why I am confident I will always have employment in IT. As I make things simpler for others to use, they in turn will find new and innovative ways of making my eyes bleed from cursed workflows that once again require professional intervention for simplicity, efficiency, and security.
  - xienze 2y ago
    > I feel like a lot of hype around Tailscale is because it vastly simplifies VPNs and their associated networking Tailscale is based on Wire Guard, isn’t it? Now there’s a piece of software that truly made VPNs simple. I have a tunnel back into my LAN by way of an EC2 instance and all it took was two super simple config files on each machine.
    - nine_k 2y ago
      Wireguard vastly simplifies the transport level, and attains high performance because it runs in the kernel. Tailscale simplifies: authentication (including OIDC), authorization (via ACLs), DNS, NAT piercing. All of that is not obvious or easy for someone without deeper expertise.
  - tjoff 2y ago
    But you demonstrately did not make it easy or simple. Of course there are tons of alternatives even if you are behind CGNAT. Nebula is but one.
- throwawaymaths 2y ago
  have you ever managed a tailnet? it's so easy.
- kajecounterhack 2y ago
  They have nice clients (e.g. for MacOS, Tizen). Ofc headscale is a thing, but if you have a company, it's also nice to have someone to yell at if your mission-critical tailnet suddenly b0rks. Imo they don't charge all that much relative to their value, depending on who you're asking.
- xarope 2y ago
  I self host tailscale with headscale [0]. [0]: https://github.com/juanfont/headscale https://github.com/juanfont/headscale
HPsquared 2y ago
If you can't trust service providers, you probably also can't trust software suppliers.
- stego-tech 2y ago
  I mean, yes? It's why Zero Trust is growing as an operations model. Supply chain attacks, vendor hostility, zero days being hoarded by nations and bad actors for exploit, the list goes on. You emphatically cannot trust vendors, suppliers, users, software, systems, or governments. Ergo, your infrastructure should be built with an appropriate risk assessment in mind, and have proper safeguards in place where feasible. That's just good OpSec.
  - deleted 2y ago
    [deleted]
- kortilla 2y ago
  Not the same. In particular you don’t need to accept software updates from software suppliers and you can also require source code or use open source. This stuff was obvious and standard in the 80s-2000s. It’s only in the last 15-20 years that it became acceptable to get updates shoved down your throat. Service providers can cut off your access any day. Software providers cannot unless you’ve given them a live update channel direct to your env.
- drdaeman 2y ago
  Definitely not true. You can audit software (it could be not easy, but ultimately doable) and skip the updates until you have capacity to audit those. You can't audit a third-party service, no matter what you do.
codetrotter 2y ago
> Is that a gigantic PITA to implement? Oh heck, you betcha it is I use my own self-hosted Wireguard VPN server. I agree with a lot of what you were saying about client certificates etc. And I plan to eventually do that sort of thing on some of my services in my own Wireguard VPN too. But in terms of Tailscale, if you are going to set up all kinds of client certificate things that will take a lot of time and effort, why not self-host Wireguard also? Setting up a Wireguard server is super simple. The only couple of things that complicate it a tiny bit is opening up a port for it for inbound connections if you host it from your home connection rather than a rented server, and managing the Wireguard public keys that are allowed to connect. But if you are going to do a whole client certificate setup on top anyway, the work of setting up your own Wireguard VPN is small in comparison. Unless like OP your ISP has put CGNAT on you.
- stego-tech 2y ago
  > But in terms of Tailscale, if you are going to set up all kinds of client certificate things that will take a lot of time and effort, why not self-host Wireguard also? Already do! I tried Tailscale initially, but ultimately decided to put in the effort of a proper Wireguard setup. It's how my personal devices always get back to my home LAN, and then exit to the internet; it's also how I make sure every DNS lookup hits the Pi-Hole, for domain blocking wherever I am. I emphatically recommend learning WireGuard (and to a lesser degree, VPN Concentration) when practical and possible. Until then, Tailscale is an excellent product.
- selfhoster 2y ago
  > Unless like OP your ISP has put CGNAT on you. I run Wireguard on a VPS and route public traffic with it over Wireguard to my home machine. Are you saying my ISP must not be CGNAT or else it wouldn't work?
  - aborsy 2y ago
    How is this a good solution, when traffic is decrypted in the cloud, all traffic goes through one node, there is no ACL, key distribution, static IP, …? Tailscale addressed those issues.
xmichael909 2y ago
Why wouldn't you just not use Tailscale? What you are describing here is, ....
HumanOstrich 2y ago
> Call me Cappy Paranoid, but I fall into the camp of "You should never trust a service provider, ever," and build infrastructure accordingly; I believe this falls into an extreme interpretation of "zero trust". That's not what Zero Trust means, at all.
- stego-tech 2y ago
  …which is why I qualified it with the phrase, “extreme interpretation of”, and made sure to encapsulate “Zero trust” in quotes to make it clear I wasn’t being technically literal in my description. Grammar and punctuation matter when you’re deliberately misusing a known term as a metaphor to make a point. That being said, the core concept of ZTA is that no user or device should be trusted by default. So yes, my statement is still generally correct even if it’s not how the term is often or commonly used.
hamandcheese 2y ago
> I'd also consider going a step further by only permitting server or resource access based on client certificate validation This is where I'm the most curious on what Tailscale will do next. So far all their products seem to contrast at the IP level, but for enterprise use cases there's a real need for application level protections as well. Cloudflare Access is a great example of what I mean.
PLG88 2y ago
Particularly as it does not include its own PKI, so E2EE is done by MITM your IdP (OICD/SAML etc) and therefore, under court order Tailscale can decrypt your traffic. We took the opposite approach with NetFoundry. (1) We open sourced the code (https://openziti.io/ https://openziti.io/), (2) we built in PKI with private keys generated at source and destination so that even if traversing NF hosted data plane, we CANNOT decrypt traffic, (3) mTLS everywhere, (4) ability to bring your own PKI, and more.
- aborsy 2y ago
  First of all, a node added to tailnet doesn’t not have the ability to decrypt the traffic in tailnet. All it can do to contact other nodes, decrypt traffic users sent to that hidden node, or modify settings in admin console. Furthermore, with tail lock, the coordination server should not be able to add nodes from outside. Can you clarify?
  - PLG88 2y ago
    Comment edited due to an incorrect understand which has been rectified.
    - aborsy 2y ago
      This is false information. Even if an attacker such as the government runs the coordination and relay servers, and the IdP, they will not be able to decrypt any traffic in tailnet. The secret keys remain on device, and traffic is end to end encrypted. There is no mechanism in the client agents to send out the secret keys. The coordination server receives the public keys and metadata. Please clarify or revise your comment!
      - PLG88 2y ago
        I see I did have a misunderstanding. I believe there is still the meta data angle, but yes, private keys on endpoints would ensure E2EE. I will update my comment.

oDot 2y ago

Yes. The best way to avoid trouble is build redundancies to it, rather than refine the troublesome part to no end

tjoff 2y ago

Then why go with tailscale in the first place? There is slacks nebula and other options that are completely self-hosted from the start. Feels like such a weird hype around tailscale.

stego-tech 2y ago
I feel like a lot of hype around Tailscale is because it vastly simplifies VPNs and their associated networking, especially for businesses, startups, or homelabs where the focus might be elsewhere or specific talent is unavailable. The problem arises when folks don't quite understand why specific decisions are being made, or use the product in nonstandard (or even negative) ways. I've seen stories of folks deploying Tailscale on every machine in their LAN, thinking that secures their traffic; using it to cross boundaries in the firewall or router between secure and insecure VLANs; and using it to connect to servers in lieu of a proper router or firewall with appropriate ACLs. Tailscale is an excellent piece of software, provided it's implemented in a way to emphasize security, and not weaken it. In OPs case, being used as an accessibility aide to a system that couldn't be secured any other way while preserving external access (in their case due to CGNAT) was an excellent use of Tailscale.
- haswell 2y ago
  Yeah, I mentally sum this up as the "Just Works" factor. As a happy Tailscale user, it's easy to see why it's so popular. I do think this simplicity is exactly what contributes to those weird and non-standard configurations.
  - stego-tech 2y ago
    > I do think this simplicity is exactly what contributes to those weird and non-standard configurations. This is why I am confident I will always have employment in IT. As I make things simpler for others to use, they in turn will find new and innovative ways of making my eyes bleed from cursed workflows that once again require professional intervention for simplicity, efficiency, and security.
- xienze 2y ago
  > I feel like a lot of hype around Tailscale is because it vastly simplifies VPNs and their associated networking Tailscale is based on Wire Guard, isn’t it? Now there’s a piece of software that truly made VPNs simple. I have a tunnel back into my LAN by way of an EC2 instance and all it took was two super simple config files on each machine.
  - nine_k 2y ago
    Wireguard vastly simplifies the transport level, and attains high performance because it runs in the kernel. Tailscale simplifies: authentication (including OIDC), authorization (via ACLs), DNS, NAT piercing. All of that is not obvious or easy for someone without deeper expertise.
- tjoff 2y ago
  But you demonstrately did not make it easy or simple. Of course there are tons of alternatives even if you are behind CGNAT. Nebula is but one.
throwawaymaths 2y ago
have you ever managed a tailnet? it's so easy.
kajecounterhack 2y ago
They have nice clients (e.g. for MacOS, Tizen). Ofc headscale is a thing, but if you have a company, it's also nice to have someone to yell at if your mission-critical tailnet suddenly b0rks. Imo they don't charge all that much relative to their value, depending on who you're asking.
xarope 2y ago
I self host tailscale with headscale [0]. [0]: https://github.com/juanfont/headscale https://github.com/juanfont/headscale

HPsquared 2y ago

If you can't trust service providers, you probably also can't trust software suppliers.

stego-tech 2y ago
I mean, yes? It's why Zero Trust is growing as an operations model. Supply chain attacks, vendor hostility, zero days being hoarded by nations and bad actors for exploit, the list goes on. You emphatically cannot trust vendors, suppliers, users, software, systems, or governments. Ergo, your infrastructure should be built with an appropriate risk assessment in mind, and have proper safeguards in place where feasible. That's just good OpSec.
- deleted 2y ago
  [deleted]
kortilla 2y ago
Not the same. In particular you don’t need to accept software updates from software suppliers and you can also require source code or use open source. This stuff was obvious and standard in the 80s-2000s. It’s only in the last 15-20 years that it became acceptable to get updates shoved down your throat. Service providers can cut off your access any day. Software providers cannot unless you’ve given them a live update channel direct to your env.
drdaeman 2y ago
Definitely not true. You can audit software (it could be not easy, but ultimately doable) and skip the updates until you have capacity to audit those. You can't audit a third-party service, no matter what you do.

codetrotter 2y ago

> Is that a gigantic PITA to implement? Oh heck, you betcha it is I use my own self-hosted Wireguard VPN server. I agree with a lot of what you were saying about client certificates etc. And I plan to eventually do that sort of thing on some of my services in my own Wireguard VPN too. But in terms of Tailscale, if you are going to set up all kinds of client certificate things that will take a lot of time and effort, why not self-host Wireguard also? Setting up a Wireguard server is super simple. The only couple of things that complicate it a tiny bit is opening up a port for it for inbound connections if you host it from your home connection rather than a rented server, and managing the Wireguard public keys that are allowed to connect. But if you are going to do a whole client certificate setup on top anyway, the work of setting up your own Wireguard VPN is small in comparison. Unless like OP your ISP has put CGNAT on you.

stego-tech 2y ago
> But in terms of Tailscale, if you are going to set up all kinds of client certificate things that will take a lot of time and effort, why not self-host Wireguard also? Already do! I tried Tailscale initially, but ultimately decided to put in the effort of a proper Wireguard setup. It's how my personal devices always get back to my home LAN, and then exit to the internet; it's also how I make sure every DNS lookup hits the Pi-Hole, for domain blocking wherever I am. I emphatically recommend learning WireGuard (and to a lesser degree, VPN Concentration) when practical and possible. Until then, Tailscale is an excellent product.
selfhoster 2y ago
> Unless like OP your ISP has put CGNAT on you. I run Wireguard on a VPS and route public traffic with it over Wireguard to my home machine. Are you saying my ISP must not be CGNAT or else it wouldn't work?
- aborsy 2y ago
  How is this a good solution, when traffic is decrypted in the cloud, all traffic goes through one node, there is no ACL, key distribution, static IP, …? Tailscale addressed those issues.

xmichael909 2y ago

Why wouldn't you just not use Tailscale? What you are describing here is, ....

HumanOstrich 2y ago

> Call me Cappy Paranoid, but I fall into the camp of "You should never trust a service provider, ever," and build infrastructure accordingly; I believe this falls into an extreme interpretation of "zero trust". That's not what Zero Trust means, at all.

stego-tech 2y ago
…which is why I qualified it with the phrase, “extreme interpretation of”, and made sure to encapsulate “Zero trust” in quotes to make it clear I wasn’t being technically literal in my description. Grammar and punctuation matter when you’re deliberately misusing a known term as a metaphor to make a point. That being said, the core concept of ZTA is that no user or device should be trusted by default. So yes, my statement is still generally correct even if it’s not how the term is often or commonly used.

hamandcheese 2y ago

> I'd also consider going a step further by only permitting server or resource access based on client certificate validation This is where I'm the most curious on what Tailscale will do next. So far all their products seem to contrast at the IP level, but for enterprise use cases there's a real need for application level protections as well. Cloudflare Access is a great example of what I mean.

PLG88 2y ago

Particularly as it does not include its own PKI, so E2EE is done by MITM your IdP (OICD/SAML etc) and therefore, under court order Tailscale can decrypt your traffic. We took the opposite approach with NetFoundry. (1) We open sourced the code (https://openziti.io/ https://openziti.io/), (2) we built in PKI with private keys generated at source and destination so that even if traversing NF hosted data plane, we CANNOT decrypt traffic, (3) mTLS everywhere, (4) ability to bring your own PKI, and more.

aborsy 2y ago
First of all, a node added to tailnet doesn’t not have the ability to decrypt the traffic in tailnet. All it can do to contact other nodes, decrypt traffic users sent to that hidden node, or modify settings in admin console. Furthermore, with tail lock, the coordination server should not be able to add nodes from outside. Can you clarify?
- PLG88 2y ago
  Comment edited due to an incorrect understand which has been rectified.
  - aborsy 2y ago
    This is false information. Even if an attacker such as the government runs the coordination and relay servers, and the IdP, they will not be able to decrypt any traffic in tailnet. The secret keys remain on device, and traffic is end to end encrypted. There is no mechanism in the client agents to send out the secret keys. The coordination server receives the public keys and metadata. Please clarify or revise your comment!
    - PLG88 2y ago
      I see I did have a misunderstanding. I believe there is still the meta data angle, but yes, private keys on endpoints would ensure E2EE. I will update my comment.