5 ms·

Post author here. I'm happy to answer any questions / respond to any rants if you like.

supermatt 5y ago
Im still a bit confused about the "without the cloud" aspect. It seems that you still need NAT traversal, which means an external STUN server, and a relay for NATs that cannot be traversed - as well as a coordination/signalling broker. Surely those require some services "in the cloud"?
- apenwarr 5y ago
  In Tailscale, much like in SDN or SD-WAN, we think of the network in two parts, the control plane and the data plane. The data plane is how the bulk of your packets get sent from one place to another, which in Tailscale is peer-to-peer (as long as your network is not completely blocking NAT traversal for some reason). Even if NAT traversal is blocked and we have to relay your data through the cloud (through our DERP network) to make it work, the data plane is still end-to-end encrypted using private keys that only exist on each node. The private keys never leave each node. The DERP servers are just relaying opaque byte streams, like any IP router would do. On the other hand, the Tailscale control plane uses a central coordination service. It's used to exchange public keys and STUN information between nodes, but this is a tiny amount of information updated rarely (and therefore reasonably cheap for us to handle at scale). These public keys are not enough for an attacker (us or anyone else) to be able to decrypt your data traffic. So when we say taildrop never sends your files through the cloud, that's because taildrop exists entirely inside the data plane, sending data through e2e encrypted tunnels that have already been established with the help of the coordination service, STUN, etc. Because the coordination service is cheap for us to run, we can have a really generous Tailscale free plan without losing all our money. The paid plans are intended to be for "corporate network" situations where people want more centralized controls, audit trails, and so on.
  - judge2020 5y ago
    Really great insight into your pricing structure (just today I was wondering if you were just burning money with the generous free plan), thanks!
  - adtac 5y ago
    >as long as your network is not completely blocking NAT traversal for some reason Out of curiosity, how often does this happen in practice? Also, how would you even do this? Isn't NAT traversal a direct consequence of how firewalls work and always possible?
    - apenwarr 5y ago
      A trivial example is networks that totally block UDP and only allow TCP traffic on port 443, say. Tailscale has an article called How NAT Traversal Works with considerably more gory details if you’re interested.
  - jimmySixDOF 5y ago
    Session Border Controllers were a big part of any carrier VoIP deployment and focused on essentially the same problem : using the signaling plane to normalize a separate direct endpoint packet flow across variable networks and devices. You are implementing this same logic in your SD-WAN or do you use like an acme packet (->oracle) box now you operate at scale ?
oezi 5y ago
Does Tailscale include automatic client node updates in any way? How to ensure all clients are up to date at any time? How do you assign devices to logical networks? How many virtual networks can be run concurrently on a single device?
- apenwarr 5y ago
  MacOS, iOS, and Linux clients can use your native OS updates. Windows needs to be updated by hand or with something like chocolatey or MDM. But more importantly, we have a policy of not breaking old clients if we can possibly avoid it. So far we have never deprecated old clients. We extend our protocols in a backward compatible way, because unilaterally breaking your network infrastructure… really sucks. The way tailscale networks (tailnets) work is probably not how you’re used to thinking about them. Each node has its own view of the world, based on which nodes and services are shared with it in particular. We have security policy settings per domain, and a node sharing UI that lets you share any of your devices with anyone else. The default model is that all devices belonging to someone in the same domain, say tailscale.com, can see each other. But we’re working on making that even more flexible since it doesn’t always do what you want for huge orgs (like universities).
  - oezi 5y ago
    > ... updates ... Do you think it is sufficient to rely on update channels via distributions? Wouldn't a bug in your code potentially expose an internal node to the internet? > Each node has its own view of the world I haven't read the docs enough, but can a node belong to many domains at once? If so, does it need one port per domain that it is shared on?
    - xena 5y ago
      > Do you think it is sufficient to rely on update channels via distributions? Tailscale employee here. Most officially supported distributions use our own package repo server (https://pkgs.tailscale.com https://pkgs.tailscale.com), which would pull Tailscale updates in your normal system updates. The other distributions that aren't in the package repo server (Alpine, Arch, Gentoo, NixOS, Void Linux, etc.) use packages made by the distribution themselves. We do our best to make sure they get updated (contacting the maintainers can be a slog at times), but we do not completely control the update process for them. > I haven't read the docs enough, but can a node belong to many domains at once? If so, does it need one port per domain that it is shared on? Not currently, follow this bug (https://github.com/tailscale/tailscale/issues/713 https://github.com/tailscale/tailscale/issues/713) to be updated on the details for this. You can sorta hack around it with node sharing (https://tailscale.com/kb/1084/sharing/ https://tailscale.com/kb/1084/sharing/), but that's unidirectional instead of bidirectional.
ignoramous 5y ago
re:localapi: Are these httpd instances? If so, are they run on-demand or are always running? Curious how this plays out on mobile devices with regards to power consumption? re:whois: I presume this is in control of a central identity service. I see that the private tailscale network is (mostly) p2p and (always) e2e, so wondering if you envision a future where the tailscale network goes decentralized without a central control àla BitTorrent / LimeWire (despite [0])? re:peerapi: Excuse my naivety, but couldn't this binary transport be used for file transfers too? Or, does http simply provide too many (file transfer) options [1] to not bother re-implementing it in a custom protocol? Thanks. [0] https://apenwarr.ca/log/20201227 https://apenwarr.ca/log/20201227 (when's vol.2 out?) [1] I can see screensharing up next, a keybase-like chat client, and even live video / event streaming?
- apenwarr 5y ago
  The localapi is indeed an http server built into the tailscaled process, which is written in Go. Since we already had an http client in there, the net new code to add an http server is quite low. And it doesn’t take any battery unless it’s being queried. I think people underestimate how cheap http can be (once you’ve paid the up front cost, anyway). You’ve correctly guessed that blog link [0] that explains the reason I don’t think we’d ever want to try making a distributed coordination service. Most importantly, corporate customers absolutely love having a single control and registration point for every corporate authorized device on their network (and thus, a way to instantly deauthorize stolen devices). What we’re going to do though is add private audit trails and tamper proofing, kind of like TLS certificate transparency, so that the central instructions can be validated in a decentralized way, if that makes sense. More on that later. :) Re: peerapi, there are lots of ways to build app layer protocols once you have tailscale making the connection itself easier. We picked http since it was the fewest lines of code and it makes an easy example. Re: live video, Jitsi already works fine on a tailscale network if you want to try that.
  - ignoramous 5y ago
    > And it doesn’t take any battery unless it’s being queried. I think people underestimate how cheap http can be... Curious about the underlying design decision on why a separate peerapi layer if a golang http/2 server is listening already (or is peerapi running over http, too)? > What we’re going to do though is add private audit trails and tamper proofing, kind of like TLS certificate transparency, so that the central instructions can be validated in a decentralized way. Exciting. Reminds me of: https://blog.okturtles.org/2014/09/the-trouble-with-certificate-transparency/ https://blog.okturtles.org/2014/09/the-trouble-with-certific... and https://book.keybase.io/docs/teams/sigchain https://book.keybase.io/docs/teams/sigchain > ...there are lots of ways to build app layer protocols once you have tailscale making the connection itself easier. True. My previous employer built an internal service similar to tailscale but it worked over bluetooth, wifi-direct in addition to ICEing NATs out. It made device discovery, cross-app, cross-device, cross-service communication super easy. Thanks again.
oarsinsync 5y ago
In a world moving towards "zero trust networking", this appears to be going in the opposite direction, where the network is now being implicitly trusted. This enables unauthenticated file transfers between hosts on the same network. Given the world we live in today is that RCE vulnerabilities are relatively common, what happens when host1 gets some malware and uses this to transfer itself onto host2? I assume this has been considered, and it's been decided that the convenience of the feature outweighs the security and reputation risk considerations?
- iso1210 5y ago
  Isn't the tailscale work more to do with authenticating hosts than the network. You trust all communication decodable by a trusted public key. That still allows malware on a trusted end device to connect to you, but then if you've got malware on the box chances are it will have access to the private keys, authentication cookies etc, anyway - at least currently.
- carlosf 5y ago
  I guess in an ideal world where everything in your org is behind MFA, app authentication and protected from denial of service, then solutions like tailscale would not be needed. But my personal experience is that most software is completely hostile to good security practices and you end up having some perimeter security as well.
- apenwarr 5y ago
  Someone else mentioned that basically malware on your machine is expected to bypass all security layers. So we’re basically saying “the ‘new’ network layer is now as trustworthy as the app layer” rather than claiming everything is perfect, you’re right. This is one reason we limited taildrop to only transfer between devices owned by a single user for now, and only to drop files into controlled locations. Tailscale also has ACL policies for when you don’t trust all the endpoints to just do anything.
JacobiX 5y ago
I use Tailscale every day I like the simplicity of setting up a new node. My problem is that I don’t know how to separate my nodes into isolated groups, so my question is does it support some form of multi-tenancy ?
- ricardbejarano 5y ago
  There is ACL support on the Admin console. Lets you choose which users/machines/tags can talk to who. Tags are like groups.
  - JacobiX 5y ago
    Thank you, yes we are using this feature but I was thinking of something like creating separate isolated network for each group of machines. For example I manage two different cloud infrastructures that don’t share anything in common. It would be more convenient if I could place them in different tenants.
ithkuil 5y ago
Awesome work, I wish I could use it. Unfortunately I use tailscale at work too and until https://github.com/tailscale/tailscale/issues/713 https://github.com/tailscale/tailscale/issues/713 I'm turned off from using tailscale also for my personal stuff since I don't want to physically switch to a different set of machines for work and personal projects.
mwcampbell 5y ago
> I wrote our Windows Taildrop GUI. I reserve the right to make fun of it. Is the GUI open source somewhere?
- mwcampbell 5y ago
  Found the answer, in the headscale readme: > Everything in Tailscale is Open Source, except the GUI clients for proprietary OS (Windows and macOS/iOS), and the 'coordination/control server'. Keeping the GUI clients for proprietary operating systems closed-source is certainly a valid business decision. It's just unfortunate that it means we don't get to fix any issues we find that may matter more to us than they currently do to the Tailscale team, e.g. accessibility. Edit: Yes, I know, to be fully consistent on this point, I should run an open-source OS. But then, the proprietary operating systems have accessibility teams, while presumably Tailscale does not.
jd3 5y ago
This is really excellent work — congrats! I previously used upspin to securely move files between devices on my network, but I really love the simplicity and ease of use that taildrop now provides. To give a concrete example, I had been dragging my feet on moving some old but useful keys/tokens between my Windows desktop and MacBook — using taildrop, the transfer was both easy and virtually instantaneous. Historically, it's always been such a pain to either have to upload/download the file(s) from a cloud service (what if I need to move private keys? that adds another layer of complexity/annoyance because then I need to encrypt them) or find a usb drive, plug it in, copy the files, eject, find the adapter for my macbook, plug it in, copy the files, eject, etc. Taildrop completely fixes all of that — it's amazing!

Im still a bit confused about the "without the cloud" aspect. It seems that you still need NAT traversal, which means an external STUN server, and a relay for NATs that cannot be traversed - as well as a coordination/signalling broker. Surely those require some services "in the cloud"?

apenwarr 5y ago
In Tailscale, much like in SDN or SD-WAN, we think of the network in two parts, the control plane and the data plane. The data plane is how the bulk of your packets get sent from one place to another, which in Tailscale is peer-to-peer (as long as your network is not completely blocking NAT traversal for some reason). Even if NAT traversal is blocked and we have to relay your data through the cloud (through our DERP network) to make it work, the data plane is still end-to-end encrypted using private keys that only exist on each node. The private keys never leave each node. The DERP servers are just relaying opaque byte streams, like any IP router would do. On the other hand, the Tailscale control plane uses a central coordination service. It's used to exchange public keys and STUN information between nodes, but this is a tiny amount of information updated rarely (and therefore reasonably cheap for us to handle at scale). These public keys are not enough for an attacker (us or anyone else) to be able to decrypt your data traffic. So when we say taildrop never sends your files through the cloud, that's because taildrop exists entirely inside the data plane, sending data through e2e encrypted tunnels that have already been established with the help of the coordination service, STUN, etc. Because the coordination service is cheap for us to run, we can have a really generous Tailscale free plan without losing all our money. The paid plans are intended to be for "corporate network" situations where people want more centralized controls, audit trails, and so on.
- judge2020 5y ago
  Really great insight into your pricing structure (just today I was wondering if you were just burning money with the generous free plan), thanks!
- adtac 5y ago
  >as long as your network is not completely blocking NAT traversal for some reason Out of curiosity, how often does this happen in practice? Also, how would you even do this? Isn't NAT traversal a direct consequence of how firewalls work and always possible?
  - apenwarr 5y ago
    A trivial example is networks that totally block UDP and only allow TCP traffic on port 443, say. Tailscale has an article called How NAT Traversal Works with considerably more gory details if you’re interested.
- jimmySixDOF 5y ago
  Session Border Controllers were a big part of any carrier VoIP deployment and focused on essentially the same problem : using the signaling plane to normalize a separate direct endpoint packet flow across variable networks and devices. You are implementing this same logic in your SD-WAN or do you use like an acme packet (->oracle) box now you operate at scale ?

oezi 5y ago

Does Tailscale include automatic client node updates in any way? How to ensure all clients are up to date at any time? How do you assign devices to logical networks? How many virtual networks can be run concurrently on a single device?

apenwarr 5y ago
MacOS, iOS, and Linux clients can use your native OS updates. Windows needs to be updated by hand or with something like chocolatey or MDM. But more importantly, we have a policy of not breaking old clients if we can possibly avoid it. So far we have never deprecated old clients. We extend our protocols in a backward compatible way, because unilaterally breaking your network infrastructure… really sucks. The way tailscale networks (tailnets) work is probably not how you’re used to thinking about them. Each node has its own view of the world, based on which nodes and services are shared with it in particular. We have security policy settings per domain, and a node sharing UI that lets you share any of your devices with anyone else. The default model is that all devices belonging to someone in the same domain, say tailscale.com, can see each other. But we’re working on making that even more flexible since it doesn’t always do what you want for huge orgs (like universities).
- oezi 5y ago
  > ... updates ... Do you think it is sufficient to rely on update channels via distributions? Wouldn't a bug in your code potentially expose an internal node to the internet? > Each node has its own view of the world I haven't read the docs enough, but can a node belong to many domains at once? If so, does it need one port per domain that it is shared on?
  - xena 5y ago
    > Do you think it is sufficient to rely on update channels via distributions? Tailscale employee here. Most officially supported distributions use our own package repo server (https://pkgs.tailscale.com https://pkgs.tailscale.com), which would pull Tailscale updates in your normal system updates. The other distributions that aren't in the package repo server (Alpine, Arch, Gentoo, NixOS, Void Linux, etc.) use packages made by the distribution themselves. We do our best to make sure they get updated (contacting the maintainers can be a slog at times), but we do not completely control the update process for them. > I haven't read the docs enough, but can a node belong to many domains at once? If so, does it need one port per domain that it is shared on? Not currently, follow this bug (https://github.com/tailscale/tailscale/issues/713 https://github.com/tailscale/tailscale/issues/713) to be updated on the details for this. You can sorta hack around it with node sharing (https://tailscale.com/kb/1084/sharing/ https://tailscale.com/kb/1084/sharing/), but that's unidirectional instead of bidirectional.

ignoramous 5y ago

re:localapi: Are these httpd instances? If so, are they run on-demand or are always running? Curious how this plays out on mobile devices with regards to power consumption? re:whois: I presume this is in control of a central identity service. I see that the private tailscale network is (mostly) p2p and (always) e2e, so wondering if you envision a future where the tailscale network goes decentralized without a central control àla BitTorrent / LimeWire (despite [0])? re:peerapi: Excuse my naivety, but couldn't this binary transport be used for file transfers too? Or, does http simply provide too many (file transfer) options [1] to not bother re-implementing it in a custom protocol? Thanks. [0] https://apenwarr.ca/log/20201227 https://apenwarr.ca/log/20201227 (when's vol.2 out?) [1] I can see screensharing up next, a keybase-like chat client, and even live video / event streaming?

apenwarr 5y ago
The localapi is indeed an http server built into the tailscaled process, which is written in Go. Since we already had an http client in there, the net new code to add an http server is quite low. And it doesn’t take any battery unless it’s being queried. I think people underestimate how cheap http can be (once you’ve paid the up front cost, anyway). You’ve correctly guessed that blog link [0] that explains the reason I don’t think we’d ever want to try making a distributed coordination service. Most importantly, corporate customers absolutely love having a single control and registration point for every corporate authorized device on their network (and thus, a way to instantly deauthorize stolen devices). What we’re going to do though is add private audit trails and tamper proofing, kind of like TLS certificate transparency, so that the central instructions can be validated in a decentralized way, if that makes sense. More on that later. :) Re: peerapi, there are lots of ways to build app layer protocols once you have tailscale making the connection itself easier. We picked http since it was the fewest lines of code and it makes an easy example. Re: live video, Jitsi already works fine on a tailscale network if you want to try that.
- ignoramous 5y ago
  > And it doesn’t take any battery unless it’s being queried. I think people underestimate how cheap http can be... Curious about the underlying design decision on why a separate peerapi layer if a golang http/2 server is listening already (or is peerapi running over http, too)? > What we’re going to do though is add private audit trails and tamper proofing, kind of like TLS certificate transparency, so that the central instructions can be validated in a decentralized way. Exciting. Reminds me of: https://blog.okturtles.org/2014/09/the-trouble-with-certificate-transparency/ https://blog.okturtles.org/2014/09/the-trouble-with-certific... and https://book.keybase.io/docs/teams/sigchain https://book.keybase.io/docs/teams/sigchain > ...there are lots of ways to build app layer protocols once you have tailscale making the connection itself easier. True. My previous employer built an internal service similar to tailscale but it worked over bluetooth, wifi-direct in addition to ICEing NATs out. It made device discovery, cross-app, cross-device, cross-service communication super easy. Thanks again.

oarsinsync 5y ago

In a world moving towards "zero trust networking", this appears to be going in the opposite direction, where the network is now being implicitly trusted. This enables unauthenticated file transfers between hosts on the same network. Given the world we live in today is that RCE vulnerabilities are relatively common, what happens when host1 gets some malware and uses this to transfer itself onto host2? I assume this has been considered, and it's been decided that the convenience of the feature outweighs the security and reputation risk considerations?

iso1210 5y ago
Isn't the tailscale work more to do with authenticating hosts than the network. You trust all communication decodable by a trusted public key. That still allows malware on a trusted end device to connect to you, but then if you've got malware on the box chances are it will have access to the private keys, authentication cookies etc, anyway - at least currently.
carlosf 5y ago
I guess in an ideal world where everything in your org is behind MFA, app authentication and protected from denial of service, then solutions like tailscale would not be needed. But my personal experience is that most software is completely hostile to good security practices and you end up having some perimeter security as well.
apenwarr 5y ago
Someone else mentioned that basically malware on your machine is expected to bypass all security layers. So we’re basically saying “the ‘new’ network layer is now as trustworthy as the app layer” rather than claiming everything is perfect, you’re right. This is one reason we limited taildrop to only transfer between devices owned by a single user for now, and only to drop files into controlled locations. Tailscale also has ACL policies for when you don’t trust all the endpoints to just do anything.

JacobiX 5y ago

I use Tailscale every day I like the simplicity of setting up a new node. My problem is that I don’t know how to separate my nodes into isolated groups, so my question is does it support some form of multi-tenancy ?

ricardbejarano 5y ago
There is ACL support on the Admin console. Lets you choose which users/machines/tags can talk to who. Tags are like groups.
- JacobiX 5y ago
  Thank you, yes we are using this feature but I was thinking of something like creating separate isolated network for each group of machines. For example I manage two different cloud infrastructures that don’t share anything in common. It would be more convenient if I could place them in different tenants.

ithkuil 5y ago

Awesome work, I wish I could use it. Unfortunately I use tailscale at work too and until https://github.com/tailscale/tailscale/issues/713 https://github.com/tailscale/tailscale/issues/713 I'm turned off from using tailscale also for my personal stuff since I don't want to physically switch to a different set of machines for work and personal projects.

mwcampbell 5y ago

> I wrote our Windows Taildrop GUI. I reserve the right to make fun of it. Is the GUI open source somewhere?

mwcampbell 5y ago
Found the answer, in the headscale readme: > Everything in Tailscale is Open Source, except the GUI clients for proprietary OS (Windows and macOS/iOS), and the 'coordination/control server'. Keeping the GUI clients for proprietary operating systems closed-source is certainly a valid business decision. It's just unfortunate that it means we don't get to fix any issues we find that may matter more to us than they currently do to the Tailscale team, e.g. accessibility. Edit: Yes, I know, to be fully consistent on this point, I should run an open-source OS. But then, the proprietary operating systems have accessibility teams, while presumably Tailscale does not.

jd3 5y ago

This is really excellent work — congrats! I previously used upspin to securely move files between devices on my network, but I really love the simplicity and ease of use that taildrop now provides. To give a concrete example, I had been dragging my feet on moving some old but useful keys/tokens between my Windows desktop and MacBook — using taildrop, the transfer was both easy and virtually instantaneous. Historically, it's always been such a pain to either have to upload/download the file(s) from a cloud service (what if I need to move private keys? that adds another layer of complexity/annoyance because then I need to encrypt them) or find a usb drive, plug it in, copy the files, eject, find the adapter for my macbook, plug it in, copy the files, eject, etc. Taildrop completely fixes all of that — it's amazing!