10 ms·

I'm one of the authors of this. Happy to answer any questions. One of the fun technical details is that, when enabled on a machine (tailscale up --ssh), the us

I'm one of the authors of this. Happy to answer any questions. One of the fun technical details is that, when enabled on a machine (tailscale up --ssh), the userspace tailscaled process takes over all TCP port 22 packets after the WireGuard decryption and doesn't even feed them into the kernel over TUN. We use gVisor's netstack to handle the TCP connections in-process. So it doesn't matter whether you have other processes (or iptables rules, etc) that would prevent the Tailscale SSH server from binding to port 22. This lets people gradually use Tailscale SSH over time without messing with their system one. The Tailscale SSH server currently only runs on Linux but there's support in git main for macOS too but it's not super well tested yet and not included in the sandboxed GUI builds currently.

jdadj 4y ago
This is neat. I've used Cloudflare's Zero-Trust SSH, but I've been frustrated that it interacts poorly with sftp and scp because of the client-side changes that they make to ~/.ssh/config Does tailscale have the same issue?
- bradfitz 4y ago
  We don't modify or require changes to your SSH client. You can use any SSH client you want.
- xena 4y ago
  Tailscale employee here. Tailscale SSH works at the target side by listening on the SSH port on that machine. Client changes aren't needed for this to work, all that is required is to use your SSH client as normal. This should allow you to use sftp and scp without issues.
- jgeralnik 4y ago
  For what it's worth I encountered the same issue and came up with a solution: https://github.com/cloudflare/cloudflared/issues/574 https://github.com/cloudflare/cloudflared/issues/574 Cloudflare have ignored the github issue (which includes a solution) but at least 3 other people seem to have found my solution helpful.
  - jdadj 4y ago
    Thanks! Make that 4.
lolsoftware 4y ago
This looks great, and I'd love to replace AWS SSM (at least for the purposes of instance access) with this! One question I have is have is around device limits. With SSM, I can easily run an agent on every instance. Tailscale has pretty tight device limits on the Team and Business plans. I have no idea what the custom pricing looks like, but I'm guessing it would exceed my budget. What's the intended way to use this with a large number of servers? A small team can easily have more devices than 5x or 10x the number of users. Should we just set up some "gateway"/"bastion" instances to access via Tailscale SSH and then use regular ssh from there? Some sort of more limited device mode that doesn't count against the device limit (for ssh only, perhaps?) would be great.
- bradfitz 4y ago
  You could do a Tailscale SSH bastion thing, yeah. But before you build a funky setup to avoid pricing concerns, at least reach out to the sales folk to see what it is. We're usually pretty flexible on exact quotas and realize that different orgs have different user/device shapes.
- manfre 4y ago
  You'll want to set up a bastion host as a subnet router. https://tailscale.com/kb/1021/install-aws/ https://tailscale.com/kb/1021/install-aws/
e12e 4y ago
> (...) the userspace tailscaled process takes over all TCP port 22 packets after the WireGuard decryption and doesn't even feed them into the kernel over TUN. We use gVisor's netstack to handle the TCP connections in-process. > So it doesn't matter whether you have other processes (or iptables rules, etc) that would prevent the Tailscale SSH server from binding to port 22. This sounds like a great feature when exploiting buggy WordPress/php apps! /s I realize this is a feature - but it's a bit sad that the standard package handling isn't up to the task; leaving (I expect) the tailscale daemon as a "magic" netcitizen - not featuring in neither "ss" or "iptables" output (why can't I login to opensshd?).
- tptacek 4y ago
  How do you figure? The idea is that Tailscale is bypassing the kernel, which it can only do for requests coming in over the tailnet --- it gets those packets raw, directly from WireGuard, unlike the normal IP packets your kernel routes to/from localhost or an egress interface.
  - password4321 4y ago
    It's not the same, but "Docker Network bypasses Firewall, no option to disable" https://github.com/moby/moby/issues/22054 https://github.com/moby/moby/issues/22054 (2016)
    - tptacek 4y ago
      You're right, it's not at all the same. The Tailscale bypass exists (1) only for traffic traversing Tailscale interfaces (by design, that's the only traffic it can impact, because Tailscale can't run a userland TCP/IP stack for non-Tailscale traffic), and (2) only for this one feature, and (3) only if you've explicitly allowed it for particular users in your Tailscale ACLs. It's not clear to me how you could screw it up to, e.g., amplify an SSRF attack.
      - password4321 4y ago
        All I'm trying to point out is that advertising "this bypasses the firewall, by design" has been abused in the past. [edit] It boils down to principle of least surprise, managing expectations, etc. - proper documentation is indeed key.
infocollector 4y ago
Hi Brad: Thanks for helping out with this feature! I've been one of the early users of tailscale. My network is around 50 machines. I've recently started having issues with ssh on some of my machines, especially from mac m1 -> some ubuntu boxes. Could this be related to this new feature? Any suggestions/pointers on how to debug these issues?
- bradfitz 4y ago
  Tailscale SSH doesn't mess with your port 22 packets if it's off so almost certainly unrelated. Have you reached out to support or filed a bug?
dstaley 4y ago
Could you share some details about the embedded SSH server? I'm curious if this would work to add SSH capabilities to devices that run Tailscale but don't include a built-in SSH server. Previously I've used dropbear, so it'd be really nice to be able to drop that requirement!
- bradfitz 4y ago
  If you're already running recent-ish Tailscale on them, they're already running an SSH server that's just disabled. Run "tailscale up --ssh" to turn it on. The code's at https://github.com/tailscale/tailscale/tree/main/ssh/tailssh https://github.com/tailscale/tailscale/tree/main/ssh/tailssh for all the details. Which details in particular are you curious about?
  - dstaley 4y ago
    Ah it's using crypto/ssh to provide the SSH server, so I think that's all the details I needed! Can't wait to give this a shot.
  - ignoramous 4y ago
    > Which details in particular are you curious about? In the linked Q&A video with Maisem, you spoke about buying books on Linux to make SSH work. Which books, if you don't mind me asking?
    - bradfitz 4y ago
      The Linux Programming Interface: https://man7.org/tlpi/ https://man7.org/tlpi/ Advanced Programming in the UNIX Environment, 3rd Edition: https://www.amazon.com/gp/product/0321637739 https://www.amazon.com/gp/product/0321637739
aidos 4y ago
Just a quick thank you to the team working on Tailscale. It’s hands down the most seamless dev experience I’ve ever seen. Every time I think “such and such would be nice”, I search the docs, and it’s already implemented better than I could have expected (eg the stateless mode for ephemeral servers). I tinkered with cloudflare before that but just couldn’t get on with the interface of the admin tooling. With Tailscale I have a lot more confidence that I’ve set up the access rules as I need them. It’s all just a lot more obvious.
dimitar 4y ago
Can you do ssh tunnelling?
- maisem 4y ago
  You can do local port forwarding today, remote port forward is still a WIP. What do you want to use it for? Disclaimer: I am one of the engineers who built Tailscale SSH.
  - dimitar 4y ago
    Accessing Clojure nREPL servers, local port forwarding works for that. This allows me to inspect and modify running applications, a Lisp superpower.
867-5309 4y ago
hello, is sftp supported? thanks
- bradfitz 4y ago
  Yes: https://github.com/tailscale/tailscale/blob/v1.26.1/ssh/tailssh/tailssh.go#L264 https://github.com/tailscale/tailscale/blob/v1.26.1/ssh/tail...
atonse 4y ago
What would we need to have open in our security groups for this to work? I think ingress wouldn't be necessary since tailscaled creates a tunnel right? But how about egress traffic? UDP for WireGuard or something else?
- bradfitz 4y ago
  Security groups where? On the Tailscale ACL side, you need to allow tcp/22 in. On your host where you're running Tailscale, usually nothing. You can keep everything locked down for ingress. Outbound UDP only, but usually cloud VMs allow outbound traffic already. (This is covered more in https://tailscale.com/kb/1082/firewall-ports/ https://tailscale.com/kb/1082/firewall-ports/)
  - atonse 4y ago
    I meant AWS, that page explains it all, thanks!
ludsan 4y ago
quick question. Does this do user (de)provisioning like Jumpcloud? I.e. if the target machine doesn't have a /home/someuser but someuser is in my tailnet ACL, will it create the account?
- mfsch 4y ago
  From [1]: “Like other SSH clients, Tailscale will only use user accounts that already exist on the host, not create new accounts.” [1]: https://tailscale.com/kb/1193/tailscale-ssh/#ensure-tailscale-ssh-is-permitted-in-acls https://tailscale.com/kb/1193/tailscale-ssh/#ensure-tailscal...
tucif 4y ago
A nice next step would be tailscale managing an ssh key that's allowed to do interact with a git(hub) repository. So that I wouldn't have to create multiple keys or setup the same key on different machines and still be able to interact with a repo from all of them. It'd be really nice just using git transparently and having tailscale take over the git ssh connection and authenticate using taliscale access controls. At least for personal projects or small teams that'd be quite convenient.
- stormbrew 4y ago
  Depending on which part of those things you find painful, you might want to look into ssh certificates? They're pretty easy to work with, much easier than most kinds of certificate systems.
zachallaun 4y ago
Just want to say thanks: This is insanely cool/easy. Combined with the VSCode Remote SSH extension and MagicDNS, it's now insanely easy to work on a project in a remote environment. I was recently reading through a relatively long post on setting up SSH through Tailscale to access a WSL2 environment, and now it's literally as easy as popping open VSCode in any environment that I have Tailscale installed on and accessing `user@my-magic-dns-machine`. Great work!
YarickR2 4y ago
So we have no way to secure this besides disabling wireguard ?
- cassianoleal 4y ago
  There's no "disabling wireguard" in Tailscale unless you don't run it at all. You can secure this by: - Not enabling the SSH feature on hosts where it's not needed - Creating ACLs so only certain clients are allowed access. So essentially, just use the same mechanisms as for everything else in Tailscale.
mike_d 4y ago
How was the decision made to roll this functionality out before announcing it to customers (we found it during a previous security audit)? While it might seem logical in your mind to bolt on extra features and add value, your customers evaluate risk based on functionality of the software they are approving. Customer buys a VPN solution, magically gets remote access that bypasses firewalls. Can we trust Tailscale to not roll out a remote file backup feature and start silently exfiltrating data (as an extreme example)?
- bradfitz 4y ago
  There are two things have have to be enabled to turn it on: (1) a target server needs to run "tailscale up --ssh" to enable the SSH server (2) your Tailscale ACLs have to permit it. Our default, if you've never set your ACLs (as is usually the case for personal users), is that you're allowed to SSH to your own untagged devices only. For an org that's already using ACLs, you won't have any SSH rules defined and thus nobody in your org can enable the SSH server. (Or rather, they can enable it but nobody can connect to it.) If your concern an org that's using the default "all packets are allowed" ACLs?
  - kortilla 4y ago
    Concern more around what looks like an ssh backdoor showing up unannounced. How would they know the subtleties of what it takes to enable it when it wasn’t announced yet?
  - delano 4y ago
    That's one part of it. I can't speak for mike_d specifically, but there is a concern with having (potentially significant) modifications made to the codebase that aren't surfaced in the release notes. I imagine closed-source projects do this on a regular basis whether customers know (or care) or not. The expectations for opensource projects are different though, particularly when it comes to system-level or near system-level components. So not being able to access the functionality is a great default but it doesn't address side effects of the changes or the desire to know about changes being made in our environments.
Syzygies 4y ago
I have Tailscale on all my Macs. I use MacOS default SSH between my machines, but only via the Tailscale interface. Nevertheless, I had to open SSH on each machine, and it's a nightmare to close up the firewall so only Tailscale gets through. You'd think this was the whole point of Tailscale; there should be a one click lock to restrict to Tailscale. But the Tailscale documentation is wanting. I actually paid for a candidate for the best firewall front end, it came with "Let us know if we can help!" and radio silence once I explained the problem. Likely, restricting to Tailscale requires a granularity one can only hand-code. I can write a firewall, I've written plenty in the past, I just couldn't find the several hours to do this as a one-off for me when it should be easy, but I was missing needed information. Tailscale is justly proud of how it connects machines through uncooperative routers and such. Tailscale SSH should do the same. The idiot's guide to securing a machine so only Tailscale SSH gets through should be to find SSH in the preferences and turn the fucker off.
- justinsaccount 4y ago
  You don't really need a firewall to do that. putting ListenAddress 100.x.x.x where 100.x.x.x is the address on the tailnet, into your sshd_config would do what you want. Unfortunately you can't specify an interface, but if you have any sort of automation in place this is easy enough to template in.
- bradfitz 4y ago
  > Tailscale SSH should do the same. It does. You can "turn the fucker off" (as you say) at the OS level and Tailscale SSH will still work. We don't send the Tailscale SSH packets through the OS for it to block them. Well, Tailscale SSH server support for macOS is still not entirely done. You can build it from source if you're brave (and set an env var to turn it on), but it's not in the product yet by default.
bhawks 4y ago
Any interest in adding mosh like features (https://mosh.org/ https://mosh.org/)? Low latency typing, session resumption etc
- raggi 4y ago
  Tailscalar here, it should just work using the normal ssh bootstrapped method.
structural 4y ago
I've been using Tailscale for years but will likely not use this feature, even though I would like to. The fundamental problem with the approach really is that connections are different over the tailnet and over the local network. Here is a specific use case that is painful: 1. There exists a cluster of machines, each with large amounts of locally attached storage. They are all on the same local network and connected with 10Gb (and likely soon 40Gb ethernet interfaces). 2. Each machine is individually on the same tailnet so they can be accessed remotely. 3. Remote users frequently need to move large amounts of data between machines. A user copying a few hundred gigabytes of data with "scp" is normal. 4. For performance reasons, it's preferred to avoid the Tailscale/wireguard overhead when copying data between adjacent machines in a rack. At this point, if I enable tailscale ssh for remote login, it appears that the problem of key management for connections between local machines (using ssh over the normal interface, not the tailnet) still remains, and in fact, the overall authentication configuration is more complex than it was before. What I would love to exist, and would make me instantly use this feature, is if the tailnet issued SSH certificates (probably injected into its own ssh-agent?), the existing tailscale SSH implemention worked just like it currently does (it's great!), AND I could manually configure servers to accept certificates issued by the tailnet. Then SSH paths like "laptop --> (over tailnet) --> server 1 --> (over local network) --> server 2" could be made to work transparently, for those machines that need it, and for regular users, it still "just works".
- bradfitz 4y ago
  I agree that'd be fun. We have something similar in the works for other protocols, but maybe SSH isn't a huge stretch to extend it to!
  - ignoramous 4y ago
    Does the current setup with magicsock mean that tailssh behaves similar to MoSH (in dealing with resuming a session, specifically)?
    - bradfitz 4y ago
JayCuthrell 4y ago
Thank you to the Tailscale team for altering my belief that VPN could only stand for Vexing Productivity Neutralizer. Features like Tailscale SSH represent the ruthless removal of annoyances. edit: grammar
linsomniac 4y ago
Hey bradfitz, guy who previously had 32150 here. :-) This looks insanely cool, a couple questions: I know it says it's linux-only right now, but is that client side or server only? Can my Windows users TailSSH into linux boxes? Would be cool if somehow it could wedge into sudo auth so you could login as a a user and sudo without password if allowed by ACLs, especally if I could add "check" to the ssh. agent pam module? One thing that has prevented me from trying Tailscale, despite the great word on the street, is I can't figure out pricing, despite contacting sales. I'd like to run it on ~120 dev+stg+prod VMs, with 10 people (devs, testers, ops). I'd like every box to talk over tailscale directly, as an overlay network, but servers I hope aren't users, that'd get expensive fast. But I need more devices than 10/user. I presume "custom" would help with that but I got no reply from sales. We are probably too small fry. Now that I'm typing this, I realize I guess we could just buy ~15-20 users despite needing only 10. I think I've resolved myself to setting up Nebula for the server overlay network, and using Tailscale for physical users, with a traditional firewall bridging them. Again, Tailscale SSH looks very nice, job well done!
- bradfitz 4y ago
  > I know it says it's linux-only right now, but is that client side or server only? Can my Windows users TailSSH into linux boxes? Linux-only on the server right. macOS support is kinda there (in git) but not entirely done and not included in the GUI builds. Windows server support is tracked in https://github.com/tailscale/tailscale/issues/4697 https://github.com/tailscale/tailscale/issues/4697. You can use any SSH client from any OS. > Would be cool if somehow it could wedge into sudo auth so you could login as a a user and sudo without password if allowed by ACLs Some of the start of that is in https://github.com/tailscale/pam https://github.com/tailscale/pam > One thing that has prevented me from trying Tailscale, despite the great word on the street, is I can't figure out pricing, despite contacting sales. I'd like to run it on ~120 dev+stg+prod VMs, with 10 people (devs, testers, ops). I'd like every box to talk over tailscale directly, as an overlay network, but servers I hope aren't users, that'd get expensive fast. But I need more devices than 10/user. I presume "custom" would help with that but I got no reply from sales. We are probably too small fry. Now that I'm typing this, I realize I guess we could just buy ~15-20 users despite needing only 10. You only pay for unique humans, not tagged role account devices. I wonder if your email got eaten as spam or something. Email me (username at tailscale) and copy sales@ and I'll make sure somebody replies. But I don't think you need a custom plan. > I think I've resolved myself to setting up Nebula for the server overlay network, and using Tailscale for physical users, with a traditional firewall bridging them. Hey, if you've got something that works, stick with it. :)
woleium 4y ago
What are the failure modes? Openssh is a well understood risk, this seems... unquantifiable?
- tptacek 4y ago
  Tailscale is a userland process built in a memory-safe language, which leaves you only the SSH protocol cryptography-type vulnerabilities, which are themselves mooted by Tailscale (see downthread for a discussion about why they didn't simply expose rsh instead of ssh). This is safer than OpenSSH. (OpenSSH, as a piece of software, is extraordinarily safe, and has one of the best records of any memory-unsafe codebases. But OpenSSH as configurable infrastructure is much less safe; people screw it up all the time.)
  - woleium 4y ago
    Sure, but how reliable is it. What are the risks that I'm going to get locked out of a production instance and suffer loss of earnings?
dataangel 4y ago
Do you build tailscale with redo? and if not why not? :)
rastignack 4y ago
Were golang’s ssh and crypto packages independently audited ?
KRuchan 4y ago
I attempted this on a VM inside a Linux host and got a lower privileged user from inside the guest VM to ssh to a root-privileged user outside on the host. Both were authenticated to Tailscale with the same gmail account, so from an OAuth perspective, this is valid. From the OS perspective though, the host SSH port is blocked, and a guest should never get full root access to the host or see the host's resources. I am not sure if I am confused about something, or maybe there are prod use-cases where the same IDP identity should have different roles/privileges depending on the machine, and Tailscale SSH breaks that?
deleted 4y ago
[deleted]
CaptainJustin 4y ago
> This lets people gradually use Tailscale SSH over time without messing with their system one. That is something I have really appreciated about Tailscale. It seems to consistently not mess with the existing environment. Considering it does networking witchcraft and it works on a variety of architectures and OSs this is quite an accomplishment. I suspect Tailscale's customers have found the same.
- ngcc_hk 4y ago
  That is not a feature it is a bug and a big hole. The firewall is the system. Just like apple bypass its own firewall and just send packet back home. Or the chinese way. Of course as said by one of the author the key is to control port 22 or rule for ssh. That is not a totally lost. Still, one that is ok … you are breaking the system by promoting a way to bypass it. Or just 1 rule. It is so hard to remember.
  - tptacek 4y ago
    No, it's not. Network access control is the whole point of Tailscale; it is the network filtering layer. It serves literally the same function that a Checkpoint Firewall-1 installation would have in 1997, and that's why people buy it. This is basic stuff from the Tailscale website; it doesn't even qualify as analysis. You really ought to understand how these things work before you describe things as "big holes".
    - ikiris 4y ago
      Because that's what we all want. Yet another place to look for ACL rules...
      - tptacek 4y ago
        If you're deploying Tailscale? Yeah, that's about right.
      - samb1729 4y ago
        Considering how simple it is to use Tailscale ACL rules with node auto-tagging, yes I absolutely want it.

jdadj 4y ago

This is neat. I've used Cloudflare's Zero-Trust SSH, but I've been frustrated that it interacts poorly with sftp and scp because of the client-side changes that they make to ~/.ssh/config Does tailscale have the same issue?

bradfitz 4y ago
We don't modify or require changes to your SSH client. You can use any SSH client you want.
xena 4y ago
Tailscale employee here. Tailscale SSH works at the target side by listening on the SSH port on that machine. Client changes aren't needed for this to work, all that is required is to use your SSH client as normal. This should allow you to use sftp and scp without issues.
jgeralnik 4y ago
For what it's worth I encountered the same issue and came up with a solution: https://github.com/cloudflare/cloudflared/issues/574 https://github.com/cloudflare/cloudflared/issues/574 Cloudflare have ignored the github issue (which includes a solution) but at least 3 other people seem to have found my solution helpful.
- jdadj 4y ago
  Thanks! Make that 4.

lolsoftware 4y ago

This looks great, and I'd love to replace AWS SSM (at least for the purposes of instance access) with this! One question I have is have is around device limits. With SSM, I can easily run an agent on every instance. Tailscale has pretty tight device limits on the Team and Business plans. I have no idea what the custom pricing looks like, but I'm guessing it would exceed my budget. What's the intended way to use this with a large number of servers? A small team can easily have more devices than 5x or 10x the number of users. Should we just set up some "gateway"/"bastion" instances to access via Tailscale SSH and then use regular ssh from there? Some sort of more limited device mode that doesn't count against the device limit (for ssh only, perhaps?) would be great.

bradfitz 4y ago
You could do a Tailscale SSH bastion thing, yeah. But before you build a funky setup to avoid pricing concerns, at least reach out to the sales folk to see what it is. We're usually pretty flexible on exact quotas and realize that different orgs have different user/device shapes.
manfre 4y ago
You'll want to set up a bastion host as a subnet router. https://tailscale.com/kb/1021/install-aws/ https://tailscale.com/kb/1021/install-aws/

e12e 4y ago

> (...) the userspace tailscaled process takes over all TCP port 22 packets after the WireGuard decryption and doesn't even feed them into the kernel over TUN. We use gVisor's netstack to handle the TCP connections in-process. > So it doesn't matter whether you have other processes (or iptables rules, etc) that would prevent the Tailscale SSH server from binding to port 22. This sounds like a great feature when exploiting buggy WordPress/php apps! /s I realize this is a feature - but it's a bit sad that the standard package handling isn't up to the task; leaving (I expect) the tailscale daemon as a "magic" netcitizen - not featuring in neither "ss" or "iptables" output (why can't I login to opensshd?).

tptacek 4y ago
How do you figure? The idea is that Tailscale is bypassing the kernel, which it can only do for requests coming in over the tailnet --- it gets those packets raw, directly from WireGuard, unlike the normal IP packets your kernel routes to/from localhost or an egress interface.
- password4321 4y ago
  It's not the same, but "Docker Network bypasses Firewall, no option to disable" https://github.com/moby/moby/issues/22054 https://github.com/moby/moby/issues/22054 (2016)
  - tptacek 4y ago
    You're right, it's not at all the same. The Tailscale bypass exists (1) only for traffic traversing Tailscale interfaces (by design, that's the only traffic it can impact, because Tailscale can't run a userland TCP/IP stack for non-Tailscale traffic), and (2) only for this one feature, and (3) only if you've explicitly allowed it for particular users in your Tailscale ACLs. It's not clear to me how you could screw it up to, e.g., amplify an SSRF attack.
    - password4321 4y ago
      All I'm trying to point out is that advertising "this bypasses the firewall, by design" has been abused in the past. [edit] It boils down to principle of least surprise, managing expectations, etc. - proper documentation is indeed key.

infocollector 4y ago

Hi Brad: Thanks for helping out with this feature! I've been one of the early users of tailscale. My network is around 50 machines. I've recently started having issues with ssh on some of my machines, especially from mac m1 -> some ubuntu boxes. Could this be related to this new feature? Any suggestions/pointers on how to debug these issues?

bradfitz 4y ago
Tailscale SSH doesn't mess with your port 22 packets if it's off so almost certainly unrelated. Have you reached out to support or filed a bug?

dstaley 4y ago

Could you share some details about the embedded SSH server? I'm curious if this would work to add SSH capabilities to devices that run Tailscale but don't include a built-in SSH server. Previously I've used dropbear, so it'd be really nice to be able to drop that requirement!

bradfitz 4y ago
If you're already running recent-ish Tailscale on them, they're already running an SSH server that's just disabled. Run "tailscale up --ssh" to turn it on. The code's at https://github.com/tailscale/tailscale/tree/main/ssh/tailssh https://github.com/tailscale/tailscale/tree/main/ssh/tailssh for all the details. Which details in particular are you curious about?
- dstaley 4y ago
  Ah it's using crypto/ssh to provide the SSH server, so I think that's all the details I needed! Can't wait to give this a shot.
- ignoramous 4y ago
  > Which details in particular are you curious about? In the linked Q&A video with Maisem, you spoke about buying books on Linux to make SSH work. Which books, if you don't mind me asking?
  - bradfitz 4y ago
    The Linux Programming Interface: https://man7.org/tlpi/ https://man7.org/tlpi/ Advanced Programming in the UNIX Environment, 3rd Edition: https://www.amazon.com/gp/product/0321637739 https://www.amazon.com/gp/product/0321637739

aidos 4y ago

Just a quick thank you to the team working on Tailscale. It’s hands down the most seamless dev experience I’ve ever seen. Every time I think “such and such would be nice”, I search the docs, and it’s already implemented better than I could have expected (eg the stateless mode for ephemeral servers). I tinkered with cloudflare before that but just couldn’t get on with the interface of the admin tooling. With Tailscale I have a lot more confidence that I’ve set up the access rules as I need them. It’s all just a lot more obvious.

dimitar 4y ago

Can you do ssh tunnelling?

maisem 4y ago
You can do local port forwarding today, remote port forward is still a WIP. What do you want to use it for? Disclaimer: I am one of the engineers who built Tailscale SSH.
- dimitar 4y ago
  Accessing Clojure nREPL servers, local port forwarding works for that. This allows me to inspect and modify running applications, a Lisp superpower.

867-5309 4y ago

hello, is sftp supported? thanks

bradfitz 4y ago
Yes: https://github.com/tailscale/tailscale/blob/v1.26.1/ssh/tailssh/tailssh.go#L264 https://github.com/tailscale/tailscale/blob/v1.26.1/ssh/tail...

atonse 4y ago

What would we need to have open in our security groups for this to work? I think ingress wouldn't be necessary since tailscaled creates a tunnel right? But how about egress traffic? UDP for WireGuard or something else?

bradfitz 4y ago
Security groups where? On the Tailscale ACL side, you need to allow tcp/22 in. On your host where you're running Tailscale, usually nothing. You can keep everything locked down for ingress. Outbound UDP only, but usually cloud VMs allow outbound traffic already. (This is covered more in https://tailscale.com/kb/1082/firewall-ports/ https://tailscale.com/kb/1082/firewall-ports/)
- atonse 4y ago
  I meant AWS, that page explains it all, thanks!

ludsan 4y ago

quick question. Does this do user (de)provisioning like Jumpcloud? I.e. if the target machine doesn't have a /home/someuser but someuser is in my tailnet ACL, will it create the account?

mfsch 4y ago
From [1]: “Like other SSH clients, Tailscale will only use user accounts that already exist on the host, not create new accounts.” [1]: https://tailscale.com/kb/1193/tailscale-ssh/#ensure-tailscale-ssh-is-permitted-in-acls https://tailscale.com/kb/1193/tailscale-ssh/#ensure-tailscal...

tucif 4y ago

A nice next step would be tailscale managing an ssh key that's allowed to do interact with a git(hub) repository. So that I wouldn't have to create multiple keys or setup the same key on different machines and still be able to interact with a repo from all of them. It'd be really nice just using git transparently and having tailscale take over the git ssh connection and authenticate using taliscale access controls. At least for personal projects or small teams that'd be quite convenient.

stormbrew 4y ago
Depending on which part of those things you find painful, you might want to look into ssh certificates? They're pretty easy to work with, much easier than most kinds of certificate systems.

zachallaun 4y ago

Just want to say thanks: This is insanely cool/easy. Combined with the VSCode Remote SSH extension and MagicDNS, it's now insanely easy to work on a project in a remote environment. I was recently reading through a relatively long post on setting up SSH through Tailscale to access a WSL2 environment, and now it's literally as easy as popping open VSCode in any environment that I have Tailscale installed on and accessing `user@my-magic-dns-machine`. Great work!

YarickR2 4y ago

So we have no way to secure this besides disabling wireguard ?

cassianoleal 4y ago
There's no "disabling wireguard" in Tailscale unless you don't run it at all. You can secure this by: - Not enabling the SSH feature on hosts where it's not needed - Creating ACLs so only certain clients are allowed access. So essentially, just use the same mechanisms as for everything else in Tailscale.

mike_d 4y ago

How was the decision made to roll this functionality out before announcing it to customers (we found it during a previous security audit)? While it might seem logical in your mind to bolt on extra features and add value, your customers evaluate risk based on functionality of the software they are approving. Customer buys a VPN solution, magically gets remote access that bypasses firewalls. Can we trust Tailscale to not roll out a remote file backup feature and start silently exfiltrating data (as an extreme example)?

bradfitz 4y ago
There are two things have have to be enabled to turn it on: (1) a target server needs to run "tailscale up --ssh" to enable the SSH server (2) your Tailscale ACLs have to permit it. Our default, if you've never set your ACLs (as is usually the case for personal users), is that you're allowed to SSH to your own untagged devices only. For an org that's already using ACLs, you won't have any SSH rules defined and thus nobody in your org can enable the SSH server. (Or rather, they can enable it but nobody can connect to it.) If your concern an org that's using the default "all packets are allowed" ACLs?
- kortilla 4y ago
  Concern more around what looks like an ssh backdoor showing up unannounced. How would they know the subtleties of what it takes to enable it when it wasn’t announced yet?
- delano 4y ago
  That's one part of it. I can't speak for mike_d specifically, but there is a concern with having (potentially significant) modifications made to the codebase that aren't surfaced in the release notes. I imagine closed-source projects do this on a regular basis whether customers know (or care) or not. The expectations for opensource projects are different though, particularly when it comes to system-level or near system-level components. So not being able to access the functionality is a great default but it doesn't address side effects of the changes or the desire to know about changes being made in our environments.

Syzygies 4y ago

I have Tailscale on all my Macs. I use MacOS default SSH between my machines, but only via the Tailscale interface. Nevertheless, I had to open SSH on each machine, and it's a nightmare to close up the firewall so only Tailscale gets through. You'd think this was the whole point of Tailscale; there should be a one click lock to restrict to Tailscale. But the Tailscale documentation is wanting. I actually paid for a candidate for the best firewall front end, it came with "Let us know if we can help!" and radio silence once I explained the problem. Likely, restricting to Tailscale requires a granularity one can only hand-code. I can write a firewall, I've written plenty in the past, I just couldn't find the several hours to do this as a one-off for me when it should be easy, but I was missing needed information. Tailscale is justly proud of how it connects machines through uncooperative routers and such. Tailscale SSH should do the same. The idiot's guide to securing a machine so only Tailscale SSH gets through should be to find SSH in the preferences and turn the fucker off.

justinsaccount 4y ago
You don't really need a firewall to do that. putting ListenAddress 100.x.x.x where 100.x.x.x is the address on the tailnet, into your sshd_config would do what you want. Unfortunately you can't specify an interface, but if you have any sort of automation in place this is easy enough to template in.
bradfitz 4y ago
> Tailscale SSH should do the same. It does. You can "turn the fucker off" (as you say) at the OS level and Tailscale SSH will still work. We don't send the Tailscale SSH packets through the OS for it to block them. Well, Tailscale SSH server support for macOS is still not entirely done. You can build it from source if you're brave (and set an env var to turn it on), but it's not in the product yet by default.

bhawks 4y ago

Any interest in adding mosh like features (https://mosh.org/ https://mosh.org/)? Low latency typing, session resumption etc

raggi 4y ago
Tailscalar here, it should just work using the normal ssh bootstrapped method.

structural 4y ago

I've been using Tailscale for years but will likely not use this feature, even though I would like to. The fundamental problem with the approach really is that connections are different over the tailnet and over the local network. Here is a specific use case that is painful: 1. There exists a cluster of machines, each with large amounts of locally attached storage. They are all on the same local network and connected with 10Gb (and likely soon 40Gb ethernet interfaces). 2. Each machine is individually on the same tailnet so they can be accessed remotely. 3. Remote users frequently need to move large amounts of data between machines. A user copying a few hundred gigabytes of data with "scp" is normal. 4. For performance reasons, it's preferred to avoid the Tailscale/wireguard overhead when copying data between adjacent machines in a rack. At this point, if I enable tailscale ssh for remote login, it appears that the problem of key management for connections between local machines (using ssh over the normal interface, not the tailnet) still remains, and in fact, the overall authentication configuration is more complex than it was before. What I would love to exist, and would make me instantly use this feature, is if the tailnet issued SSH certificates (probably injected into its own ssh-agent?), the existing tailscale SSH implemention worked just like it currently does (it's great!), AND I could manually configure servers to accept certificates issued by the tailnet. Then SSH paths like "laptop --> (over tailnet) --> server 1 --> (over local network) --> server 2" could be made to work transparently, for those machines that need it, and for regular users, it still "just works".

bradfitz 4y ago
I agree that'd be fun. We have something similar in the works for other protocols, but maybe SSH isn't a huge stretch to extend it to!
- ignoramous 4y ago
  Does the current setup with magicsock mean that tailssh behaves similar to MoSH (in dealing with resuming a session, specifically)?
  - bradfitz 4y ago

JayCuthrell 4y ago

Thank you to the Tailscale team for altering my belief that VPN could only stand for Vexing Productivity Neutralizer. Features like Tailscale SSH represent the ruthless removal of annoyances. edit: grammar

linsomniac 4y ago

Hey bradfitz, guy who previously had 32150 here. :-) This looks insanely cool, a couple questions: I know it says it's linux-only right now, but is that client side or server only? Can my Windows users TailSSH into linux boxes? Would be cool if somehow it could wedge into sudo auth so you could login as a a user and sudo without password if allowed by ACLs, especally if I could add "check" to the ssh. agent pam module? One thing that has prevented me from trying Tailscale, despite the great word on the street, is I can't figure out pricing, despite contacting sales. I'd like to run it on ~120 dev+stg+prod VMs, with 10 people (devs, testers, ops). I'd like every box to talk over tailscale directly, as an overlay network, but servers I hope aren't users, that'd get expensive fast. But I need more devices than 10/user. I presume "custom" would help with that but I got no reply from sales. We are probably too small fry. Now that I'm typing this, I realize I guess we could just buy ~15-20 users despite needing only 10. I think I've resolved myself to setting up Nebula for the server overlay network, and using Tailscale for physical users, with a traditional firewall bridging them. Again, Tailscale SSH looks very nice, job well done!

bradfitz 4y ago
> I know it says it's linux-only right now, but is that client side or server only? Can my Windows users TailSSH into linux boxes? Linux-only on the server right. macOS support is kinda there (in git) but not entirely done and not included in the GUI builds. Windows server support is tracked in https://github.com/tailscale/tailscale/issues/4697 https://github.com/tailscale/tailscale/issues/4697. You can use any SSH client from any OS. > Would be cool if somehow it could wedge into sudo auth so you could login as a a user and sudo without password if allowed by ACLs Some of the start of that is in https://github.com/tailscale/pam https://github.com/tailscale/pam > One thing that has prevented me from trying Tailscale, despite the great word on the street, is I can't figure out pricing, despite contacting sales. I'd like to run it on ~120 dev+stg+prod VMs, with 10 people (devs, testers, ops). I'd like every box to talk over tailscale directly, as an overlay network, but servers I hope aren't users, that'd get expensive fast. But I need more devices than 10/user. I presume "custom" would help with that but I got no reply from sales. We are probably too small fry. Now that I'm typing this, I realize I guess we could just buy ~15-20 users despite needing only 10. You only pay for unique humans, not tagged role account devices. I wonder if your email got eaten as spam or something. Email me (username at tailscale) and copy sales@ and I'll make sure somebody replies. But I don't think you need a custom plan. > I think I've resolved myself to setting up Nebula for the server overlay network, and using Tailscale for physical users, with a traditional firewall bridging them. Hey, if you've got something that works, stick with it. :)

woleium 4y ago

What are the failure modes? Openssh is a well understood risk, this seems... unquantifiable?

tptacek 4y ago
Tailscale is a userland process built in a memory-safe language, which leaves you only the SSH protocol cryptography-type vulnerabilities, which are themselves mooted by Tailscale (see downthread for a discussion about why they didn't simply expose rsh instead of ssh). This is safer than OpenSSH. (OpenSSH, as a piece of software, is extraordinarily safe, and has one of the best records of any memory-unsafe codebases. But OpenSSH as configurable infrastructure is much less safe; people screw it up all the time.)
- woleium 4y ago
  Sure, but how reliable is it. What are the risks that I'm going to get locked out of a production instance and suffer loss of earnings?

dataangel 4y ago

Do you build tailscale with redo? and if not why not? :)

rastignack 4y ago

Were golang’s ssh and crypto packages independently audited ?

KRuchan 4y ago

I attempted this on a VM inside a Linux host and got a lower privileged user from inside the guest VM to ssh to a root-privileged user outside on the host. Both were authenticated to Tailscale with the same gmail account, so from an OAuth perspective, this is valid. From the OS perspective though, the host SSH port is blocked, and a guest should never get full root access to the host or see the host's resources. I am not sure if I am confused about something, or maybe there are prod use-cases where the same IDP identity should have different roles/privileges depending on the machine, and Tailscale SSH breaks that?

deleted 4y ago

[deleted]

CaptainJustin 4y ago

> This lets people gradually use Tailscale SSH over time without messing with their system one. That is something I have really appreciated about Tailscale. It seems to consistently not mess with the existing environment. Considering it does networking witchcraft and it works on a variety of architectures and OSs this is quite an accomplishment. I suspect Tailscale's customers have found the same.

ngcc_hk 4y ago
That is not a feature it is a bug and a big hole. The firewall is the system. Just like apple bypass its own firewall and just send packet back home. Or the chinese way. Of course as said by one of the author the key is to control port 22 or rule for ssh. That is not a totally lost. Still, one that is ok … you are breaking the system by promoting a way to bypass it. Or just 1 rule. It is so hard to remember.
- tptacek 4y ago
  No, it's not. Network access control is the whole point of Tailscale; it is the network filtering layer. It serves literally the same function that a Checkpoint Firewall-1 installation would have in 1997, and that's why people buy it. This is basic stuff from the Tailscale website; it doesn't even qualify as analysis. You really ought to understand how these things work before you describe things as "big holes".
  - ikiris 4y ago
    Because that's what we all want. Yet another place to look for ACL rules...
    - tptacek 4y ago
      If you're deploying Tailscale? Yeah, that's about right.
    - samb1729 4y ago
      Considering how simple it is to use Tailscale ACL rules with node auto-tagging, yes I absolutely want it.