12 ms·
CVE-2022-41924 – tailscaled can be used to remotely execute code on Windows
- ghuntley 4y agoTechnical write up by the security researcher at https://emily.id.au/tailscale https://emily.id.au/tailscale ps. she's looking an employer rn // hire her!
- kbenson 4y agoAlways nice to see a vendor respond quickly and adequately. Even better when they go above and beyond, as seems to be indicated here by the timeline posted and the assessment of what fixes were put in place by the researcher(s). Good job Tailscale.
- dang 4y agoThat article has so much more information in it that I've changed the URL to that from https://tailscale.com/security-bulletins/#ts-2022-004 https://tailscale.com/security-bulletins/#ts-2022-004. Thanks!
- archon810 4y agoI submitted it 7 hours ago, but it didn't get any traction with the original title https://news.ycombinator.com/item?id=33695800 https://news.ycombinator.com/item?id=33695800.
- dang 4y agoAh sorry - we're eventually going to implement karma sharing to take care of cases like this. Often I search to see if someone else has already submitted the 'better' URL but I forgot to do that here. There's a lot of randomness in which submissions get noticed and achieve liftoff from /newest. At least it evens out over time if you continue to post good submissions!
- deleted 4y ago[deleted]
- beanjuiceII 4y agosounds like tailscale should
- ghuntley 4y agothat would be a really cool outcome (tm)
- deleted 4y ago[deleted]
- erdaniels 4y agocan you DM me her email? I can't find it anywhere and I'd love to start a chat with on a security position at my work.
- er4hn 4y agowhere does she say she is looking for an employer? Would be worthwhile to start a conversation with her.
- ghuntley 4y agoEm interned with me earlier this year. Have passed the threads over to her.
- Loic 4y agoThe quality of her writing is wonderful, clear, to the point, funny and of course all the technical details are well explained. For a non native English reader like me (I am French), a real pleasure to read. You can congratulate her!
- LVB 4y agohttps://emily.id.au/tailscale#co-authored-by https://emily.id.au/tailscale#co-authored-by mentions looking for work.
- IshKebab 4y agoVery well written. Also quite worrying given they're supposed to be a security company and these kinds of issues are well known. Then again it does seem like the entire universe applies "eh probably nobody will try and hack it" to services listening on local TCP interfaces. They certainly don't care about multi-user machines, though I suppose there are so many local root exploits these days you're basically trusting your users anyway in that situation.
- dathinab 4y ago> Also quite worrying Linux is used by by docent "security companies" and still has vulnerabilities from time to time. So does Apple in the products where their care about security, etc. More important are three things to realize: 1. their handling of the incident, which wasn't just fast but you could say absurdly fast to a point that I'm pretty sure multiple employees dropped everything the moment they read the mail to solely focused on fixing and analyzing it 2. it's Windows only (at least it's main problem is), coming from a workaround for a feature "missing" in windows from a company which is relative young and started out in the Linux/UNIX space. 3. it is exploitable due to a fundamental design flaw of browsers Or what I'm trying to say: It isn't that surprising (or worrying) that such a thing happened, what matters is how they handle it and make sure that it will not happen again.
- IshKebab 4y ago> a feature "missing" in windows It's not missing. Windows has named pipes. They just don't use the same API as Unix sockets so you would have to do some work to do it properly (or use an existing library that abstracts both interfaces). In any case Windows has had support for Unix sockets since 2017 so there's really no excuse. > Linux is used by by docent "security companies" and still has vulnerabilities from time to time. So does Apple in the products where their care about security, etc. Right but neither of them are companies whose sole product is a security product. I agree their response time is good - they probably realised how bad it looks!
- oefnak 4y agoWell, since tailscale doesn't auto update, it doesn't matter how fast the response is. Updating it will take a long time anyway.
- intelVISA 4y agono love for our homie Jamie, Geoffrey?
- mikehotel 4y agoBest exploit write up I’ve read. All the details, interspersed with nonchalant humor. “None of these words are in the Bible. This is certainly true of the King James Version, but many of these words can be found in what might as well be the Old Testament of Same-Origin Policy: the WhatWG Fetch Standard, which defines the CORS rules we are being accused of violating.”
- jakedata 4y agoThe client app is not indicating that 1.32.3 for Windows is available yet but the download link on the site has been updated. Tailscale client downloads are extremely slow at the moment, so I suggest you distribute one copy manually around your tailnet rather than bogging down their servers even more.
- mayakacz 4y agoTailscalar here. The Windows client caches the current version for a while, so may not yet have v1.32.3 available on your device. In that case, you can still pull the latest release from http://pkgs.tailscale.com/stable http://pkgs.tailscale.com/stable.
- 4y ago
- meibo 4y agoDo they have enough logs to reach out to people that were affected? As far as vulnerabilities go, this set is one is one of the worst ones I've seen this decade, and they seem rather straightforward. Would be nice to get a blog post from them that goes a bit into impact, not just a report that tells you to update. It's nice that they responded quickly, but I feel like this shouldn't have happened in the first place for a network security company and it makes the Windows client feel like a bit of an afterthought. Looks like they have a PR open to switch it to named pipes, I hope that is properly reviewed by someone that knows Windows APIs before it's merged.
- arkadiyt 4y agoedit: I stand corrected as pointed out by the replies below. Curious what logs they had to prove this! Original comment: > Do they have enough logs to reach out to people that were affected? It happens on the client, there are no server logs that Tailscale could check
- mlyle 4y ago"Reviewing all logs confirms this vulnerability was not triggered or exploited."
- deleted 4y ago[deleted]
- bradfitz 4y agoReconfiguring the local client daemon (tailscaled) is reported to Tailscale log servers so there's server-side evidence if it's exploited.
- cmeacham98 4y ago> Curious what logs they had to prove this! My guess is the client sends some kind of "goodbye" message when it gets reconfigured to another coordination server, and that message has enough information to determine if it originated from this attack.
- tailspin2019 4y ago> The speed and quality of Tailscale's response to our report is unlike any vendor interaction I have experienced, and suggests a deep commitment to keeping their customers safe. I have mixed feelings here as a Tailscale customer. Yes a quick response is great, but this actual security issue is pretty terrible IMHO. Anything other than an immediate response would have been akin to lighting their company on fire and walking away.
- bombcar 4y agoYour username would have been a good option for a cutesy name for the vulnerability, however.
- hellcow 4y ago> Anything other than an immediate response would have been akin to lighting their company on fire and walking away. Have we forgotten Zoom, who reinstalled itself secretly on user machines with an RCE-vulnerable server, which they described as “working as intended?” They’re still wildly popular today with organizations despite the insane lack of regard for security and their users’ safety. Mistakes happen. I applaud Tailscale for moving so quickly and doing the right thing.
- xmodem 4y agoI would caution against grading on a curve
- bobbylarrybobby 4y agoThe users of tailscale and of zoom expect very different things from their respective software vendors
- deleted 4y ago[deleted]
- acuteaura 4y agoAs the reporting party of the much less severe (and much less interesting) TS-2022-003[1] I can confirm that vendor interaction is also swift when not responding would not light the company on fire. In fact there were several other vendors that were affected that have yet to mitigate it. [1]: https://notes.acuteaura.net/posts/github-enterprise-security/ https://notes.acuteaura.net/posts/github-enterprise-security...
- cassianoleal 4y agoTo potentially save you a click: Who is affected? All Windows clients prior to version v1.32.3 are affected.
- ferbivore 4y agoReleasing a patch and a detailed write-up on the same day seems like a bit of an unfortunate choice, especially for a WTF!! vulnerability like this. In software that doesn't auto-update, no less...
- meibo 4y agoEspecially as the fixes seemingly have been going into their public GitHub branch for days, since the report. I wonder if that was a conscious choice or negligience, maybe I'm missing something? I would expect these to be released as patches/merged in when the vulnerability is published, like a lot of other security-critical open source software does it.
- jeroenhd 4y agoLooking at the timeline, it looks like Tailscale opted to allow for public release on the day of the patch: > Sat 19 November: Coordinated Disclosure time proposed by Tailscale, accepted by us, Tailscale shares planned Security Bulletins and blog post > Tue 22 November, 5:06AM: Blog draft shared with Tailscale (a bit last minute, sorry!!!) > Tue 22 November, 7:00AM: Coordinated disclosure time Because the code is open source anyway, I'm guessing they assume attackers would see the announcement of a vulnerability, browse the recent pull requests and figure it all out themselves anyway. Delaying publishing of the details saves maybe a few days of exposure to risk for motivated attackers, especially as the author seems to have done her work together with one other person in just over a week. They've also sent out emails it seems, so people know they should update ASAP and why. With the extremely limited amount of people running Tailscale (and the even smaller subgroup running it on Windows specifically) I don't think it's an attack hackers will rush to roll out. Mitigations also exist (i.e. block access from the browser to 100.100.100.100) so even in situations where you cannot update you can protect yourself.
- amluto 4y agoI don't see a writeup of how this was fixed. Merely checking the Host header is insufficient -- the vulnerability would still be wide open to anyone who can open TCP sockets to localhost. Windows has APIs (named pipes, DCOM (eww) and such) that allow authenticated local access to services. Unixes have unix sockets.
- Arnavion 4y agoWindows from W10 onwards has Unix sockets too.
- monocasa 4y agoThe windows implementation lacks facilities like SCM_RIGHTS though to ask the kernel who's on the other side.
- Arnavion 4y agoYes, but their existing TCP implementation wouldn't have been doing any auth either. So presumably they don't need it. (I don't know anything about Tailscale so I'm just going on first principles.)
- JJJollyjim 4y ago[co-author of the research here] They actually approximate this functionality in the Windows implementation: It checks netstat to enforce that incoming TCP connections are from the expected Windows user! https://github.com/tailscale/tailscale/blob/2a991a3541ae5d567a5bf26c6fe60a7349bda9bb/ipn/ipnserver/server.go#L141 https://github.com/tailscale/tailscale/blob/2a991a3541ae5d56... That's why we were happy with the solution they implemented as a stopgap, until they could switch to named pipes (which there is now an open PR for).
- say_it_as_it_is 4y agoI've read the description several times and find it hard to follow: ..an attacker-controlled website visited by the node..rebinds DNS for the peer API to an attacker-controlled DNS server making peer API requests in the client, including accessing the node’s Tailscale environment variables
- 0xbadcafebee 4y agoI really appreciate the Superfluous GraphViz.
- fulafel 4y agoThis kind of graph is also known as an attack tree. Agreed, it's good visualization of this. https://en.wikipedia.org/wiki/Attack_tree https://en.wikipedia.org/wiki/Attack_tree
- ruuda 4y ago> If you visit my website, I am granted the honour and the privilege of executing arbitrary Javascript on your computer. > > This is a pretty bad idea This is why I disable javascript by default, but I suspect that on this page it's needed to fix the theme or something, because the text is light grey on a white background, and all monospace sections are completely illegible. Edit: I don't mean to hate on the author, the content of the article is really interesting!
- bheadmaster 4y ago> I suspect that on this page it's needed to fix the theme or something, because the text is light grey on a white background, and all monospace sections are completely illegible. You seem to be correct. I found a single <script> tag in the source, with the following code: (() => { let v = localStorage.getItem("color-scheme"), a = window.matchMedia("(prefers-color-scheme: dark)").matches, cl = document.documentElement.classList, setColorScheme = v => (!v || v === "auto" ? a : v === "dark") ? cl.add("dark") : cl.remove("dark"); setColorScheme(v); window.setColorScheme = v => { setColorScheme(v); localStorage.setItem("color-scheme", v) }; })(); Though I don't see what's the point of this since the "light" theme, as you've pointed out, is completely illegible.
- adql 4y agomy FF is set to tell webpage to use dark theme and I get light gray on white (Chrome without that displays it fine). Looks fine if I inspect and set the bg color to black so I think author just didn't test dark theme properly...
- edf13 4y agoThat’s a selective quote… > This is a pretty bad idea, but luckily even the web browser has its limits.
- deleted 4y ago[deleted]
- cesarb 4y ago> If you run non-HTTPS web services on your Tailnet, and those services are unauthenticated or rely on Tailscale for authentication, implement an allowlist of expected HTTP Host headers to prevent malicious Javascript from accessing these services. In my opinion, this should be done not only for non-HTTPS services, but for all services: the "default" virtual host (used where there is no Host header, or when it has an unexpected value) should have nothing except a static 4xx error page. This not only avoids DNS rebinding attacks, but also avoids automated attacks in which the attacker doesn't know the correct hostname for the service (mostly automated scans for vulnerable PHP scripts and similar).
- judge2020 4y agoIn addition, it's basically required if you're using a reverse proxy service, eg. Cloudflare or Akamai. CF websites have been found via Shodan or Censys because site info is left open via https ://[ip]:443.
- radicaldreamer 4y agoThey should immediately blacklist the affected client versions.
- Komodai 4y ago
- nix23 4y agoWas just a matter of time...and much more will come.
- jalino23 4y agowhat do you mean?
- saghm 4y agoI'm not sure I've ever seen a detailed technical writeup of a vulnerability before that started with such clear and concise instructions on the exact steps needed to defend against it at the start of the article before. In particular, making clear the priority of what to patch is excellent. If I'm a user of a product where a bug was found, I'm definitely interested in learning about what the bug was, how it was discovered, and whether I should be worried about other bugs in the future, but the absolute first thing I want is to do whatever I can to make sure I'm not affected by it. Listing what to patch and/or change in code might be more "boring" than the narrative of how the bug was found, and it might spoil the surprise, but I think sometimes we focus so much on the fun of the process of finding the bugs or revel in the cleverness of an attack (and those things are fun!) that we forget that the real point to it is to make our stuff safer. There's plenty of time for fun, but make sure you patch things first!
- deleted 4y ago[deleted]
- paddlepop 4y agoI call this the "Vulnerability view" instead of a "Remediation view" and its something I feel a lot of Security people and tooling gets wrong when sharing information with those outside our bubble. It is dead easy to export a vulnerability scan or penetration test report and throw it at the developers, but you will get much better outcomes and better rapport if you tell them what they need to do (i.e. patch to version x.x.x) versus telling them what is wrong ("the sky is falling!").
- jacobr1 4y agoAlso interesting was this recommendation: "Keep using Tailscale! The speed and quality of Tailscale's response to our report is unlike any vendor interaction I have experienced, and suggests a deep commitment to keeping their customers safe." Every product has security issues now and then. The real challeng is building robust processes remediate them (and to ensure that class of issue doesn't reoccur). The teams that deliver, by being transparent and fixing their stuff in a timely way, get my business.
- deleted 4y ago
- arc-in-space 4y agoI... didn't get an email? Very cool to find out by looking at hn
- enneff 4y agoI did. Are you running an affected node?
- arc-in-space 4y agoYeah, I do, it may have been down at the time though - unsure
- deleted 4y ago[deleted]
- adam_arthur 4y agoDoes this mean we won’t get spammed with tailscale articles every day now?
- afhammad 4y agoThis means you will see more of Tailscale. Vulnerabilities are inevitable, the actions taken in the hours (ideally) and days following the discovery is what matters most.
- beanjuiceII 4y agoi would still rather see less considering where tailscale software sits in my privacy/security. at some point i'd ask why do i pay to use this swiss cheese ? (not saying thats the case, but if I were to continue to see more issues)
- afhammad 4y agoYep agreed. I think/hope an incident such as this will create a step change in their security processes. As you said, it only takes a few of these incidents to make it very difficult for the business to recover, especially given the tech savvy customer base.
- ev1 4y agoThe biggest shock to me here is "aarch64 Windows doesn't have calc.exe"
- snake_plissken 4y agoThe concept of DNS rebinding and DNS records pointing to a private/localhost IP address is particularly interesting and I remember when I first came across it in the wild. It's not exactly re-binding in the classic attack sense described in the article: some US sportsbooks make you download a geolocation service that verifies your location in order to place bets. The sportsbook's front end communicates with it through a DNS record pointing back to 127.0.0.1, and opens up a WebSocket to talk to the service. I imagine the WebSocket is used to bypass the same-origin policy but perhaps someone more knowledgeable can speak to that.
- freedinosaur 4y ago> In theory, there is no path for a malicious Tailscale control plane to remotely execute code on your machine, unless you happen to run network services that are designed to allow it, like an SSH server with Tailscale-backed authentication. Now I feel less crazy for not using Tailscale SSH for similar reasons. I'd like to see a security evaluation of Tailscale, on a per feature basis. I'd like to see tailscaled run with far fewer privileges. Is there a Tailscale alternative that just does Wireguard + NAT traversal and doesn't try to do key management?
- mikestaub 4y agoI have a task to investigate https://www.firezone.dev https://www.firezone.dev
- klabb3 4y ago> Is there a Tailscale alternative that just does Wireguard + NAT traversal and doesn't try to do key management? I really wish there was a NAT traversal protocol or library that wasn't overly complex and focused on the 90% cases. It would help not just tailscale's but anyone building p2p tech.
- anderspitman 4y agoI believe libp2p has some NAT traversal stuff: https://docs.libp2p.io/concepts/nat/ https://docs.libp2p.io/concepts/nat/
- freedinosaur 4y agohttps://github.com/hyprspace/hyprspace https://github.com/hyprspace/hyprspace is built on top of that. It's remarkably simple: libp2p's DHT + libp2p's NAT punching + TUN device. I don't think it offers authn/authz, but that's fine: neither does my ISP. I just want SSH reachability.
- doublepg23 4y agoI wonder if IPv6 will ever be the “path of least resistance” vs. NAT punching.
- semi-extrinsic 4y agoSuper interesting article, and TIL Firefox does not implement PNA (Private Network Access). Does anyone know why? It seems like an obviously good thing to have. https://wicg.github.io/private-network-access/ https://wicg.github.io/private-network-access/
- angry_octet 4y agoThe usual incompetence. It would break some existing use etc.
- ale42 4y agoAny reference about this? (i.e. somebody saying that it would break something?) Curious to see how they'd justify it.
- klabb3 4y agoRelated: note also that tailscale's tailnet 100.* subnet is som form of CGNAT public ip block. I think Tailscale thought long and hard about this, and landed on it because it was a path of lesser resistance to break fewer things. And if you squint they fit the stated purpose. But even if browsers now implement PNA the tailnet itself is public address space, so that vector still exists. I wonder if browsers (and eventually standards) will be pressured to treat those blocks as private.
- api 4y agoThe real takeaway here is that you should never treat any network boundary as critical for security. This is true whether it's a physical boundary or a virtual one (with TS being one example of the latter). If your private net is full of trivial to access things with no access control or horribly insecure services, that's a huge problem. There are many many many ways to hop over firewalls. Hostile JS on web sites is just one. Network boundaries are only first lines of defense in what should be a defense in depth strategy. Never depend on any one single boundary completely. My personal criteria is: if it's not secure enough to be connected directly to the Internet with no firewall, it's broken. Make it that secure and then put it on a secure network.
- loo 4y agoHeads up that I had to update through the UI twice - first brought me from 1.30.x 1.32.2, then second to 1.32.3.
- a-dub 4y agoi might be mistaken but i think there was something else weird about the windows loopback interface. i can't remember what it was, but something like binding the loopback interface would bind on all interfaces by default maybe?
- deleted 4y ago[deleted]
- amluto 4y ago> We can ask Tailscale to open a path on an SMB share. Windows being Windows, it will send your username (and a hash of your login password) to this server, unprompted, despite having no reason to consider the server trustworthy. Wow, I used to think Linux security was miles ahead of Windows security more than 20 years ago because of insanity like this. Fast forward 20 years. NTLMv2 is common, so cracking a password actually requires guessing the entire password instead of just 8 characters. But password guesses are much cheaper, so we haven’t gained much. Microsoft, how long will it take you to fix this for real? Opening a URL or UNC path should not, without an opt-in, authenticate at all. If configured to authenticate, it should prove, zero-knowledge, to the server that the supplied password (e.g. the logged-in user password) matches the server’s expected password. No further information should be leaked.
- deleted 4y ago[deleted]
- adql 4y agoSSH has solution for that forever too, server certs. Server cert change or (if you're fancy) is not signed by right CA and you get an alert.
- amluto 4y agoServer certs are a different issue. If OpenSSH, by default, sent SHA256(logged in user’s password) to the server, even after verifying the cert, it would get laughed out of the toolbox of security-conscious users.
- erosenbe0 4y agoI enjoyed the explanation very much. Even though I don't use mesh VPNs (yet), the architectural discussion of the vulnerability entailed numerous useful bits of background on browser and network infrastructure. Commendable work!
- zvrba 4y agoSo, lazy developers using loopback device instead of platform-specific named pipes on windows.
- mrjin 4y agoAmazing. Was about to sign up but gave up after reading their T&C.
- chaz6 4y agoThere does not appear to be a setting available to prevent known vulnerable clients from connecting to your tailnet.