8 ms·
Improving Tailscale via Apple’s open source
- 0xbadcafebee 4y agoIsn't there usually vendor documentation of internal system calls? I know Microsoft used to provide it, though I think you had to pay for MSDN? I assume Apple has something similar. In terms of finding a syscall, you don't usually need to dig through source code. On Linux, strace and gdb will show you a program's active system calls, and on MacOS it'd be dtruss.
- macshome 4y agoThings like this are documented in the Apple sources and header files. Even if there are stubs of documentation they pretty much just give you the names of files, or man pages, to go read. Once you get up out of the POSIX layers the Apple documentation has improved a lot lately.
- raggi 4y agoYou need both at depth. strace and dtruss translate binary information to a textual representation but they’re only as good as their coverage. When you get closer to the edges of common use they fall down. On macOS / iOS using dtruss for all programs comes with extra burdens, too. We (tailscalar here!) have another post out today on more throughput optimizations and the story there is related in this way: when we started working on that code path the docs didn’t exist! (There are some docs now, in Linux). But either way the stuff is seldom used outside of specialist implementations - so when you need to understand what’s going on you’re headed for the source, or a debugger, or a symbolic tracer rather than an interpretive one.
- EMIRELADERO 4y agoNice blog post! I like the way this problem is approached and solved. Peeking under the "table" to see how the internals work is actually creative when trying to fix things "upstream" on an upper abstraction layer. Unfortunately, I still don't love Tailscale. I do like it, a lot even. But their refusal to open-source all their clients (and the server) is baffling, especially considering that they have an employee contributing to Headscale, the community-led FOSS tailscale server. At that point just open-source the damn thing! Issues aside, it's still a great product. It actually felt like magic when I first used it in a way few technologies have.
- jzelinskie 4y agoI don't think that their choice to keep their server software proprietary is baffling. They're going to architect it so that it's easiest/best for their team to run which is not the same what's easiest/best for EVERYONE to run as is the typical goal of open source software.
- ceejayoz 4y agoThere's two concerns: "can I run it myself", and "what does this security-critical third-party code do?" Open sourcing helps with the latter, even if the code's not very well optimized for general purpose use for the former case.
- judge2020 4y agoThe concern is that people will assume the OSS is for both scenarios, so even if you close issues and have a PR bot rejecting PRs, you'll still get both support tickets and (good|bad) PR from people trying to run the mainline system themselves. Supporting development on a made-for-self-hosting OSS project is the most user friendly option and allows them to avoid needing to handle the woes of public code.
- ehPReth 4y agowoah, they have an employee contributing to head scale now?
- drewg123 4y agoShared / Open source is super helpful when you're trying to figure out how to do something unusual that's not on the paved path covered by the docs and examples. In the early 2000s, when doing Mac drivers for an OS-bypass HPC NIC (Myrinet, a sort of pre-cursor to inifiband/roce), I spent 90% of my time reading the Apple source and about 10% of my time coding. In my case the first goal was to figure out how to use BSD ioctls rather than Mach based userclient stuff, as we had tons of shared kernel and userspace code for ioctls (across linux, dec unix, solaris, freebsd, windows, etc). And the second was figuring out exactly which iomemory descriptor variant would work well for registering memory.
- jamesfmilne 4y agoInteresting background. I've recently been toying with the idea of writing some open-source Mac DriverKit drivers for Mellanox cards, so I'm not forced to pay through the nose for ATTO cards (which are just rebranded Mellanox cards.) Sadly all the DriverKit stuff appears to be completely closed source. At least its user-space though, so perhaps a bit less painful than normal kernel driver development...
- ignoramous 4y ago> Shared / Open source is super helpful when you're trying to figure out how to do something unusual that's not on the paved path covered by the docs and examples. Not just the unusual, but the usual too: Just this past month or so, a user on Fly.io forums helped debug just why reading from stdin / writing to stdout didn't work for non-root users (https://community.fly.io/t/10375/8 https://community.fly.io/t/10375/8). It all started from the fact that Fly.io had open sourced a snapshot-in-time of their init process back in 2021. Also, it isn't uncommon in the Android world for developers to routinely find the right APIs to use by reading the OS source code.
- iudqnolq 4y agoA very meaningful point in my learning as a junior developer was when I realized I should regularly click go to definition on standard library functions. I needed to tweak the behavior of an Android standard library function, so I copied it's source code (~20 lines) and made some edits. Blew my mind at the time it was that simple.
- heyflyguy 4y agoI love tailscale, but an unusual thing happened when I upgraded my home pc to Windows 11. Now my tailscale still connects to all of my other computers and servers, but I can't browse any network shares. I guess authentication changed in Windows 11 and is not backward-compatible. Tailscale worked marvelously for me for ages, but Windows 11 for sure threw a wrench in the gears!
- unraveller 4y agoI had to manually enable workstation service on my win11 install to get my home network shares working and before that in win10 I had force guest mode just to browse freely and remote desktop. AllowInsecureGuestAuth 1 HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\LanmanWorkstation\Parameters
- heyflyguy 4y agoYou are quite a person and I thank you
- skrtskrt 4y agotheir clients have to do a lot of platform specific configuration to make it appear magic- OS/distro-specific network configuration is always going to be a hairy, moving target - open an issue on their GitHub and they'll probably help you start poking at it
- i386 4y agoProduct person here, I’ve been using and loving Tail scale for about a year now and I don’t understand when I have to pay and why and I really don’t understand why any of it is OSS. Where’s the revenue and where’s the moat?
- stingraycharles 4y agoWe’re a customer and paying Tailscale a decent amount of $ a month and planning on expanding our usage more and more. They’re probably just using a generous free tier as a growth tool.
- dgentry 4y agohttps://tailscale.com/blog/free-plan/ https://tailscale.com/blog/free-plan/ Being a mesh means most connections are direct and do not go through any infrastructure which costs Tailscale money, making a Free tier economically workable.
- spiantino 4y agoI wonder if this will help with the iOS app's battery drain issues. VPNs of any kind come with a battery penalty, but some of these issues might have exacerbated that
- neilalexander 4y agoUsing regular Wireguard with on-demand enabled and persistent keepalives disabled is about as good as VPN on iOS can possibly get. Without the keepalives, there is no idle protocol traffic and it is therefore barely consuming any battery power until traffic is sent over the tunnel.
- bradfitz 4y agoThis is part of it. There are about five more parts.
- omneity 4y agoSlightly related, I use Zerotier for some personal stuff as a managed VPN, including connecting several Apple devices. I never see it mentioned in HN (Zerolove for Zerotier lol), does anyone who used both knows what am I missing out for not using Tailscale?
- stingraycharles 4y agoWe recently migrated away from our old OpenVPN deployment, and Zerotier was on the table. In the end we picked Tailscale because it’s based on open protocols and more transparent in how they work. From a different perspective, it appears Zerotier is investing a lot of engineering efforts in maintaining its own protocol, with arguable benefits. Tailscale seems to be deploying new features on an almost weekly basis and are having huge velocity, and to me it just appeared their trade-offs made more sense, and I have more confidence in their business doing great on a long term.
- sconi 4y agoyeah, different protocol. Similar concepts. Tailscale has captured mindshare, reminds me some of Cloudflare. There are many solutions out there offering unique value props relative to what Tailscale is doing, and it all comes down to customer needs. I'm not sure how broadly Tailscale has been adopted outside of the dev / HN crowd, so that's always something to dig into.
- stingraycharles 4y agoLet’s just say that as the person who introduced Tailscale within our org, it has been one of the most easiest solutions I ever had to convince my org of. Given the relative intrusiveness of switching a VPN provider (it touches a lot of core infrastructure), I expected a lot more resistance, but everybody was on board immediately. In all honesty, Tailscale being so good, we never seriously looked at Zerotier. We only evaluated it on paper, not actually tried to use it.
- ElectricalUnion 4y agoZerotier doesn't have support for runnning without TUN devices without their "Business Source License"d libzt. Being able to run Tailscale under otherwise very restricted userland environments, without changing too much code is nice.
- skrtskrt 4y agofunny side note about Tailscale/iOS: one of the first things I tried with Tailscale was TailDrop-ing a random file to my iPad. I chose a dotfile like .gitconfig, but then the iOS Files app wouldn't show it to me. It would show me that there was one item of nonzero size in the folder, but I could not see the actual file, and now (I assume) I can't delete it either unless I delete the folder.
- api 4y agoHeh... we went spelunking into Apple's open source for a not too different reason a few years ago! https://www.zerotier.com/blog/how-zerotier-eliminated-kernel-extensions-on-macos/ https://www.zerotier.com/blog/how-zerotier-eliminated-kernel... https://apple.stackexchange.com/questions/337715/fake-ethernet-interfaces-feth-if-fake-anyone-ever-seen-this https://apple.stackexchange.com/questions/337715/fake-ethern... We did this to eliminate the need for kernel extensions to create L2 interfaces.
- EdSchouten 4y agoI see this was eventually fixed by manually declaring a structure that matches struct ifreq. Instead of doing that, I would recommend submitting a patch against golang.org/x/sys to get that autogenerated. The Go folks tend to merge such PRs very quickly (<48h).
- dcow 4y agoThere are Apple-supported ways to solve both these issues built in to the `NEPacketTunnelProvider` framework. No need to drop into obscure `ioctls`! The correct way to solve the looping issue is by configuring the `excludedRoutes` property on the `NEIPv4|6Settings` object associated with your network extension. See: https://developer.apple.com/documentation/networkextension/neipv4settings/1406267-excludedroutes https://developer.apple.com/documentation/networkextension/n.... I imagine this is an important property for a dynamic mesh network to keep updated regularly as the topology changes. The traffic egressing the wrong interface issue happens because the system handles swapping between WiFi and Cellular radios quite judiciously, and the BSD socket API isn't plugged into this management (AFAIK there's no way to get notified when the delegated interface changes, so you don't really know when to re-bind). Instead, you should use the provided `packetFlow` to write IP packets to the tunnel, which will always egress via the "correct" (whichever one the system decides is best at the moment) interface. Tailscale's go library would need to implement hooks to call out to the platform implementation for Apple vs others, which is annoying. But, it's the right way and avoids these weird edge cases and bugs.
- ayewo 4y agoWould be interesting to hear the thoughts of Tailscale folks regarding your suggested fix.
- ayewo 4y agoIn addition to bradfitz returning to comment, I managed to get a response out of apenwarr on Twitter https://twitter.com/apenwarr/status/1647058229304651782 https://twitter.com/apenwarr/status/1647058229304651782
- pjmlp 4y agoIt is the usual issue that Linux folks keep looking at NeXTSTEP derivatives as yet another UNIX clone, instead of what is on top of it.
- dcow 4y agoAlso note there's an Android analog: https://developer.android.com/reference/android/net/VpnService#protect(int) https://developer.android.com/reference/android/net/VpnServi.... You call `protect` on sockets that should not send data through your tunnel. It's a bit more flexible and straightforward IMHO (: