16 ms·
Our User-Mode WireGuard Year
- iqanq 5y agoCan someone explain to me why wireguard is implemented as a kernel module? Yes I get it, more performance. But isn't it completely and absolutely insane to run a complicated piece of software that is open to outside connections with kernel privileges?
- remram 5y agoPerformance.
- Hendrikto 5y agoIt also helps with availability. If you got a recent kernel, it‘s already there.
- tptacek 5y agoThe WireGuard protocol is deliberately designed to be straightforward to run in the kernel. In steady state, it doesn't even require dynamic memory allocation. It uses timers in lieu of extra statekeeping. It has a simplified networking model ("cryptokey routing") that defers to the host TCP/IP stack a bunch of stuff that other VPN protocols take upon themselves to build. It has just one keying mechanism and an API to build more interesting authentication features (like SSO integration) on top of it, rather than having it invade the core design. It helps that it was designed and implemented by a kernel exploit author.
- robertlagrant 5y agoIsn't most networking in the kernel? It's pretty complicated already.
- miloignis 5y agoRunning complex software open to outside connections in the kernel is pretty standard - the TCP/IP stack is in the kernel too!
- renewiltord 5y agoIncredible blog post. Thank you very much for sharing. Signing up now. Also, this post makes me update my prior on top HN contributors being unproductive (i.e. that they spend their time on this board all the time instead of working).
- pier25 5y agoNot even 30 mins ago I set up WireGuard to connect to a PG instance on Fly. I expected this to be a headache but it took less than 5 mins to download WG, generate the conf with the fly CLI and paste it into WG. Done.
- tptacek 5y agoYou shouldn't have to do this unless you want to (for instance, to make a permanent WireGuard connection) --- you can just run `flyctl proxy` to set up a connection to 5432/tcp.
- pier25 5y agoReally? I followed the instructions in their docs and that's what was recommended. https://fly.io/docs/reference/postgres/#connecting-to-postgres-from-outside-fly https://fly.io/docs/reference/postgres/#connecting-to-postgr... Also, wouldn't I need to install Wireguard anyway to use fly proxy?
- tptacek 5y agoNope! That's what this whole post is about. :)
- api 5y agoWireGuard is just a transport protocol, so of course you could use it in place of SSL/TLS if you wanted. Interesting though, and I prefer it to SSL/TLS because X509 certs suck.
- tptacek 5y agoWireGuard isn't really the interesting bit here, it's running TCP/IP over it in userland. You cannot straightforwardly do that with SSL/TLS, but it is in fact the API that WireGuard provides.
- sho_hn 5y agoThis is where the article lost me a little bit. I (think I) technically got the part of running a TCP/IP stack in an unprivileged user process, so you don't have to elevate privilege for adding a network interface and using the host OS TCP/IP stack. And maybe that's already very cool. But: - What other benefits does it give you? - This isn't a new problem and presumably has prior best practices for mitigation. What is this replacing, what was the landscape like before this? What was the most similar already? - Have people been looking for a better solution like this for some time? - What's the over/under on the maintenance cost? You added in another TCP/IP stack to look after. You maybe save on static configuration and can make your system more dynamic. Pros, cons ... let's list them. The talks a bit about problems they were trying to address, but not in a way that clearly answered the above for me. It's of course valid to write a piece with a more informed audience in mind, but in something that aims to spread the virtues of an idea I think it could do more.
- tptacek 5y agoBecause 'sho_hn brought this up, here's a stab at a pro/con list of building TCP/IP directly into our API the way `flyctl` does: Pro: + Can just run "native" SSH directly over it (or, in our case, use x/crypto/ssh, without modification). + Lets `flyctl` offers a `flyctl proxy` command to users, so they can plug their own programs into whatever application they need to use, without asking us to change some proxy we run in our infrastructure. + Offers a single security and access control model (IPv6 private networking), rather than something we have to think about on a per-app basis. + In theory, we get all this right and never have to think about another network protocol in our infrastructure. + Allows existing network management tools and libraries to function directly with Fly.io infrastructure. + With the WebSockets gateway, we can do all of this stuff directly in browsers as well; that is, we can present TCP/IP as an API to browser Javascript to do UI stuff (and we're doing more and more UI stuff these days, in Elixir.) + Puts more IPv6 in the world. + Get to talk to Jason Donenfeld more. + Get to write blog posts like this. Con: - Spends one (maybe multiple) innovation tokens or whatever you want to call them. - Way more things can go wrong; relies on state synchronization and on a clear network path between our users and our gateways. Right now, we have to care whether you can speak 51820/udp. - User-mode TCP/IP via Netstack is probably significantly slower than a simple TCP proxy would be. - Required `flyctl` to run a background agent process to manage multiple connections through WireGuard. - The agent process adds to the list of things that can go wrong (hopefully we're ironed most of them out now). I can probably come up with more cons. ... It's buried in the middle of the post but I want to say it again because I think it's important: this sort of started out as a stunt; it's what I put together to allow people to SSH into their instances without having to install WireGuard locally, and that's all it was. I don't have to write a soul-searching pro/con list on stunts I use to give people SSH access, because lots of providers have super janky "pop a private terminal" setups. But all this stuff took on greater importance when we used it to run Docker for our remote builders. I like the approach we're taking a lot! I don't... regret it? I don't think? I think I'm happy with it. But it's complicated.
- dolmen 5y agoThis is an answer to this question by sho_hn: https://news.ycombinator.com/item?id=30276877 https://news.ycombinator.com/item?id=30276877
- marcus_cemes 5y agoFly.io's blog posts are incredible, they really seem to really enjoy what they do and want to share what they've made with everyone else. I love them for that. I wish that more companies could be like this and skip the corporate BS, it shows that they really have something outstanding to offer.
- deleted 5y ago[deleted]
- samwillis 5y agoI think their super power here is employing a renowned security expert who is an incredibly good communicator! (And happens to be a top HN contributor)
- myth_drannon 5y agoAnd they also employ Phoenix framework creator.
- Karrot_Kream 5y agoI'm pretty sure their experience running an ISP helps too heh.
- tptacek 5y agoIt's definitely all my talent that keeps this place running. I'm definitely not just a noisy message board guy who got hired after most of this infrastructure was built and deployed and then just proceeded to make a bunch of message board noise about it.
- swyx 5y agosee, this is why the people love reading what you write. keep giving credit but also having fun!
- philosopher1234 5y ago
- anderspitman 5y agoUsermode WireGuard would be a big deal. I maintain a list[0] of tunneling solutions, and one of the only limitations of systems built on WireGuard is the requirement for admin privileges. Even with the performance hit from running outside the kernel, UDP-based tunnels have a lot of advantages for multiplexing channels. Pretty much your only mainstream options today are QUIC and WireGuard, and only QUIC is intended to run in userspace. I'd have to dig into the details more, but something like this might allow you to implement a simple tunneling system based on WireGuard that runs in the kernel if you have the privileges, otherwise falls back to usermode and is no worse than QUIC in terms of performance. That would be awesome. [0]: https://github.com/anderspitman/awesome-tunneling https://github.com/anderspitman/awesome-tunneling
- pmarreck 5y agoIs https://tailscale.com/ https://tailscale.com/ not "usermode WireGuard"? I've been playing with it for a while now (it has a fairly generous free tier) and am quite impressed. I can access any of my LAN machines (my servers, my NAS, etc.) from anywhere that is also connected to the same network, and the names work for DNS as well.
- tptacek 5y ago1. Tailscale is amazing. I hate them so much. (We use Tailscale and are very happy with it.) 2. Tailscale is user-mode WireGuard. 3. "User-mode WireGuard" in the sense this post uses the term is a misnomer and refers to the fact that we run TCP/IP itself in userland (Tailscale normally runs through a tunnel device and uses your native TCP/IP stack). 4. But Tailscale also has code to do user-mode TCP/IP (they've got it running in a browser with wasm).
- pmarreck 5y agoFascinating!
- anderspitman 5y agoLast I heard[0] they were experimenting but hadn't shipped it. AFAIK their client still requires root, no? Running on wasm sounds awesome. This[1] looks like it. Do you know how they're doing the actual networking? WebRTC tunnel? [0]: https://news.ycombinator.com/item?id=24483173 https://news.ycombinator.com/item?id=24483173 [1]: https://twitter.com/bradfitz/status/1451423386777751561?lang=en https://twitter.com/bradfitz/status/1451423386777751561?lang...
- mwcampbell 5y ago> being able to pop a shell on a running app was table-stakes for the platform. Tangent: that's debatable IMO. In my company's current AWS infrastructure, there's no shell access to either the production containers or the host machines. I did write a script to create an ephemeral container that lets me (and future staff) run a shell inside the production network. And the thing I usually do in that shell is run psql; I suppose that's not ideal for auditability. But still, I can't poke around in the live containers or the host machines; in theory they could be distroless, with no shell at all. I'm trying to take immutable infrastructure to the max here; this seems like a good thing for security. It does mean that for debugging production problems, I can only go by what I find in logs and the database. But that has been acceptable so far. Edit: Given tptacek's security background, I was surprised that he considered production shell access essential for a new app platform.
- remram 5y agoIt's useful when developing. Not being able to shell into the production system is fine if you can shell into a staging/development system. If you are running Docker containers and you can shell into local containers, that is usually "close enough" that you can do useful troubleshooting. But fly.io (and CloudFlare workers, etc) are different enough from off-the-shelf containers that it is very important to be able to poke at containers when they break, even if they are not the actual production containers.
- tptacek 5y agoIt's funny you bring this up, because I had the same thought when I was first implementing SSH (the initial implementation relied on native WireGuard, and just plugged a client certificate into your running SSH agent). I thought people might not want to enable SSH access, and so I made the provisioning of the root SSH certificate optional: you have to run `flyctl ssh establish` to tell us to set up a root cert for your organization. It's turning out to have been a misfeature that confuses people more than it helps anyone, and we may get to a place soon where we just automatically provision a root cert for new organizations.
- mwcampbell 5y ago
- lyeager 5y ago> The Consul cluster would hold an Entmoot Hah! That one got me. They know their audience :)
- jonathanoliver 5y agoThis one made me laugh out loud and I had to come to HN to post this. You beat me to it.
- fuzzybear3965 5y agoHelp me out. This threw me for a loop and Google wasn't helpful.
- zellyn 5y agoWeird. For me, every Google result for [entmoot] is appropriate and basically gives the answer to your question I would have given. It's a Lord of the Rings reference.
- fuzzybear3965 5y agoRight - I understood that it was the walking tree from Lord of the Rings. But, what does it mean for Consul to hold a walking tree? Is it something like: "Consul bears a heavy weight (has a lot of responsibility and so responds slowly)"?
- zellyn 5y agoA walking tree is an Ent. A meeting of Ents, with ponderous deliberation — ponderous on _Ent_ timescales, that is — is an Entmoot.
- fuzzybear3965 5y agoOhhhh. Their clocks run slowly. A lot of time passes between events. Totally makes sense. Thanks!
- born-jre 5y agoi had something similar in mind but using libp2p[0]. use as a universal mesh network as library but without central control server, better p2p/NAT traversal, no need to mess with keys*. [0]: https://github.com/libp2p/ https://github.com/libp2p/
- drunner 5y agoHow do they manage their mesh? I've just been doing research on setting up my own wireguard mesh (currently using a spoke/hub setup with pi-hole/pivpn). I found https://github.com/HarvsG/WireGuardMeshes https://github.com/HarvsG/WireGuardMeshes today which is awesome, but I'm curious what fly.io / other readers here may be using.
- ohyeshedid 5y agoI usually build my own solutions, but I've played with Netmaker and it seems solid. https://github.com/gravitl/netmaker https://github.com/gravitl/netmaker
- ignoramous 5y agoNetmaker is SSPLd. Careful using it with anything at all (or, casually recommending it!). If your personal project connects to your proprietary server, then you're in a tough spot in terms of license compliance.
- anonymousisme 5y agoBack in the day (nearly 30 years ago) people would run a user-mode stack to obtain Internet connectivity via a (dial-up) Unix shell account. The program was "slirp" which was named after SLIP/CSLIP, but then upgraded to support PPP once that became a thing. https://en.wikipedia.org/wiki/Slirp https://en.wikipedia.org/wiki/Slirp
- rhn_mk1 5y agoHistory likes to repeat itself: https://github.com/rootless-containers/slirp4netns https://github.com/rootless-containers/slirp4netns
- jacob019 5y agoI was using SLIRP a few years ago to tunnel traffic through a cheap OpenVZ VPS, (for downloading Linux ISOs). I had to manually patch the TCP window size and recompile the binaries to get decent speed as the original code doesn't support TCP Window Size Scaling. I tried to upstream the patch to Debian but the maintainer wouldn't reply. It worked well enough but now I use wireguard. Modified SLIRP code is also found in VirtualBox, Qemu, UML and other virtualization software, for sharing the host connection in NAT mode.
- xanaxagoras 5y agoWhat a trip back in time. I used this and TIA [1] as a youth. [1] https://en.wikipedia.org/wiki/The_Internet_Adapter https://en.wikipedia.org/wiki/The_Internet_Adapter
- chrissnell 5y agoSLIP was a problem for us when I worked at an ISP in 1995. Cheapsakes would buy our barebones "shell access only" package and then run SLIP on our shell server. We eventually wrote a cron job to kill their processes when we found them. Sneaky ones just renamed the binary.
- croon 5y agoMan I miss the old internet when things were more playful and that was the extent of harm being done. Not that I condone cheapskates stealing your bandwidth. Maybe it's just rosy nostalgia.
- tedunangst 5y agoMan, where are all those virtualization fanboys who said usermode linux was a dead end? :)
- vbitz 5y agoWell gVisor uses mostly the same method of system call emulation (PTRACE_SYSEMU). It's also one of the three major projects that use it besides User-mode Linux and rr.
- ulzeraj 5y agoI was using wireguard-go on FreeBSD jail running on top of an APU2C2 board. Torrenting from my laptop caused wireguard-go cpu usage to spike to high loads and 30-50% CPU usage. Loading wireguard-kmod on the host machine plus some devfs rules dropped the CPU load to 0s. Not sure what happened there. The processor seems to score less than an RPi4 on Geekbench.
- bityard 5y agoI use one of these as a firewall (running OPNSense) and they're very nice but the CPU is indeed _slow_. It's plenty good enough for everything the firewall does but booting it up takes minutes and that's saying something for FreeBSD.
- hedora 5y agoOdd. I run openbsd on a similar one, and booting is reasonably fast. I even have a linux vm running on it in vmd, and haven’t noticed performance issues with that either.
- ulzeraj 5y agoPure FreeBSD boots in less than a minute from the SDCard. Have you tried plugging in a serial cable to check what service is slowing down the boot?
- MisterTea 5y ago> The processor seems to score less than an RPi4 on Geekbench. The apu2 is an embedded AMD quad core 1GHz SoC consuming 5W. It is not a powerful system by any means and not surprised rivaled by a >1GHz quad core Arm.
- mastax 5y agoThe AMD Jaguar cores came out in 2013. On the other hand, the Cortex A53 came out in 2012 so it's still a bit embarrassing for AMD. That was before the AMD renaissance, though.
- thundergolfer 5y agoThis post pairs nicely with Julia Evans’ post on why most people use the Linux kernel’s TCP/IP stack and why a few others would bother with a userland stack. In her post she doesn’t mention fly.io’s motivation for doing userland TCP-IP: a nicer end user client experience. If I read this correctly then fly.io did all this work to make their CLI user experience markedly better. That’s pretty cool. Twist yourself in interesting knots to make your user’s lives better, maybe even in ways they won’t notice!
- mayli 5y agoI always love the style how fly.io's blog is written, and a big fan of their freemium product. I am see a group of enthusiastic hackers behind the product, and keeps improving it in a reasonable way, or a cool way that doesn't sound boring.
- symlinkk 5y agoI thought wireguard-go requires the kernel tun device. See for example this Docker image for wireguard-go which maps the tun device and requires elevated permissions - https://github.com/masipcat/wireguard-go-docker https://github.com/masipcat/wireguard-go-docker
- est 5y agoBeen using tunsafe for years. It's solid wireguard-compatible, works on user-space, and had extra features like TCP handshake+UDP data, https obfuscation, etc. Author is @strigeus of uTorrent/Spotify fame.
- tarasglek 5y agoFirst off, you guys rule for doing this. I been dreaming about a general purpose userspace/unprivileged wireguard wrapper, this gets us closer to that. 1. Could you include more info on which userspace tcp/ip stack you use and why? I presume doing userspace UDP is relatively trivial/fast compared to what slirp had to do with tcp. 2. How does flyctl hijack syscalls across Linux and Windows? Is there some abstraction to do that? Wasn't even aware this was a pattern on Windows. I realize I could read the code, but would appreciate some direction. Kudos for the websocket proxy too! Would be really cool if this and the unprivileged wireguard became standard parts of wireguard toolset.
- tptacek 5y agoWe use netstack. It's great. https://pkg.go.dev/inet.af/netstack https://pkg.go.dev/inet.af/netstack I could try to give you reasons we use it but the truth is that I mentioned something about wanting a user-mode TCP and Jason Donenfeld said netstack was there, and then got it to work. We don't do any system call hijacking! Don't have to. Go abstracts Dialers and Listeners, and the WireGuard part itself is just a vanilla UDP protocol spoken over a PacketConn.
- sandGorgon 5y agois it possible to force fly.io to use a particular region ? India has these strict data residency laws for financial/healthcare companies (and I know that singapore has them too). So we need assured guarantees that data is not traversing outside of the region. most cloud vendors have specific data residency compliance for India - https://aws.amazon.com/compliance/india-data-protection/ https://aws.amazon.com/compliance/india-data-protection/
- michaeldwan 5y agoYou control which region your app and it’s volumes are in. Metrics, logs, and volume snapshots end up on servers in the USA. We haven’t addressed data residency for those platform services yet, but we might someday if there’s enough interest.
- the_duke 5y agoI really recommend tackling this. Many EU companies are increasingly concerned about data location, with some going into full panic mode. > Metrics, logs, and volume snapshots end up on servers in the USA This will unfortunately prevent me from using/recommending Fly for EU customers at the moment. It should also be pretty prominently stated in the docs.
- aszs 5y agoI feel like the lede is buried here... If I'm reading correctly I can use flyctl to set up an SSH encrypted tunnel over user-space TCP/IP over wireguard encrypted IP packets over UDP over WebSockets over HTTP over TLS encrypted TCP/IP? And not even just for fun, but to solve real problems in production. I can see why you might have mixed feelings about that!
- withinboredom 5y agoWhat’s even funnier is that if I read it right, it’s ipv6 only, and WSL in Windows doesn’t support IPV6 routing over WireGuard. The kernel is missing some important CONFIG flags when it was compiled to route the packets properly.
- mst 5y agoIt's running its own TCP stack in userspace though so the hosting kernel's routing abilities aren't an issue. I am both impressed and horrified, and I mean that in the best possible way.
- dolmen 5y ago> I am both impressed and horrified, and I mean that in the best possible way. This sentence style sounded familiar in my head. Looking at the username. Oh, of course, that's him!
- mst 5y agomstcat approves of this :D (for those who don't know, http://trout.me.uk/mstcat.jpg http://trout.me.uk/mstcat.jpg )
- pxeger1 5y ago> It looks like all of Fly.io isn’t working, when in fact the only part of Fly.io that isn’t working is the part that allows you to use it. What's wrong with that? ;)
- the_biot 5y agoIt's a little surreal to see all this cheering and adoration of fly.io, but they can't even keep their blog webserver up. It's not a great look for a cloud-related company.
- mrkurt 5y agoOur blog webserver is fine. What kind of error did you see?
- the_biot 5y agoTimes out, rarely gets the page out.
- mrkurt 5y agoDoes debug.fly.dev work? Can you run `curl -v https://fly.io/blog/ https://fly.io/blog/ -sS -o /dev/null -D-` and post what you see?
- the_biot 5y agodebug times out as well. curl output: flake: curl -v https://fly.io/blog/ -sS -o /dev/null -D- * Trying 2a09:8280:1::a:791... * TCP_NODELAY set * Connected to fly.io (2a09:8280:1::a:791) port 443 (#0) * ALPN, offering h2 * ALPN, offering http/1.1 * successfully set certificate verify locations: * CAfile: /etc/ssl/certs/ca-certificates.crt CApath: /etc/ssl/certs } [5 bytes data] * TLSv1.3 (OUT), TLS handshake, Client hello (1): } [512 bytes data] * OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to fly.io:443 * stopped the pause stream! * Closing connection 0 curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to fly.io:443 Seems to work when forced to IPv4 though.
- mrkurt 5y agoOk last thing, can you try `curl https://debug.fly.dev https://debug.fly.dev --ipv4` and tell me what the `Fly-Region` header says? Thank you for helping with this!
- efitz 5y agoTFA is surprisingly unapologetic about the Rube Goldberg SSH solution. On the other hand fly.io looks really interesting, I want to try it out. The infrastructure described (other than the SSH hack) feels like how a modern cloud platform should be built.
- paradox242 5y agoAs I read this, the tone and some of the topics started to ring a bell. Fly? Wireguard? I scrolled back up to the top to see who the author was and sure enough it was Thomas from the Security, Cryptography, Whatever podcast that I've been listening to for the last few months. For anyone that enjoyed this kind content you should also definitely check out the podcast he is a part of, the other hosts are great as well.
- dolmen 5y agoThanks for the pointer to that podcast. https://securitycryptographywhatever.buzzsprout.com/ https://securitycryptographywhatever.buzzsprout.com/