10 ms·
We improved the performance of a userspace TCP stack in Go
- wmf 2y ago"Asking for elevated permissions inside secure clusters at regulated financial enterprises or top secret government networks is at best a big delay and at worst a nonstarter." But exfiltrating data with a userspace VPN is totally fine? I'm also wondering why not use TLS.
- tazjin 2y agoYeah, the optimisations are cool of course, but (maybe due to being unfamiliar with the tool?!) I didn't understand why they can't just `listen(2)`.
- vlovich123 2y agoIt’s answered in the opening paragraph although I’ll admit I’m still unclear. > We are committed to keeping your data safe through end-to-end encryption and to making Coder easy to run across a wide variety of systems from client laptops and desktops to VMs, containers, and bare metal. If we used the TCP implementation in the OS, we’d need a way for the TCP packets to get from the operating system back into Coder for encryption. This is called a TUN device in unix-style operating systems and creating one requires elevated permissions, limiting who can run Coder and where. Asking for elevated permissions inside secure clusters at regulated financial enterprises or top secret government networks is at best a big delay and at worst a nonstarter. The specific part that’s unclear is why encryption needs to be applied at the TCP layer and at that point if they need it at the transport layer why they’re not using something like QUIC which has a much more mature user-space implementation.
- immibis 2y agoOr TLS. It seems to be a remote cloud desktop type of product, so why not use TLS like every other one?
- neonsunset 2y agoThe quote - is this yet another issue caused by abysmal FFI overhead in Go?
- tazjin 2y agoThere's nothing related to FFI calls in this quote.
- zer00eyz 2y agohttps://www.reddit.com/r/golang/comments/12nt2le/when_dealing_with_c_when_is_go_slow/ https://www.reddit.com/r/golang/comments/12nt2le/when_dealin... If your C doesn't fight the scheduler it isn't that bad.
- kbolino 2y agoGreat find! Specifically: On a goroutine not locked to an OS thread (the default), don't take more than 1 microsecond in a single C call. If you need to take longer in C, lock the goroutine to an OS thread (runtime.LockOSThread), but then don't do things in Go that would park that goroutine (time.Sleep, blocking channel read/write, etc).
- cricketlover 2y agoAgree. Very unclear why they won't simply use a secure socket or why a user space tunnel will be needed. I surmise that the reason might be that a user space tunnel might be faster (like maybe they can do UDP over TCP or something to gain speed improvements). Good post nevertheless.
- dpeckett 2y agoI think the key insight behind this approach (and I'm biased here having written something similar) is that the difference between QUIC and (wireguard + network stack) is A LOT less than you might think.
- Xelynega 2y agoI'm confused on why they would need a TUN device for a client or server application, so why they would need this solution in the first place(even with their explanation). As I understand the only reason you'd use a TUN interface is if you want to send/receive raw IP packets. Their marketing doesn't make it very clear what their product does, but I can't see a reason it would need to send/receive raw IP packets rather than TCP/UDP packets over a specific port...
- tptacek 2y agoEvery connection you make to a remote service "exfiltrates data". Modern TLS is just as opaque to middleboxes as WireGuard is, unless you add security telemetry directly to endpoints --- and then you don't care about the network anyways, so just monitor the endpoint. The reason you'd use WireGuard rather than TLS is that it allows you to talk directly to multiple services, using multiple protocols (most notably, things like Postgres and Redis) without having to build custom serverside "gateways" for each of those protocols.
- taeric 2y agoI think the point was more that doing this as a way to avoid the red tape of getting permission to open a new connection is odd?
- tptacek 2y agoI understand the impulse, but I think it misconstrues the "red tape" this method avoids. It's sidestepping a quirky OS limitation, which dates back to an era of "privileged ports" and multi-user machines. It's not really sidestepping any sort of modern policy boundary. For instance: you could do the exact same thing with WebSockets (and people do).
- taeric 2y agoI was thinking websockets; though, I thought those largely hit the same criticisms? That is, tons of things moved to them specifically to avoid any firewall rules about what they were allowed to send over a network. I'll fully grant that that seems to be the norm for everything browser related. Policies got difficult to install new software, just point your browser to this url and call it a day.
- mindcrime 2y agoI was thinking websockets; though, I thought those largely hit the same criticisms? That is, tons of things moved to them specifically to avoid any firewall rules about what they were allowed to send over a network. Arguably, this basic phenomenon has been going on for 20+ years. A lot of people by 2005-2007 or so had come to belive (and probably correctly) that a lot of the impetus for adopting SOAP based web-services over the preceding few years was simply because everything ran over ports 80 and 443 which were already open in the firewall. So deploying a remote service this way was more tractable than submitting a request to allow access to yet another port in firewall, and deal with the inevitable bureaucratic nightmare of getting that approved.
- anyfoo 2y agoYou can't control what information flows through an outbound connection, not even in trivial cases. Even if you straight go ahead and say "I allow you to make this connection, but I'm not even allowing you to send any data", you have timing sidechannels to deal with. In any more reasonable case, an almost infinite number of things can be used to exfiltrate any data you want, even if you think you have not only full application-level inspection, but even application-level rewrite. Pretty much the only thing you can do is somewhat filter out known-bad, not directly motivated outbound traffic, such as malware payloads with very clear signatures. This only works if it's "not directly motivated", because as soon as there's a person who wants to do it, they can skirt around it again.
- raggi 2y agofwiw, you technically don't need a privileged container to use tun, you just need suitable permissions on the kernel tun interfaces.
- convolvatron 2y agois this part of the open source releases? I looked at the coder.com github, but couldn't find it. I haven't written a compatible TCP, but a different reliable transport in go userspace. fairness aside, i wonder why we dont see this more often. would love to take a look
- tazjin 2y agoThey upstreamed their gVisor changes: https://github.com/google/gvisor/pull/10287 https://github.com/google/gvisor/pull/10287
- jijji 2y agoit's a solution looking for a problem
- pantalaimon 2y agoThe obvious question is: How does it compare to the in-Kernel TCP stack?
- syzcowboy99 2y agogVisor's netstack is still much slower than the kernel's (and likely always will be). The goal of this userspace netstack is not to compete with the kernel on performance, but offer an alternative that is more portable and secure.
- raggi 2y agofor some definition of portable which is deeply tied to the go runtime
- Xelynega 2y agoHow is it more portable or secure than an API that's been stable for decades, and getting constant security fixes? I see an explanation in their blog about avoiding TUN devices since they require elevated permissions, but why would you need a TUN device to send data to/from an application? I can't understand what their product does from the marketing material but it doesn't look like it would require constructing raw IP packets instead of TCP/UDP packets and letting the OS wrap them in the other layers.
- raggi 2y agoYou can have multiple layers of security boundary on most of the customer-exposed surface area, and avoid more risky surface areas in the kernel. Portable is a bit of a weird word here because for many of us with gray beards the word means architectures, kernels and systems, but I think in this context it tends to more mean "can run just as easily on my macbook as in a cloud container", but in practice the software isn't that portable, as Go isn't that portable - at least not in the context of vs. a niche C "portable network stack" that would build roughly anywhere that there's a working C toolchain, which is almost everywhere. Constant security fixes for the kernel are a real pain in deployments unless you follow upstream kernels closely. If your business is in shipping Linux runtimes with a high packing density, you really need to find ways to minimize the exposed Linux surface area, or organize to be able to ship kernel upstream updates at an extremely high frequency (relative to normal infrastructure upgrade rates for kernels / mandatory reboots) (and I would not consider kexec safe in this kind of context, at all). An alternative approach might be firecracker / microvms and so on, but those have their own tradeoffs too. The core point is that you want more than one layer between the host machines and the user code that wants to interact with Linux features.
- nynx 2y agoDoesn’t creating a raw socket need elevated permissions?
- tptacek 2y agoThey're not creating raw sockets†. The neat thing about WireGuard is that it runs over vanilla UDP, and presents to the "client" a full TCP/IP interface. We normally plug that interface directly into the kernel, but you don't have to; you can just write a userspace program that speaks WireGuard directly, and through it give a TCP/IP stack interface directly to your program. † I don't think? I didn't see them say that, and we do the same thing and we don't create raw sockets.
- vlovich123 2y agoSo it tunnels TCP/IP over Wireguard UDP?
- tptacek 2y agoCorrect (I mean, that's fundamentally what WireGuard is: a UDP TCP/IP tunnel, with strong modern encryption).
- ignoramous 2y agoYes; also see: https://github.com/WireGuard/wireguard-go/blob/12269c2761734b15625017d8565745096325392f/tun/netstack/examples/http_client.go https://github.com/WireGuard/wireguard-go/blob/12269c2761734...
- andrewstuart 2y agoIf you’re tunneling a better connection configuration isn’t the tunnel what defines the latency?
- andrewstuart 2y agoI have a problem right now which is that it’s slow to copy large files from one side of the earth to the other. Is this the basis of a solution to that maybe?
- dpe82 2y agoWhat do you think are the current problems contributing to your slow transfers?
- andrewstuart 2y agoWindow and buffer size is a problem on high latency links.
- dpe82 2y agoWhy do you suspect a user space implementation of TCP would improve those issues beyond existing kernel implementations?
- 392 2y agoNo. Profile first. Make sure you've tried tweaking params like batch sizes.
- raggi 2y agonot enough detail here to provide a good answer, but I can tell you explicitly that if you're using SMB you're likely not going to get good performance here even if your network stack is has tons of space to overcome bdp and congestion challenges.
- parhamn 2y agoI don't know anything about Coder, but Gvisor proliferation is annoying. It's a boon for cloud providers, helping them find another way to get a large multiple performance decrease per dollar spent in exchange for questionable security benefits. And I'm seeing it everywhere now.
- kccqzy 2y agoThere are still products from cloud providers that don't use gvisor. Basics like EC2 or GCE. Sounds like you chose the wrong cloud product.
- tptacek 2y agoAre you referring to gVisor the container runtime, or gVisor/netstack, the TCP/IP stack? I see more uptick in netstack. I don't see proliferation of gVisor itself. "Security" is much more salient to gVisor than it is to netstack.
- parhamn 2y agoIn the issue of abysmal performance on cloud-compute/PaaS Im talking about the container runtime (most Paas is gVisor or Firecracker, no?) cloudrun, DO, modal, etc. But given this article is about improving gvisors userland tcp performance significantly, it seems like the netstack stuff causes major performance losses too. I saw a github link in another top article today https://github.com/misprit7/computerraria https://github.com/misprit7/computerraria where the Readme's Pitch section feels very relevant to gvisor.
- tptacek 2y agoI don’t believe many PAAS run gVisor; a surprising number just run multitenant docker. The netstack stuff here has nothing to do with the rest of gVisor.
- parhamn 2y ago> The netstack stuff here has nothing to do with the rest of gVisor. How so? Besides being part of it, it is at least similar in the group of "bloated slow userland implementation of things the kernel handles well"
- dpeckett 2y agoReally cool to see others hacking on netstack, bit of a shame it's tied up in the gVisor monorepo (and all the Bazel idiosyncracies) but it's a very neat piece of kit. I've actually been hacking on a similar FOSS project lately, with a focus on building what I'm calling a layer 3 service mesh for the edge. More or less came out of my learned hatred for managing mTLS at scale and my dislike for shoving everything through a L7 proxy (insane protocol complexity, weird bugs, and you still have the issue of authenticating you are actually talking to the proxy you expect). Last week I got the first release of the userspace router shipped, worth taking a look if you want to play around with a completely userspace and unprivileged WireGuard compatible VPN server. https://github.com/noisysockets/nsh/blob/main/docs/router.md https://github.com/noisysockets/nsh/blob/main/docs/router.md
- iangudger 2y agoIf you want to use netstack without Bazel, just use the go branch: https://github.com/google/gvisor/tree/go https://github.com/google/gvisor/tree/go go get gvisor.dev/gvisor/pkg/tcpip@go The go branch is auto generated with all of the generated code checked in.
- dave78 2y agoI did this once for an experimental project and found it really difficult to keep the version of gVisor I was using up to date, since it seems like the API is extremely volatile. Anyone else had this experience? If so, is there some way around it that I don't know? Or did I just try it at a bad point in the development timeline?
- ignoramous 2y agoThe API is indeed prone to change without notice, but it isn't anything terribly unmanageable. > really difficult to keep the version of gVisor I was using up to date For our project, we update gvisor whenever Tailscale does.
- iangudger 2y agoIt could be that you happened to find a period of rapid change, but it is also possible that you ran into the issue that raggi mentioned in the sibling comment.
- deleted 2y ago[deleted]
- deleted 2y ago[deleted]
- raggi 2y agoIt's great to see this, I know the team went on a long journey through this and the blog makes it almost look shorter and simpler than it was. I'm hoping one day we can all integrate the support for GSO that's been landing in gvisor too, but so far we've (tailscale) not had a chance to look deeply into that yet. It was really effective for our tun and UDP interfaces though.
- ignoramous 2y ago> one day we can all integrate the support for GSO that's been landing in gvisor Google engs recently rewrote the GSO bit, but unlike Tailscale, it is only for TCP, though. Besides, gvisor has had "software" & "hardware" GSO support for as long as I can remember.
- kylecarbs 2y agoAt Coder we’re fans and users of Tailscale, so very happy to have these changes be consumed upstream as well!
- zxt_tzx 2y agoI met one of the founders of Coder.com, he's a really cool dude. It's a pity that it is a product aimed more at enterprises than individual developers, else it would have far more developer mindshare. Unlike, say, GitHub Codespaces, running something like this on your own infra means your incentives and Coder.com's are aligned, i.e. both of you want to reduce your cloud costs (as opposed to, say, GitHub running on Azure gives them an opportunity and incentive to mark up on Azure cloud costs).
- santiagobasulto 2y agoIt seems like a great product. I'm wondering why they don't offer more "startup-oriented" plans. It's like either Self Hosted or "Talk to sales". Is it maybe to not compete against Github codespaces?
- kylecarbs 2y agoFounder of Coder here. Many small (or teams at big) companies use Coder for free with <=150 devs just using our open-source. We’ve tried to align our pricing with the value of the product. In small teams the productivity gains seem to be much lower, so we target Enterprise!
- withinboredom 2y agoSpeaking of ... https://coder.com/docs https://coder.com/docs -- what's next is empty.
- Narhem 2y ago[flagged]
- yencabulator 2y agotl;dr Increased TCP receive buffer size, implemented HyStart instead of traditional TCP slow start in gVisor's netstack, changed an in-process packet queue from drop-when-full to block-when-full.
- jiveturkey 2y agohelp me understand something. > we’d need a way for the TCP packets to get from the operating system back into Coder for encryption. yes, this is commonly done via OpenSSL for example. > This is called a TUN device in unix-style operating systems and creating one requires elevated permissions waitasec, wut? sure you could use a TUN device I guess, but assuming some kind of multi-tenant separation is an underlying assumption they didn't mention in their intro, couldn't you also use cgroup'd containers? sorry if I'm not fluent in the terminology. i'm struggling to understand the constraints that push them towards gVisor. simply needing to do encryption doesn't seem like justification. i'm sure they have very good reasons, but needing to satisfy a financial regulator seems orthogonal at best. i would just like to understand those reasons.