5 ms·
Hi! This is a blog post sharing some low-level Linux networking we're doing at Modal with WireGuard. As a serverless platform we hit a bit of a tricky tradeoff
by ekzhang 2y ago
Hi! This is a blog post sharing some low-level Linux networking we're doing at Modal with WireGuard.
As a serverless platform we hit a bit of a tricky tradeoff: we run multi-tenant user workloads on machines around the world, and each serverless function is an autoscaling container pool. How do you let users give their functions static IPs, but also decouple them from compute resource flexibility?
We needed a high-availability VPN proxy for containers and didn't find one, so we built our own on top of WireGuard and open-sourced it at https://github.com/modal-labs/vprox https://github.com/modal-labs/vprox
Let us know if you have thoughts! I'm relatively new to low-level container networking, and we (me + my coworkers Luis and Jeffrey + others) have enjoyed working on this.
- xxpor 2y agoYou're using containers as a multi-tenancy boundary for arbitrary code?
- ekzhang 2y agoWe use gVisor! It's an open-source application security sandbox spun off from Google. We work with the gVisor team to get the features we need (notably GPUs / CUDA support) and also help test gVisor upstream https://gvisor.dev/users/ https://gvisor.dev/users/ It's also used by Google Kubernetes Engine, OpenAI, and Cloudflare among others to run untrusted code.
- deleted 2y ago[deleted]
- doctorpangloss 2y agoAre these the facts? - You are using a container orchestrator like Kubernetes - You are using gVisor as a container runtime - Two applications from different users, containerized, are scheduled on the same node. Then, which of the following are true? (1) Both have shared access to an NVIDIA GPU (2) Both share access to the NVIDIA GPU via CUDA MPS (3) If there were 2 or more MIGs on the node with a MIG-supporting GPU, the NVIDIA container toolkit shim assigned a distinct MIG to each application
- ekzhang 2y agoWe don't use Kubernetes to run user workloads, we do use gVisor. We don't use MIG (multi-instance GPU) or MPS. If you run a container on Modal using N GPUs, you get the entire N GPUs. If you'd like to learn more, you can check out our docs here: https://modal.com/docs/guide/gpu https://modal.com/docs/guide/gpu Re not using Kubernetes, we have our own custom container runtime in Rust with optimizations like lazy loading of content-addressed file systems. https://www.youtube.com/watch?v=SlkEW4C2kd4 https://www.youtube.com/watch?v=SlkEW4C2kd4
- doctorpangloss 2y agoSuppose I ask for two H100s. Will I have GPU P2P capabilities?
- thundergolfer 2y agoYes it will. (I work at Modal.)
- ekzhang 2y agoYep! This is something we have internal tests for haha, you have good instincts that it can be tricky. Here's an example of using that for multi-GPU training https://modal.com/docs/examples/llm-finetuning https://modal.com/docs/examples/llm-finetuning
- doctorpangloss 2y agoOkay, well think very deeply about what you are saying about isolation; the topology of the hardware; and why NVIDIA does not allow P2P access even in vGPU settings except in specific circumstances that are not yours. I think if it were as easy to make the isolation promises you are making, NVIDIA would already do it. Malformed NVLink messages make GPUs fall off the bus even in trusted applications.
- deleted 2y ago[deleted]
- deleted 2y ago[deleted]
- crishoj 2y agoNeat. I am curious what notable differences there are between Modal and Tailscale.
- ekzhang 2y agoThanks. We did check out Tailscale, but they didn't quite have what we were looking for: some high-availability custom component that plugs into a low-level container runtime. (Which makes sense, it's pretty different from their intended use case.) Modal is actually a happy customer of Tailscale (but for other purposes). :D
- mrbluecoat 2y agoSo if a company only needs an outbound VPN for their road warriors and not an inbound VPN to access internal servers, vprox could be a simpler alternative to Tailscale?
- dangoodmanUT 2y agoAs a predominantly rust shop, why choose go for this?
- klooney 2y agoWireguard's premiere user space implementation is in go.
- ekzhang 2y agoThis is it. I like Rust a lot, but you gotta pick the right tool for the job sometimes
- rendaw 2y agoPremiere = official? Do you know how it compares to this by Cloudflare? https://github.com/cloudflare/boringtun https://github.com/cloudflare/boringtun
- ekzhang 2y agoSorry, just realized the misunderstanding. To clarify Modal still uses the kernel WireGuard module. The userspace part that’s in Go and not in other languages that we use is the wgctrl library.