6 ms·
Static IPs for Serverless Containers
- deleted 2y ago[deleted]
- deleted 2y ago[deleted]
- eqvinox 2y agoI guess my first question is, why is this built on IPv4 rather than IPv6...
- ekzhang 2y agoYeah, great question. This came up at the beginning of design. A lot of our customers specifically needed IPv4 whitelisting. For example, MongoDB Atlas (a very popular database vendor) only supports IPv4. https://www.mongodb.com/community/forums/t/does-mongodb-atlas-support-ipv6-only-client-for-m2-instance/276117 https://www.mongodb.com/community/forums/t/does-mongodb-atla... The architecture of vprox is pretty generic though and could support IPv6 as well.
- eqvinox 2y agoI guess that works until other customers need access to IPv6-only resources… (e.g.: we've stopped rolling IPv4 to any of our CI. No IPv6, no build artifacts…) In a perfect world I'd also be asking whether you considered NAT64, but unfortunately I'm well aware that's a giant world of pain to get to work on Linux (involving either out-of-tree Jool, or full-on VPP)
- ekzhang 2y agoYeah, you hit the nail on the head. We considered NAT64 as well and looked at some implementations including eBPF-based ones like Cilium. Glad to know that IPv6-only is working well for you. "In a perfect world…" :)
- eqvinox 2y agoIt is what it is :/ … I do periodically ask these questions to track how v4-vs-v6 things are developing, and they're moving, albeit at a snail's pace. (FTR, it works for us because our CI is relatively self-contained. And we have local git mirrors… f***ing github…)
- lowpro 2y agoAt my company (Fortune 100), we've been selling a lot of our public v4 space to implement... RFC1918 space. We've re-IP'd over 50,000 systems so far to private space. We just implemented NAT for the first time ever. I was surprised to see how far behind some companies are.
- eqvinox 2y agoProgress is coming from the weirdest corners… US DoD and NATO require IPv6 feature-parity to IPv4 nowadays, no full IPv6 = no bidding on tenders… (I would already have expected this to be quite effective in forcing IPv6, but tbh I'm still surprised just how effective.)
- ekzhang 2y agoHi! This is a blog post sharing some low-level Linux networking we're doing at Modal with WireGuard. As a serverless platform we hit a bit of a tricky tradeoff: we run multi-tenant user workloads on machines around the world, and each serverless function is an autoscaling container pool. How do you let users give their functions static IPs, but also decouple them from compute resource flexibility? We needed a high-availability VPN proxy for containers and didn't find one, so we built our own on top of WireGuard and open-sourced it at https://github.com/modal-labs/vprox https://github.com/modal-labs/vprox Let us know if you have thoughts! I'm relatively new to low-level container networking, and we (me + my coworkers Luis and Jeffrey + others) have enjoyed working on this.
- xxpor 2y agoYou're using containers as a multi-tenancy boundary for arbitrary code?
- ekzhang 2y agoWe use gVisor! It's an open-source application security sandbox spun off from Google. We work with the gVisor team to get the features we need (notably GPUs / CUDA support) and also help test gVisor upstream https://gvisor.dev/users/ https://gvisor.dev/users/ It's also used by Google Kubernetes Engine, OpenAI, and Cloudflare among others to run untrusted code.
- deleted 2y ago[deleted]
- doctorpangloss 2y agoAre these the facts? - You are using a container orchestrator like Kubernetes - You are using gVisor as a container runtime - Two applications from different users, containerized, are scheduled on the same node. Then, which of the following are true? (1) Both have shared access to an NVIDIA GPU (2) Both share access to the NVIDIA GPU via CUDA MPS (3) If there were 2 or more MIGs on the node with a MIG-supporting GPU, the NVIDIA container toolkit shim assigned a distinct MIG to each application
- jimmyl02 2y agothis is a really neat writeup! the design choice to make each "exit node" control the local wireguard connections instead of a global control plane is pretty neat. an unfinished project I worked on (https://github.com/redpwn/rvpn https://github.com/redpwn/rvpn) was a bit more ambitious with a global control plane and I quickly learned supporting multiple clients especially anything networking related is a tarpit. the focus on linux / aws specifically here and the results achievable from it are nice to see. networking is challenging and this was a nice deep dive into some networking internals, thanks for sharing the details :)
- ekzhang 2y agoThanks for sharing. I'm interested in seeing what a global control plane might look like, seems like authentication might be tricky to get right! Controlling our worker environment (like `net.ipv4.conf.all.rp_filter` sysctl) is a big help for us since it means we don't have to deal with the fullness of all possible network configurations.
- cactacea 2y agoStatic IPs for allowlists need to die already. Its 2024, come on, surely we can do better than this
- ekzhang 2y agoWhat would you suggest as an alternative?
- thatfunkymunki 2y agoa more modern, zero-trust solution like mTLS authentication
- ekzhang 2y agoThat makes sense, mTLS is great. Some services like Google Cloud SQL are really good about support for it. https://cloud.google.com/sql/docs/mysql/configure-ssl-instance https://cloud.google.com/sql/docs/mysql/configure-ssl-instan... It's not quite a zero-trust solution though due to the CA chain of trust. mTLS is security at a different layer though than IP source whitelisting. I'd say that a lot of companies we spoke to would want both as a defense-in-depth measure. Even with mTLS, network whitelisting is relevant. If your certificate were to be exposed for instance, an attacker would still need to be able to forge a source IP address to start a connection.
- thatfunkymunki 2y agoI'd put it in the zero-trust category if the server (or owner of the server, etc) is the issuer of the client certificate and the client uses that certificate to authenticate itself, but I'll admit this is a pedantic point that adds nothing of substance. The idea being that you trust your issuance of the certificate and the various things that can be asserted based on how it was issued (stored in TPM, etc), rather than any parameter that could be controlled by the remote party.
- PLG88 2y agoIf mTLS is combined with outbound connections, then IP source whitelisting is irrelevant; the external network cannot connect to your resources. This (and more) is exactly what we (I work on it) built with open source OpenZiti, a zero trust networking platform. Bonus points, it includes SDKs so you can embed ZTN into the serverless function, a colleague demonstrated it with a Python workload on AWS - https://blog.openziti.io/my-intern-assignment-call-a-dark-webhook-from-aws-lambda https://blog.openziti.io/my-intern-assignment-call-a-dark-we....
- ATechGuy 2y ago> Modal has an isolated container runtime that lets us share each host’s CPU and memory between workloads. Looks like Modal hosts workloads in Containers, not VMs. How do you enforce secure isolation with this design? A single kernel vulnerability could lead to remote execution on the host, impacting all workloads . Am I missing anything?
- ekzhang 2y agoI mentioned this in another comment thread, but we use gVisor to enforce isolation. https://gvisor.dev/users/ https://gvisor.dev/users/ It's also used by Google Kubernetes Engine, OpenAI, and Cloudflare among others to run untrusted code.
- fusjdffddddddds 2y agoIt's going to take years for orgs to adopt IPv6 and mTLS+JWT/OIDC. Even longer for QUIC/H3.
- klysm 2y agoI’m not convinced that mTLS or OIDc are good ideas
- fusjdffddddddds 2y ago... Are you going to say why?
- nodesocket 2y agoCouldn't a NAT instance in-front of containers accomplish this as well (assuming only needed for outbound traffic)? The open source project fck-nat[1] looks amazing for this purpose. [1] https://fck-nat.dev/stable/ https://fck-nat.dev/stable/
- ekzhang 2y agoRight, vprox servers act as multiplexed NAT instances with a VPN attached. You do still need the VPN part though since our containers run around the world, in multiple regions and availability zones. Setting the gateway to a machine running fck-nat would only work if that machine is in the same subnet (e.g., for AWS, in one availability zone). The other features that were hard requirements for us were multi-tenancy and high availability / failover. By the way, fck-nat is just a basic shell script that sets the `ip_forward` and `rp_filter` sysctls and adds an IP masquerade rule. If you look at vprox, we also do this but build a lot on top of it. https://github.com/modal-labs/vprox https://github.com/modal-labs/vprox
- nodesocket 2y agoAhh that makes sense. I do think that a single fck-nat instance can service multiple AZ's though in a AWS region. Just need to adjust the VPC routing table. Thanks for the reply and info.
- qianli_cs 2y agoThanks for sharing. This new feature is neat! It might sound a bit out there, but here's a thought: could you enable assigning unique IP addresses to different serverless instances? For certain use cases, like web scraping, it's helpful to simulate requests coming from multiple locations instead of just one. I think allowing requests to originate from a pool of IP addresses would be doable given this proxy model.
- handfuloflight 2y agoDo you block certain ports?
- heinternets 2y agoSo much work seems to go into working around the limitations of IPv4 instead of towards a fully IPv6 capable world.
- klysm 2y agoUnfortunately we gotta do both. Overlay networks like wireguard might be a good stepping stone to move software towards IPv6 anyway
- klysm 2y agoWhy is it important to have a static outbound ip address?
- stuckkeys 2y agoThis is just what I needed. Chefs kiss.
- techn00 2y agoside question: what do you use to make the diagrams?