14 ms·
Container networking is simple (2020)
- avmich 6y ago> Working with containers always feels like magic. Containers are just application - which existed for many decades - in a single-process (roughly) operating system. Like MS DOS. Nothing magical except containerized processes can't talk to each other directly - there are security boundaries.
- dangerbird2 6y ago> in a single-process (roughly) operating system. Containers don't (generally) run in their own operating system, just their own userland. It can actually be a source of security vulnerabilities to assume that containers are completely isolated from each other, such as running a root user container in production, assuming it can't get privileged access to the host. It's less similar to a bare-metal MS DOS application than it is a glorified chroot jail
- avmich 6y ago> Containers don't (generally) run in their own operating system Right, but containerized application can't (ideally) talk to other applications on the same machine, that's how it's similar to a single-process OS with a single app running. Of course there are details like a single application may still contain multiple processes from OS standpoint, but the overall comparison stands. > It's less similar to a bare-metal MS DOS application than it is a glorified chroot jail These two cases are similar enough from containerized application standpoint (only OS services are different than those of MS DOS).
- z3t4 6y agoYou can have a lot of fun with network namespaces in Linux, like running some programs in their own network/VPN.
- m463 6y agoIt is if you use: --net=host
- 29athrowaway 6y agoWhich defeats the purpose of containerization because now the container can sniff traffic on the host, hijack TCP connections, etc.
- stingraycharles 6y agoI know many people use single tenant hardware to run containers, so I wouldn’t say that it defeats the purpose of containerization, just one of the benefits.
- jeffbee 6y agoThe confusion and disagreement on this topic is rooted in faulty assumptions about other organizations' requirements. Many people use containers with host networking, without uid namespaces, without pid namespaces, without bind mounts. What you think is "a container" may not be at all universal. Also the idea that host networking grants all processes CAP_NET_RAW is just weird and wrong.
- MaxBarraclough 6y agoIf you're running untrusted code in your container, you've pretty much already lost. Containers are useful for deployment and configuration, they are not a robustly secure sandbox. For that you still need to go with a VM. No cloud provider will offer to run your containers alongside other customer's containers, on a shared kernel. Your containers always run within your own VM.
- jeffbee 6y agoI have to say, it would be cool if they did. If I could get a dirt-cheap rate for running batch workloads in a potentially antagonistic environment, I could make use of that. Not all data is sensitive.
- NexRebular 6y agoIndeed... VIMAGE and Crossbow are such great technologies for virtualizing container network stacks. Couldn't imagine life without them anymore.
- gyre007 6y agoThis blog post reminded me of the post [1] I wrote about 7 years ago o_O [1] https://cybernetist.com/2013/11/19/lxc-networking/ https://cybernetist.com/2013/11/19/lxc-networking/
- jnwatson 6y agoA lot of discussion around Docker in general is rehashing stuff that lxc did a long time ago, so this fits the mold.
- gorgoiler 6y agoBrilliant write up. Lots of this topic is so much easier to understand when it’s presented from first principles, without any of the LXC or Docker helpfulness hiding the details. (If the author is reading, thank you! I’ll likely use this material for the pupils in my computer club.) Managing an IPv6 stack alongside IPv4 is also very informative. IPv6 is still not widely deployed — SMTP is likely tied to v4 for all eternity — but it’s incredibly useful for managing multiple sites of inventory over the internet. Seeing RFC1918 style 10.x.y.z private addresses and IPAM in use by internal ops and IT in 2021 brings tears to my eyes. Adding a section on using conntrack to watch the way in which the kernel handles MASQUERADE and DNAT would be illustrative as well. I really like the diagrams too.
- iximiuz 6y agoThank you very much for your feedback! I appreciate it a lot because at the end of the day that's what keeps me motivated!
- gorgoiler 6y agoThe ip tool chain is a worthy thing to promote. I especially like all of the replace verbs, which I wish I’d known about sooner. They make idempotency much simpler to express without any if ! ip thing get <long list of route args>; then ip add <same list of args> ; fi stuff.
- john-tells-all 6y agoThat's cool! Here's a synopsis of `ip replace`: "replace will either modify an existing address or create a new one if the specified address does not exist" https://serverfault.com/questions/476926/understanding-ip-addr-change-and-ip-addr-replace-commands/666521#666521 https://serverfault.com/questions/476926/understanding-ip-ad...
- gegtik 6y agoThis is the first article i see on HN with (2020) in parens to show that it's a dated article =\
- caymanjim 6y agoThis is...not simple. I agree that container networking is not all that much different from other Linux networking, but that doesn't make it simple. A lot of application developers are switching to working with containers and haven't historically had to do any manual network configuration. It's all new to them. Linux networking conventions change every few years and simplying keeping up with the basics is a chore. netplan is the current flavor of the week but it's still new to plenty of people who've never had to worry about anything more than auto-configured DHCP or cloud-provider-default VPS networking. There are also important security considerations with container networking. Docker, by default, punches massive holes through your firewall in non-obvious ways. People don't realize that with a default Docker configuration, containers are ignoring any normal firewall rules you may have setup with iptables or ufw. Locking that down is only easy if you already know iptables well, and even if you do, managing it is a pain. The article doesn't touch on Kubernetes, but that's a whole other can of worms. You have to pick a CNI and manually configure it; DNS doesn't just magically work; default CIDR allocations often conflict with existing networks; load balancer ingress for a development/single-host/non-cloud environment is a horror show. This is a good and helpful article, but container networking is not simple by any stretch.
- mauflows 6y agoI hope k8s (or some managed subset) eventually provides a stable networking abstraction. There is so much to consider when it comes to network hardening and I'd like it to not have to constantly stress about it, feeling like I don't know enough.
- Hackbraten 6y agoI think it’s time for us to get past “X is simple” or “X is easy.” It’s a highly subjective thing to say, no matter the topic. It’s never helpful, it just makes other people feel bad.
- iximiuz 6y agoWell, to be honest, some level of sarcasm was meant to be in this title because it precedes a 4000+ words explanation of the topic... But seems like I'm pretty bad at writing good titles. For sure I didn't mean to make anyone feel bad.
- zekica 6y agoWhy does no one talk about IPv6 this day and age?
- ggm 6y agoCame here to say this. ULA would have been stacks better than this.
- CameronNemo 6y agoMaybe they find it too cumbersome to memorize and type an ipv6 address. Or grew up and cut their teeth on v4.
- yjftsjthsd-h 6y agoBenefits don't yet outweigh the cost for many, and years of familiarity/habit mean that it would need some really compelling advantages to get used.
- fulafel 6y agoWith IPv6 there's not enough tinkering with service meshes, port forwardings, workarounds for private ip address conflicts and VPCs to write articles about.
- otterley 6y agoIPv6 doesn't solve the difficulties of container networking. Containers have private network namespaces, which means that a container can't communicate with other hosts without (1) a dedicated interface or (2) a bridge (basically a software switch) for host-to-container communication. These problems are Layer 2 problems, while IPv4 and IPv6 are at layer 3.
- cooervo 6y agohow is that simple? :O
- jVinc 6y agoI know a guy, he hated networking, so he decided to set up a kubernetes cluster on his home server to abstract all his networking difficulties away. Now he occasionally complains that the network inside the network inside the network is annoying because it doesn't play nicely with the network outside the network on top of the main network. Overall I feel the rate at which he talks about networking issues has increased, but his enthusiasm for talking about them has gone up, so I guess that's a win.
- dan_quixote 6y agoYeah, I feel cursed by the same habit. I get to about a 75% understanding of a domain and feel like I'm about to run out of things to learn...so I expand the domain and continue until I'm at that 75% mark again...and repeat...
- KaiserPro 6y agoThis is a very good article. Container networking is as simple as linux networking, which isn't. Its as much of a faff now as it was when I was doing KVM virtualisation professionally (don't, pay and use VMware. You'll be much happier, and have lots of free time) To debug its a massive arse and lacks any useful and friendly debugging tools. One thing that does help is either to use VLANs or a second adaptor to separate container traffic from control. It makes debugging slightly easier. By easier, I mean that when you accidentally misconfigure it, there is a better chance that you'll be able to get control of the host still.
- CameronNemo 6y agoBesides VLANs, another way to separate control and data plane traffic is using VRFs. https://people.kernel.org/dsahern/management-vrf-and-dns https://people.kernel.org/dsahern/management-vrf-and-dns https://people.kernel.org/dsahern/docker-and-management-vrf https://people.kernel.org/dsahern/docker-and-management-vrf
- waynesonfire 6y agoi'd rather learn the real network stack. containers are not at all simple. in fact, it's made me adopt freebsd for my next project and i love it.
- musicale 6y agoIt's not rocket science, but it's not entirely simple because there are a lot of moving parts, because IP networking is annoyingly complicated, and because we are stuck with IP addresses and interfaces that they are attached to, with many of these low-level details percolating up to user code. I like Linux's a-la-carte container APIs but their flexibility seems to increase complexity, as you end up dealing with multiple namespaces as well as cgroups, etc.. Then there are Kubernetes and Docker which add a lot of complexity on top, while providing some levels of abstraction which don't entirely eliminate the need for understanding the underlying internals. Having some more mid-level container and networking APIs would be nice, but I'm not sure it would solve the complexity problem. Guaranteed employment for networking/container/linux experts I guess.
- mongol 6y agoAre macvlan and macvtap not container things? Are they more for VM networking?