9 ms·

This is...not simple. I agree that container networking is not all that much different from other Linux networking, but that doesn't make it simple. A lot of ap

This is...not simple. I agree that container networking is not all that much different from other Linux networking, but that doesn't make it simple. A lot of application developers are switching to working with containers and haven't historically had to do any manual network configuration. It's all new to them. Linux networking conventions change every few years and simplying keeping up with the basics is a chore. netplan is the current flavor of the week but it's still new to plenty of people who've never had to worry about anything more than auto-configured DHCP or cloud-provider-default VPS networking. There are also important security considerations with container networking. Docker, by default, punches massive holes through your firewall in non-obvious ways. People don't realize that with a default Docker configuration, containers are ignoring any normal firewall rules you may have setup with iptables or ufw. Locking that down is only easy if you already know iptables well, and even if you do, managing it is a pain. The article doesn't touch on Kubernetes, but that's a whole other can of worms. You have to pick a CNI and manually configure it; DNS doesn't just magically work; default CIDR allocations often conflict with existing networks; load balancer ingress for a development/single-host/non-cloud environment is a horror show. This is a good and helpful article, but container networking is not simple by any stretch.

mauflows 6y ago
I hope k8s (or some managed subset) eventually provides a stable networking abstraction. There is so much to consider when it comes to network hardening and I'd like it to not have to constantly stress about it, feeling like I don't know enough.
Hackbraten 6y ago
I think it’s time for us to get past “X is simple” or “X is easy.” It’s a highly subjective thing to say, no matter the topic. It’s never helpful, it just makes other people feel bad.
- iximiuz 6y ago
  Well, to be honest, some level of sarcasm was meant to be in this title because it precedes a 4000+ words explanation of the topic... But seems like I'm pretty bad at writing good titles. For sure I didn't mean to make anyone feel bad.
  - a1369209993 6y ago
    FWIW, I thought it was obviously either stupidity[0] or sarcasm based on the title, and the 4000+ words explanation confirmed it as the latter. Can't speak for anyone else, though. Edit: 0: well, or lies / container vendor shilling
  - Hackbraten 6y ago
    Appreciate the response. Thank you.
stonesweep 6y ago
> netplan is the current flavor of the week ... for Ubuntu, and Ubuntu only. It is an invention of Canonical not adopted by the rest of Linux distros (except Ubuntu derivatives), generally speaking.
- pnutjam 6y ago
  Long live wicked, easily managed with Yast on OpenSuse.
  - mongol 6y ago
    Yes, but how do you manage it without yast?
    - pnutjam 6y ago
      try: wicked --help ... but the beautiful thing about yast, is you get the interface via gui or console and it's largely identical.
skrtskrt 6y ago
Any good books on Linux networking? I have started spinning up “bare metal” k8s on a cloud VM and it’s not that hard to get going until you get to anything networking related then I feel like I’ve just jumped off a cliff. I have no knowledge there and the online resources seem to be nonexistent because you’re expected to just use a prebaked solution from cloud providers. I ended up just installing k3s but I have yet to figure out where the Traefik packaged in k3s can listen directly on port 80 and 443, but the basic Traefik installed via Helm cannot
- kazen44 6y ago
  To be fair, starting with something like a BSD might be easier in terms of networking. Mainly because the tooling hasn't been all over the place in the last 2 decades. Also, a lot of network knowledge is not OS specific. Learning about IP, ethernet, routing protocols etc is valuable no matter which OS you use.
  - skrtskrt 6y ago
    I have done some learning about more generalized network concepts (ip addresses, protocols, routing) but I want to make the jump to actually apply it in some VMs now I think my next stop is the network section of Unix and Linux System Administration Handbook
  - tiew9Vii 6y ago
    I agree. I've had years of Linux experience and decided my home network + router was quite poor and I'd set up my own home network router and use OpenBSD to do it as an excuse to look at a BSD and fix my network issues. OpenBSD has been an absolute pleasure to work with. The man pages are well written and complete, the filesystem is well organized, having no issues that people have with Systemd the simplicity of OpenBSD's init system showed you don't need complex init systems. Then you have PF which is far easier to deal with than IP tables. It took me a few hours of playing to set OpenBSD up with multiple vlan's, dhcp server, firewall, cross vlan routing, mdns etc. The hardest part was figuring out the correct parameters for my ISP broadband connection as the ISP didn't publish some of the information so needed to sniff the network traffic to find that i needed to set a vlan for my isp. The simplicity of OpenBSD makes it very easy to learn networking and if you are familiar with Linux you will become familiar with BSD very quickly as BSD's are less fragmented than Linux by their nature.
kenniskrag 6y ago
> through your firewall in non-obvious ways. People don't realize that with a default Docker configuration, containers are ignoring any normal firewall rules you may have setup with iptables or ufw. What do you mean by non-obvious? If I bind a port of the container to the host eg. -p 8080:80 only this port is a hole. Do you have something different in mind? (I'm a docker beginner)
- mmgutz 6y ago
  Docker's custom iptables chains, upon restart of network or docker, can and most likely WILL clobber rules that used to work. Docker adds the DOCKER-USER user chain for the purpose of readding rules back after docker has set up its rules. Most engineers never have to deal with it unless they are deploying to production boxes.
- caymanjim 6y ago
  That alone is non-obvious. Docker adds prerouting and masquerade changes. They don't show up in the default iptables rule list (iptables -L). If you don't know about it creating the other tables it adds, and how to list them explicitly, you won't see these rules added. Docker binds to the public interface by default, rather than to much-safer private/localhost defaults. If you have an iptables INPUT DENY rule, it doesn't come into play at all, and that's certainly untuitive for people who aren't iptables experts. For example, if you're using ufw (which most non-power-users rely on) and have it set to DENY by default, that won't protect you from the example you provided. The way Docker works is simply wrong. It should not mess with iptables at all. It does so because that makes it "easier" for novice users, but that ease of use is a real security risk.
  - still_grokking 6y ago
    > The way Docker works is simply wrong. And that's not only the networking. Everything about Docker is like that. It's a mess. Don't use.
speakspokespok 6y ago
A -- "A lot of application developers are switching to working with containers and haven't historically had to do any manual network configuration. It's all new to them." B -- "Linux networking conventions change every few years and simplying keeping up with the basics is a chore. netplan is the current flavor of the week but it's still new to plenty of people who've never had to worry about anything more than auto-configured DHCP or cloud-provider-default VPS networking." My perspective has been that "A" is directly leading to "B" for similar reasons new javascript frameworks keep being released. Linux networking conventions don't change rapidly (5 - 10 years) but new management tooling does get created alongside to sit next to existing tooling, already abstracted out OS tools have new tooling to further abstract out system management which just creates further unnecessary complexity.
jasonjayr 6y ago
> Linux networking conventions change every few years and simplying keeping up with the basics is a chore. Debian has used /etc/network/interfaces for as long as I know. (at least 15+ years) It's sane, short and straightforward. For servers, this has been more than ideal for every setup I could conceive.

mauflows 6y ago

I hope k8s (or some managed subset) eventually provides a stable networking abstraction. There is so much to consider when it comes to network hardening and I'd like it to not have to constantly stress about it, feeling like I don't know enough.

Hackbraten 6y ago

I think it’s time for us to get past “X is simple” or “X is easy.” It’s a highly subjective thing to say, no matter the topic. It’s never helpful, it just makes other people feel bad.

iximiuz 6y ago
Well, to be honest, some level of sarcasm was meant to be in this title because it precedes a 4000+ words explanation of the topic... But seems like I'm pretty bad at writing good titles. For sure I didn't mean to make anyone feel bad.
- a1369209993 6y ago
  FWIW, I thought it was obviously either stupidity[0] or sarcasm based on the title, and the 4000+ words explanation confirmed it as the latter. Can't speak for anyone else, though. Edit: 0: well, or lies / container vendor shilling
- Hackbraten 6y ago
  Appreciate the response. Thank you.

stonesweep 6y ago

> netplan is the current flavor of the week ... for Ubuntu, and Ubuntu only. It is an invention of Canonical not adopted by the rest of Linux distros (except Ubuntu derivatives), generally speaking.

pnutjam 6y ago
Long live wicked, easily managed with Yast on OpenSuse.
- mongol 6y ago
  Yes, but how do you manage it without yast?
  - pnutjam 6y ago
    try: wicked --help ... but the beautiful thing about yast, is you get the interface via gui or console and it's largely identical.

skrtskrt 6y ago

Any good books on Linux networking? I have started spinning up “bare metal” k8s on a cloud VM and it’s not that hard to get going until you get to anything networking related then I feel like I’ve just jumped off a cliff. I have no knowledge there and the online resources seem to be nonexistent because you’re expected to just use a prebaked solution from cloud providers. I ended up just installing k3s but I have yet to figure out where the Traefik packaged in k3s can listen directly on port 80 and 443, but the basic Traefik installed via Helm cannot

kazen44 6y ago
To be fair, starting with something like a BSD might be easier in terms of networking. Mainly because the tooling hasn't been all over the place in the last 2 decades. Also, a lot of network knowledge is not OS specific. Learning about IP, ethernet, routing protocols etc is valuable no matter which OS you use.
- skrtskrt 6y ago
  I have done some learning about more generalized network concepts (ip addresses, protocols, routing) but I want to make the jump to actually apply it in some VMs now I think my next stop is the network section of Unix and Linux System Administration Handbook
- tiew9Vii 6y ago
  I agree. I've had years of Linux experience and decided my home network + router was quite poor and I'd set up my own home network router and use OpenBSD to do it as an excuse to look at a BSD and fix my network issues. OpenBSD has been an absolute pleasure to work with. The man pages are well written and complete, the filesystem is well organized, having no issues that people have with Systemd the simplicity of OpenBSD's init system showed you don't need complex init systems. Then you have PF which is far easier to deal with than IP tables. It took me a few hours of playing to set OpenBSD up with multiple vlan's, dhcp server, firewall, cross vlan routing, mdns etc. The hardest part was figuring out the correct parameters for my ISP broadband connection as the ISP didn't publish some of the information so needed to sniff the network traffic to find that i needed to set a vlan for my isp. The simplicity of OpenBSD makes it very easy to learn networking and if you are familiar with Linux you will become familiar with BSD very quickly as BSD's are less fragmented than Linux by their nature.

kenniskrag 6y ago

> through your firewall in non-obvious ways. People don't realize that with a default Docker configuration, containers are ignoring any normal firewall rules you may have setup with iptables or ufw. What do you mean by non-obvious? If I bind a port of the container to the host eg. -p 8080:80 only this port is a hole. Do you have something different in mind? (I'm a docker beginner)

mmgutz 6y ago
Docker's custom iptables chains, upon restart of network or docker, can and most likely WILL clobber rules that used to work. Docker adds the DOCKER-USER user chain for the purpose of readding rules back after docker has set up its rules. Most engineers never have to deal with it unless they are deploying to production boxes.
caymanjim 6y ago
That alone is non-obvious. Docker adds prerouting and masquerade changes. They don't show up in the default iptables rule list (iptables -L). If you don't know about it creating the other tables it adds, and how to list them explicitly, you won't see these rules added. Docker binds to the public interface by default, rather than to much-safer private/localhost defaults. If you have an iptables INPUT DENY rule, it doesn't come into play at all, and that's certainly untuitive for people who aren't iptables experts. For example, if you're using ufw (which most non-power-users rely on) and have it set to DENY by default, that won't protect you from the example you provided. The way Docker works is simply wrong. It should not mess with iptables at all. It does so because that makes it "easier" for novice users, but that ease of use is a real security risk.
- still_grokking 6y ago
  > The way Docker works is simply wrong. And that's not only the networking. Everything about Docker is like that. It's a mess. Don't use.

speakspokespok 6y ago

A -- "A lot of application developers are switching to working with containers and haven't historically had to do any manual network configuration. It's all new to them." B -- "Linux networking conventions change every few years and simplying keeping up with the basics is a chore. netplan is the current flavor of the week but it's still new to plenty of people who've never had to worry about anything more than auto-configured DHCP or cloud-provider-default VPS networking." My perspective has been that "A" is directly leading to "B" for similar reasons new javascript frameworks keep being released. Linux networking conventions don't change rapidly (5 - 10 years) but new management tooling does get created alongside to sit next to existing tooling, already abstracted out OS tools have new tooling to further abstract out system management which just creates further unnecessary complexity.

jasonjayr 6y ago

> Linux networking conventions change every few years and simplying keeping up with the basics is a chore. Debian has used /etc/network/interfaces for as long as I know. (at least 15+ years) It's sane, short and straightforward. For servers, this has been more than ideal for every setup I could conceive.