13 ms·
I just learned: Docker edits firewall rules for you
- teekert 4y agoOh no I only expose ports on my tailnet. Or, I guess not then? How to check the status of the real firewall them, if ufw can mislead?
- justinclift 4y agoIf you're legally allowed to run nmap against your host (eg some network places will have a real problem with it!) then it can be a useful way to check the "real world" reachability of ports.
- teekert 4y agoAfter some testing, the port really only seems to be open on the tailnet, luckily.
- berkle4455 4y agoIf you’re exposing ports on your tailnet then you’re presumably binding to those specific interfaces, right?
- teekert 4y agoNope I use ufw to only allow incoming on the tailnet [0]. Guess that I’ll ben nmapping some hosts soon… [0]: https://blog.hmrt.nl/posts/first_steps_arch_box/#set-up-tailscale-and-restict-ssh-to-the-tailnet-using-ufw https://blog.hmrt.nl/posts/first_steps_arch_box/#set-up-tail...
- patrick451 4y agoIt's fascinating that the bulk of the responses here amount to "you're holding it wrong". Somehow I think the conversation would look a lot different with the same argument applied to a memory CVE in a c++ application.
- zdw 4y agoDocker was not designed with security in mind, and on any host exposed to the internet a massive security risk because of this. Putting someone in the `docker` group is nearly equivalent to giving them root permissions. K8s at least offers more fine-grained control over networking, between port range restrictions and having to enable a NodePort.
- raesene9 4y agos/nearly// From https://zwischenzugs.com/2015/06/24/the-most-pointless-docker-command-ever/ https://zwischenzugs.com/2015/06/24/the-most-pointless-docke... anyone with Docker access gets root with that command. TBH out of the box Kubernetes is as bad, or worse than docker, in that anyone with create pod permissions can get root on every worker node in the cluster (and the control plane nodes if it's unmanaged Kubernetes) https://raesene.github.io/blog/2019/04/01/The-most-pointless-kubernetes-command-ever/ https://raesene.github.io/blog/2019/04/01/The-most-pointless...
- fuuffsjccgf 4y agoI love how all use cases for this is to simply run some command as a user without write permission to anything. We evolved sysadmin oposing thumbs called virtual machines just to deal with our environment making it cheaper to emulate an entire other computer instead of dealing with having a user without write access, guaranteed, on the FS.
- bionade24 4y agoThe 1st thing only happens when users explicitly misconfigure docker, the other one is a real security problem when docker runs in root mode.
- dns_snek 4y agoThere's nothing "explicit" about this sort of misconfiguration, that's exactly why it catches so many people out. It's a bad/insecure default that serves no practical purpose. If Docker bound to 127.0.0.1 by default and required you to explicitly bind to "0.0.0.0" to expose the container to the outside world, the frequency of dangerous misconfigurations would likely be reduced by 95%+. It would still catch people who bind to 0.0.0.0 and expect the traffic to be blocked by UFW, but that's a different issue. `iptables: false` is a very crude solution, they should offer something like `iptables: "manual"` where Docker only touches its own iptables chains and lets you wire them up to your own UFW/iptables chains in a sane way.
- bionade24 4y agoWhile docker's port exposure should be more explicit, he should also mention that he used docker without reading the parts of the doc he neeeded to, blindly trusting some dangerous half-knowledge from a forum or other persons. If he'd bind the host port to localhost or put caddy in a container in the same vlan, it wouldn't have happened. From the blogpost I'm not even sure if he's aware of the binding option. If you argue that a software shouldn't behave this way after being configured so explicitly, Archlinux not preconfiguring iptables to limit exposure like Debian or Ubuntu is worse, because it happens implicitly.
- chaps 4y agoConsidering how aggressively docker has been pushed down peoples' throats over the past 10 years, from bootcamps to tutorial after tutorial teaching how to use docker... it seems pretty unreasonable to think that everybody is going to read the all of the docker docs that describe this problem. Considering the risks involved, it's frankly silly for them not to include a warning. Hell, in the "container networking" docs, it even says it doesn't: "By default, when you create or run a container using docker create or docker run, the container doesn’t expose any of it’s ports to the outside world." And archlinux not preconfiguring iptables is a very, very archlinux-y thing to do -- not sure what your point on that is supposed to be there.
- Godel_unicode 4y agoIt doesn’t say that it doesn’t do that, it says that it doesn’t do that by default. If you explicitly tell it to expose a port, how can you possibly be surprised when it does so? If you don’t read the docs you don’t get to complain when you don’t understand the behavior. Edit: the second paragraph on the first search result for “docker networking” says that because it’s trying to present things in a platform independent way the overview won’t cover iptables specifics and then links to the detailed docs of how it uses iptables. If you can’t read two paragraphs maybe don’t try to be an engineer.
- ethbr0 4y agoIf something is designed and marketed as being usable without reading docs, it'd damn well better have defaults that "First, do no harm." It's insane to build something that optimizes ease of use, and then require users to understand it in depth to avoid footgunning.
- nubinetwork 4y agoThe fact that ufw doesn't play well with iptables sounds like a good reason not to use it. (Yes yes I know iptables is antiquated but they should still be compatible with each other at the kernel level)
- Volundr 4y agoUFW works fine with iptables. The issue occurs because docker adds an iptables chain that takes precedence over the UFW rules.
- nubinetwork 4y agoIf ufw worked fine, they shouldn't have had to run "iptables -L" to find out that docker put in a rule that "ufw status" couldn't see. The commands should just be interchangeable.
- Volundr 4y agoUFW works by creating iptables rules. It intentionally does not expose the full functionality of iptables. The u is for "uncomplicated". The fact that you can add iptables chains that take precedence over the UFW managed chain is a feature of iptables, not a bug of UFW. You'll find the same is true of any Linux firewall that sits on top of iptables or nftables.
- lazka 4y agoCould UFW detect/warn that there are iptables rules interfering with its own rules?
- Volundr 4y agoThat's a good question actually. Given the complexity and power of iptables and nftables (which is the reason UFW, firewalld and the like exist) I'd guess probably not. Trying to determine if a rule on another chain (or really a combination of rules across a combination of chains) changes the behavior of one of yours would be... challenging. I suspect the best it could do reliably is detect that rules in other chains exist and warn about that, but that's also a pretty common case. Pretty much every VPS in a cloud would have this warning for example. Personally I think the real problem here is docker. Injecting rules into iptables is a bit like blindly injecting lines of code into someone else's program sight unseen. I'm not aware of any other tools that are nearly so willing to do so.
- vbezhenar 4y agoI can also note that I didn't find any supported way to prevent that from happening. So it's one thing to find out that docker edit firewall rules. It's another thing to build a proper set of firewall rules which will prevent incoming connections. I came up with some rules including DOCKER-USER chain but I'd qualify them as hack relying on undocumented configuration. And I just have no idea how to configure firewalld because it puts another layer of complexity on top of iptables.
- ethbr0 4y agoFYI, afaik firewalld as a high level interface is on the way out in RHEL* land (if that's what you're using), to be replaced by nftables.
- 2OEH8eoCRo0 4y agofirewalld is a high level interface to both nftables and iptables.
- ethbr0 4y agoAfaik, firewalld doesn't speak nftables (the tooling, not the backend). At least, in the sense that if you use nftables (the high level tooling/config) and firewalld (the high level tooling/config), there will be some incompatibilities. So it's either/or.
- yackback 4y agoThe docs, always the first place to look: https://docs.docker.com/network/iptables/ https://docs.docker.com/network/iptables/
- cratermoon 4y agoAlso https://docs.docker.com/config/containers/container-networking/ https://docs.docker.com/config/containers/container-networki...
- NoZebra120vClip 4y agoRemember UPnP? Good times.
- josephcsible 4y agoThe difference between this and UPnP is that this is a program opening firewall rules on the computer it's running on, whereas UPnP let programs open firewall rules on other devices.
- NoZebra120vClip 4y agoA... program? You know that Docker containers are self-contained operating systems? The whole point of containers like Docker is to separate the allocation and control of resources via a middle way that is not quite virtualization. So a monolithic OS running on one computer is at one end of the spectrum, and a "separate hardware device" accessible by LAN is further along the spectrum, but it's a continuum. So I feel like Docker's control of the host OS presents a similar surprise to us today as we felt when UPnP was in control of networked devices a few decades ago. To a Docker or UPnP expert, it's just another fact of provisioning the application's resources. UPnP has often been deployed at the hands of inexperienced users, and so it is with Docker.
- JohnFen 4y ago> You know that Docker containers are self-contained operating systems? That doesn't qualify as a program?
- NoZebra120vClip 4y agoWell, if it's 1983, and the Master Control Program is after you, be careful not to get de-rezzed. "Hey, I got a new program to install on my computer!" "What's your new program called?" "This program is... Ubuntu Linux."
- shudza 4y agoYou can also add `iptables: false` to docker's daemon json and docker will stop messing with your iptables conf.
- deleted 4y ago[deleted]
- wswope 4y agoThe docs strongly advise against this: https://docs.docker.com/network/iptables/ https://docs.docker.com/network/iptables/
- 2bluesc 4y agoThis is on my todo list after wrestling docker almost in to compliance with nftables. Even so, I still have some issue with ports forwarded to docker services not NATting correctly and instead show up with a source IP of the docker bridge. Switching to nftables exclusively (docker is using iptables-nft) and preventing docker from doing this should resolve my issues. I can't think of another application on any of my systems that muck with firewall rules behind the scenes like this.
- deleted 4y ago[deleted]
- dawnerd 4y agoOther solution is to simply not expose ports directly with rocket compose and instead use something like Traefik to proxy them. I’ve stopped using ports after having a redis instance taken over by some crypto miner.
- Gordonjcp 4y agoTraefik is a good idea for other reasons, but there's no problem with using Docker to directly expose ports. You just need to ensure that you're only exposing the *right* ports, and aren't just blindly opening everything up to the world. By default nothing is exposed to the world. You do not need to expose your Redis server's port to anything.
- Animats 4y agoWhy does Docker have the privilege of editing the iptables? Is it running as root?
- KaiserPro 4y agoit also fiddles with your container's resolv.conf, which can cock stuff up really badly if you are using search domain.
- bionade24 4y agoThere is a option to hardcode DNS and also an option to don't abstract networking and use host mode. Fiddling with your container's resolv.conf is necessary for docker networks between containers.
- KaiserPro 4y agoIndeed its configurable, but its not obvious! it was only when I was doing TCP dump on the host did I realise that docker was routing our containers to 8.8.8.8, rather than the proper DNS that we had specified.
- ysleepy 4y agoIt is a long standing issue and is a gotcha for everyone. https://github.com/moby/moby/issues/22054 https://github.com/moby/moby/issues/22054 It's completely asinine.
- Gordonjcp 4y agoWhat's asinine is "I configured it wrong and it broke, it's a bug in docker".
- justinclift 4y agoDocker's networking behaviour famously changes between point releases, sometimes without even being mentioned in the release notes. Very much a case of "make sure you use some sort of overarching firewall solution" wrapping your Docker hosts, otherwise you can be in for a world of hurt. :/ Their support for IPv6 used to be extremely shitty too, with the documented switch to enable it... not working at all. Heh. ;) The IPv6 thing may or may not be better now, but I haven't tried Docker with IPv6 in years so no idea.
- Gordonjcp 4y agoNo, you don't need a firewall. Docker doesn't expose anything to the public interface unless you specifically request that it does. If you don't understand the implications of this, you should not be running servers. It is absolute shite at doing IPv6, which is annoying.
- justinclift 4y ago> Docker doesn't expose anything to the public interface unless you specifically request that it does. Heh Heh Heh. Having directly been involved in the clean up of it doing exactly that a few years ago, I'll just say that your confidence is dangerously misplaced. :(
- Gordonjcp 4y ago
- quectophoton 4y agoI only learned about this a few years ago by coincidence, because I happened to open HN when there was a story[1] about NewsBlur having been hacked "thanks" to this Docker "feature". [1]: https://news.ycombinator.com/item?id=27670058 https://news.ycombinator.com/item?id=27670058
- bionade24 4y agoIf such an obvious thing happens to a company because they're too smart to read documentation, they would have run into big problems with LXD or sd-machined, too.
- spiffytech 4y agoI disagree that this is obvious or reflects an inappropriate attitude. This is a case where a tool you trust (ufw) lies to you because Docker not only unexpectedly adds iptables rules for simple scenarios, but adds them at a higher priority than your firewall. Software doesn't normally modify your firewall when it binds to a port, and firewalls exist to restrict port exposure. Docker adding iptables rules is understandable once you realize the full power of Docker networking, but I feel it's reasonable for people to not predict that behavior, given such power is beyond many people's use cases. (FWIW, rootless podman doesn't alter iptables, validating the proposition that using Docker in simple scenarios doesn't lend itself to a mental model that predicts iptables changes). If Docker was a simple tool with brief docs, I could agree, but Docker's docs are so extensive that knowing everything it does isn't practical. One just has to hope they stumble into docs on footguns. It's hard for me to call this behavior "obvious" without basically invalidating footguns as an idea.
- quectophoton 4y ago> Docker adding iptables rules is understandable once you realize the full power of Docker networking, but I feel it's reasonable for people to not predict that behavior, given such power is beyond many people's use cases. (FWIW, rootless podman doesn't alter iptables, validating the proposition that using Docker in simple scenarios doesn't lend itself to a mental model that predicts iptables changes). Adding to this. I really don't blame people (not even my past self) for: * Expecting `-p 8001:8002` to behave the same way as the unprivileged `ssh -L 0.0.0.0:8001:${CONTAINER_IP_IN_ONE_DOCKER_SUBNET}:8002`. * Expecting `ufw` to pickup changes in firewall rules. * Trusting the offline `man 1 docker-run`. Even knowing what I know now (I'm still pretty ignorant, but a little bit less than in the past), I still consider this a bad default from the Docker CLI. Yes, `docker run` needs privileges for other stuff, but since the forwarding part can be done without punching a hole, I don't think it's unreasonable to expect it to be done that way. Even just mentioning the word "iptables" in `docker-run(1)` would go a long way.
- skerit 4y agoI learned this a few months ago too. It's such a weird, stupid system. But everyone seems to think this is pretty normal. I think the firewall is my business, and nothing else should automatically open ports in it. Especially not docker.
- izacus 4y agoYep, learned that the hard way when it trashed my server UFW config as well. It felt kinda wierd to have a tool change machine configuration in such a way (even VMs generally don't do that?), but what do I know, I still prefer to deploy single binary services.
- bionade24 4y agoThis is not about being unnormal, this is a design decision. Just because people use docker in the "tradional admin way", too doesn't mean it's a wrong decision to do this. Wishful thinking about what a command of a software does without looking it up is generally dangerous. There are abstractions to help people that don't want to bother, e.g. portainer.
- Brian_K_White 4y agoViolates least surprise.
- xboxnolifes 4y agoAccomplishes least resistance.
- Brian_K_White 4y agoLeast resistance is of no legitimate value by itself. It's a goal only within the bounds of other constraints. The examples are literally countless.
- josephcsible 4y agoHow else would you expect it to NAT traffic into and out of the containers' network namespaces?
- logimus 4y agoUse podman and ditch docker, you can even alias it.
- mardifoufs 4y agoI thought podman did the same?
- logimus 4y agoMuch more natively integrated into Linux.
- anotherhue 4y agoI've been burned and infuriated by this, but let me ask honestly, is there other (common-ish) software that does this? The novelty to me was half the issue.
- nickstinemates 4y agoHow else did OP expect traffic to get to his containers? Magic? edit to add: after reading the article completely, a few things stand out. 1. It is clear the author does not understand the details of the compose file and is new to Docker. 2. Proposed solution is also incomplete 3. Not understanding why you would want to wrap caddy (even though it's a static binary) in the same system is missing the point. 4. Blaming Docker for clear user error is crazy.
- longitudinal93 4y agoThis is a trend that needs to be sharply curtailed. The last time I _re-started_ my Big Blue Button server it automatically installed UFW along with it's own ruleset completely locking me out as I run SSHD on a non-standard port. All without asking or even telling me it was going to do this. Aargh!
- fulafel 4y agoThis is the kind of case that is a good argument for regulated liability even on open source or gratis software components in the egregrious cases - the actor is a big tech company and clearly exposing users to compromise breakins in a calculated, fully aware way, for ~7 years. [1] https://github.com/moby/moby/issues/22054 https://github.com/moby/moby/issues/22054