12 ms·
Permissive forwarding rule leads to unintentional exposure of containers (2021)
- geenat 4y agoRootless Docker is currently the only version that does not touch iptables. Modifying iptables was a massive oversight by Docker- they really should deprecate that.
- k8sToGo 4y agoOr you just pass --iptables=false to dockerd. Then it will also not do anything.
- password4321 4y agoIn context, https://github.com/moby/moby/issues/22054#issuecomment-962202433 https://github.com/moby/moby/issues/22054#issuecomment-96220... and following: > Docker containers in the 172.16.0.0/12 ip range are definitely reachable from external hosts. All it requires is a custom route on the attacker's machine that directs traffic bound for 172.16.0.0/12 through the victim's machine. > Notice that the common advice of binding the published port to 127.0.0.1 does not protect the container from external access. ... > You don't even have to --publish.
- koolba 4y agoWould this attack require having control of a machine on the same local network? Otherwise how would you route that packet to the target machine?
- infotogivenm 4y agoYes, I believe so, although you need “privileged network access” on the box - meaning you can craft random IP packets, not just “control” of the box. So on the LAN or otherwise in-path for the route for that subnet’s responses.
- jandrese 4y agoAren't you just crafting a packet to a 172.17.0.0/16 address on port 80? That requires no special permission whatsoever.
- infotogivenm 4y agoIf your compromised box has a route already for that subnet and machine yes. That would be odd though I think.
- jandrese 4y agoRight, you aren't going to send that packet over the Internet, and you'll only getting it through local routers that have the routes you need, which should not be common.
- sigg3 4y agoMight be common with IoT devices, cloud network equipment etc. :|
- ranger_danger 4y agoyes
- infotogivenm 4y agoI believe this is only true when ipv4 forwarding kernel tunable is enabled, but am not sure. Does anyone know?
- ranger_danger 4y agoIf you disable it at least, your containers will no longer have internet access.
- infotogivenm 4y agoHad to look it up, you’re right: https://github.com/moby/moby/issues/490 https://github.com/moby/moby/issues/490 It’s required by docker, ugh.
- metadat 4y agoSee sibling comment: https://news.ycombinator.com/item?id=31841487 https://news.ycombinator.com/item?id=31841487
- cxcorp 4y agoThis is even worse than falsely binding to 127.0.0.1. If I don't publish a port in a container, I expect to *not publish any ports*.
- tinco 4y agoEh, isn't the real story then that Linux will route random incoming traffic on any interface? Even if you haven't configured it as a router? I know whoever was in charge of configuring Dockers iptables routes should have known this and messed up, but that is fucked up.
- jeroenhd 4y agoI'm a bit confused; as far as I know Linux only forwards packets on interfaces where you've enabled routing, doesn't it? If you set up: net.ipv4.ip_forward=0 net.ipv4.conf.eth0.forwarding=0 net.ipv4.conf.docker.forwarding=1 I'm pretty sure this isn't an issue, right? My guess is that the real real story is probably that every guide on the internet says to just set net.ipv4.ip_forward=1 and that nobody bothers to stop and read up on the sysctl parameters they're copy/pasting from the internet. For this attack to succeed, the attacker also needs to be on the same network or have their upstream ISPs accept plain external traffic towards internal networks. Executing the PoC on Linux without being in the same subnet won't even be accepted (though raw sockets may still send out traffic towards the host that will probably get filtered somewhere along the way).
- forbiddenlake 4y agoThis doesn't work. Breaks traffic out (below) and in. # sysctl net.ipv4.ip_forward net.ipv4.conf.enp8s0.forwarding net.ipv4.conf.docker0.forwarding net.ipv4.ip_forward = 1 net.ipv4.conf.enp8s0.forwarding = 1 net.ipv4.conf.docker0.forwarding = 1 # docker run --rm alpine/curl -sI --max-time 5 http://1.1.1.1 | head -n2 HTTP/1.1 301 Moved Permanently Server: cloudflare # sysctl net.ipv4.conf.enp8s0.forwarding=0 net.ipv4.conf.enp8s0.forwarding = 0 # docker run --rm alpine/curl -I --max-time 5 http://1.1.1.1 | head -n2 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- 0:00:05 --:--:-- 0 curl: (28) Connection timed out after 5000 milliseconds
- lmm 4y ago> Linux will route random incoming traffic on any interface? Even if you haven't configured it as a router? "configuring Linux as a router" is exactly what "adding an iptables rule that routes" is. That's what it's for, that's how you do it.
- Spivak 4y agoWell yeah, you turned on ip forwarding and created a network on your host that can be reached. Imagine this was a VM host and those networks were meant to be routable, how else could it work? Are people just now discovering the FORWARD chain?
- theptip 4y agoLike the OP said, > most Linux users do not know how to configure their firewalls and have not added any rules to DOCKER-USER. The few users that do know how to configure their firewalls are likely to be unpleasantly surprised that their existing FORWARD rules have been preceded by Docker's own forwarding setup
- mark-wagner 4y agoI'm confused. Does docker change the routing table? Otherwise there will be no route back to the attacker's machine. Also, if 172.16.0.0/12 is bound to the docker0 interface then the kernel will see packets from IPs in that range that come in on other interfaces as martians and drop them.
- blueflow 4y agoIs this still the realm of "mistakes can happen sometimes" or is it neglience? Where is the limit?
- k8sToGo 4y agoIs it neglience if it is a documented feature? Bad UX does not imply neglience?
- JamesSwift 4y agoIts documented, but still extremely surprising even for experienced docker users. The only place you will read about this is is you actually go to the docs page for the `-p` flag. But as I've said before, why would I do that? I already know what `-p` does (spoiler: I didn't know what it did). It was multiple years before I realized I was exposing my services like this, after it came up on HN a while back.
- k8sToGo 4y agoI am not saying that this does not suck. I was a victim of this as well. I am just saying that it isn't really negligence, just bad architecture.
- JamesSwift 4y agoRight, but theres bad architecture and then theres "this is a security risk and every tutorial in the wild + every app in production uses this in an insecure way and we haven't done anything about it". I just realized I posted my thoughts on this github issue [1] which is now _six_ years old. There have been no updates / changes made as far as I can tell. [1] - https://github.com/moby/moby/issues/22054 https://github.com/moby/moby/issues/22054
- k8sToGo 4y agoMhh good point. I guess if you keep bad architecture long enough it becomes negligence.
- jck 4y agoSecuring docker, with it's weird iptables shenanigans is a nightmare. I prefer to use rootless podman, and also avoid exposing ports (by using internal routing and if needed, a reverse proxy with only one published port)
- 2OEH8eoCRo0 4y agoSame here. Are there even legit reasons to not run all containers rootless? I run rootless everything, as well as things that need privileged ports. The design choices by the Docker team are just baffling.
- withinboredom 4y agoI believe, at the time, setting up a container required root. As in the kernel didn't expose the right pieces for a non-root user to ever dream of starting a container.
- BeefySwain 4y agoYeah it's nuts. The best solution I've found is some kind of cloud firewall, whether that be an off the self service that you use with whatever cloud provider you are using, or rolling your own by routing all traffic through another host that doesn't have Docker nuking all your firewall rules every time it restarts.
- deleted 4y ago[deleted]
- iasay 4y agoThis. You need a last resort security control against stuff like this anyway. Even an automation failure or misunderstanding of a ruleset can leave you exposed. Security must be layered.
- notimetorelax 4y agoTo be fair either in the cloud or on premise, firewall is a must. It’s just one of the layers of security.
- TekMol 4y agoSo every time you sit down in an internet cafe and connecting to the wifi, you are fucked because all the docker containers that run on your machine are open to everybody? Or - depending on the lan settings - at least to the wifi provider? Also - shouldn't the web be full off vurnurable database servers then?
- R0b0t1 4y ago> Also - shouldn't the web be full off vurnurable database servers then? It already was, but yes.
- albatruss 4y ago> Also - shouldn't the web be full off vurnurable database servers then? No, the docker bridge network is not on a routable subnet.
- TekMol 4y agoDoes it have to? The attack looks like it would also work over the internet: 2. [ATTACKER] Route all packets destined for 172.16.0.0/12 through the victim's machine. ip route add 172.16.0.0/12 via 192.168.0.100 Here, "192.168.0.100" could be exchange for any ip address I guess?
- gcbirzan 4y agoExcept you would have to be on the same layer 2 network as the "victim" for this to work.
- thedougd 4y agoIf that's true, you can then send packets to it, but not receive replies. That's still a problem.
- jandrese 4y agoThat will only work if you are on the same subnet. When you craft a packet for that address, the stack will see that route and send an ARP "who has" request out whatever interface you assigned when you did that IP route rule (probably your default ethernet). If nobody responds than the packet dies in the stack.
- contingencies 4y agoOuch. This is the security posture that half a billion dollars, hundreds of engineers, multiple layers of abstraction to a firewall ruleset and each Stream-aligned team with an embedded Product Manager and UX Designer[0] buys you. I too contributed security fixes to Docker in the early stages but soon gave up. Docker is a fascinating study in hype. Containers already existed, docker wrapped them very badly, raised half a billion dollars, popularized their wrapper (I understand kubernetes is the fad-wrapper now) and then failed to obtain a revenue stream. I didn't see them return much to open source, though perhaps they contributed to projects I don't see. Honestly, the whole thing has a plotline like a Shakespearean tragedy. Someone could do a film, starting with interviews with the guys who wrote the original Linux containers and namespacing kernel code at IBM, the maintainers of jails in FreeBSD, and an early hacker at VMWare. [0] https://www.docker.com/blog/building-stronger-happier-engineering-teams-with-team-topologies/ https://www.docker.com/blog/building-stronger-happier-engine...
- xvector 4y agoHow is K8S a fad?
- natpalmer1776 4y agoOn a long enough timeline, everything can be considered a fad!
- blowski 4y agoAsk me in 10 years, I'll tell you whether it was a fad or not.
- languageserver 4y agoa 20 year run in tech is not a "fad".
- 867-5309 4y ago"tech" was a fad to your future AI overlords
- rootsudo 4y agoNice, that's interesting indeed!
- wiredfool 4y agoThe other fun thing here, at least with docker-compose, is that it behaves really badly with IPv6, essentially, if you don't manually set a IPv6 address, then all IPv6 to the host address is masqueraded to IPv4, with the visible address being the router address for the network. So, Any external IPv6 connections come in looking like they're from the local network.
- staticassertion 4y agoWith vanilla Docker ipv6 is just disabled by default I believe.
- 0xbadcafebee 4y agoIt should, however, restrict the source ip address range to 127.0.0.1/8 and the in-interface to the loopback interface: Chain DOCKER (2 references) pkts bytes target prot opt in out source destination 0 0 ACCEPT tcp -- lo docker0 127.0.0.1/8 172.17.0.2 tcp dpt:5432 I think this would break intra-container network communication, since containers on docker1 (172.17.1.2) would not be coming from source 127.0.0.0/8 or device lo. Docker would need to create an explicit rule matrix from/to each container in each network (does it already do this?)
- infotogivenm 4y agoSetting aside the (rather scary) issue of crafted external packets being bridged into the internal docker subnet - the “0.0.0.0” fw-bypassing default for published ports in a developer-oriented tool is a complete nightmare for infosec teams at software engineering companies. I’ve seen of thousands of employee hours wasted chasing that down.
- M0r13n 4y agoNice to see that "Docker and Kubernetes are in first and second place as the most loved and wanted tools" according to the Stack Overflow developer survey.
- Matl 4y agoDocker is definitely a 'good enough' frontend for containers in that it did a lot to make them more accessible, hence why it tops that survey, (presumably). This can be quite separate from Docker being a security nightmare at the same time.
- asimpson 4y agoAh yes, this footgun. Sometime last year I discovered our QA databases had all been locked with a ransom demand, how were they exposed? We had `ufw` blocking the db port?! Ah yes, Docker -p. smh.
- dewlinedew2 4y agoGive us money or were keeping your test data
- dekhn 4y agoDocker was definitely a fork in the road that led to unhappiness. I was tracking linux containers for several years (specifically, with the goal of doing cross-machine container migration) and they were doing great. Then docker came along and... pretty much every experience Ive had with it involves realizing core decisions were made by people who had no experience and we're all paying for it, long-term.
- yencabulator 4y agoWhen Docker was still new, a Go IRC channel saw one employee after another coming to ask about combining ungzip and untar in a streaming fashion inside a Go process. We tried to help, but that was roughly when I realized Docker will never gain technical excellence. It never was in their culture.
- maxboone 4y agoRelated thread on the GitHub repo of Moby: - https://github.com/moby/moby/issues/22054#issuecomment-1099773429 https://github.com/moby/moby/issues/22054#issuecomment-10997...
- Spivak 4y agoDocker's behavior is unintutivie but makes sense given how container networking works. If you use UFW read https://github.com/chaifeng/ufw-docker https://github.com/chaifeng/ufw-docker and follow the guide. Then configuring firewall rules to containers is as easy as - name: Open HTTPS ufw: rule: allow proto: tcp route: true port: 443
- aspyct 4y agoI am (was) using portainer on my host to manage my docker containers. Turns out everything was, indeed, publicly accessible. I could fix it for my custom containers, by binding on 127.0.0.1:<port>, but I can't edit portainer's config. Anyone has a solution for this?
- DavideNL 4y agoYou can't fix this issue by binding to 127.0.0.1 My solution has always been to just use some external firewall (outside of the docker host machine.) Often cloud providers have this as a feature, to configure a firewall for your vps/network.
- aspyct 4y agoDoes that really protect me against the "routing through" my machine they explain?
- foresto 4y agoThis conversation makes me thankful that I spent time on the extra hoops to get Docker working in rootless mode. (That mode wasn't officially released yet; I imagine it's easier by now.) It also reminds me of Podman as a promising alternative. Now that it's in Debian Stable, I think it's about time I give it a try.
- guns 4y agoDocker users on the Mac are not affected by this issue, but they should be aware that the "Automatically allow downloaded signed software to receive incoming connections" option in the firewall settings must be unchecked in order to block incoming connections to container ports published to 0.0.0.0. This is necessary because Docker Desktop for Mac is presumably signed by Apple.
- nickjj 4y agoCan someone explain how that proof of concept in the gist applies to a production environment? For example, let's say I take this exact command from the gist: docker run -e POSTGRES_PASSWORD=password -p 127.0.0.1:5432:5432 postgres And run that on a VPS somewhere out in the world, let's say it's on a public IP address of 106.12.52.111 (I completely made up this IP address btw). How can you sitting on your dev box on a different network open a psql connection to 106.12.52.111 on port 5432 to the point where you haven't been blocked by iptables? The part I don't get is the gist mentions: > An attacker that sends traffic to 172.17.0.2:80 through the docker host will match the rule above and successfully connect to the container, obviating any security benefit of binding the published port on the host to 127.0.0.1. What's worse, users who bind their published ports to 127.0.0.1 operate under a false sense of security and may not bother taking further precautions against unintentional exposure. In the above example how is someone going to send traffic to 172.17.0.2:80 through the Docker host from a box on a different network than the Docker host? Also is this still exploitable if you drop everything at the iptables level before you start using Docker? For example all of my iptables configs start with: :INPUT DROP [0:0] :FORWARD DROP [0:0] :OUTPUT ACCEPT [0:0]
- guns 4y ago> In the above example how is someone going to send traffic to 172.17.0.2:80 through the Docker host from a box on a different network than the Docker host? The attacker and host will generally need to be on the same network so that the attacker's packets are not dropped because they are addressed to a non-routable private IP address. You could access the containers at 106.12.52.111 if you were in the same network (e.g. 106.12.52.0/24) and the packets did not have to traverse a router. > Also is this still exploitable if you reject everything at the iptables level before you start using Docker? Yes. Docker appends the FORWARD chain with custom rules that explicitly forward traffic to published ports.
- nickjj 4y ago> You could access the containers at 106.12.52.111 if you were in the same network (e.g. 106.12.52.0/24) and the packets did not have to traverse a router. Ok thanks, that's sort of what I thought (you had to be on the same network) but I wasn't 100% on that because networking has a lot of rabbit holes. Your gist is very well written and a great find but based on the scope of the vulnerability this wouldn't be classified as a catastrophic event right? If it's only limited to the attacker and the Docker host being on the same network while packets never go through a router then it's not an issue for the common case of someone hosting their web app or service on a VPS somewhere on the internet and have used 127.0.0.1:XXXX:XXXX to publish a port (perhaps their web app is published to localhost so nginx running directly on the Docker host can reverse proxy it -- this is what I've done for years now).
- naniwaduni 4y agoTakeaway: NAT around netnses is as about as bad a substitute for explicit filtering + proper bind discipline as it always was for local networks. So we should treat containers the same way we've long known to treat the hosts behind a NAT: sure, the NAT itself provides a little bit of defense in depth, but that's basically accidental and services should still be independently secured.
- SergeAx 4y agoI would love to see a version of portscanner which scans for Docker containers on my network, just for fun.
- netsectoday 4y agoThis is a valid concern when running Dockerized containers and it caught me years ago when I was deploying my first containerized servers to the wild. It was a "holy shit" moment, but anyone who is trusted (getting paid) to secure servers should stumble upon this pretty quickly. 1) Add firewall rules to UFW 2) Test firewall rules 3) Notice you can bypass those rules (they didn't take) 4) iptables -vL 5) Why the fuck is there a DOCKER-USER chain at the top of FORWARD, and it's above all of my UFW entries? 6) Adjust iptables without UFW 7) Realize adjustment didn't work after reboot because Docker dynamically adds the iptables entries on start (above my entries) 8) Add my chain to the bottom of the DOCKER-USER chain, before the RETURN all 9) Verify firewall rules 10) Reboot server 11) Verify firewall rules This is probably a huge issue today because most people can't do their job correctly. They are pulling that sweet salary, bullshitting their stand-ups, zoning out on video calls, and squirming until they can turn netflix back on while working remotely. You just need to have enough attention span to verify your work and dig in when things are broken.
- hda111 4y agoYou should try podman instead. The rootless version obviously can’t bypass UFW. When I tested podman as root it also didn’t punch holes into UFW. Not sure how it is today however.