15 ms·

Docker has a known security issue with port exposure in that it punches holes through the firewall without asking your permission, see https://github.com/moby/m

Docker has a known security issue with port exposure in that it punches holes through the firewall without asking your permission, see https://github.com/moby/moby/issues/4737 https://github.com/moby/moby/issues/4737 I usually expose ports like `127.0.0.1:1234:1234` instead of `1234:1234`. As far as I understand, it still punches holes this way but to access the container, an attacker would need to get a packet routed to the host with a spoofed IP SRC set to `127.0.0.1`. All other solutions that are better seem to be much more involved.

dawnerd 2y ago
We’ve found this out a few times when someone inexperienced with docker would expose a redis port and run docker compose up on a public accessible machine. Would only be minutes until that redis would be infected. Also blame redis for having the ability to run arbitrary code without auth by default.
bluedino 2y ago
Containers are widely used at our company, by developers who don't understand underlying concepts, and they often expose services on all interfaces, or to all hosts. You can explain this to them, they don't care, you can even demonstrate how you can access their data without permission, and they don't get it. Their app "works" and that's the end of it. Ironically enough even cybersecurity doesn't catch them for it, they are too busy harassing other teams about out of date versions of services that are either not vulnerable, or already patched but their scanning tools don't understand that.
- dijit 2y ago
  This is pretty common, developers are focused on making things that work. Sysadmins were always the ones who focused on making things secure, and for a bunch of reasons they basically don’t exist anymore. EDIT: what guidelines did I break?
  - ocdtrekkie 2y ago
    I suspect you'll find a lot of intersection between the move to "devops" outfits who "don't need IT anymore" and "there's a lot more security breaches now", but hey, everyone's making money so who cares?
  - smarx007 2y ago
    I don't think you broke any (did not downvote). But you wrote something along the lines "Sysadmins were always the ones who focused on making things secure, and for a bunch of reasons they basically don’t exist anymore. I guess this is fine." before you edited the last bit out. I think those who downvoted you think that this is plain wrong. I guess it's fine if you get rid of sysadmins and have dev splitting their focus across dev, QA, sec, and ops. It's also fine if you have devs focus on dev, QA, code part of the sec and sysadmins focus on ops and network part of the sec. Bottom line is - someone needs to focus on sec :) (and on QAing and DBAing)
  - bluedino 2y ago
    > This is pretty common, developers are focused on making things that work. True, but over the last twenty years, simple mistakes by developers have caused so many giant security issues. Part of being a developer now is knowing at least the basics on standard security practices. But you still see people ignoring things as simple as SQL injection, mainly because it's easy and they might not even have been taught otherwise. Many of these people can't even read a Python error message so I'm not surprised. And your cybersecurity department likely isn't auditing source code. They are just making sure your software versions are up to date.
    - bt1a 2y ago
      and many of these people havent debugged messages more complex than a Python error message. tastelessly jabbing at needing to earn your marks by slamming into segfaults and pushing gdb
  - harrall 2y ago
    Sometimes when you work less rigidly as a team, covering for others when it’s convenient for you, everyone gets more things done with less stress and less trouble. And you go home at 5pm and had a good work day.
- malfist 2y ago
  Checklist security at it's finest. My team where I work is responsible for sending frivolous newsletters via email and sms to over a million employees. We use an OTP for employees to verify they gave us the right email/phone number to send them to. Security sees "email/sms" and "OTP" and therefor, tickets us at the highest "must respond in 15 minutes" priority ticket every time an employee complains about having lost access to an email or phone number. Doesn't matter that we're not sending anything sensitive. Doesn't matter that we're a team of 4 managing more than a million data points. Every time we push back security either completely ignores us and escalates to higher management, or they send us a policy document about security practices for communication channels that can be used to send OTP codes. Security wields their checklist like a cudgel. Meanwhile, our bug bounty program, someone found a dev had opened a globally accessible instance of the dev employee portal with sensitive information and reported it. Security wasn't auditing for those, since it's not on their checklist.
  - dfsegoat 2y ago
    I feel this. Recently implemented a very trivial “otp to sign an electronic document” function in our app. Security heard “otp” and forced us through a 2 month security/architecture review process for this sign-off feature that we built with COTs libraries in a single sprint.
    - malfist 2y ago
      Oh I know that feeling. We got in hot water because the codes were 6 digits long and security decided we needed to make them eight digits. We pushed back and initially they agreed with us and gave us an exception, but about a year later some compliance audit told them it was no longer acceptable and we had to change it ASAP. About a year after that they told us it needed to be ten characters alphanumeric and we did a find and replace in the code base for "verification code" and "otp" and called them verification strings, and security went away.
      - dfsegoat 2y ago
        Heh. We also got treated to the digit thing. That topic alone was about 30 mins of mtg. time with a vp of eng and 2 seniors in the mtg.
- calvinmorrison 2y ago
  Turns out devsecops was just a layoff scheme for sysadmins
- queuebert 2y ago
  > Ironically enough even cybersecurity doesn't catch them for it, they are too busy harassing other teams about out of date versions of services that are either not vulnerable, or already patched but their scanning tools don't understand that. Wow, this really hits home. I spend an inordinate amount of time dealing with false positives from cybersecurity.
- ropable 2y ago
  "Everybody gangsta 'bout infosec until their machine is cryptolockered." (some CISO, probably).
- nitwit005 2y ago
  There are certainly people that don't care about security out there, but the biggest issue is just how much people are expected to know. Docker, AWS, Kubernetes, some wrapper they've put around Kubernetes, a bunch of monitoring tools, etc. And none of it will be their main job, so they'll just try to get something working by copying a working example, or reading a tutorial.
globular-toast 2y ago
This is only an issue if you run Docker on your firewall, which you absolutely should not.
- smarx007 2y ago
  Ideally, yes. But in reality, this means that if you just want to have 1 little EC2 VM on AWS running Docker, you now need to create a VM, a VPC, an NLB/ALB in front of the VPC ($20/mo+, right?) and assign a public IP address to that LB instead. For a VM like t4g.nano, it could mean going from a $3/mo bill to $23/mo ($35 in case of a NAT gateway instead of an LB?) bill, not to mention the hassle of all that setup. Hetzner, on the other hand, has a free firewall included.
  - Fnoord 2y ago
    There's no good reason a VM or container on Hetzner cannot use a firewall like IPTables. If that makes the service too expensive you increase cost or otherwise lower resources. A firewall is a very simple, essential part of network security. Every simple IoT device running Linux can run IPTables, too.
    - akerl_ 2y ago
      Docker by default modifies iptables rules to allow traffic when you use the options to launch a container with port options. If you have your own firewall rules, docker just writes its own around them.
      - Fnoord 2y ago
        I always have to define 'external: true' at the network. Which I don't do with databases. I link it to an internal network, shared with application. You can do the same with your web application, thereby only needing auth on reverse proxy. Then you use whitelisting on that port, or you use a VPN. But I also always use a firewall where OCI daemon does not have root access on.
anthropodie 2y ago
And this was one of the reason why I switched to Podman. I haven't looked back since.
- MortyWaves 2y ago
  I want to use Podman but I keep reading the team feels podman-compose to be some crappy workaround they don’t really want to keep. This is daunting because: Take 50 random popular open source self-hostable solutions and the instructions are invariably: normal bare installation or docker compose. So what’s the ideal setup when using podman? Use compose anyway and hope it won’t be deprecated, or use SystemD as Podman suggests as a replacement for Compose?
  - somebehemoth 2y ago
    podman rootless running services with quadlet is not a bad start.
    - pahae 2y ago
      Quadlets are pretty nice but require podman > 4.4 to function properly. Debian 12, for example, still only has podman ~4.3 in its repos.
    - smarx007 2y ago
      Is there a tool/tutorial that assumes that I already have a running docker compose setup instead of starting with some toy examples? Basically, I am totally excited about using systemd that I already have on my system instead of adding a new daemon/orchestrator but I feel that the gap between quadlet 101 and migrating quite a complex docker compose YAML to podman/quadlet is quite large.
      - somebehemoth 2y ago
        There was not such a tool when I learned how to do this. Quadlet is relatively new (podman 5) so lots of podman/systemd documentation refers to podman commands that generate systemd unit files. I agree there is a gap.
aaomidi 2y ago
Tbh I prefer not exposing any ports directly, and then throwing Tailscale on the network used by docker. This automatically protects everything behind a private network too.
- smarx007 2y ago
  I would love to read a write-up on that! Are you doing something like https://tailscale.com/blog/docker-tailscale-guide https://tailscale.com/blog/docker-tailscale-guide ?
  - aaomidi 2y ago
    Yep! That's very similar to what I do. I have a tailscale container, and a traefik container. Then I use labels with all my other containers to expose themselves on Traefik.
- yx827ha 2y ago
  I agree, Tailscale FTW! You didn't even need to integrate it with docker. Just add a subnet route and evening just works. It's a great product.
- Manouchehri 2y ago
  Another option is using Cloudflare Tunnels (`cloudflared`), and stacking Cloudflare Access on top (for non-public services) to enforce authentication.
  - aaomidi 2y ago
    just fyi cloudflare closes any idle connection thats been around longer than 10 seconds.
  - sieabahlpark 2y ago
    [dead]
- diggan 2y ago
  FOSS alternative is to throw up a $5 VPS on some trusted host, then use Wireguard (FOSS FTW) to do basically exactly the same, but cheaper, without giving away control and with better privacy. There is bunch of software that makes this easier than trivial too, one example: https://github.com/g1ibby/auto-vpn/ https://github.com/g1ibby/auto-vpn/
geye1234 2y ago
I am not a security person at all. Are you really saying that it could potentially cause Iptables to open ports without an admin's knowing? Is that shockingly, mind-bogglingly bad design on Docker's part, or is it just me? Worse, the linked bug report is from a DECADE ago, and the comments underneath don't seem to show any sense of urgency or concern about how bad this is. Have I missed something? This seems appalling.
- smarx007 2y ago
  Your understanding is correct, unfortunately. Not only that, the developers are also reluctant to make 127.0.0.1:####:#### the default in their READMEs and docker-compose.yml files because UsEr cOnVeNiEnCe, e.g. https://github.com/louislam/uptime-kuma/pull/3002 https://github.com/louislam/uptime-kuma/pull/3002 closed WONTFIX
  - geye1234 2y ago
    Amazing. I just don't know what to say, except that anyone who doesn't know how to open a firewall port has no business running Docker, or trying to understand containerization. As someone says in that PR, "there are many beginners who are not aware that Docker punches the firewall for them. I know no other software you can install on Ubuntu that does this." Anyone with a modicum of knowledge can install Docker on Ubuntu -- you don't need to know a thing about ufw or iptables, and you may not even know what they are. I wonder how many machines now have ports exposed to the Internet or some random IoT device as a result of this terrible decision?
- pahae 2y ago
  Correct. This can be disabled [0] but you need to work around this then. Usually you can "just" use host-networking and manage iptable rules manually. Not pretty but in that case you at least know what's done to the system and iptables. [0] https://docs.docker.com/engine/network/packet-filtering-firewalls/#prevent-docker-from-manipulating-iptables https://docs.docker.com/engine/network/packet-filtering-fire...
- tomjen3 2y ago
  To run Docker, you need to be an admin or in the Docker group, which warns you that it is equivalent to having sudo rights, AKA be an admin. As for it not being explicitly permitted, no ports are exposed by default. You must provide the docker run command with -p, for each port you want exposed. From their perspective, they're just doing exactly what you told them to do. Personally, I think it should default to giving you an error unless you specified what IPs to listen to, but this is far from a big of an issue as people make it out to be. The biggest issue is that it is a ginormous foot gun for people who don't know Docker.
znpy 2y ago
I avoid most docker problems by running unprivileged containers via rootless podman, on a rocky-linux based host with selinux enabled. At this point docker should be considered legacy technology, podman is the way to go.
- diggan 2y ago
  Would that actually save you in this case? OP had their container exposed to the internet, listening for incoming remote connections. Wouldn't matter in that case if you're running a unprivileged container, podman, rocky-linux or with selinux, since everything is just wide open at that point.
  - smarx007 2y ago
    I think podman does not punch holes in the firewall as opposed to docker. I.e., to expose a container on port 8080 on the WAN in podman, you need to both expose 8080:8080 and use, for example, firewalld to open port 8080. Which I consider a correct behaviour.
    - diggan 2y ago
      Sure, but the issue here wasn't because the default behavior surprised OP. OP needed a service that was accessible from a remote endpoint, so they needed to have some connection open. They just (for some reason) chose to do it over public internet instead of a private network. But regardless of software used, it would have led to the same conclusion, a vulnerable service running on the open internet.
  - dboreham 2y ago
    I think it's more about whether traffic is bound to localhost or a routable interface. Podman has different behavior vs Docker.
    - smarx007 2y ago
      I think exposing 8080:8080 would result in sockets bound to 0.0.0.0:8080 in either Docker or Podman. You still need 127.0.0.1:8080:8080 for the socket binding to be 127.0.0.1:8080 in Podman. The only difference is that Podman would not punch holes in the firewall after binding on 0.0.0.0:8080, thus preventing an unintended exposure given that the firewall is set up to block all incoming connections except on 443, for example. Edit: just confirmed this to be sure. $ podman run --rm -p 8000:80 docker.io/library/nginx:mainline $ podman ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 595f71b33900 docker.io/library/nginx:mainline nginx -g daemon o... 40 seconds ago Up 41 seconds 0.0.0.0:8000->80/tcp youthful_bouman $ ss -tulpn | rg 8000 tcp LISTEN 0 4096 *:8000 *:* users:(("rootlessport",pid=727942,fd=10))
veyh 2y ago
I wonder how many people realize you can use the whole 127.0.0.0/8 address space, not just 127.0.0.1. I usually use a random address in that space for all of a specific project's services that need to be exposed, like 127.1.2.3:3000 for web and 127.1.2.3:5432 for postgres.
- number6 2y ago
  TIL I always thought it was /32
  - deleted 2y ago
    [deleted]
- jerf 2y ago
  Also a great way around code that tries to block you from hitting resources local to the box. Lots of code out there in the world blocking the specific address "127.0.0.1" and maybe if you were lucky "localhost" but will happily connect to 127.6.243.88 since it isn't either of those things. Or the various IPv6 localhosts. Relatedly, a lot of systems in the world either don't block local network addresses, or block an incomplete list, with 172.16.0.0/12 being particularly poorly known.
- 9dev 2y ago
  Also, many people don’t remember that those zeros in between numbers in IPs can be slashed, so pinging 127.1 works fine. This is also the reason why my home network is a 10.0.0.0/24—don’t need the bigger address space, but reaching devices at 10.1 sure is convenient!
  - diggan 2y ago
    I had no idea about this, and been computing for almost 20 years now, thanks! Trying to get ping to ping `0.0.0.0` was interesting $ ping -c 1 "" ping: : Name or service not known $ ping -c 1 "." ping: .: No address associated with hostname $ ping -c 1 "0." ^C $ ping -c 1 ".0" ping: .0: Name or service not known $ ping -c 1 "0" PING 0 (127.0.0.1) 56(84) bytes of data. 64 bytes from 127.0.0.1: icmp_seq=1 ttl=64 time=0.028 ms $ ping -c 1 "0.0" PING 0.0 (127.0.0.1) 56(84) bytes of data. 64 bytes from 127.0.0.1: icmp_seq=1 ttl=64 time=0.026 ms
Timber-6539 2y ago
Not a security issue. Docker explain it very clearly on their documentation. https://docs.docker.com/engine/network/packet-filtering-firewalls/#docker-and-iptables-chains https://docs.docker.com/engine/network/packet-filtering-fire... If you have opened up a port in your network to the public, the correct assumption is to direct outside connections to your application as per your explicit request.
plagiarist 2y ago
The ports thing is what convinced me to transition to Podman. I don't need a container tool doing ports on my behalf. Why am I running containers as a user that needs to access the Docker socket anyway? Also, shoutout to the teams that suggest easy setup running their software in a container by adding the Docker socket into its file system.
adriancr 2y ago
securing is straightforward, too bad it's not by default: https://docs.docker.com/engine/network/packet-filtering-firewalls/ https://docs.docker.com/engine/network/packet-filtering-fire...
- smarx007 2y ago
  Do I understand the bottom two sections correctly? If I am using ufw as a frontend, I need to switch to firewalld instead and modify the 'docker-forwarding' policy to only forward to the 'docker' zone from loopback interfaces? Would be good if the page described how to do it, esp. for users who are migrating from ufw. More confusingly, firewalld has a different feature to address the core problem [1] but the page you linked does not mention 'StrictForwardPorts' and the page I linked does not mention the 'docker-forwarding' policy. [1]: https://firewalld.org/2024/11/strict-forward-ports https://firewalld.org/2024/11/strict-forward-ports
  - adriancr 2y ago
    I'm not sure about ufw/firewalld. Maybe docs aren't clear there either I configured iptables and had no trouble blocking WAN access to docker... In addition to that there's the default host in daemon.json plus specifying bindings to local host directly in compose / manually.
  - jeroenhd 2y ago
    UFW and Docker don't work well together. Both of them call iptables (or nftables) in a way that assumes they're in control of most of the firewall, which means they can conflict or simply not notice each other's rules. For instance, UFW's rules to block all traffic get overriden by Docker's rules, because there is no active block rule (that's just the default, normally) and Docker just added a rule. UFW doesn't know about firewall chains it didn't create (even though it probably should start listing Docker ports at some point, Docker isn't exactly new...) so `ufw list` will show you only the manually configured UFW rules. What happens when you deny access through UFW and permit access through Docker depends entirely on which of the two firewall services was loaded first, and software updates can cause them to reload arbitrarily so you can't exactly script that easily. If you don't trust Docker at all, you should move away from Docker (i.e. to podman) or from UFW (i.e. to firewalld). This can be useful on hosts where multiple people spawn containers, so others won't mess up and introduce risks outside of your control as a sysadmin. If you're in control of the containers that get run, you can prevent container from being publicly reachable by just not binding them to any public ports. For instance, in many web interfaces, I generally just bind containers to localhost (-p 127.0.0.1:8123:80 instead of -p 80) and configure a reverse proxy like Nginx to cache/do permission stuff/terminate TLS/forward requests/etc. Alternatively, binding the port to your computer's internal network address (-p 192.168.1.1:8123:80 instead of -p 80) will make it pretty hard for you to misconfigure your network in such a way that the entire internet can reach that port. Another alternative is to stuff all the Docker containers into a VM without its own firewall. That way, you can use your host firewall to precisely control what ports are open where, and Docker can do its thing on the virtual machine.
josephcsible 2y ago
It only exposes ports if you pass the command-line flag that says to do so. How is that "without asking your permission"?
- dizhn 2y ago
  It should have the proxy set up but leave opening the port to the user. No other sever software that I know of touches the firewall to make its own services accessible. Though I am aware that the word being used is "expose". I personally only have private IPs on my docker hosts when I can and access them with wireguard.
spr-alex 2y ago
-p 127.0.0.1: might not offer all of the protections the way you would expect, and is arguably a bug in dockers firewall rules they're failing to address. they choose to instead say hey we dont protect against L2, and have an open issue here: https://github.com/moby/moby/issues/45610 https://github.com/moby/moby/issues/45610. this secondary issue with docker is a bit more subtle, it's that they don't respect the bind address when they do forwarding into the container. the end result is that machines one hop away can forward packets into the docker container. for a home user the impact could be that the ISP can reach into the container. depending on risk appetite this can be a concern (salt typhoon going after ISPs). more commonly it might end up exposing more isolated work related systems to related networks one hop away
- smarx007 2y ago
  What about cloud VMs? I would love to read more about "they don't respect the bind address when they do forwarding into the container" and "machines one hop away can forward packets into the docker container" if you could be so kind! Upd: thanks for a link, looks quite bad. I am now thinking that an adjacent VM in a provider like Hetzner or Contabo could be able to pull it off. I guess I will have to finally switch remaining Docker installations to Podman and/or resort to https://firewalld.org/2024/11/strict-forward-ports https://firewalld.org/2024/11/strict-forward-ports
  - spr-alex 2y ago
    i cant speak to hetzner, contabo. i have tested this attack on aws, gcp a while back and their L2 segmentation was solid. VMs/containers should be VLANd across customers/projects on most mature providers. On some it may not be though. if theres defense in depth it may be worth checking out L2 forwarding within a project for unexpected pivots an attacker could use. we've seen this come up in pentests I work on SPR, we take special care in our VPN to avoid these problems as well, by not letting docker do the firewalling for us. (one blog post on the issue: https://www.supernetworks.org/pages/blog/docker-networking-containment https://www.supernetworks.org/pages/blog/docker-networking-c...). as an aside there's a closely related issue with one-hop attacks with conntrack as well, that we locked down in October.
disambiguation 2y ago
Interesting trick, it was definitely an "Oh shit" moment when I learned the hard way that ports were being exposed. I think setting an internal docker network is another simple fix for this, though it complicates talking to other machines in the same firewall.
wutwutwat 2y ago
From the linked issue > by running docker images that map the ports to my host machine If you start a docker container and map port 8080 of the container to port 8080 on the host machine, why would you expect port 8080 on the host machine to not be exposed? I don't think you understand what mapping and opening a port does if you think that when you tell docker to expose a port on the host machine that it's a bug or security issue when docker then exposes a port on the host machine... docker supports many network types, vlans, host attached, bridged, private, etc. There are many options available to run your containers on if you don't want to expose ports on the host machine. A good place to start: If you don't want ports exposed on the host machine then probably should not start your docker container up with host networking and a port exposed on that network... Regardless of that, your container host machines should be behind a load balancer w/ firewall and/or a dedicated firewall, so containers poking holes (because you told them to and then got mad at it) shouldn't be an issue
- rpcope1 2y ago
  I think the unintuitive thing is that by "port mapping", Docker is doing DNAT which doesn't trigger the input firewall rules. Unless you're relatively well versed in the behavior of iptables or notables, you probably expect the "port mapping" to work like a regular old application proxy (which would obey a firewall rules blocking all inputs) and not use NAT and firewall rules (and all of the attendant complexity that brings).
sureglymop 2y ago
I use rootless docker and then also do this for my traefik container. Then, a bit of nftables config to expose to the outside world.

dawnerd 2y ago

We’ve found this out a few times when someone inexperienced with docker would expose a redis port and run docker compose up on a public accessible machine. Would only be minutes until that redis would be infected. Also blame redis for having the ability to run arbitrary code without auth by default.

bluedino 2y ago

Containers are widely used at our company, by developers who don't understand underlying concepts, and they often expose services on all interfaces, or to all hosts. You can explain this to them, they don't care, you can even demonstrate how you can access their data without permission, and they don't get it. Their app "works" and that's the end of it. Ironically enough even cybersecurity doesn't catch them for it, they are too busy harassing other teams about out of date versions of services that are either not vulnerable, or already patched but their scanning tools don't understand that.

dijit 2y ago
This is pretty common, developers are focused on making things that work. Sysadmins were always the ones who focused on making things secure, and for a bunch of reasons they basically don’t exist anymore. EDIT: what guidelines did I break?
- ocdtrekkie 2y ago
  I suspect you'll find a lot of intersection between the move to "devops" outfits who "don't need IT anymore" and "there's a lot more security breaches now", but hey, everyone's making money so who cares?
- smarx007 2y ago
  I don't think you broke any (did not downvote). But you wrote something along the lines "Sysadmins were always the ones who focused on making things secure, and for a bunch of reasons they basically don’t exist anymore. I guess this is fine." before you edited the last bit out. I think those who downvoted you think that this is plain wrong. I guess it's fine if you get rid of sysadmins and have dev splitting their focus across dev, QA, sec, and ops. It's also fine if you have devs focus on dev, QA, code part of the sec and sysadmins focus on ops and network part of the sec. Bottom line is - someone needs to focus on sec :) (and on QAing and DBAing)
- bluedino 2y ago
  > This is pretty common, developers are focused on making things that work. True, but over the last twenty years, simple mistakes by developers have caused so many giant security issues. Part of being a developer now is knowing at least the basics on standard security practices. But you still see people ignoring things as simple as SQL injection, mainly because it's easy and they might not even have been taught otherwise. Many of these people can't even read a Python error message so I'm not surprised. And your cybersecurity department likely isn't auditing source code. They are just making sure your software versions are up to date.
  - bt1a 2y ago
    and many of these people havent debugged messages more complex than a Python error message. tastelessly jabbing at needing to earn your marks by slamming into segfaults and pushing gdb
- harrall 2y ago
  Sometimes when you work less rigidly as a team, covering for others when it’s convenient for you, everyone gets more things done with less stress and less trouble. And you go home at 5pm and had a good work day.
malfist 2y ago
Checklist security at it's finest. My team where I work is responsible for sending frivolous newsletters via email and sms to over a million employees. We use an OTP for employees to verify they gave us the right email/phone number to send them to. Security sees "email/sms" and "OTP" and therefor, tickets us at the highest "must respond in 15 minutes" priority ticket every time an employee complains about having lost access to an email or phone number. Doesn't matter that we're not sending anything sensitive. Doesn't matter that we're a team of 4 managing more than a million data points. Every time we push back security either completely ignores us and escalates to higher management, or they send us a policy document about security practices for communication channels that can be used to send OTP codes. Security wields their checklist like a cudgel. Meanwhile, our bug bounty program, someone found a dev had opened a globally accessible instance of the dev employee portal with sensitive information and reported it. Security wasn't auditing for those, since it's not on their checklist.
- dfsegoat 2y ago
  I feel this. Recently implemented a very trivial “otp to sign an electronic document” function in our app. Security heard “otp” and forced us through a 2 month security/architecture review process for this sign-off feature that we built with COTs libraries in a single sprint.
  - malfist 2y ago
    Oh I know that feeling. We got in hot water because the codes were 6 digits long and security decided we needed to make them eight digits. We pushed back and initially they agreed with us and gave us an exception, but about a year later some compliance audit told them it was no longer acceptable and we had to change it ASAP. About a year after that they told us it needed to be ten characters alphanumeric and we did a find and replace in the code base for "verification code" and "otp" and called them verification strings, and security went away.
    - dfsegoat 2y ago
      Heh. We also got treated to the digit thing. That topic alone was about 30 mins of mtg. time with a vp of eng and 2 seniors in the mtg.
calvinmorrison 2y ago
Turns out devsecops was just a layoff scheme for sysadmins
queuebert 2y ago
> Ironically enough even cybersecurity doesn't catch them for it, they are too busy harassing other teams about out of date versions of services that are either not vulnerable, or already patched but their scanning tools don't understand that. Wow, this really hits home. I spend an inordinate amount of time dealing with false positives from cybersecurity.
ropable 2y ago
"Everybody gangsta 'bout infosec until their machine is cryptolockered." (some CISO, probably).
nitwit005 2y ago
There are certainly people that don't care about security out there, but the biggest issue is just how much people are expected to know. Docker, AWS, Kubernetes, some wrapper they've put around Kubernetes, a bunch of monitoring tools, etc. And none of it will be their main job, so they'll just try to get something working by copying a working example, or reading a tutorial.

globular-toast 2y ago

This is only an issue if you run Docker on your firewall, which you absolutely should not.

smarx007 2y ago
Ideally, yes. But in reality, this means that if you just want to have 1 little EC2 VM on AWS running Docker, you now need to create a VM, a VPC, an NLB/ALB in front of the VPC ($20/mo+, right?) and assign a public IP address to that LB instead. For a VM like t4g.nano, it could mean going from a $3/mo bill to $23/mo ($35 in case of a NAT gateway instead of an LB?) bill, not to mention the hassle of all that setup. Hetzner, on the other hand, has a free firewall included.
- Fnoord 2y ago
  There's no good reason a VM or container on Hetzner cannot use a firewall like IPTables. If that makes the service too expensive you increase cost or otherwise lower resources. A firewall is a very simple, essential part of network security. Every simple IoT device running Linux can run IPTables, too.
  - akerl_ 2y ago
    Docker by default modifies iptables rules to allow traffic when you use the options to launch a container with port options. If you have your own firewall rules, docker just writes its own around them.
    - Fnoord 2y ago
      I always have to define 'external: true' at the network. Which I don't do with databases. I link it to an internal network, shared with application. You can do the same with your web application, thereby only needing auth on reverse proxy. Then you use whitelisting on that port, or you use a VPN. But I also always use a firewall where OCI daemon does not have root access on.

anthropodie 2y ago

And this was one of the reason why I switched to Podman. I haven't looked back since.

MortyWaves 2y ago
I want to use Podman but I keep reading the team feels podman-compose to be some crappy workaround they don’t really want to keep. This is daunting because: Take 50 random popular open source self-hostable solutions and the instructions are invariably: normal bare installation or docker compose. So what’s the ideal setup when using podman? Use compose anyway and hope it won’t be deprecated, or use SystemD as Podman suggests as a replacement for Compose?
- somebehemoth 2y ago
  podman rootless running services with quadlet is not a bad start.
  - pahae 2y ago
    Quadlets are pretty nice but require podman > 4.4 to function properly. Debian 12, for example, still only has podman ~4.3 in its repos.
  - smarx007 2y ago
    Is there a tool/tutorial that assumes that I already have a running docker compose setup instead of starting with some toy examples? Basically, I am totally excited about using systemd that I already have on my system instead of adding a new daemon/orchestrator but I feel that the gap between quadlet 101 and migrating quite a complex docker compose YAML to podman/quadlet is quite large.
    - somebehemoth 2y ago
      There was not such a tool when I learned how to do this. Quadlet is relatively new (podman 5) so lots of podman/systemd documentation refers to podman commands that generate systemd unit files. I agree there is a gap.

aaomidi 2y ago

Tbh I prefer not exposing any ports directly, and then throwing Tailscale on the network used by docker. This automatically protects everything behind a private network too.

smarx007 2y ago
I would love to read a write-up on that! Are you doing something like https://tailscale.com/blog/docker-tailscale-guide https://tailscale.com/blog/docker-tailscale-guide ?
- aaomidi 2y ago
  Yep! That's very similar to what I do. I have a tailscale container, and a traefik container. Then I use labels with all my other containers to expose themselves on Traefik.
yx827ha 2y ago
I agree, Tailscale FTW! You didn't even need to integrate it with docker. Just add a subnet route and evening just works. It's a great product.
Manouchehri 2y ago
Another option is using Cloudflare Tunnels (`cloudflared`), and stacking Cloudflare Access on top (for non-public services) to enforce authentication.
- aaomidi 2y ago
  just fyi cloudflare closes any idle connection thats been around longer than 10 seconds.
- sieabahlpark 2y ago
  [dead]
diggan 2y ago
FOSS alternative is to throw up a $5 VPS on some trusted host, then use Wireguard (FOSS FTW) to do basically exactly the same, but cheaper, without giving away control and with better privacy. There is bunch of software that makes this easier than trivial too, one example: https://github.com/g1ibby/auto-vpn/ https://github.com/g1ibby/auto-vpn/

geye1234 2y ago

I am not a security person at all. Are you really saying that it could potentially cause Iptables to open ports without an admin's knowing? Is that shockingly, mind-bogglingly bad design on Docker's part, or is it just me? Worse, the linked bug report is from a DECADE ago, and the comments underneath don't seem to show any sense of urgency or concern about how bad this is. Have I missed something? This seems appalling.

smarx007 2y ago
Your understanding is correct, unfortunately. Not only that, the developers are also reluctant to make 127.0.0.1:####:#### the default in their READMEs and docker-compose.yml files because UsEr cOnVeNiEnCe, e.g. https://github.com/louislam/uptime-kuma/pull/3002 https://github.com/louislam/uptime-kuma/pull/3002 closed WONTFIX
- geye1234 2y ago
  Amazing. I just don't know what to say, except that anyone who doesn't know how to open a firewall port has no business running Docker, or trying to understand containerization. As someone says in that PR, "there are many beginners who are not aware that Docker punches the firewall for them. I know no other software you can install on Ubuntu that does this." Anyone with a modicum of knowledge can install Docker on Ubuntu -- you don't need to know a thing about ufw or iptables, and you may not even know what they are. I wonder how many machines now have ports exposed to the Internet or some random IoT device as a result of this terrible decision?
pahae 2y ago
Correct. This can be disabled [0] but you need to work around this then. Usually you can "just" use host-networking and manage iptable rules manually. Not pretty but in that case you at least know what's done to the system and iptables. [0] https://docs.docker.com/engine/network/packet-filtering-firewalls/#prevent-docker-from-manipulating-iptables https://docs.docker.com/engine/network/packet-filtering-fire...
tomjen3 2y ago
To run Docker, you need to be an admin or in the Docker group, which warns you that it is equivalent to having sudo rights, AKA be an admin. As for it not being explicitly permitted, no ports are exposed by default. You must provide the docker run command with -p, for each port you want exposed. From their perspective, they're just doing exactly what you told them to do. Personally, I think it should default to giving you an error unless you specified what IPs to listen to, but this is far from a big of an issue as people make it out to be. The biggest issue is that it is a ginormous foot gun for people who don't know Docker.

znpy 2y ago

I avoid most docker problems by running unprivileged containers via rootless podman, on a rocky-linux based host with selinux enabled. At this point docker should be considered legacy technology, podman is the way to go.

diggan 2y ago
Would that actually save you in this case? OP had their container exposed to the internet, listening for incoming remote connections. Wouldn't matter in that case if you're running a unprivileged container, podman, rocky-linux or with selinux, since everything is just wide open at that point.
- smarx007 2y ago
  I think podman does not punch holes in the firewall as opposed to docker. I.e., to expose a container on port 8080 on the WAN in podman, you need to both expose 8080:8080 and use, for example, firewalld to open port 8080. Which I consider a correct behaviour.
  - diggan 2y ago
    Sure, but the issue here wasn't because the default behavior surprised OP. OP needed a service that was accessible from a remote endpoint, so they needed to have some connection open. They just (for some reason) chose to do it over public internet instead of a private network. But regardless of software used, it would have led to the same conclusion, a vulnerable service running on the open internet.
- dboreham 2y ago
  I think it's more about whether traffic is bound to localhost or a routable interface. Podman has different behavior vs Docker.
  - smarx007 2y ago
    I think exposing 8080:8080 would result in sockets bound to 0.0.0.0:8080 in either Docker or Podman. You still need 127.0.0.1:8080:8080 for the socket binding to be 127.0.0.1:8080 in Podman. The only difference is that Podman would not punch holes in the firewall after binding on 0.0.0.0:8080, thus preventing an unintended exposure given that the firewall is set up to block all incoming connections except on 443, for example. Edit: just confirmed this to be sure. $ podman run --rm -p 8000:80 docker.io/library/nginx:mainline $ podman ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 595f71b33900 docker.io/library/nginx:mainline nginx -g daemon o... 40 seconds ago Up 41 seconds 0.0.0.0:8000->80/tcp youthful_bouman $ ss -tulpn | rg 8000 tcp LISTEN 0 4096 *:8000 *:* users:(("rootlessport",pid=727942,fd=10))

veyh 2y ago

I wonder how many people realize you can use the whole 127.0.0.0/8 address space, not just 127.0.0.1. I usually use a random address in that space for all of a specific project's services that need to be exposed, like 127.1.2.3:3000 for web and 127.1.2.3:5432 for postgres.

number6 2y ago
TIL I always thought it was /32
- deleted 2y ago
  [deleted]
jerf 2y ago
Also a great way around code that tries to block you from hitting resources local to the box. Lots of code out there in the world blocking the specific address "127.0.0.1" and maybe if you were lucky "localhost" but will happily connect to 127.6.243.88 since it isn't either of those things. Or the various IPv6 localhosts. Relatedly, a lot of systems in the world either don't block local network addresses, or block an incomplete list, with 172.16.0.0/12 being particularly poorly known.
9dev 2y ago
Also, many people don’t remember that those zeros in between numbers in IPs can be slashed, so pinging 127.1 works fine. This is also the reason why my home network is a 10.0.0.0/24—don’t need the bigger address space, but reaching devices at 10.1 sure is convenient!
- diggan 2y ago
  I had no idea about this, and been computing for almost 20 years now, thanks! Trying to get ping to ping `0.0.0.0` was interesting $ ping -c 1 "" ping: : Name or service not known $ ping -c 1 "." ping: .: No address associated with hostname $ ping -c 1 "0." ^C $ ping -c 1 ".0" ping: .0: Name or service not known $ ping -c 1 "0" PING 0 (127.0.0.1) 56(84) bytes of data. 64 bytes from 127.0.0.1: icmp_seq=1 ttl=64 time=0.028 ms $ ping -c 1 "0.0" PING 0.0 (127.0.0.1) 56(84) bytes of data. 64 bytes from 127.0.0.1: icmp_seq=1 ttl=64 time=0.026 ms

Timber-6539 2y ago

Not a security issue. Docker explain it very clearly on their documentation. https://docs.docker.com/engine/network/packet-filtering-firewalls/#docker-and-iptables-chains https://docs.docker.com/engine/network/packet-filtering-fire... If you have opened up a port in your network to the public, the correct assumption is to direct outside connections to your application as per your explicit request.

plagiarist 2y ago

The ports thing is what convinced me to transition to Podman. I don't need a container tool doing ports on my behalf. Why am I running containers as a user that needs to access the Docker socket anyway? Also, shoutout to the teams that suggest easy setup running their software in a container by adding the Docker socket into its file system.

adriancr 2y ago

securing is straightforward, too bad it's not by default: https://docs.docker.com/engine/network/packet-filtering-firewalls/ https://docs.docker.com/engine/network/packet-filtering-fire...

smarx007 2y ago
Do I understand the bottom two sections correctly? If I am using ufw as a frontend, I need to switch to firewalld instead and modify the 'docker-forwarding' policy to only forward to the 'docker' zone from loopback interfaces? Would be good if the page described how to do it, esp. for users who are migrating from ufw. More confusingly, firewalld has a different feature to address the core problem [1] but the page you linked does not mention 'StrictForwardPorts' and the page I linked does not mention the 'docker-forwarding' policy. [1]: https://firewalld.org/2024/11/strict-forward-ports https://firewalld.org/2024/11/strict-forward-ports
- adriancr 2y ago
  I'm not sure about ufw/firewalld. Maybe docs aren't clear there either I configured iptables and had no trouble blocking WAN access to docker... In addition to that there's the default host in daemon.json plus specifying bindings to local host directly in compose / manually.
- jeroenhd 2y ago
  UFW and Docker don't work well together. Both of them call iptables (or nftables) in a way that assumes they're in control of most of the firewall, which means they can conflict or simply not notice each other's rules. For instance, UFW's rules to block all traffic get overriden by Docker's rules, because there is no active block rule (that's just the default, normally) and Docker just added a rule. UFW doesn't know about firewall chains it didn't create (even though it probably should start listing Docker ports at some point, Docker isn't exactly new...) so `ufw list` will show you only the manually configured UFW rules. What happens when you deny access through UFW and permit access through Docker depends entirely on which of the two firewall services was loaded first, and software updates can cause them to reload arbitrarily so you can't exactly script that easily. If you don't trust Docker at all, you should move away from Docker (i.e. to podman) or from UFW (i.e. to firewalld). This can be useful on hosts where multiple people spawn containers, so others won't mess up and introduce risks outside of your control as a sysadmin. If you're in control of the containers that get run, you can prevent container from being publicly reachable by just not binding them to any public ports. For instance, in many web interfaces, I generally just bind containers to localhost (-p 127.0.0.1:8123:80 instead of -p 80) and configure a reverse proxy like Nginx to cache/do permission stuff/terminate TLS/forward requests/etc. Alternatively, binding the port to your computer's internal network address (-p 192.168.1.1:8123:80 instead of -p 80) will make it pretty hard for you to misconfigure your network in such a way that the entire internet can reach that port. Another alternative is to stuff all the Docker containers into a VM without its own firewall. That way, you can use your host firewall to precisely control what ports are open where, and Docker can do its thing on the virtual machine.

josephcsible 2y ago

It only exposes ports if you pass the command-line flag that says to do so. How is that "without asking your permission"?

dizhn 2y ago
It should have the proxy set up but leave opening the port to the user. No other sever software that I know of touches the firewall to make its own services accessible. Though I am aware that the word being used is "expose". I personally only have private IPs on my docker hosts when I can and access them with wireguard.

spr-alex 2y ago

-p 127.0.0.1: might not offer all of the protections the way you would expect, and is arguably a bug in dockers firewall rules they're failing to address. they choose to instead say hey we dont protect against L2, and have an open issue here: https://github.com/moby/moby/issues/45610 https://github.com/moby/moby/issues/45610. this secondary issue with docker is a bit more subtle, it's that they don't respect the bind address when they do forwarding into the container. the end result is that machines one hop away can forward packets into the docker container. for a home user the impact could be that the ISP can reach into the container. depending on risk appetite this can be a concern (salt typhoon going after ISPs). more commonly it might end up exposing more isolated work related systems to related networks one hop away

smarx007 2y ago
What about cloud VMs? I would love to read more about "they don't respect the bind address when they do forwarding into the container" and "machines one hop away can forward packets into the docker container" if you could be so kind! Upd: thanks for a link, looks quite bad. I am now thinking that an adjacent VM in a provider like Hetzner or Contabo could be able to pull it off. I guess I will have to finally switch remaining Docker installations to Podman and/or resort to https://firewalld.org/2024/11/strict-forward-ports https://firewalld.org/2024/11/strict-forward-ports
- spr-alex 2y ago
  i cant speak to hetzner, contabo. i have tested this attack on aws, gcp a while back and their L2 segmentation was solid. VMs/containers should be VLANd across customers/projects on most mature providers. On some it may not be though. if theres defense in depth it may be worth checking out L2 forwarding within a project for unexpected pivots an attacker could use. we've seen this come up in pentests I work on SPR, we take special care in our VPN to avoid these problems as well, by not letting docker do the firewalling for us. (one blog post on the issue: https://www.supernetworks.org/pages/blog/docker-networking-containment https://www.supernetworks.org/pages/blog/docker-networking-c...). as an aside there's a closely related issue with one-hop attacks with conntrack as well, that we locked down in October.

disambiguation 2y ago

Interesting trick, it was definitely an "Oh shit" moment when I learned the hard way that ports were being exposed. I think setting an internal docker network is another simple fix for this, though it complicates talking to other machines in the same firewall.

wutwutwat 2y ago

From the linked issue > by running docker images that map the ports to my host machine If you start a docker container and map port 8080 of the container to port 8080 on the host machine, why would you expect port 8080 on the host machine to not be exposed? I don't think you understand what mapping and opening a port does if you think that when you tell docker to expose a port on the host machine that it's a bug or security issue when docker then exposes a port on the host machine... docker supports many network types, vlans, host attached, bridged, private, etc. There are many options available to run your containers on if you don't want to expose ports on the host machine. A good place to start: If you don't want ports exposed on the host machine then probably should not start your docker container up with host networking and a port exposed on that network... Regardless of that, your container host machines should be behind a load balancer w/ firewall and/or a dedicated firewall, so containers poking holes (because you told them to and then got mad at it) shouldn't be an issue

rpcope1 2y ago
I think the unintuitive thing is that by "port mapping", Docker is doing DNAT which doesn't trigger the input firewall rules. Unless you're relatively well versed in the behavior of iptables or notables, you probably expect the "port mapping" to work like a regular old application proxy (which would obey a firewall rules blocking all inputs) and not use NAT and firewall rules (and all of the attendant complexity that brings).

sureglymop 2y ago

I use rootless docker and then also do this for my traefik container. Then, a bit of nftables config to expose to the outside world.