6 ms·

> automatically add a rule to open the firewall on linux There's no way this is true. This completely defeats the purpose of a firewall. If it is happening, t

by junon 6y ago

> automatically add a rule to open the firewall on linux There's no way this is true. This completely defeats the purpose of a firewall. If it is happening, then it's Docker doing this - not the linux firewall that comes stock with most distribution (iptables and the like). They would never simply add rules to the chains just because something was listening on that port. Really, the best security advice for using Docker is to not use Docker. Unfortunately, there aren't very many "hold your hand" alternatives available. Aside from LXD and that family of technologies, which are criminally underused.

Nullabillity 6y ago
Docker uses iptables for port forwarding, and those rules typically end up ahead of the rules inserted by your firewall (firewalld/ufw/manual scripts/whatever). It's not so much that they explicitly open a firewall rule, as that they take a networking path that isn't really covered by traditional firewalls. Another way of viewing it is that Docker "is" your firewall for your container workloads, and that adding a port-forward is equivalent to adding a firewall rule. Of course, that doesn't change that public-by-default is a bad default.
- junon 6y ago
  This is right, I remember now - docker does mangle your iptables chains. I remember fighting with this a while back. Terrible practice, in my opinion. Docker shouldn't be touching firewall stuff.
  - krab 6y ago
    Iptables magic is essential to how a lot of container networking stuff is implemented, though.
    - peterwwillis 6y ago
      This is (imho) a huge flaw in the concept of a "container". I don't think most people comprehend how much crap is going on in the background. For most container purposes, host networking and the default process namespace is absolutely fine, and reduces a lot of problems with interacting with containerized apps. 95% of the use case of containers is effectively just a chroot wrapper. If you need more features, this should be optional. This would also make rootless federated containerized apps just work. But nobody wants to go back to incremental features if Docker gives them everything at once.
      - thr0w3345 6y ago
        If you think that’s bad, wait til you see what the iptables-save output is like on an istio-proxy sidecar ;)
        junon 6y ago
        Kubernetes as well. We ran into instances where iptables contention was so bad during outage recovery that things just stalled. iptables-save looked like a bomb went off.
  - unilynx 6y ago
    I've resorted to adding my own firewall rules to the 'raw' table, which pretty much preempts all the rules Docker or the distribution inserts. It's not as powerful as the later tables in the chain (see https://upload.wikimedia.org/wikipedia/commons/3/37/Netfilter-packet-flow.svg https://upload.wikimedia.org/wikipedia/commons/3/37/Netfilte... ) but a lot more robust.
- l3s2d 6y ago
  This has been a major pain point for me. Despite my `firewalld` configuration only allowing specific traffic, all my containers were exposed. My current policy is to set `"iptables": false` in Docker's `daemon.json` on any public machine. I don't understand why this isn't the default.
  - Nullabillity 6y ago
    > My current policy is to set `"iptables": false` in Docker's `daemon.json` on any public machine. I don't understand why this isn't the default. If you don't muck with iptables then you need a (slow) userspace proxy to expose your pods. That also means losing things like the source IP address for any incoming connections.
    - l3s2d 6y ago
      Interesting, I haven't noticed any slowdown, but I am running fairly low traffic services. I do see that RemoteAddr is from a private IP range. Luckily I'm not using this information anywhere, but good to know.
acdha 6y ago
If this is a surprise to you, scan your servers and see what else has helpful behavior which you didn’t expect. Services like shodan.io will even do this regularly and email you when things change. It’s easy to blame Docker but I’ve seen this failure mode happen many times over the years - even experienced admins make mistakes. As always, defense in depth and continuous monitoring trumps raging about software.
- junon 6y ago
  I don't disagree with any of your points, but they aren't relevant to anything I've said. I never said Docker is the only one doing this. It's also not a surprise Docker in particular is doing this - Docker has a long history of doing bad things.
  - acdha 6y ago
    It is relevant because you said “there's no way this is true” when it is in fact true, which means that your understanding of how the system works doesn't match the actual behaviour. I mentioned the importance of scanning to catch those situations quickly.
- cpuguy83 6y ago
  I'm a Docker maintainer and have been for years. The default is terrible. It got into 1.0 and just got stuck there :( Changing defaults on such widely used software is, unfortunately, hard. For whatever reason Docker prepends rules. There are things you can do to add your own filtering (Docker forwards to the "DOCKER-USER" chain where you can put your rules), but it requires people to know what is happening I order to use it securely. It would be really nice to be in a more secure situation by default... open to suggestions and contributions for Docker 21.
  - Godel_unicode 6y ago
    Have you considered at least logging to the terminal on container start whenever a port gets exposed to the internet because of a 3306:3306 (i.e. without an explicit ip to bind to)? Part of the issue seems to be that people haven't read the docs and so don't really understand what that snippet they copied from that helpful blog, you know, does
kspacewalk2 6y ago
>Aside from LXD and that family of technologies, which are criminally underused. Criminally underused indeed. I have no idea why it's not more popular for 'average' users/orgs. I don't know what issues may come up with scaling this up, but in our small org we've been running 20-30 (mostly unprivileged) LXD containers in production for several years now for all sorts of intranet and external-facing services (auth, DB, web, etc). Sure, it requires a bit more thought to set up than Docker, but it's well-documented (for most people's uses at least), secure, stable and lightweight.
- scns 6y ago
  >I have no idea why it's not more popular for 'average' users/orgs. Maybe because many devs use Macs / Windows? Maybe WSL may tilt the balance in LXDs favour, but on OSX? Run it a VM yourself without the conveniences of docker-compose up? As a Linuxuser myself, i looked at the competing solutions (podman, LXD, and the one by Canonical whose name i forgot) and thought: "Ain't gonna fly in a mixed environment." They may be technologically superior, worse is better once again i guess. Would prefer them to Docker too.
  - krimeo 6y ago
    Isn't docker running on OS X in VM anyway?
    - bruce_one 6y ago
      Yes it does, but it's very transparent - until you run into a few things that make it obvious, eg if you disk-image size limits, or memory limits. I work with a few people who believed it was native, and used it for quite a while, until things started going wrong and we logged in to the vm to fix them.
      - krimeo 6y ago
        Kinda my point. The reason docker is popular is simply that most of the devs are lazy, not very knowledgeable in the actual underlying technology and very short-sighted. And docker marketing has very effectively used that to their advantage.
hxtk 6y ago
It has to do with the implementation of the networking. The DOCKER chain in the nat table gets hit by inbound traffic on the PREROUTING chain before UFW's chains on the filter table. IIRC, can get around this by directing UFW to write its rules to the DOCKER-USER chain. Firewalld is implemented differently and will exhibit the expected blocking behavior: traffic targeting docker ports will encounter firewalld before it encounters docker.
deleted 6y ago
[deleted]

by junon 6y ago

Nullabillity 6y ago

Docker uses iptables for port forwarding, and those rules typically end up ahead of the rules inserted by your firewall (firewalld/ufw/manual scripts/whatever). It's not so much that they explicitly open a firewall rule, as that they take a networking path that isn't really covered by traditional firewalls. Another way of viewing it is that Docker "is" your firewall for your container workloads, and that adding a port-forward is equivalent to adding a firewall rule. Of course, that doesn't change that public-by-default is a bad default.

junon 6y ago
This is right, I remember now - docker does mangle your iptables chains. I remember fighting with this a while back. Terrible practice, in my opinion. Docker shouldn't be touching firewall stuff.
- krab 6y ago
  Iptables magic is essential to how a lot of container networking stuff is implemented, though.
  - peterwwillis 6y ago
    This is (imho) a huge flaw in the concept of a "container". I don't think most people comprehend how much crap is going on in the background. For most container purposes, host networking and the default process namespace is absolutely fine, and reduces a lot of problems with interacting with containerized apps. 95% of the use case of containers is effectively just a chroot wrapper. If you need more features, this should be optional. This would also make rootless federated containerized apps just work. But nobody wants to go back to incremental features if Docker gives them everything at once.
    - thr0w3345 6y ago
      If you think that’s bad, wait til you see what the iptables-save output is like on an istio-proxy sidecar ;)
      - junon 6y ago
        Kubernetes as well. We ran into instances where iptables contention was so bad during outage recovery that things just stalled. iptables-save looked like a bomb went off.
- unilynx 6y ago
  I've resorted to adding my own firewall rules to the 'raw' table, which pretty much preempts all the rules Docker or the distribution inserts. It's not as powerful as the later tables in the chain (see https://upload.wikimedia.org/wikipedia/commons/3/37/Netfilter-packet-flow.svg https://upload.wikimedia.org/wikipedia/commons/3/37/Netfilte... ) but a lot more robust.
l3s2d 6y ago
This has been a major pain point for me. Despite my `firewalld` configuration only allowing specific traffic, all my containers were exposed. My current policy is to set `"iptables": false` in Docker's `daemon.json` on any public machine. I don't understand why this isn't the default.
- Nullabillity 6y ago
  > My current policy is to set `"iptables": false` in Docker's `daemon.json` on any public machine. I don't understand why this isn't the default. If you don't muck with iptables then you need a (slow) userspace proxy to expose your pods. That also means losing things like the source IP address for any incoming connections.
  - l3s2d 6y ago
    Interesting, I haven't noticed any slowdown, but I am running fairly low traffic services. I do see that RemoteAddr is from a private IP range. Luckily I'm not using this information anywhere, but good to know.

acdha 6y ago

If this is a surprise to you, scan your servers and see what else has helpful behavior which you didn’t expect. Services like shodan.io will even do this regularly and email you when things change. It’s easy to blame Docker but I’ve seen this failure mode happen many times over the years - even experienced admins make mistakes. As always, defense in depth and continuous monitoring trumps raging about software.

junon 6y ago
I don't disagree with any of your points, but they aren't relevant to anything I've said. I never said Docker is the only one doing this. It's also not a surprise Docker in particular is doing this - Docker has a long history of doing bad things.
- acdha 6y ago
  It is relevant because you said “there's no way this is true” when it is in fact true, which means that your understanding of how the system works doesn't match the actual behaviour. I mentioned the importance of scanning to catch those situations quickly.
cpuguy83 6y ago
I'm a Docker maintainer and have been for years. The default is terrible. It got into 1.0 and just got stuck there :( Changing defaults on such widely used software is, unfortunately, hard. For whatever reason Docker prepends rules. There are things you can do to add your own filtering (Docker forwards to the "DOCKER-USER" chain where you can put your rules), but it requires people to know what is happening I order to use it securely. It would be really nice to be in a more secure situation by default... open to suggestions and contributions for Docker 21.
- Godel_unicode 6y ago
  Have you considered at least logging to the terminal on container start whenever a port gets exposed to the internet because of a 3306:3306 (i.e. without an explicit ip to bind to)? Part of the issue seems to be that people haven't read the docs and so don't really understand what that snippet they copied from that helpful blog, you know, does

kspacewalk2 6y ago

>Aside from LXD and that family of technologies, which are criminally underused. Criminally underused indeed. I have no idea why it's not more popular for 'average' users/orgs. I don't know what issues may come up with scaling this up, but in our small org we've been running 20-30 (mostly unprivileged) LXD containers in production for several years now for all sorts of intranet and external-facing services (auth, DB, web, etc). Sure, it requires a bit more thought to set up than Docker, but it's well-documented (for most people's uses at least), secure, stable and lightweight.

scns 6y ago
>I have no idea why it's not more popular for 'average' users/orgs. Maybe because many devs use Macs / Windows? Maybe WSL may tilt the balance in LXDs favour, but on OSX? Run it a VM yourself without the conveniences of docker-compose up? As a Linuxuser myself, i looked at the competing solutions (podman, LXD, and the one by Canonical whose name i forgot) and thought: "Ain't gonna fly in a mixed environment." They may be technologically superior, worse is better once again i guess. Would prefer them to Docker too.
- krimeo 6y ago
  Isn't docker running on OS X in VM anyway?
  - bruce_one 6y ago
    Yes it does, but it's very transparent - until you run into a few things that make it obvious, eg if you disk-image size limits, or memory limits. I work with a few people who believed it was native, and used it for quite a while, until things started going wrong and we logged in to the vm to fix them.
    - krimeo 6y ago
      Kinda my point. The reason docker is popular is simply that most of the devs are lazy, not very knowledgeable in the actual underlying technology and very short-sighted. And docker marketing has very effectively used that to their advantage.

hxtk 6y ago

It has to do with the implementation of the networking. The DOCKER chain in the nat table gets hit by inbound traffic on the PREROUTING chain before UFW's chains on the filter table. IIRC, can get around this by directing UFW to write its rules to the DOCKER-USER chain. Firewalld is implemented differently and will exhibit the expected blocking behavior: traffic targeting docker ports will encounter firewalld before it encounters docker.

deleted 6y ago

[deleted]