9 ms·

Wait until they learn about Docker ignoring iptable rules. https://www.baeldung.com/linux/docker-container-published-port-ignoring-ufw-rules https://www.baeldu

Wait until they learn about Docker ignoring iptable rules. https://www.baeldung.com/linux/docker-container-published-port-ignoring-ufw-rules https://www.baeldung.com/linux/docker-container-published-po...

Muromec 3y ago
But docker doesn't ignore iptables, it adds iptables rule to forward packets to docker table in iptables.
- yjftsjthsd-h 3y ago
  Okay, docker doesn't ignore firewall rules, it bypasses them. Still annoying.
  - masom 3y ago
    It does not bypasses them, it uses the firewall to add forwarding rules to the container. If you read https://www.baeldung.com/linux/docker-container-published-port-ignoring-ufw-rules https://www.baeldung.com/linux/docker-container-published-po... there's a bunch of ways to prevent this from happening. By enabling port forwarding and exposing, you're essentially asking docker to configure the host for it. It's likely a surprising behaviour, but seems like ufw has a bug if you have multiple chains and groups instead of just the default one.
    - 3abiton 3y ago
      For the average users, it goes over the head
      - deleted 3y ago
        [deleted]
    - yjftsjthsd-h 3y ago
      > It does not bypasses them, it uses the firewall to add forwarding rules to the container. ... yes, it adds a forwarding rule. Which skips over the rest of my firewall rules. One might even say that it bypasses them. > By enabling port forwarding and exposing, you're essentially asking docker to configure the host for it. No, I'm asking docker to listen on that port and pass it through to the container. If I tell nginx to listen on port 80, would you expect it to take that as a "do anything you can to make sure you get port 80, including rewriting the firewall tables to ignore the rule that blocks port 80"?
      - tinco 3y ago
        GP is being annoying and I agree with you for the most part and I feel Docker should have tried to play nicer with the firewall. That said it is a mistake to consider Docker in the same class as nginx. Docker is a system for telling the Linux kernel how to set up the environments of processes. Doing whatever it can at the absolute lowest level of abstraction the kernel offers is kind of its entire gig.
        yjftsjthsd-h 3y ago
        That's a fair argument - I disagree, but it's a valid perspective and probably the root of the difference in views. I personally expect docker to be a piece that fits into the system to run containers. If docker expects to effectively be the system at least for things in its domain, then that would explain why it does things that I consider out of its scope. (There's an echo here of the argument about systemd; if you expect systemd to be everything between the kernel and userspace, then of course it includes ex. its own ntpd and cron replacement, whereas if you expect it to be a service manager then every extra function looks like an overreach)
        masom 3y ago
        The reason docker is integrating this way is to easily group the rules it introduced. > I feel Docker should have tried to play nicer with the firewall. The problem is on the UFW side not setting up properly when other chains are present. UFW is just a front-end for iptables, and docker integrates with iptables. Can the situation be made better? Likely on the UFW side. https://docs.docker.com/network/packet-filtering-firewalls/#add-iptables-policies-before-dockers-rules https://docs.docker.com/network/packet-filtering-firewalls/#... The docker behaviour is documented... UFW seems buggy in that it only operates on set of prefixed groups instead of looking at all the groups in iptables. Now, looking at the UFW code: https://git.launchpad.net/ufw/tree/src/backend_iptables.py?h=ubuntu/master https://git.launchpad.net/ufw/tree/src/backend_iptables.py?h... it seems to setup it's own chains and ignores everything else. It even filters out all other chains unless they're part of UFW.
    - edvinbesic 3y ago
      Upvoted because you are correct. Calling it a bypass almost implies something more nefarious vs saying docker automatically adds its own forwarding rules.
      - southerntofu 3y ago
        It's not intended to be nefarious, but it can have bad consequences. I know the docker devs had good intentions to help people expose services, when they introduced this. However, they also "helped" people unwillingly expose services, oops. Just because it was meant to be useful and harmless doesn't mean they didn't actually bypass the firewall rules by adding their own rules.
  - jeroenhd 3y ago
    I don't think that's fair. Linux users misunderstand nftables/iptables and UFW pretends to have complete control when it really doesn't. It's annoying, but the fault lies squarely with the lack of reporting of other chains in UFW.
    - IshKebab 3y ago
      If everyone is misunderstanding how something works it's because it doesn't work the right way. Clearly people want firewall rules that take priority over everything else.
      - jeroenhd 3y ago
        I agree, and tools that try to make something complex (nftables) simple and easy to use (UFW, Firewalld) should make that possible. Different lists are a powerful nftables feature that is used in many system services and applications for things like routing and specialized forwarding. You can't just disable that without breaking tons of software. Instead, tools like UFW should make it clear which ports are or aren't open and what firewall rules actually apply to the current system. They exist to save users from nftables and eBPF programming, and they should work as advertised. Ufw advertises itself with "Ufw stands for Uncomplicated Firewall, and is program for managing a netfilter firewall" not "Ufw stands for Uncomplicated Firewall, which lets you add some firewall rules but doesn't manage all of them so you need to be extra careful if you use it."
    - nextaccountic 3y ago
      The real problem is that iptables rules aren't composable. In this thread alone there are several tools that add iptables rules (kubernetes, docker, ufw) and they are just dumped without regards to each other. This could be solved by more tooling.
      - jeroenhd 3y ago
        I disagree that we need more tooling. We need better tooling. Docker and Kubernetes adding network filters makes total sense, they're useless without forwarding packets between devices. UFW, on the other hand, pretends to make firewalls simple and easy to use when they're really not. I'm not expecting UFW to start managing Docker's/K8s'/libvirtd's firewall rules, I'm just expecting them to add a section of "ports opened by other applications" that show the open ports that UFW can't control.
  - drdaeman 3y ago
    Set `{"iptables": false}` in daemon.json (or pass `--iptables=false` to the daemon) and it won't do it. The defaults are convenient enshittification for your casual security-ignorant developer.
    - hackyhacky 3y ago
      Yes but this breaks the very container routing that we need Docker for. An "all or nothing" approach is not ideal.
      - drdaeman 3y ago
        I'm not sure how much this routing is really needed. I use nftables instead of iptables, so I told Docker to not touch anything. I'm gradually getting rid of Docker (NixOS containers powered by systemd-nspawn are more convenient for me at the moment), so I don't have any fancy networking for it - but I still run it. I have a bunch of static nftables rules for the bridge (`iifname "docker0" ct state established,related counter accept` etc.) so the containers can access what they need. And then Docker uses docker-proxy for the exposed ports (paired with appropriate firewall rules when it's not on `lo`, where I have an umbrella `iif "lo" accept`) and it seem to works just fine.
  - ilyt 3y ago
    Not really. Just ufw wasn't made to deal with it.
- TacticalCoder 3y ago
  Several containers and VMs by default just go a bit "overboard" I'd say, adding network interfaces and firewalling rules left and right. It takes people installing these for the first time by surprise.
- yrro 3y ago
  It would be fairer to say that docker rudely stomps all over your iptables configuration. ;)
jwitthuhn 3y ago
This bit me the first time I deployed using docker. Everything but 80/443 blocked by the firewall. "Surely it is safe to run my app server on port 8000 because no one can access that port externally." Docker stepped in to help by making sure people could access that port.
- justsomehnguy 3y ago
  My favourite part in Docker is what if you have 127.0.0.1 configured as your resolver (eg you are running Unbound) then Docker helpfully configure containers to use 1.1.1.1 instead. Thanks, my ass, and give back my time trying to diagnose [a container with no diagnostic tools, because it's a lightweight container] why it can't connect to someservice.on.my.lan.
  - judge2020 3y ago
    Wouldn't 127.0.0.1 be the loopback for the container itself? So you'd need to set the DNS to the gateway IP or dynamically by resolving `host.docker.internal` (assuming `--add-host=host.docker.internal:host-gateway`)
    - justsomehnguy 3y ago
      Yes. > So you'd need to set the DNS to the gateway IP This is the least friction way. Run your own resolver there to resolve names for containers, forward to the host's loopback for everything else. > or dynamically by resolving `host.docker.internal` Resolving. Name. Of the DNS server. Right?
      - judge2020 3y ago
        I mean dynamically as in a startup script that resolves that hostname, then updates the resolver to the IP it returns.
        justsomehnguy 3y ago
gnfargbl 3y ago
Not just docker, either. If you configure a NodePort on your kubernetes cluster, you'll run into exactly the same issue: ufw-sourced iptables rules are overriden by kubernetes-sourced rules. Here's the only fix I was able to find, which doesn't seem well documented. Run kubectl -n kube-system edit configmap kube-proxy and edit in this: nodePortAddresses: - 192.168.0.0/16 ...where the CIDR block should correspond to the local network on your machine.
alexey-salmin 3y ago
This helps https://github.com/chaifeng/ufw-docker https://github.com/chaifeng/ufw-docker
ilyt 3y ago
It's probably ufw putting its rules in INPUT and OUTPUT chain and docker needing them in FORWARD. Don't blame ufw incompetence on iptables. There is separate issue with bridging aptly described here https://wiki.libvirt.org/Net.bridge.bridge-nf-call_and_sysctl.conf.html https://wiki.libvirt.org/Net.bridge.bridge-nf-call_and_sysct... but IIRC docker doesn't work in that config by default. IIRC Centos/RHEL comes with bridging iptables pre-broken because they're bad at making firewall rules there.
timost 3y ago
I think podman rootless behaves much better than docker regarding firewall "bypassing"
yrro 3y ago
Developers learning how their system actually works? I'll believe it when I see it... ;) More seriously, I believe nftables has the capacity to help here: now there can be several chains which all attach to the same hook. So docker can put rules into the FORWARD base chain of the filter table (belonging to iptables, mine is empty because I don't use iptables): # nft list chain ip filter FORWARD table ip filter { chain FORWARD { type filter hook forward priority filter; policy accept; } } ... but then the filter_FORWARD base chain of the firewalld table (belonging to firewalld) _also_ gets to process packets. Note the numerically higher filter priority, which means that this chain runs later: # nft list chain inet firewalld filter_FORWARD table inet firewalld { chain filter_FORWARD { type filter hook forward priority filter + 10; policy accept; ct state { established, related } accept ct status dnat accept iifname "lo" accept ip6 daddr { ::/96, ::ffff:0.0.0.0/96, 2002::/24, 2002:a00::/24, 2002:7f00::/24, 2002:a9fe::/32, 2002:ac10::/28, 2002:c0a8::/32, 2002:e000::/19 } log prefix "RFC3964_IPv4_REJECT: " reject with icmpv6 type addr-unreachable jump filter_FORWARD_POLICIES_pre jump filter_FORWARD_IN_ZONES jump filter_FORWARD_OUT_ZONES jump filter_FORWARD_POLICIES_post ct state { invalid } meta pkttype host log prefix "STATE_INVALID_DROP: " ct state { invalid } drop meta pkttype host log prefix "FINAL_REJECT: " reject with icmpx type admin-prohibited } } Now if only tools would learn to stay in their lane, cease assuming they are the only tool managing the nftables state, we'd have ufw managing its own table, docker managing its own table, firewalld managing its own...
bandyaboot 3y ago
They can learn about what Docker does with iptables the traditional way, by reading Docker’s networking documentation. https://docs.docker.com/network/packet-filtering-firewalls/ https://docs.docker.com/network/packet-filtering-firewalls/ It says explicitly that docker and ufw should be considered incompatible. Docker also has a configuration key to prevent it from modifying iptables.
- Dylan16807 3y ago
  People only have so much time to read documentation. You can't expect them to read an entire textbook before they start using Docker.
  - pests 3y ago
    And people wonder why we aren't taken seriously like other engineering fields. Imagine a civil engineer telling someone they don't have time to read the documentation on the bolts their using to hold up a bridge.
    - Dylan16807 3y ago
      Imagine a bolt seller saying you have to read this entire book to understand the bolts, instead of having a couple pages containing the important specifications.
      - pests 3y ago
        The book _is already the important information_. This is evident that not reading it casued problems. Then imagine getting upset because the bolts don't work as you thought they did. You go do additional research and find the information right in that book you ignored! Too late, 100,000 bolts have already been used so now we gotta deal with workarounds and other additonal problems. Maybe should have read the book.
        Dylan16807 3y ago
zelphirkalt 3y ago
Isn't that simply because Docker adds its own firewall rules, in order to realize its docker networks?

Muromec 3y ago

But docker doesn't ignore iptables, it adds iptables rule to forward packets to docker table in iptables.

yjftsjthsd-h 3y ago
Okay, docker doesn't ignore firewall rules, it bypasses them. Still annoying.
- masom 3y ago
  It does not bypasses them, it uses the firewall to add forwarding rules to the container. If you read https://www.baeldung.com/linux/docker-container-published-port-ignoring-ufw-rules https://www.baeldung.com/linux/docker-container-published-po... there's a bunch of ways to prevent this from happening. By enabling port forwarding and exposing, you're essentially asking docker to configure the host for it. It's likely a surprising behaviour, but seems like ufw has a bug if you have multiple chains and groups instead of just the default one.
  - 3abiton 3y ago
    For the average users, it goes over the head
    - deleted 3y ago
      [deleted]
  - yjftsjthsd-h 3y ago
    > It does not bypasses them, it uses the firewall to add forwarding rules to the container. ... yes, it adds a forwarding rule. Which skips over the rest of my firewall rules. One might even say that it bypasses them. > By enabling port forwarding and exposing, you're essentially asking docker to configure the host for it. No, I'm asking docker to listen on that port and pass it through to the container. If I tell nginx to listen on port 80, would you expect it to take that as a "do anything you can to make sure you get port 80, including rewriting the firewall tables to ignore the rule that blocks port 80"?
    - tinco 3y ago
      GP is being annoying and I agree with you for the most part and I feel Docker should have tried to play nicer with the firewall. That said it is a mistake to consider Docker in the same class as nginx. Docker is a system for telling the Linux kernel how to set up the environments of processes. Doing whatever it can at the absolute lowest level of abstraction the kernel offers is kind of its entire gig.
      - yjftsjthsd-h 3y ago
        That's a fair argument - I disagree, but it's a valid perspective and probably the root of the difference in views. I personally expect docker to be a piece that fits into the system to run containers. If docker expects to effectively be the system at least for things in its domain, then that would explain why it does things that I consider out of its scope. (There's an echo here of the argument about systemd; if you expect systemd to be everything between the kernel and userspace, then of course it includes ex. its own ntpd and cron replacement, whereas if you expect it to be a service manager then every extra function looks like an overreach)
      - masom 3y ago
        The reason docker is integrating this way is to easily group the rules it introduced. > I feel Docker should have tried to play nicer with the firewall. The problem is on the UFW side not setting up properly when other chains are present. UFW is just a front-end for iptables, and docker integrates with iptables. Can the situation be made better? Likely on the UFW side. https://docs.docker.com/network/packet-filtering-firewalls/#add-iptables-policies-before-dockers-rules https://docs.docker.com/network/packet-filtering-firewalls/#... The docker behaviour is documented... UFW seems buggy in that it only operates on set of prefixed groups instead of looking at all the groups in iptables. Now, looking at the UFW code: https://git.launchpad.net/ufw/tree/src/backend_iptables.py?h=ubuntu/master https://git.launchpad.net/ufw/tree/src/backend_iptables.py?h... it seems to setup it's own chains and ignores everything else. It even filters out all other chains unless they're part of UFW.
  - edvinbesic 3y ago
    Upvoted because you are correct. Calling it a bypass almost implies something more nefarious vs saying docker automatically adds its own forwarding rules.
    - southerntofu 3y ago
      It's not intended to be nefarious, but it can have bad consequences. I know the docker devs had good intentions to help people expose services, when they introduced this. However, they also "helped" people unwillingly expose services, oops. Just because it was meant to be useful and harmless doesn't mean they didn't actually bypass the firewall rules by adding their own rules.
- jeroenhd 3y ago
  I don't think that's fair. Linux users misunderstand nftables/iptables and UFW pretends to have complete control when it really doesn't. It's annoying, but the fault lies squarely with the lack of reporting of other chains in UFW.
  - IshKebab 3y ago
    If everyone is misunderstanding how something works it's because it doesn't work the right way. Clearly people want firewall rules that take priority over everything else.
    - jeroenhd 3y ago
      I agree, and tools that try to make something complex (nftables) simple and easy to use (UFW, Firewalld) should make that possible. Different lists are a powerful nftables feature that is used in many system services and applications for things like routing and specialized forwarding. You can't just disable that without breaking tons of software. Instead, tools like UFW should make it clear which ports are or aren't open and what firewall rules actually apply to the current system. They exist to save users from nftables and eBPF programming, and they should work as advertised. Ufw advertises itself with "Ufw stands for Uncomplicated Firewall, and is program for managing a netfilter firewall" not "Ufw stands for Uncomplicated Firewall, which lets you add some firewall rules but doesn't manage all of them so you need to be extra careful if you use it."
  - nextaccountic 3y ago
    The real problem is that iptables rules aren't composable. In this thread alone there are several tools that add iptables rules (kubernetes, docker, ufw) and they are just dumped without regards to each other. This could be solved by more tooling.
    - jeroenhd 3y ago
      I disagree that we need more tooling. We need better tooling. Docker and Kubernetes adding network filters makes total sense, they're useless without forwarding packets between devices. UFW, on the other hand, pretends to make firewalls simple and easy to use when they're really not. I'm not expecting UFW to start managing Docker's/K8s'/libvirtd's firewall rules, I'm just expecting them to add a section of "ports opened by other applications" that show the open ports that UFW can't control.
- drdaeman 3y ago
  Set `{"iptables": false}` in daemon.json (or pass `--iptables=false` to the daemon) and it won't do it. The defaults are convenient enshittification for your casual security-ignorant developer.
  - hackyhacky 3y ago
    Yes but this breaks the very container routing that we need Docker for. An "all or nothing" approach is not ideal.
    - drdaeman 3y ago
      I'm not sure how much this routing is really needed. I use nftables instead of iptables, so I told Docker to not touch anything. I'm gradually getting rid of Docker (NixOS containers powered by systemd-nspawn are more convenient for me at the moment), so I don't have any fancy networking for it - but I still run it. I have a bunch of static nftables rules for the bridge (`iifname "docker0" ct state established,related counter accept` etc.) so the containers can access what they need. And then Docker uses docker-proxy for the exposed ports (paired with appropriate firewall rules when it's not on `lo`, where I have an umbrella `iif "lo" accept`) and it seem to works just fine.
- ilyt 3y ago
  Not really. Just ufw wasn't made to deal with it.
TacticalCoder 3y ago
Several containers and VMs by default just go a bit "overboard" I'd say, adding network interfaces and firewalling rules left and right. It takes people installing these for the first time by surprise.
yrro 3y ago
It would be fairer to say that docker rudely stomps all over your iptables configuration. ;)

jwitthuhn 3y ago

This bit me the first time I deployed using docker. Everything but 80/443 blocked by the firewall. "Surely it is safe to run my app server on port 8000 because no one can access that port externally." Docker stepped in to help by making sure people could access that port.

justsomehnguy 3y ago
My favourite part in Docker is what if you have 127.0.0.1 configured as your resolver (eg you are running Unbound) then Docker helpfully configure containers to use 1.1.1.1 instead. Thanks, my ass, and give back my time trying to diagnose [a container with no diagnostic tools, because it's a lightweight container] why it can't connect to someservice.on.my.lan.
- judge2020 3y ago
  Wouldn't 127.0.0.1 be the loopback for the container itself? So you'd need to set the DNS to the gateway IP or dynamically by resolving `host.docker.internal` (assuming `--add-host=host.docker.internal:host-gateway`)
  - justsomehnguy 3y ago
    Yes. > So you'd need to set the DNS to the gateway IP This is the least friction way. Run your own resolver there to resolve names for containers, forward to the host's loopback for everything else. > or dynamically by resolving `host.docker.internal` Resolving. Name. Of the DNS server. Right?
    - judge2020 3y ago
      I mean dynamically as in a startup script that resolves that hostname, then updates the resolver to the IP it returns.
      - justsomehnguy 3y ago

gnfargbl 3y ago

Not just docker, either. If you configure a NodePort on your kubernetes cluster, you'll run into exactly the same issue: ufw-sourced iptables rules are overriden by kubernetes-sourced rules. Here's the only fix I was able to find, which doesn't seem well documented. Run kubectl -n kube-system edit configmap kube-proxy and edit in this: nodePortAddresses: - 192.168.0.0/16 ...where the CIDR block should correspond to the local network on your machine.

alexey-salmin 3y ago

This helps https://github.com/chaifeng/ufw-docker https://github.com/chaifeng/ufw-docker

ilyt 3y ago

It's probably ufw putting its rules in INPUT and OUTPUT chain and docker needing them in FORWARD. Don't blame ufw incompetence on iptables. There is separate issue with bridging aptly described here https://wiki.libvirt.org/Net.bridge.bridge-nf-call_and_sysctl.conf.html https://wiki.libvirt.org/Net.bridge.bridge-nf-call_and_sysct... but IIRC docker doesn't work in that config by default. IIRC Centos/RHEL comes with bridging iptables pre-broken because they're bad at making firewall rules there.

timost 3y ago

I think podman rootless behaves much better than docker regarding firewall "bypassing"

yrro 3y ago

Developers learning how their system actually works? I'll believe it when I see it... ;) More seriously, I believe nftables has the capacity to help here: now there can be several chains which all attach to the same hook. So docker can put rules into the FORWARD base chain of the filter table (belonging to iptables, mine is empty because I don't use iptables): # nft list chain ip filter FORWARD table ip filter { chain FORWARD { type filter hook forward priority filter; policy accept; } } ... but then the filter_FORWARD base chain of the firewalld table (belonging to firewalld) _also_ gets to process packets. Note the numerically higher filter priority, which means that this chain runs later: # nft list chain inet firewalld filter_FORWARD table inet firewalld { chain filter_FORWARD { type filter hook forward priority filter + 10; policy accept; ct state { established, related } accept ct status dnat accept iifname "lo" accept ip6 daddr { ::/96, ::ffff:0.0.0.0/96, 2002::/24, 2002:a00::/24, 2002:7f00::/24, 2002:a9fe::/32, 2002:ac10::/28, 2002:c0a8::/32, 2002:e000::/19 } log prefix "RFC3964_IPv4_REJECT: " reject with icmpv6 type addr-unreachable jump filter_FORWARD_POLICIES_pre jump filter_FORWARD_IN_ZONES jump filter_FORWARD_OUT_ZONES jump filter_FORWARD_POLICIES_post ct state { invalid } meta pkttype host log prefix "STATE_INVALID_DROP: " ct state { invalid } drop meta pkttype host log prefix "FINAL_REJECT: " reject with icmpx type admin-prohibited } } Now if only tools would learn to stay in their lane, cease assuming they are the only tool managing the nftables state, we'd have ufw managing its own table, docker managing its own table, firewalld managing its own...

bandyaboot 3y ago

They can learn about what Docker does with iptables the traditional way, by reading Docker’s networking documentation. https://docs.docker.com/network/packet-filtering-firewalls/ https://docs.docker.com/network/packet-filtering-firewalls/ It says explicitly that docker and ufw should be considered incompatible. Docker also has a configuration key to prevent it from modifying iptables.

Dylan16807 3y ago
People only have so much time to read documentation. You can't expect them to read an entire textbook before they start using Docker.
- pests 3y ago
  And people wonder why we aren't taken seriously like other engineering fields. Imagine a civil engineer telling someone they don't have time to read the documentation on the bolts their using to hold up a bridge.
  - Dylan16807 3y ago
    Imagine a bolt seller saying you have to read this entire book to understand the bolts, instead of having a couple pages containing the important specifications.
    - pests 3y ago
      The book _is already the important information_. This is evident that not reading it casued problems. Then imagine getting upset because the bolts don't work as you thought they did. You go do additional research and find the information right in that book you ignored! Too late, 100,000 bolts have already been used so now we gotta deal with workarounds and other additonal problems. Maybe should have read the book.
      - Dylan16807 3y ago

zelphirkalt 3y ago

Isn't that simply because Docker adds its own firewall rules, in order to realize its docker networks?