5 ms·

Did they sort out rootless networking? I've tried podman twice but simply cannot get networking to work properly when using a docker-compose file.

2OEH8eoCRo0 4y ago
What was wrong with it?
- misnome 4y ago
  It is very difficult getting containers that aren’t in a pod together to be able to talk to each other, with rootless pods, and without just host-binding everything. None of the normal hostname conveniences are in place, and if it’s possible to do it isn’t default and probably needed much more underlying knowledge than I had when I tried.
  - yrro 4y ago
    I think the new network implementation in Podman 4 ("netavark") helps with that sort of thing. So the following works: [sam@isis ~]$ podman network create foo foo [sam@isis ~]$ podman run -d --name=host1 --network=foo --init registry.access.redhat.com/ubi9/ubi-minimal sleep inf 1f7ffa0e474f7116d782ffa4d8dd9c0454893e9dac54b2cd9e61481a7f86a2ff [sam@isis ~]$ podman run -d --name=host2 --network=foo --init registry.access.redhat.com/ubi9/ubi-minimal sleep inf d8ffa58cb9de77305545b9f85043c016d03193adb6543b1768a68623b044b0e5 [sam@isis ~]$ podman exec host1 getent ahosts host2 10.89.0.3 STREAM host2.dns.podman 10.89.0.3 DGRAM 10.89.0.3 RAW [sam@isis ~]$ podman exec host2 getent ahosts host1 10.89.0.2 STREAM host1.dns.podman 10.89.0.2 DGRAM 10.89.0.2 RAW i.e., we have a network called 'foo' to which containers 'host1' and 'host2' are attached; they can resolve each others hostnames. It's worth noting if you are trying this on a system where you previously ran podman <= 3, I think you have to 'podman system reset' to get a new configuration with the new Podman 4 network implementation. (Personally I just run all containers with --host=net, this is just for local development, who cares) I think this will work with docker-compose as long as you 'systemctl --user start podman.socket' and 'export DOCKER_HOST=unix:/$XDG_RUNTIME_DIR/podman/podman.sock'.
  - kuratkull 4y ago
    Had the same issue, the containers did not resolve each other by hostname. This fixed it (run on the host): apt install containernetworking-plugins podman-plugins
    - yrro 4y ago
      That's not needed/used since Podman 4, where the networking stack was rewritten; one of the features of the new "netavark" stack is that it has DNS resolution for containers within a network by default, no plugin needed.
limaho 4y ago
I haven’t had issues with rootless networking but I run pretty simple apps. I gave up on podman-compose though and run individual commands to create podman objects. Once rootless containers are in the same pod you can access other containers in the same pod over localhost
- kirbyfan64sos 4y ago
  Podman has a Docker-compatible service API, so you can use it via docker-compose.
- itsmartapuntocm 4y ago
  I embraced pods and now I don’t think I would want to go back to docker without them. Once I get everything built the way I want I dump the config to a yaml with “podman generate kube” for future reference. It’s also nice being able to use systemd service files to start/stop everything, and podman has a command to generate those too.
galleywest200 4y ago
I too have issues with this, but mainly trying to run protected ports (port 53 for DNS with PiHole) as rootless Podman -- it just did not work. Went back to docker and have not had an issue since.
- candiddevmike 4y ago
  Just don't use rootless. You can still do user namespace containers (the real benefit to rootless, IMO) as a root user. I predict we will find all sorts of fun, niche security bugs that were made to get rootless containers working, especially networking--linux really doesn't like unprivileged changes to the networking stack. If you already have sudo/root access for your user, rootless is just going to cause more problems.
- mbreese 4y ago
  I think that’s by design. Non-root users aren’t supposed to be able to listen on protected ports. If you need to run PiHole as a non-root user, you could start the container rootless to listen on 5353/udp (arbitrary). Then (as root), add a firewall rule to redirect traffic from 5353/udp -> 53/udp. That probably the only way to make this work while still running the container as a non-root user.
  - yrro 4y ago
    There are two ways around this: Frist, you can run the container with --cap-add=net_bind_service which allows processes to bind to privileged ports: $ podman run --cap-add=net_bind_service --rm -it --user=1 localhost/socat bash-5.1$ id uid=1(bin) gid=1(bin) groups=1(bin) bash-5.1$ capsh --print Current: cap_net_bind_service=eip Bounding set =cap_chown,cap_dac_override,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_net_bind_service,cap_sys_chroot,cap_setfcap Ambient set =cap_net_bind_service Current IAB: !cap_dac_read_search,!cap_linux_immutable,^cap_net_bind_service,!cap_net_broadcast,!cap_net_admin,!cap_net_raw,!cap_ipc_lock,!cap_ipc_owner,!cap_sys_module,!cap_sys_rawio,!cap_sys_ptrace,!cap_sys_pacct,!cap_sys_admin,!cap_sys_boot,!cap_sys_nice,!cap_sys_resource,!cap_sys_time,!cap_sys_tty_config,!cap_mknod,!cap_lease,!cap_audit_write,!cap_audit_control,!cap_mac_override,!cap_mac_admin,!cap_syslog,!cap_wake_alarm,!cap_block_suspend,!cap_audit_read,!cap_perfmon,!cap_bpf,!cap_checkpoint_restore Securebits: 00/0x0/1'b0 (no-new-privs=0) secure-noroot: no (unlocked) secure-no-suid-fixup: no (unlocked) secure-keep-caps: no (unlocked) secure-no-ambient-raise: no (unlocked) uid=1(bin) euid=1(bin) gid=1(bin) groups=1(bin) Guessed mode: UNCERTAIN (0) bash-5.1$ socat -dd - TCP-LISTEN:22 2023/02/21 09:19:34 socat[82] N reading from and writing to stdio 2023/02/21 09:19:34 socat[82] W ioctl(5, IOCTL_VM_SOCKETS_GET_LOCAL_CID, ...): Inappropriate ioctl for device 2023/02/21 09:19:34 socat[82] N listening on AF=2 0.0.0.0:22 Second, you can run the container with --sysctl=net.ipv4.ip_unprivileged_port_start=0 which tells the kernel to allow any process to bind to ports above 0 instead of the default of 1024: $ podman run --sysctl=net.ipv4.ip_unprivileged_port_start=0 --rm -it --user=1 registry.access.redhat.com/ubi9/ubi-minimal bash-5.1$ id uid=1(bin) gid=1(bin) groups=1(bin) bash-5.1$ socat -dd - TCP-LISTEN:22 2023/02/21 09:26:55 socat[59] N reading from and writing to stdio 2023/02/21 09:26:55 socat[59] W ioctl(5, IOCTL_VM_SOCKETS_GET_LOCAL_CID, ...): Inappropriate ioctl for device 2023/02/21 09:26:55 socat[59] N listening on AF=2 0.0.0.0:22
seized 4y ago
Podman 4.0 brought a new network stack for rootless called Netavark. Works nicely for me.
aaravchen 4y ago
Since docker-compose is designed to exclusively work with rootful containers (basically the only thing docker supports), it's not a surprise you're having problems trying to run it rootless. Podman added an optional rootful daemon a few releases ago that you can run in place of/next to the docker daemon that (theoretically) supports all the same docker network functionality specifcally so docker and docker compose commands can be run unchanged on podman.

2OEH8eoCRo0 4y ago

What was wrong with it?

misnome 4y ago
It is very difficult getting containers that aren’t in a pod together to be able to talk to each other, with rootless pods, and without just host-binding everything. None of the normal hostname conveniences are in place, and if it’s possible to do it isn’t default and probably needed much more underlying knowledge than I had when I tried.
- yrro 4y ago
  I think the new network implementation in Podman 4 ("netavark") helps with that sort of thing. So the following works: [sam@isis ~]$ podman network create foo foo [sam@isis ~]$ podman run -d --name=host1 --network=foo --init registry.access.redhat.com/ubi9/ubi-minimal sleep inf 1f7ffa0e474f7116d782ffa4d8dd9c0454893e9dac54b2cd9e61481a7f86a2ff [sam@isis ~]$ podman run -d --name=host2 --network=foo --init registry.access.redhat.com/ubi9/ubi-minimal sleep inf d8ffa58cb9de77305545b9f85043c016d03193adb6543b1768a68623b044b0e5 [sam@isis ~]$ podman exec host1 getent ahosts host2 10.89.0.3 STREAM host2.dns.podman 10.89.0.3 DGRAM 10.89.0.3 RAW [sam@isis ~]$ podman exec host2 getent ahosts host1 10.89.0.2 STREAM host1.dns.podman 10.89.0.2 DGRAM 10.89.0.2 RAW i.e., we have a network called 'foo' to which containers 'host1' and 'host2' are attached; they can resolve each others hostnames. It's worth noting if you are trying this on a system where you previously ran podman <= 3, I think you have to 'podman system reset' to get a new configuration with the new Podman 4 network implementation. (Personally I just run all containers with --host=net, this is just for local development, who cares) I think this will work with docker-compose as long as you 'systemctl --user start podman.socket' and 'export DOCKER_HOST=unix:/$XDG_RUNTIME_DIR/podman/podman.sock'.
- kuratkull 4y ago
  Had the same issue, the containers did not resolve each other by hostname. This fixed it (run on the host): apt install containernetworking-plugins podman-plugins
  - yrro 4y ago
    That's not needed/used since Podman 4, where the networking stack was rewritten; one of the features of the new "netavark" stack is that it has DNS resolution for containers within a network by default, no plugin needed.

limaho 4y ago

I haven’t had issues with rootless networking but I run pretty simple apps. I gave up on podman-compose though and run individual commands to create podman objects. Once rootless containers are in the same pod you can access other containers in the same pod over localhost

kirbyfan64sos 4y ago
Podman has a Docker-compatible service API, so you can use it via docker-compose.
itsmartapuntocm 4y ago
I embraced pods and now I don’t think I would want to go back to docker without them. Once I get everything built the way I want I dump the config to a yaml with “podman generate kube” for future reference. It’s also nice being able to use systemd service files to start/stop everything, and podman has a command to generate those too.

galleywest200 4y ago

I too have issues with this, but mainly trying to run protected ports (port 53 for DNS with PiHole) as rootless Podman -- it just did not work. Went back to docker and have not had an issue since.

candiddevmike 4y ago
Just don't use rootless. You can still do user namespace containers (the real benefit to rootless, IMO) as a root user. I predict we will find all sorts of fun, niche security bugs that were made to get rootless containers working, especially networking--linux really doesn't like unprivileged changes to the networking stack. If you already have sudo/root access for your user, rootless is just going to cause more problems.
mbreese 4y ago
I think that’s by design. Non-root users aren’t supposed to be able to listen on protected ports. If you need to run PiHole as a non-root user, you could start the container rootless to listen on 5353/udp (arbitrary). Then (as root), add a firewall rule to redirect traffic from 5353/udp -> 53/udp. That probably the only way to make this work while still running the container as a non-root user.
- yrro 4y ago
  There are two ways around this: Frist, you can run the container with --cap-add=net_bind_service which allows processes to bind to privileged ports: $ podman run --cap-add=net_bind_service --rm -it --user=1 localhost/socat bash-5.1$ id uid=1(bin) gid=1(bin) groups=1(bin) bash-5.1$ capsh --print Current: cap_net_bind_service=eip Bounding set =cap_chown,cap_dac_override,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_net_bind_service,cap_sys_chroot,cap_setfcap Ambient set =cap_net_bind_service Current IAB: !cap_dac_read_search,!cap_linux_immutable,^cap_net_bind_service,!cap_net_broadcast,!cap_net_admin,!cap_net_raw,!cap_ipc_lock,!cap_ipc_owner,!cap_sys_module,!cap_sys_rawio,!cap_sys_ptrace,!cap_sys_pacct,!cap_sys_admin,!cap_sys_boot,!cap_sys_nice,!cap_sys_resource,!cap_sys_time,!cap_sys_tty_config,!cap_mknod,!cap_lease,!cap_audit_write,!cap_audit_control,!cap_mac_override,!cap_mac_admin,!cap_syslog,!cap_wake_alarm,!cap_block_suspend,!cap_audit_read,!cap_perfmon,!cap_bpf,!cap_checkpoint_restore Securebits: 00/0x0/1'b0 (no-new-privs=0) secure-noroot: no (unlocked) secure-no-suid-fixup: no (unlocked) secure-keep-caps: no (unlocked) secure-no-ambient-raise: no (unlocked) uid=1(bin) euid=1(bin) gid=1(bin) groups=1(bin) Guessed mode: UNCERTAIN (0) bash-5.1$ socat -dd - TCP-LISTEN:22 2023/02/21 09:19:34 socat[82] N reading from and writing to stdio 2023/02/21 09:19:34 socat[82] W ioctl(5, IOCTL_VM_SOCKETS_GET_LOCAL_CID, ...): Inappropriate ioctl for device 2023/02/21 09:19:34 socat[82] N listening on AF=2 0.0.0.0:22 Second, you can run the container with --sysctl=net.ipv4.ip_unprivileged_port_start=0 which tells the kernel to allow any process to bind to ports above 0 instead of the default of 1024: $ podman run --sysctl=net.ipv4.ip_unprivileged_port_start=0 --rm -it --user=1 registry.access.redhat.com/ubi9/ubi-minimal bash-5.1$ id uid=1(bin) gid=1(bin) groups=1(bin) bash-5.1$ socat -dd - TCP-LISTEN:22 2023/02/21 09:26:55 socat[59] N reading from and writing to stdio 2023/02/21 09:26:55 socat[59] W ioctl(5, IOCTL_VM_SOCKETS_GET_LOCAL_CID, ...): Inappropriate ioctl for device 2023/02/21 09:26:55 socat[59] N listening on AF=2 0.0.0.0:22

seized 4y ago

Podman 4.0 brought a new network stack for rootless called Netavark. Works nicely for me.

aaravchen 4y ago

Since docker-compose is designed to exclusively work with rootful containers (basically the only thing docker supports), it's not a surprise you're having problems trying to run it rootless. Podman added an optional rootful daemon a few releases ago that you can run in place of/next to the docker daemon that (theoretically) supports all the same docker network functionality specifcally so docker and docker compose commands can be run unchanged on podman.