6 ms·

Anyone have experience with this? I love the idea of it being daemonless.

edwintorok 6y ago
If you run it as non-root it is significantly slower than docker as root. Docker can us the overlay2 kernel driver, whereas podkan would use fuse-overlayfs in userspace. This has a high CPU overhead (e.g. don't try to run AFL inside podman), and a 1024 FD limit for the entire container (so a 'make -j40' usually dies). There are ways around it: raise the ulimit for your user and run new enough podman to raise limit for fuse-overlayfs, use the 'vfs' driver (it has other perf issues). I heard (but haven't tested yet) that the 'btrfs' driver avoids all these problems and works from userspace. Obviously requires an FS formatted as btrfs... There are also compatibility issues with Docker. E.g. one container was running sshfs inside Docker just fine, but fails with a permission error on /dev/fuse with podman.
- stuaxo 6y ago
  Are these fuse limitations or fuse-overlayfs limitations ?
  - edwintorok 6y ago
    I think FUSE limitations: fuse is served by a single userspace process, which is limited the same way as any other userspace process by ulimit. It is not a fundamental limitation of podman, just of podman's default rootless behaviour.
- SEJeff 6y ago
  You can run podman as root, but it doesn't default to it, for generally sensible security reasons. Also, docker runs as root, so it won't have permissions problems. You can change the permissions of /dev/fuse if you want to allow podman containers to access it or update the group of the user launching podman.
- eriksjolund 6y ago
  If I understand correctly support for native rootless mounts is currently under development: https://github.com/containers/storage/pull/816 https://github.com/containers/storage/pull/816 The functionality requires Linux kernel 5.11 (soon to be released)
  - fatherlinnux 6y ago
    That is correct. We're targeting that work for RHEL 8.5 in November-ish, so you'll likely see that drop in Fedora and other Linux distros sooner.
snickms 6y ago
I've used podman-compose for a few web projects. Rootless means you can't bind to privileged ports etc. but after a bit of fiddling I can now spin up my choice of DB/server/mock pretty quickly - just like I could with docker. The lack of 'cannot connect to dockerd' mysteries makes for a much-improved developer experience if you ask me.
sevagh 6y ago
I have a Makefile for a Rust project which binds the local repository to a Docker volume, builds it in the container using muslrust, and then does a chown to change the target directory back from root ownership to my own user. All I had to do was 's/docker/podman/g' and remove the chown hack and it works fine: https://github.com/sevagh/pq/commit/6acf6d05a094ac2959567a9ab989cca1e9e78736#diff-76ed074a9305c04054cdebb9e9aad2d818052b07091de1f20cad0bbac34ffb52 https://github.com/sevagh/pq/commit/6acf6d05a094ac2959567a9a... It understands Dockerfiles and can pull images from Dockerhub.
thechao 6y ago
I’m a GPU driver dev and you guys are talking ~alien~, right now. What does “daemonless” mean? Why is that good?
- deleted 6y ago
  [deleted]
- shaded-enmity 6y ago
  In Docker each container process is a child of the Docker daemon process. If you need to apply security patches to the Docker daemon it kills all your running containers.
  - raesene9 6y ago
    Technically the parent process for a contained process with Docker is containerd-shim. Also, Docker does support live restore if you want to keep containers running over daemon restarts https://docs.docker.com/config/containers/live-restore/ https://docs.docker.com/config/containers/live-restore/
raesene9 6y ago
Out of curiosity, why do you prefer daemonless setups?
- georgyo 6y ago
  If I have 30 containers running, why should a single daemon being restarted cause all 30 to shutdown as well? Similarly, the docker security model is that there isn't a security model. If you can talk to the docker socket you have what ever privileges the daemon is running as.
  - raesene9 6y ago
    So Docker supports live restore https://docs.docker.com/config/containers/live-restore/ https://docs.docker.com/config/containers/live-restore/ which addresses the first point. Second point, yep if you run Docker as root and someone can access the socket file they get root. If that's a concern, you can run Docker rootless. And as we're talking file permissions on a local host to allow that access, the same applies to podman containers does it not? If there are permission issues allowing another user to use access the container filesystems, you have the same problem.
    - zaarn 6y ago
      Rootless Docker is basically a joke, I've tried to run production workloads on it for about a year before I gave up. Numerous docker images will refuse to work, a bunch of things will subtly fill your error logs with warnings and it doesn't mesh well with running docker swarm at all.
      - raesene9 6y ago
        Rootless docker only left experimental status with 20.10 which came out in December 2020, so maybe they would have addressed some of those issues... As to swarm, I was comparing Docker rootless to podman, which is more a developer use case than prod. container clusters.
- 6y ago
igneo676 6y ago
I had pretty negative experiences with it. I'm not sure how much of my experience is _actually_ the fault of Podman and how much of it was around using it as a drop-in replacement for Docker. It baffled me though and I'm a long time Docker and Linux user. I used Arch Linux and followed their docs[1], mainly because I wanted it to also run without root. Big mistake. Attempting to run a simple postgres database w/ and w/o root permissions for Podman resulted in my system getting into an inconsistent state. I was unable to relaunch the container due to the state of my system. I mucked around with runc/crun and I ended up literally nuking both the root and non-root directories that stored them on my computer. I reset my computer back to Podman for root only, hoping that I had just chosen a niche path. It still would leave my system broken, requiring drastic measures to recover. No thanks. After much debugging, finally switching back to Docker, I realized my mistake: I had forgotten a required environment variable. Silly mistake. Docker surfaced the problem immediately. Podman did not. Docker recovered gracefully from the error. Podman left my system in an inconsistent state, unable to relaunch the container and unable to remove it Again, I'm not sure how much of my experience was just trying to force Podman into a square hole? I'm sure there are other people that make it work just fine for their use cases Edit: I should note that I used it as a docker-compose replacement, which is probably another off-the-beaten path usage that made this more dramatic than it should have been 1. https://wiki.archlinux.org/index.php/Podman https://wiki.archlinux.org/index.php/Podman
- dnautics 6y ago
  I don't know when you tried this but I recall it having sucked in the summer when I tried it and I gave up. New job in the fall, decided to give podman a shot "if it doesn't work on the first try I'm going to use docker", then podman-compose 3rd party instructions (I can't remember which) just works out of the box. I'm running a fleet that includes postgres and redis.
perlgeek 6y ago
One huge advantage over docker is that if you mount a directory into the container, and the container writes files there, on the host system they always have the owner of the user that started the container. That makes it much more convenient for build environments, without having to hard-code user IDs both in the container and on the host.
- Spivak 6y ago
  If you're on a Linux system you can actually make this work better with sssd. So sssd's architecture is actually client-server over a unix socket. So all you need to do is create a very simple base container layer that just installs sssd-client, and wires up /etc/nsswitch.conf to use it (your package manager will almost surely do this automatically). Then just bind mount the sssd socket into your container and boom, all your host users are in the container. If you already log in with sssd you're done. But if you only use local users then you'll need to configure the proxy provider so that sssd reads from your passwd on your host. In this case the host system doesn't actually have to use sssd for anything.
  - perlgeek 6y ago
    Sounds interesting, do you have a link to an example where this is done/demonstrated? (Also I'm not sure how that's better (and not just different), except maybe it allows more than one host user in the container, but I haven't had a use case for that).
  - kortex 6y ago
    This is glorious and just the thing I was looking for. I am trying to move towards an even more container based dev environment, basically shell into long running containers. Maybe even window manager in docker . Totally eliminates dependency hell, e.g. ROS heavy workflows where it wants to control every part of your environment.
shaicoleman 6y ago
I tried it, and went back to Docker with BuildKit. * The newer version of Docker+BulidKit supports bind-mounts/caching/secrets/memory filesystem etc. we used it to significantly speed up builds. You could probably find a way in buildah to achieve the same things, but it's not standard. * Parallel builds - Docker+BulidKit builds Dockerfile in parallel, in podman things run serially. The combination of caching and parallel builds and Docker being faster even for single-threaded builds, Docker builds ended up an order of magnitude faster than the Podman ones. * Buggy caching - There were a lot of caching bugs (randomly rebuilding when nothing changed, and reusing cached layers when files have changed). These issues are supposedly fixed, but I've lost trust in the Dockerfile builds. * Various bugs when used as a drop-in replacement for Docker. * Recurring issues on Ubuntu. It seemed all the developers were on Fedora/RHEL, and there were recurring issues with the Ubuntu builds. Things might be better now. * non-root containers require editing /etc/subuid, which you can't do unless you have root access. More information about the new BuildKit features in Docker: https://pythonspeed.com/articles/docker-buildkit/ https://pythonspeed.com/articles/docker-buildkit/
- feanaro 6y ago
  FWIW, I'm using Podman and I'm happy with it given that it's fast enough for my purposes and I haven't encountered any caching bugs.
- lima 6y ago
  Buildkit's caches are a killer feature. They go a long way towards making Docker builds usable for iterative development. We also migrated back to Docker from Podman because of Buildkit and the general bugginess of Podman.
jka 6y ago
From limited experience: podman and the associated toolset (buildah, skopeo) show a lot of promise. That said, they are evolving and may currently require administrative care and attention when used in production. If your environment would benefit from smaller, isolated containerization tools then my two cents would be that it's worth keeping an eye on these as they mature, and perhaps perform early evaluation (bearing in mind that you may not be able to migrate completely, yet). The good: - Separation of concerns; individual binaries and tools that do not all have to be deployed in production - A straightforward migration path to build containers from existing Dockerfiles (via "buildah bud") - Progressing-and-planned support for a range of important container technology, including rootless and multi-architecture builds
dnautics 6y ago
I've been using my company's docker compose script transparently with podman-compose for three months now and I have basically forgotten that I've switched. I actually didn't use docker before this, so I can't comment on what the difference is. I feel like the podman experience is binary, either it works perfectly or it sucks balls. My suggestion is to give it a try, if it fails, then maybe file a bug report, fail fast and fall back on docker.

If you run it as non-root it is significantly slower than docker as root. Docker can us the overlay2 kernel driver, whereas podkan would use fuse-overlayfs in userspace. This has a high CPU overhead (e.g. don't try to run AFL inside podman), and a 1024 FD limit for the entire container (so a 'make -j40' usually dies). There are ways around it: raise the ulimit for your user and run new enough podman to raise limit for fuse-overlayfs, use the 'vfs' driver (it has other perf issues). I heard (but haven't tested yet) that the 'btrfs' driver avoids all these problems and works from userspace. Obviously requires an FS formatted as btrfs... There are also compatibility issues with Docker. E.g. one container was running sshfs inside Docker just fine, but fails with a permission error on /dev/fuse with podman.

stuaxo 6y ago
Are these fuse limitations or fuse-overlayfs limitations ?
- edwintorok 6y ago
  I think FUSE limitations: fuse is served by a single userspace process, which is limited the same way as any other userspace process by ulimit. It is not a fundamental limitation of podman, just of podman's default rootless behaviour.
SEJeff 6y ago
You can run podman as root, but it doesn't default to it, for generally sensible security reasons. Also, docker runs as root, so it won't have permissions problems. You can change the permissions of /dev/fuse if you want to allow podman containers to access it or update the group of the user launching podman.
eriksjolund 6y ago
If I understand correctly support for native rootless mounts is currently under development: https://github.com/containers/storage/pull/816 https://github.com/containers/storage/pull/816 The functionality requires Linux kernel 5.11 (soon to be released)
- fatherlinnux 6y ago
  That is correct. We're targeting that work for RHEL 8.5 in November-ish, so you'll likely see that drop in Fedora and other Linux distros sooner.

snickms 6y ago

I've used podman-compose for a few web projects. Rootless means you can't bind to privileged ports etc. but after a bit of fiddling I can now spin up my choice of DB/server/mock pretty quickly - just like I could with docker. The lack of 'cannot connect to dockerd' mysteries makes for a much-improved developer experience if you ask me.

sevagh 6y ago

I have a Makefile for a Rust project which binds the local repository to a Docker volume, builds it in the container using muslrust, and then does a chown to change the target directory back from root ownership to my own user. All I had to do was 's/docker/podman/g' and remove the chown hack and it works fine: https://github.com/sevagh/pq/commit/6acf6d05a094ac2959567a9ab989cca1e9e78736#diff-76ed074a9305c04054cdebb9e9aad2d818052b07091de1f20cad0bbac34ffb52 https://github.com/sevagh/pq/commit/6acf6d05a094ac2959567a9a... It understands Dockerfiles and can pull images from Dockerhub.

thechao 6y ago

I’m a GPU driver dev and you guys are talking ~alien~, right now. What does “daemonless” mean? Why is that good?

deleted 6y ago
[deleted]
shaded-enmity 6y ago
In Docker each container process is a child of the Docker daemon process. If you need to apply security patches to the Docker daemon it kills all your running containers.
- raesene9 6y ago
  Technically the parent process for a contained process with Docker is containerd-shim. Also, Docker does support live restore if you want to keep containers running over daemon restarts https://docs.docker.com/config/containers/live-restore/ https://docs.docker.com/config/containers/live-restore/

raesene9 6y ago

Out of curiosity, why do you prefer daemonless setups?

georgyo 6y ago
If I have 30 containers running, why should a single daemon being restarted cause all 30 to shutdown as well? Similarly, the docker security model is that there isn't a security model. If you can talk to the docker socket you have what ever privileges the daemon is running as.
- raesene9 6y ago
  So Docker supports live restore https://docs.docker.com/config/containers/live-restore/ https://docs.docker.com/config/containers/live-restore/ which addresses the first point. Second point, yep if you run Docker as root and someone can access the socket file they get root. If that's a concern, you can run Docker rootless. And as we're talking file permissions on a local host to allow that access, the same applies to podman containers does it not? If there are permission issues allowing another user to use access the container filesystems, you have the same problem.
  - zaarn 6y ago
    Rootless Docker is basically a joke, I've tried to run production workloads on it for about a year before I gave up. Numerous docker images will refuse to work, a bunch of things will subtly fill your error logs with warnings and it doesn't mesh well with running docker swarm at all.
    - raesene9 6y ago
      Rootless docker only left experimental status with 20.10 which came out in December 2020, so maybe they would have addressed some of those issues... As to swarm, I was comparing Docker rootless to podman, which is more a developer use case than prod. container clusters.
6y ago

igneo676 6y ago

I had pretty negative experiences with it. I'm not sure how much of my experience is _actually_ the fault of Podman and how much of it was around using it as a drop-in replacement for Docker. It baffled me though and I'm a long time Docker and Linux user. I used Arch Linux and followed their docs[1], mainly because I wanted it to also run without root. Big mistake. Attempting to run a simple postgres database w/ and w/o root permissions for Podman resulted in my system getting into an inconsistent state. I was unable to relaunch the container due to the state of my system. I mucked around with runc/crun and I ended up literally nuking both the root and non-root directories that stored them on my computer. I reset my computer back to Podman for root only, hoping that I had just chosen a niche path. It still would leave my system broken, requiring drastic measures to recover. No thanks. After much debugging, finally switching back to Docker, I realized my mistake: I had forgotten a required environment variable. Silly mistake. Docker surfaced the problem immediately. Podman did not. Docker recovered gracefully from the error. Podman left my system in an inconsistent state, unable to relaunch the container and unable to remove it Again, I'm not sure how much of my experience was just trying to force Podman into a square hole? I'm sure there are other people that make it work just fine for their use cases Edit: I should note that I used it as a docker-compose replacement, which is probably another off-the-beaten path usage that made this more dramatic than it should have been 1. https://wiki.archlinux.org/index.php/Podman https://wiki.archlinux.org/index.php/Podman

dnautics 6y ago
I don't know when you tried this but I recall it having sucked in the summer when I tried it and I gave up. New job in the fall, decided to give podman a shot "if it doesn't work on the first try I'm going to use docker", then podman-compose 3rd party instructions (I can't remember which) just works out of the box. I'm running a fleet that includes postgres and redis.

perlgeek 6y ago

One huge advantage over docker is that if you mount a directory into the container, and the container writes files there, on the host system they always have the owner of the user that started the container. That makes it much more convenient for build environments, without having to hard-code user IDs both in the container and on the host.

Spivak 6y ago
If you're on a Linux system you can actually make this work better with sssd. So sssd's architecture is actually client-server over a unix socket. So all you need to do is create a very simple base container layer that just installs sssd-client, and wires up /etc/nsswitch.conf to use it (your package manager will almost surely do this automatically). Then just bind mount the sssd socket into your container and boom, all your host users are in the container. If you already log in with sssd you're done. But if you only use local users then you'll need to configure the proxy provider so that sssd reads from your passwd on your host. In this case the host system doesn't actually have to use sssd for anything.
- perlgeek 6y ago
  Sounds interesting, do you have a link to an example where this is done/demonstrated? (Also I'm not sure how that's better (and not just different), except maybe it allows more than one host user in the container, but I haven't had a use case for that).
- kortex 6y ago
  This is glorious and just the thing I was looking for. I am trying to move towards an even more container based dev environment, basically shell into long running containers. Maybe even window manager in docker . Totally eliminates dependency hell, e.g. ROS heavy workflows where it wants to control every part of your environment.

shaicoleman 6y ago

I tried it, and went back to Docker with BuildKit. * The newer version of Docker+BulidKit supports bind-mounts/caching/secrets/memory filesystem etc. we used it to significantly speed up builds. You could probably find a way in buildah to achieve the same things, but it's not standard. * Parallel builds - Docker+BulidKit builds Dockerfile in parallel, in podman things run serially. The combination of caching and parallel builds and Docker being faster even for single-threaded builds, Docker builds ended up an order of magnitude faster than the Podman ones. * Buggy caching - There were a lot of caching bugs (randomly rebuilding when nothing changed, and reusing cached layers when files have changed). These issues are supposedly fixed, but I've lost trust in the Dockerfile builds. * Various bugs when used as a drop-in replacement for Docker. * Recurring issues on Ubuntu. It seemed all the developers were on Fedora/RHEL, and there were recurring issues with the Ubuntu builds. Things might be better now. * non-root containers require editing /etc/subuid, which you can't do unless you have root access. More information about the new BuildKit features in Docker: https://pythonspeed.com/articles/docker-buildkit/ https://pythonspeed.com/articles/docker-buildkit/

feanaro 6y ago
FWIW, I'm using Podman and I'm happy with it given that it's fast enough for my purposes and I haven't encountered any caching bugs.
lima 6y ago
Buildkit's caches are a killer feature. They go a long way towards making Docker builds usable for iterative development. We also migrated back to Docker from Podman because of Buildkit and the general bugginess of Podman.

jka 6y ago

From limited experience: podman and the associated toolset (buildah, skopeo) show a lot of promise. That said, they are evolving and may currently require administrative care and attention when used in production. If your environment would benefit from smaller, isolated containerization tools then my two cents would be that it's worth keeping an eye on these as they mature, and perhaps perform early evaluation (bearing in mind that you may not be able to migrate completely, yet). The good: - Separation of concerns; individual binaries and tools that do not all have to be deployed in production - A straightforward migration path to build containers from existing Dockerfiles (via "buildah bud") - Progressing-and-planned support for a range of important container technology, including rootless and multi-architecture builds

dnautics 6y ago

I've been using my company's docker compose script transparently with podman-compose for three months now and I have basically forgotten that I've switched. I actually didn't use docker before this, so I can't comment on what the difference is. I feel like the podman experience is binary, either it works perfectly or it sucks balls. My suggestion is to give it a try, if it fails, then maybe file a bug report, fail fast and fall back on docker.