23 ms·

Podman: A Daemonless Container Engine

321 points by lobo_tuerto 6y ago

zeroxfe 6y ago
Anyone have experience with this? I love the idea of it being daemonless.
- edwintorok 6y ago
  If you run it as non-root it is significantly slower than docker as root. Docker can us the overlay2 kernel driver, whereas podkan would use fuse-overlayfs in userspace. This has a high CPU overhead (e.g. don't try to run AFL inside podman), and a 1024 FD limit for the entire container (so a 'make -j40' usually dies). There are ways around it: raise the ulimit for your user and run new enough podman to raise limit for fuse-overlayfs, use the 'vfs' driver (it has other perf issues). I heard (but haven't tested yet) that the 'btrfs' driver avoids all these problems and works from userspace. Obviously requires an FS formatted as btrfs... There are also compatibility issues with Docker. E.g. one container was running sshfs inside Docker just fine, but fails with a permission error on /dev/fuse with podman.
  - stuaxo 6y ago
    Are these fuse limitations or fuse-overlayfs limitations ?
    - edwintorok 6y ago
      I think FUSE limitations: fuse is served by a single userspace process, which is limited the same way as any other userspace process by ulimit. It is not a fundamental limitation of podman, just of podman's default rootless behaviour.
  - SEJeff 6y ago
    You can run podman as root, but it doesn't default to it, for generally sensible security reasons. Also, docker runs as root, so it won't have permissions problems. You can change the permissions of /dev/fuse if you want to allow podman containers to access it or update the group of the user launching podman.
  - eriksjolund 6y ago
    If I understand correctly support for native rootless mounts is currently under development: https://github.com/containers/storage/pull/816 https://github.com/containers/storage/pull/816 The functionality requires Linux kernel 5.11 (soon to be released)
    - fatherlinnux 6y ago
      That is correct. We're targeting that work for RHEL 8.5 in November-ish, so you'll likely see that drop in Fedora and other Linux distros sooner.
- snickms 6y ago
  I've used podman-compose for a few web projects. Rootless means you can't bind to privileged ports etc. but after a bit of fiddling I can now spin up my choice of DB/server/mock pretty quickly - just like I could with docker. The lack of 'cannot connect to dockerd' mysteries makes for a much-improved developer experience if you ask me.
- sevagh 6y ago
  I have a Makefile for a Rust project which binds the local repository to a Docker volume, builds it in the container using muslrust, and then does a chown to change the target directory back from root ownership to my own user. All I had to do was 's/docker/podman/g' and remove the chown hack and it works fine: https://github.com/sevagh/pq/commit/6acf6d05a094ac2959567a9ab989cca1e9e78736#diff-76ed074a9305c04054cdebb9e9aad2d818052b07091de1f20cad0bbac34ffb52 https://github.com/sevagh/pq/commit/6acf6d05a094ac2959567a9a... It understands Dockerfiles and can pull images from Dockerhub.
- thechao 6y ago
  I’m a GPU driver dev and you guys are talking ~alien~, right now. What does “daemonless” mean? Why is that good?
  - deleted 6y ago
    [deleted]
  - shaded-enmity 6y ago
    In Docker each container process is a child of the Docker daemon process. If you need to apply security patches to the Docker daemon it kills all your running containers.
    - raesene9 6y ago
      Technically the parent process for a contained process with Docker is containerd-shim. Also, Docker does support live restore if you want to keep containers running over daemon restarts https://docs.docker.com/config/containers/live-restore/ https://docs.docker.com/config/containers/live-restore/
- raesene9 6y ago
  Out of curiosity, why do you prefer daemonless setups?
  - georgyo 6y ago
    If I have 30 containers running, why should a single daemon being restarted cause all 30 to shutdown as well? Similarly, the docker security model is that there isn't a security model. If you can talk to the docker socket you have what ever privileges the daemon is running as.
    - raesene9 6y ago
      So Docker supports live restore https://docs.docker.com/config/containers/live-restore/ https://docs.docker.com/config/containers/live-restore/ which addresses the first point. Second point, yep if you run Docker as root and someone can access the socket file they get root. If that's a concern, you can run Docker rootless. And as we're talking file permissions on a local host to allow that access, the same applies to podman containers does it not? If there are permission issues allowing another user to use access the container filesystems, you have the same problem.
      - zaarn 6y ago
        Rootless Docker is basically a joke, I've tried to run production workloads on it for about a year before I gave up. Numerous docker images will refuse to work, a bunch of things will subtly fill your error logs with warnings and it doesn't mesh well with running docker swarm at all.
        raesene9 6y ago
        Rootless docker only left experimental status with 20.10 which came out in December 2020, so maybe they would have addressed some of those issues... As to swarm, I was comparing Docker rootless to podman, which is more a developer use case than prod. container clusters.
  - 6y ago
- igneo676 6y ago
  I had pretty negative experiences with it. I'm not sure how much of my experience is _actually_ the fault of Podman and how much of it was around using it as a drop-in replacement for Docker. It baffled me though and I'm a long time Docker and Linux user. I used Arch Linux and followed their docs[1], mainly because I wanted it to also run without root. Big mistake. Attempting to run a simple postgres database w/ and w/o root permissions for Podman resulted in my system getting into an inconsistent state. I was unable to relaunch the container due to the state of my system. I mucked around with runc/crun and I ended up literally nuking both the root and non-root directories that stored them on my computer. I reset my computer back to Podman for root only, hoping that I had just chosen a niche path. It still would leave my system broken, requiring drastic measures to recover. No thanks. After much debugging, finally switching back to Docker, I realized my mistake: I had forgotten a required environment variable. Silly mistake. Docker surfaced the problem immediately. Podman did not. Docker recovered gracefully from the error. Podman left my system in an inconsistent state, unable to relaunch the container and unable to remove it Again, I'm not sure how much of my experience was just trying to force Podman into a square hole? I'm sure there are other people that make it work just fine for their use cases Edit: I should note that I used it as a docker-compose replacement, which is probably another off-the-beaten path usage that made this more dramatic than it should have been 1. https://wiki.archlinux.org/index.php/Podman https://wiki.archlinux.org/index.php/Podman
  - dnautics 6y ago
    I don't know when you tried this but I recall it having sucked in the summer when I tried it and I gave up. New job in the fall, decided to give podman a shot "if it doesn't work on the first try I'm going to use docker", then podman-compose 3rd party instructions (I can't remember which) just works out of the box. I'm running a fleet that includes postgres and redis.
- perlgeek 6y ago
  One huge advantage over docker is that if you mount a directory into the container, and the container writes files there, on the host system they always have the owner of the user that started the container. That makes it much more convenient for build environments, without having to hard-code user IDs both in the container and on the host.
  - Spivak 6y ago
    If you're on a Linux system you can actually make this work better with sssd. So sssd's architecture is actually client-server over a unix socket. So all you need to do is create a very simple base container layer that just installs sssd-client, and wires up /etc/nsswitch.conf to use it (your package manager will almost surely do this automatically). Then just bind mount the sssd socket into your container and boom, all your host users are in the container. If you already log in with sssd you're done. But if you only use local users then you'll need to configure the proxy provider so that sssd reads from your passwd on your host. In this case the host system doesn't actually have to use sssd for anything.
    - perlgeek 6y ago
      Sounds interesting, do you have a link to an example where this is done/demonstrated? (Also I'm not sure how that's better (and not just different), except maybe it allows more than one host user in the container, but I haven't had a use case for that).
    - kortex 6y ago
      This is glorious and just the thing I was looking for. I am trying to move towards an even more container based dev environment, basically shell into long running containers. Maybe even window manager in docker . Totally eliminates dependency hell, e.g. ROS heavy workflows where it wants to control every part of your environment.
- shaicoleman 6y ago
  I tried it, and went back to Docker with BuildKit. * The newer version of Docker+BulidKit supports bind-mounts/caching/secrets/memory filesystem etc. we used it to significantly speed up builds. You could probably find a way in buildah to achieve the same things, but it's not standard. * Parallel builds - Docker+BulidKit builds Dockerfile in parallel, in podman things run serially. The combination of caching and parallel builds and Docker being faster even for single-threaded builds, Docker builds ended up an order of magnitude faster than the Podman ones. * Buggy caching - There were a lot of caching bugs (randomly rebuilding when nothing changed, and reusing cached layers when files have changed). These issues are supposedly fixed, but I've lost trust in the Dockerfile builds. * Various bugs when used as a drop-in replacement for Docker. * Recurring issues on Ubuntu. It seemed all the developers were on Fedora/RHEL, and there were recurring issues with the Ubuntu builds. Things might be better now. * non-root containers require editing /etc/subuid, which you can't do unless you have root access. More information about the new BuildKit features in Docker: https://pythonspeed.com/articles/docker-buildkit/ https://pythonspeed.com/articles/docker-buildkit/
  - feanaro 6y ago
    FWIW, I'm using Podman and I'm happy with it given that it's fast enough for my purposes and I haven't encountered any caching bugs.
  - lima 6y ago
    Buildkit's caches are a killer feature. They go a long way towards making Docker builds usable for iterative development. We also migrated back to Docker from Podman because of Buildkit and the general bugginess of Podman.
- jka 6y ago
  From limited experience: podman and the associated toolset (buildah, skopeo) show a lot of promise. That said, they are evolving and may currently require administrative care and attention when used in production. If your environment would benefit from smaller, isolated containerization tools then my two cents would be that it's worth keeping an eye on these as they mature, and perhaps perform early evaluation (bearing in mind that you may not be able to migrate completely, yet). The good: - Separation of concerns; individual binaries and tools that do not all have to be deployed in production - A straightforward migration path to build containers from existing Dockerfiles (via "buildah bud") - Progressing-and-planned support for a range of important container technology, including rootless and multi-architecture builds
- dnautics 6y ago
  I've been using my company's docker compose script transparently with podman-compose for three months now and I have basically forgotten that I've switched. I actually didn't use docker before this, so I can't comment on what the difference is. I feel like the podman experience is binary, either it works perfectly or it sucks balls. My suggestion is to give it a try, if it fails, then maybe file a bug report, fail fast and fall back on docker.
Datenstrom 6y ago
Glad to see Podman here, worked with it at a past job where services were deployed on RedHat. Converting all existing deployment scripts to podman was as easy as do a recursive search and replace `s/docker/podman/` and there was no learning curve as all the commands were the same. By default root is not required which is great also.
ddtaylor 6y ago
I liked podman for what I was working on with it. The only hangup I found was that it couldn't do networking related routing? Does anyone know more?
- dathinab 6y ago
  As far as I know it can do so but not when running rootless, I also don't thing it's currently possible to be done rootless. You still can run podman as root without damon. Or you can run a podman deamon (through systemd) which is even compatible with docker-compose but has most of the drawbacks of running docker.
- gvalkov 6y ago
  Quite a lot is possible with CNI [1]. For example, we use this setup to give real IPs to containers: # /etc/cni/net.d/testnet.conflist { "cniVersion": "0.4.0", "name": "testnet", "plugins": [ { "type": "bridge", "bridge": "br0", # main host interface is part of this bridge "ipam": { "type": "host-local", "subnet": "10.0.0.0/16", "gateway": "10.0.0.1", "routes": [{ "dst": "0.0.0.0/0"}] } } ] } You can then start a container and operate on its network namespace for added flexibility: podman run -it --net testnet --ip 10.0.0.2 ... ns=$(basename $(podman inspect $id | jq -r '.[0] .NetworkSettings .SandboxKey')) ip netns exec $ns ip route add ... [1]: https://github.com/containernetworking/cni https://github.com/containernetworking/cni
daitangio 6y ago
Happy to see docker alternative, but the road is still long for it to replace docker
- dathinab 6y ago
  I don't think so. It's doing a better job for most of the use cases I had recently. You can run it rootless which makes it a far better use cases for a lot of dev use cases (where you don't need to emulate a lot of network aspects or heavy filesystem usage). Also you can run it as root without deamon which is much better for a bunch of use cases and eliminates any problems with fuse-overlayfs. This is the best way to e.g. containerize systemd services and similar (and security wise still better as e.g. it still works well with the audit subsystem). If worse comes to worse you can run it as a deamon in which case it's compatible with docker compose, and has more or less the same problems. In my experience besides some integration testing use-cases is either already a good docker replacement without running it as a deamon, or the use case shouldn't be handled by docker either way (sure there are always some exceptions). Lastly I had far less problems with podman and firewalls then with docker but I haven't looked into why that's the case. (But it seems to be related to me using nftables instead of iptables).
- long_warmup 6y ago
  I use podman in my projects, no problems. And since cli is compatible with docker all my work I have done using docker simply runs with podman.
shyim 6y ago
I still miss an easy way to setup multiple containers in a single network like with docker-compose. podman-compose is not really useable.
- Spivak 6y ago
  An Ansible playbook with the podman_container module is my go-to. It's not as easy because it's less magic but it's also less magic which I count as a win.
- dathinab 6y ago
  You can use docker-compose with podman. Through it has similar drawbacks as using docker-compose with docker. As far as I know there is currently no rootless way on Linux to setup custom internal networks like docker compose does (hence why podman-compose doesn't support such things).
  - chippiewill 6y ago
    There definitely is a way as creating custom networks works perfectly fine with rootless docker. I'm surprised if podman doesn't support that since it uses rootlesskit (same as rootless docker) which does the heavy lifting for rootless networking.
- pydry 6y ago
  What's wrong with it?
  - dathinab 6y ago
    With docker all container have their own ip address and (potential) internal DNS name. But this can't be done rootless. So with rootless podman all container map to the same ip address but different ports. This is for some use cases (e.g. spinning up a DB for integration testing) not a problem at all. For others it is. More over you can run multiple groups of docker containers in separate networks, you can't do so with rootless podman. Through you can manage networks with rootfull podman (which still has no deamon and as such works better with e.g. capabilities and the audit sub system then docker does). Through to get the full docker-compose experience you need to run it as a deamon (through systemd) in which case you can use docker-compose with podman but it has most of the problems docker has.
osipov 6y ago
Any plans for Docker to offer a daemonless version?
acec 6y ago
It is almost 100% compatible with docker. I only had to do minimal changes in my Dockerfiles to use it.
- lmeyerov 6y ago
  ... but not docker-compose
- lima 6y ago
  But not Buildkit and docker-compose.
  - westurner 6y ago
    Podman v3 is compatible with docker-compose (but not yet swarm mode, FWIU), has a socket and a daemon that services it. Buildah (`podman buildx`, `buildah bud --arch arm64`) just gained multiarch build support; so also building arm64 containers from the same Dockerfile is easy now. https://github.com/containers/buildah/issues/1590 https://github.com/containers/buildah/issues/1590 IDK what BuildKit features should be added to Buildah, too?
    - lima 6y ago
      > IDK what BuildKit features should be added to Buildah, too? Cache mounts are what we rely on for incremental rebuilds.
Xophmeister 6y ago
Isn't Singularity[1] also daemonless? [1] https://sylabs.io/ https://sylabs.io/
- SEJeff 6y ago
  Yes, but singularity is a gigantic suid root binary where as podman uses user namespaces for unprivileged stuff.
geerlingguy 6y ago
What's the status on ease of running on Mac? I know last time I seriously considered testing it out in my workflow, it was kind of crazy that I had to have a separate machine (VM or real) just to run container images... I see in the docs [1]: "Podman is a tool for running Linux containers. You can do this from a MacOS desktop as long as you have access to a linux box either running inside of a VM on the host, or available via the network. You need to install the remote client and then setup ssh connection information." And yes, I know that Docker (and any other Linux container-based tool) also runs a VM—but it sets it up almost 100% transparently for me. I install docker, I run `docker run` and I have a container running. [1] https://podman.io/getting-started/installation#macos https://podman.io/getting-started/installation#macos
- throwaway894345 6y ago
  The docker machine or whatever it’s called these days is pretty impressive but when we’ve tried to use it in developer workflows on mac it would eat all of the CPU allotted to it in order to marshal filesystem events over the host/guest VM boundary. We tried a number of workarounds but none solved the problem. I’m sure it’s possible, but we ultimately gave up and moved back to developing against native processes on MacOS. For other use cases I imagine it’s just fine.
  - dzolvd 6y ago
    this is an issue I have battled as well. Docker is great for normalizing the execution environment; which should be a huge boon for developer tooling but on MacOS having your laptop sound like a hovercraft just to have some files watched for hot rebuilds is no bueno.
- eadmund 6y ago
  > I know last time I seriously considered testing it out in my workflow, it was kind of crazy that I had to have a separate machine (VM or real) just to run container images... Why does that seem 'kind of crazy' to you? A container is really just a namespaced Linux process, so … you either need a running Linux kernel or a good-enough emulation thereof. What seems kind of crazy to me is that so many folks who are deploying on Linux develop on macOS. That's not a dig against macOS, but one against the inevitable pains which arise when developing in one environment and deploying in another. Even though I much prefer a Linux environment, it would seem similarly crazy to develop on Linux and deploy on Windows or macOS. Heck, I think it is kind of crazy to develop on Ubuntu and deploy on Debian, and they are generally very very close!
corty 6y ago
Even less IPv6 support than docker. With docker you can at least get it to work somehow, even if it is totally different from IPv4, weirdly. Podman just has no IPv6 support to speak of.
- 3np 6y ago
  For airgapped networks, this is great.
- flowless 6y ago
  Not entirely true. I was researching this few hours ago and according to https://github.com/containers/podman/issues/6114#issuecomment-632409804 https://github.com/containers/podman/issues/6114#issuecommen... it just works when you add another network. Docker registry having no IPv6 is another fun story tho.
  - corty 6y ago
    Well, from that issue from 2020-12-24: "--ipv6 has not landed yet", so you cannot assign static external v6 adresses to a container. No activity on that bug since. There is tons of stuff in docker that works automatically, where in podman it is like "Please be aware you have to configure ip6tables (or whatever your OS firewall is) for forwarding.". Yeah, well, if i have to do everything by hand or program a ton of scripts around it, it is quite useless, isn't it? IPv6 support is always like this: half-baked at best, someone got it to work somehow, developers declare it done since it worked for someone. Then crickets... IPv6 support isn't done until you can replace IPv4 with it, with no changes to your config except addresses. Even docker isn't there yet. And podman's is still in a larval state.
lifeisstillgood 6y ago
I redid some docker builds yesterday with podman, and filled up my laptops 99GB entirely with layers. Docker was handling it somehow better. I used --squash, cleaned up the various apt-gets but boy it made me realise that for every abstraction we get leakage - sometimes serious leakage
SEJeff 6y ago
Under the covers, both podman AND docker use runc. Redhat is writing a new version named "crun" which is lower overhead and faster: https://github.com/containers/crun https://github.com/containers/crun
- cashewchoo 6y ago
  Adduser vs useradd is older than I am and I still can't keep them straight. Just put this with the rest of the fire, I guess, heh!
  - lrossi 6y ago
    Indeed. After 30 years of Linux, we should have got better at naming things.
  - SEJeff 6y ago
    I think it is sort of a play on words. Since runc is written in go and is slower, crun is written in c and is faster. Stupid nerd joke I know.
- lrossi 6y ago
  Upside: At least they used a different name. On other occasions, they just hijack the name. Like they did with dstat.
  - dralley 6y ago
    That was a misunderstanding where they saw that there had been no commit activity for 3 years, and no responses to issues/PRs filed in the past 18 months, and assumed the upstream was dead. Making that assumption was reasonable - moving forwards without at least making an attempt to contact the maintainer "just in case" was not. It would have been courteous to at least try. But it's not exactly the deliberate hostile takeover that you make it out to be. https://bugzilla.redhat.com/show_bug.cgi?id=1614277#c9 https://bugzilla.redhat.com/show_bug.cgi?id=1614277#c9 >> To my knowledge, there is no need (legally) to obtain consent to use the name 'dstat' for a replacement command providing the same functionality. It might be a nice thing to do from a community perspective, however - if there was someone to discuss with upstream. >> However, dstat is dead upstream. There have been no updates for years, no responses at all to any bug reports in the months I've been following the github repo now, and certainly no attempt to begin undertaking a python3 port. >> Since there is nobody maintaining the original dstat code anymore, it seemed a futile exercise to me so I've not attempted to contact the original author. And as pcp-dstat is now well advanced beyond the original dstat - implementing features listed in dstat's roadmap for many years, and with multiple active contributors - I think moving on with the backward-compatible name symlink is the best we can do.
runeks 6y ago
Sincere question: why would I want to use Podman over Docker? What does being "daemonless" actually buy me in practice?
- lmeyerov 6y ago
  $ for your favorite infra oss startup or volunteer rhel ~forces you to do podman in rhel 8+. They make it hard enough to do a docker install that corp policies will reject docker (non-standard centos 7 repos etc), while allowing ibm's thing. Ex: Locks podman over docker for a lot of US gov where rhel 8 is preferred and with only standard repos, and anything else drowns in special approvals. subtle and orthogonal to podman's technical innovations.. but as a lot of infra oss funding ultimately comes from what enterprise policies allow, this a significant anti-competitive adoption driver for podman by rhel/ibm.
  - op00to 6y ago
    Because people can’t install non-Red Hat software due to internal policy, that’s somehow anti-competitive?
    - lmeyerov 6y ago
      The main repos have plenty of non-rhel sw, and people can expend social capital working around IT policies, but that doesn't change it being against the grain. We are fine because we use Ubuntu for the generally better GPU support, but I don't have control over the reality of others. I repeatedly get on calls with teams where this comes up, and I feel sorry for the individuals who are stuck paying the time etc. cost for this. Docker for RHEL 8 is literally installing Docker for Centos 7, so this isn't a technology thing, but political: A big corporate abusing their trusted neutral oss infra position to push non-neutral decisions in order to prop up another part of its business. This is all bizarre to operators who barely know Docker to beginwith, and care more about say keeping the energy grid running. ("The only authorized thing will likely not work b/c incompatibilities, and I have to spend 2w getting the old working thing approved?") And again, none of this is technical. Docker is great but should keep improving, so competitive pressure and alternatives are clearly good... but anti-competitive stuff isn't.
williesleg 6y ago
Yay IBM!
fillest 6y ago
Well, "daemonless" is kind of marketing - there is still this daemon-per-container 'conmon' thing https://github.com/containers/conmon https://github.com/containers/conmon and I don't get why it is needed because 1) who actually needs to re-attach anyway? 2) container's streams are already properly handled by whatever supervisor (e.g. systemd). You can't disable conmon and I'm not sure if its usage is not hardcoded throughout the codebase. I would very much like to use Podman as a finally proper container launcher in production (non-FAANG scale - at which you maybe start to need k8s), but having an unnecessary daemon moving part in thousands lines of C makes me frown so far.
op00to 6y ago
Every complaint here boils down to: “it’s not docker, it doesn’t work exactly like docker, therefore it’s bad”.
- jeffbee 6y ago
  Which is a remarkable position considering how bad docker is. You’d think that not being docker would be a feature by itself.
curt15 6y ago
The ability to start containers as a normal user ("rootless mode") is quite interesting and imo the most significant feature missing from LXD, which requires you to either use sudo or be a part of the "lxc" group (equivalent to having sudo privileges).
- mixmastamyk 6y ago
  Wouldn't the lxc group have much more focused permissions than sudo?
  - ta988 6y ago
    You can mount the root fs in a container and with uid mapping and because the daemon is running as root you can pretty much do whatever you want.
- hda111 6y ago
  You can afair configure LXD to disallow escalating privileges via group membership.
rq1 6y ago
What’s the difference with systemd-nspawn? (with or without -U)
- eeZah7Ux 6y ago
  ...that nspawn is already available on 90% of Linux systems.
- Klasiaster 6y ago
  Podman can run under your user and use your UID (e.g., 1000) as UID 0 for the root user in the container. Any other user in the container gets mapped to a temporary unprivileged UID on your system (the range to pick from is specified in /etc/subuid).
- mekster 6y ago
  How are you meant to pull package from docker hub with nspawn?
  - rq1 6y ago
    I don’t. I’m using Nix containers.
yamrzou 6y ago
I you'd like to know what it is like to use Podman, I've found those terminal recordings by Matthew Heon (Podman contributor) quite helpful: https://asciinema.org/~mheon https://asciinema.org/~mheon
symlinkk 6y ago
Who cares? It’s one command to start the daemon.
superbcarrot 6y ago
Can anyone explain what daemonless means in this case and what the advantages are? (I don't work in this space and my knowledge ends with knowing what a container is.)
- _joel 6y ago
  You don't need root access to run the containers
- 8fingerlouie 6y ago
  IIRC, docker by default runs a daemon as root, which spawns all your containers. It is possible to run it rootless as well (https://docs.docker.com/engine/security/rootless/ https://docs.docker.com/engine/security/rootless/), though you’d need a separate daemon for each user you’d want to run containers as. Podman doesn’t have that. It spawns containers without the help of a controlling daemon, and can spawn containers both as root and rootless. Rootless is of course a fairly big deal considering if you run docker containers as root, and runc has a vulnerability, you could potentially escape the container and become root, where a rootless installation would just let you escape to whatever user is running the container.
  - mekster 6y ago
    So, it's only about a tiny security concern but it got much bigger usability problems and what's worse RedHat is forcing people to use podman since RHEL 8 when it's not even ready.
yobert 6y ago
I've been trying to use podman in production and it is not working very well. I'm excited for this technology but it is not ready. - podman layer caching during builds do not work. When we switched back to docker-ce, builds went from 45 minutes to 3 minutes with no changes to our Dockerfile - fuse-overlayfs sits at 100% CPU all day, every day on our production servers - podman loses track of containers. Sometimes they are running but don't show in podman ps - sometimes podman just gets stuck and can't prune or stop a container (edit: formatting)
- yobert 6y ago
  Reading through the comments though it looks like I should try making a BTRFS mount for the container storage and that might help the fuse-overlayfs problems though.
  - tw04 6y ago
    I know it tends to be a controversial technology around here, but I've had great luck with OpenZFS for container storage.
  - znpy 6y ago
    It feels weird that a containerisation technology imposes requirements on the underlying storage technology to work well, tho. Now I understand that btrfs is not really a requirement. But I've been thinking about using podman to replace docker-in-docker on our fleet of gitlab runners where we build our images, and running btrfs is a deal breaker. We really don't want to add complexity to the mix. Fuse-overlay burning through the whole CPU is not really good looking either. Mh. I still have to properly dig deep into this, but I'm keeping my hopes high.
    - carlosf 6y ago
      Have you tried Kaniko? That's how I got hid of DinD in my Gitlab Runners.
      - horstmeyer 6y ago
Cu3PO42 6y ago
I have recently found out you can actually run Docker containers with LXC as well. I've yet to get deep into it, but it seems to work fine*. * Manual config tweaking may be required.
- linkdd 6y ago
  IIRC, Docker's first versions were only a high-level API to LXC. containerd/runc/... came after.
Brian_K_White 6y ago
100 years ago I wrote a simple sysv init script that started/stopped/queried/enabled/disabled lxc containers, graceful host shutdown (all containers shut down gracefully before the host shuts down), you could start/stop/query/enable/disable individual containers as well as the the meta-service of all containers, all from a few k of bash, and no daemon, not even in the form of a shell loop that stayed running. Right about then systemd came along (as in, my distro at the time adopted it) and essentially required a service to monitor (some process, tcp port, or file to monitor), but my efficient little script had no service by design, because none was needed and I hate making and running anything that isn't needed to get a given job done. (Plus about that time my company went whole-hog into vmware and a managed service provider and so my infant home-grown lxc containers on colo hardware system never grew past that initial tiny poc) I laugh so many years later reading "Hey! Check it out! You don't need no steeenkeeeng daemon to run some containers!"
maweki 6y ago
One very interesting piece of tech coming from it, is toolbox (https://github.com/containers/toolbox https://github.com/containers/toolbox). Basically throwaway (or keeparound) rootless containers with their own root directory but shared HOME. Install hundreds of dev-dependencies to build this one piece of software? Yeah, not gonna install those packages permanently. Spin up a toolbox, build it, install it in my home/.local. You have root in the container without having root in the host system. That takes care of a lot of issues as well.
- johnchristopher 6y ago
  I usually share a volume between containers (eg:: a volume for wp-cli cache, another for -g npm_modules). What benefits would toolbox add ?
  - dnautics 6y ago
    a lot of software expects a usable home directory. There was another type of containerization (syos) which was designed for HPC use cases (think deploying 10k nodes doing one thing many times in parallel divvying up and pulling a shared dataset on a academic or industry (pharma), cluster with a high performance distributed filesystem underneath) that did this, however syos is not appropriate for most webscale use.
  - maweki 6y ago
    You could, for example, host the whole dev environment for a project in a container and still develop on the code in home.
    - johnchristopher 6y ago
      Ah, I see ; put the whole dev toolchain in the container. I use bindfs to mount the volume. I have a $HOME/Dev folders with WPProjectA, WPProjectB folders. Each has a volume subfolder mounted like that (the script has more variables but that's the gist of it): /usr/bin/bindfs \ --force-user=johnchristopher \ --force-group=johnchristopher \ --create-for-user=www-data \ --create-for-group=www-data \ /var/lib/docker/volumes/WPProjectA-web/_data \ $HOME/Dev/WPProjectA/volume This setup allows using VSCode+xdebug and editing the code in the mounted volume while running the container and remote debugging.
minieggs 6y ago
I have a container I stick in Syncthing that I do 100% of my development within across all my machines. One day I decided to switch to podman on a whim and forgot I made the switch. That was about nine-ish months ago.
westurner 6y ago
Is the title of this page out of date? AFAIU, Podman v3 has a docker-compose compatible socket and there's a daemon; so "Daemonless Container Engine" is no longer accurate. "Using Podman and Docker Compose" https://podman.io/blogs/2021/01/11/podman-compose.html https://podman.io/blogs/2021/01/11/podman-compose.html
- Conan_Kudo 6y ago
  Podman does not run in a daemon-mode by default. There's a dbus-activated service unit that activates the daemon when you try to use Docker Compose with it.
TwoNineFive 6y ago
I recently passed the RHCSA exam, which as of last October added a containers/podman section to the test. I usually work with lxcs under libvirt and directly, so working with podman was new to me. The test requirements are very simplistic, but I went further and spent a decent amount of time finding annoyances and issues. I am not impressed with podman. It's buggy and slow. Documentation is very basic and the underlaying mechanics are not covered where needed. Want to reconfigure the command parameters your start your container with to add a exported variable? Gotta remake the entire setup because there's no edit command.
- ta988 6y ago
  I really enjoy working with LXC. And the fact that I can switch now between container mode or VM mode makes it even better. That and best ng able to do lxc in lxc easily to test an infrastructure locally.
gvalkov 6y ago
It's great to see how far Podman (and its sister projects) have come. I think it's a reliable tool and I'm a happy user both personally and professionally. We make heavy use of Podman in our infrastructure and it's mostly a pleasure. My current pet peeves are that: 1) Ansible's podman_container module is not as polished as docker_container. I regularly run into idempotency issues with it (so lots of needlessly restarted containers). 2) Gitlab's Docker executor doesn't support Podman and all our CI agents run on CentOS 8. I ended up writing a custom executor for it and it's working quite well though (we're probably not going back to the container executor even if it supported Podman, since the custom executor offers so much more flexibility). 3) GPU support is easier/more documented on Docker. For this reason, the GPU servers we have are all Ubuntu 20.04 + Docker since it's the more beaten path. 4) Podman-compose just needs more work. Luckily for us, it seems that Podman 3.x will support docker-compose natively [1]. As mentioned, our CI environment is very dependent on Podman. The first step of every Gitlab pipelines is to build the container image in which the rest of the jobs will run. I find that it's simpler to have a shell executor in a unprivileged, restricted environment (i.e. can only run `podman build`) than setting up dind just for building images. All jobs that follow are ran in rootless containers, for that nice added layer of security. Wishing all the best to the Podman, Buildah and Skopeo teams. [1]: https://www.redhat.com/sysadmin/podman-docker-compose https://www.redhat.com/sysadmin/podman-docker-compose
barosl 6y ago
I tried to use Podman a few months ago, but the time spent to execute the "podman" command was considerably higher than that of the "docker" command. I am worried if this is a fundamental problem of the daemonless design. If that is the case, it would not be possible to solve the speed problem of Podman.
hardwaresofton 6y ago
I was a big fan of Podman and switched to it wholeheartedly[0][1] but after a while I ran into a few issues that made me have to switch off of it and back to docker: - image unpacking issues (invalid tar header) - linking containers isn't supported [EDIT] - I wrote up the post real quick[2] [0]: https://vadosware.io/post/rootless-containers-in-2020-on-arch-linux/ https://vadosware.io/post/rootless-containers-in-2020-on-arc... [1]: https://vadosware.io/post/bits-and-bobs-with-podman/ https://vadosware.io/post/bits-and-bobs-with-podman/ [2]: https://vadosware.io/post/back-to-docker-after-issues-with-podman/ https://vadosware.io/post/back-to-docker-after-issues-with-p...

321 points by lobo_tuerto 6y ago

zeroxfe 6y ago

Anyone have experience with this? I love the idea of it being daemonless.

edwintorok 6y ago
If you run it as non-root it is significantly slower than docker as root. Docker can us the overlay2 kernel driver, whereas podkan would use fuse-overlayfs in userspace. This has a high CPU overhead (e.g. don't try to run AFL inside podman), and a 1024 FD limit for the entire container (so a 'make -j40' usually dies). There are ways around it: raise the ulimit for your user and run new enough podman to raise limit for fuse-overlayfs, use the 'vfs' driver (it has other perf issues). I heard (but haven't tested yet) that the 'btrfs' driver avoids all these problems and works from userspace. Obviously requires an FS formatted as btrfs... There are also compatibility issues with Docker. E.g. one container was running sshfs inside Docker just fine, but fails with a permission error on /dev/fuse with podman.
- stuaxo 6y ago
  Are these fuse limitations or fuse-overlayfs limitations ?
  - edwintorok 6y ago
    I think FUSE limitations: fuse is served by a single userspace process, which is limited the same way as any other userspace process by ulimit. It is not a fundamental limitation of podman, just of podman's default rootless behaviour.
- SEJeff 6y ago
  You can run podman as root, but it doesn't default to it, for generally sensible security reasons. Also, docker runs as root, so it won't have permissions problems. You can change the permissions of /dev/fuse if you want to allow podman containers to access it or update the group of the user launching podman.
- eriksjolund 6y ago
  If I understand correctly support for native rootless mounts is currently under development: https://github.com/containers/storage/pull/816 https://github.com/containers/storage/pull/816 The functionality requires Linux kernel 5.11 (soon to be released)
  - fatherlinnux 6y ago
    That is correct. We're targeting that work for RHEL 8.5 in November-ish, so you'll likely see that drop in Fedora and other Linux distros sooner.
snickms 6y ago
I've used podman-compose for a few web projects. Rootless means you can't bind to privileged ports etc. but after a bit of fiddling I can now spin up my choice of DB/server/mock pretty quickly - just like I could with docker. The lack of 'cannot connect to dockerd' mysteries makes for a much-improved developer experience if you ask me.
sevagh 6y ago
I have a Makefile for a Rust project which binds the local repository to a Docker volume, builds it in the container using muslrust, and then does a chown to change the target directory back from root ownership to my own user. All I had to do was 's/docker/podman/g' and remove the chown hack and it works fine: https://github.com/sevagh/pq/commit/6acf6d05a094ac2959567a9ab989cca1e9e78736#diff-76ed074a9305c04054cdebb9e9aad2d818052b07091de1f20cad0bbac34ffb52 https://github.com/sevagh/pq/commit/6acf6d05a094ac2959567a9a... It understands Dockerfiles and can pull images from Dockerhub.
thechao 6y ago
I’m a GPU driver dev and you guys are talking ~alien~, right now. What does “daemonless” mean? Why is that good?
- deleted 6y ago
  [deleted]
- shaded-enmity 6y ago
  In Docker each container process is a child of the Docker daemon process. If you need to apply security patches to the Docker daemon it kills all your running containers.
  - raesene9 6y ago
    Technically the parent process for a contained process with Docker is containerd-shim. Also, Docker does support live restore if you want to keep containers running over daemon restarts https://docs.docker.com/config/containers/live-restore/ https://docs.docker.com/config/containers/live-restore/
raesene9 6y ago
Out of curiosity, why do you prefer daemonless setups?
- georgyo 6y ago
  If I have 30 containers running, why should a single daemon being restarted cause all 30 to shutdown as well? Similarly, the docker security model is that there isn't a security model. If you can talk to the docker socket you have what ever privileges the daemon is running as.
  - raesene9 6y ago
    So Docker supports live restore https://docs.docker.com/config/containers/live-restore/ https://docs.docker.com/config/containers/live-restore/ which addresses the first point. Second point, yep if you run Docker as root and someone can access the socket file they get root. If that's a concern, you can run Docker rootless. And as we're talking file permissions on a local host to allow that access, the same applies to podman containers does it not? If there are permission issues allowing another user to use access the container filesystems, you have the same problem.
    - zaarn 6y ago
      Rootless Docker is basically a joke, I've tried to run production workloads on it for about a year before I gave up. Numerous docker images will refuse to work, a bunch of things will subtly fill your error logs with warnings and it doesn't mesh well with running docker swarm at all.
      - raesene9 6y ago
        Rootless docker only left experimental status with 20.10 which came out in December 2020, so maybe they would have addressed some of those issues... As to swarm, I was comparing Docker rootless to podman, which is more a developer use case than prod. container clusters.
- 6y ago
igneo676 6y ago
I had pretty negative experiences with it. I'm not sure how much of my experience is _actually_ the fault of Podman and how much of it was around using it as a drop-in replacement for Docker. It baffled me though and I'm a long time Docker and Linux user. I used Arch Linux and followed their docs[1], mainly because I wanted it to also run without root. Big mistake. Attempting to run a simple postgres database w/ and w/o root permissions for Podman resulted in my system getting into an inconsistent state. I was unable to relaunch the container due to the state of my system. I mucked around with runc/crun and I ended up literally nuking both the root and non-root directories that stored them on my computer. I reset my computer back to Podman for root only, hoping that I had just chosen a niche path. It still would leave my system broken, requiring drastic measures to recover. No thanks. After much debugging, finally switching back to Docker, I realized my mistake: I had forgotten a required environment variable. Silly mistake. Docker surfaced the problem immediately. Podman did not. Docker recovered gracefully from the error. Podman left my system in an inconsistent state, unable to relaunch the container and unable to remove it Again, I'm not sure how much of my experience was just trying to force Podman into a square hole? I'm sure there are other people that make it work just fine for their use cases Edit: I should note that I used it as a docker-compose replacement, which is probably another off-the-beaten path usage that made this more dramatic than it should have been 1. https://wiki.archlinux.org/index.php/Podman https://wiki.archlinux.org/index.php/Podman
- dnautics 6y ago
  I don't know when you tried this but I recall it having sucked in the summer when I tried it and I gave up. New job in the fall, decided to give podman a shot "if it doesn't work on the first try I'm going to use docker", then podman-compose 3rd party instructions (I can't remember which) just works out of the box. I'm running a fleet that includes postgres and redis.
perlgeek 6y ago
One huge advantage over docker is that if you mount a directory into the container, and the container writes files there, on the host system they always have the owner of the user that started the container. That makes it much more convenient for build environments, without having to hard-code user IDs both in the container and on the host.
- Spivak 6y ago
  If you're on a Linux system you can actually make this work better with sssd. So sssd's architecture is actually client-server over a unix socket. So all you need to do is create a very simple base container layer that just installs sssd-client, and wires up /etc/nsswitch.conf to use it (your package manager will almost surely do this automatically). Then just bind mount the sssd socket into your container and boom, all your host users are in the container. If you already log in with sssd you're done. But if you only use local users then you'll need to configure the proxy provider so that sssd reads from your passwd on your host. In this case the host system doesn't actually have to use sssd for anything.
  - perlgeek 6y ago
    Sounds interesting, do you have a link to an example where this is done/demonstrated? (Also I'm not sure how that's better (and not just different), except maybe it allows more than one host user in the container, but I haven't had a use case for that).
  - kortex 6y ago
    This is glorious and just the thing I was looking for. I am trying to move towards an even more container based dev environment, basically shell into long running containers. Maybe even window manager in docker . Totally eliminates dependency hell, e.g. ROS heavy workflows where it wants to control every part of your environment.
shaicoleman 6y ago
I tried it, and went back to Docker with BuildKit. * The newer version of Docker+BulidKit supports bind-mounts/caching/secrets/memory filesystem etc. we used it to significantly speed up builds. You could probably find a way in buildah to achieve the same things, but it's not standard. * Parallel builds - Docker+BulidKit builds Dockerfile in parallel, in podman things run serially. The combination of caching and parallel builds and Docker being faster even for single-threaded builds, Docker builds ended up an order of magnitude faster than the Podman ones. * Buggy caching - There were a lot of caching bugs (randomly rebuilding when nothing changed, and reusing cached layers when files have changed). These issues are supposedly fixed, but I've lost trust in the Dockerfile builds. * Various bugs when used as a drop-in replacement for Docker. * Recurring issues on Ubuntu. It seemed all the developers were on Fedora/RHEL, and there were recurring issues with the Ubuntu builds. Things might be better now. * non-root containers require editing /etc/subuid, which you can't do unless you have root access. More information about the new BuildKit features in Docker: https://pythonspeed.com/articles/docker-buildkit/ https://pythonspeed.com/articles/docker-buildkit/
- feanaro 6y ago
  FWIW, I'm using Podman and I'm happy with it given that it's fast enough for my purposes and I haven't encountered any caching bugs.
- lima 6y ago
  Buildkit's caches are a killer feature. They go a long way towards making Docker builds usable for iterative development. We also migrated back to Docker from Podman because of Buildkit and the general bugginess of Podman.
jka 6y ago
From limited experience: podman and the associated toolset (buildah, skopeo) show a lot of promise. That said, they are evolving and may currently require administrative care and attention when used in production. If your environment would benefit from smaller, isolated containerization tools then my two cents would be that it's worth keeping an eye on these as they mature, and perhaps perform early evaluation (bearing in mind that you may not be able to migrate completely, yet). The good: - Separation of concerns; individual binaries and tools that do not all have to be deployed in production - A straightforward migration path to build containers from existing Dockerfiles (via "buildah bud") - Progressing-and-planned support for a range of important container technology, including rootless and multi-architecture builds
dnautics 6y ago
I've been using my company's docker compose script transparently with podman-compose for three months now and I have basically forgotten that I've switched. I actually didn't use docker before this, so I can't comment on what the difference is. I feel like the podman experience is binary, either it works perfectly or it sucks balls. My suggestion is to give it a try, if it fails, then maybe file a bug report, fail fast and fall back on docker.

Datenstrom 6y ago

Glad to see Podman here, worked with it at a past job where services were deployed on RedHat. Converting all existing deployment scripts to podman was as easy as do a recursive search and replace `s/docker/podman/` and there was no learning curve as all the commands were the same. By default root is not required which is great also.

ddtaylor 6y ago

I liked podman for what I was working on with it. The only hangup I found was that it couldn't do networking related routing? Does anyone know more?

dathinab 6y ago
As far as I know it can do so but not when running rootless, I also don't thing it's currently possible to be done rootless. You still can run podman as root without damon. Or you can run a podman deamon (through systemd) which is even compatible with docker-compose but has most of the drawbacks of running docker.
gvalkov 6y ago
Quite a lot is possible with CNI [1]. For example, we use this setup to give real IPs to containers: # /etc/cni/net.d/testnet.conflist { "cniVersion": "0.4.0", "name": "testnet", "plugins": [ { "type": "bridge", "bridge": "br0", # main host interface is part of this bridge "ipam": { "type": "host-local", "subnet": "10.0.0.0/16", "gateway": "10.0.0.1", "routes": [{ "dst": "0.0.0.0/0"}] } } ] } You can then start a container and operate on its network namespace for added flexibility: podman run -it --net testnet --ip 10.0.0.2 ... ns=$(basename $(podman inspect $id | jq -r '.[0] .NetworkSettings .SandboxKey')) ip netns exec $ns ip route add ... [1]: https://github.com/containernetworking/cni https://github.com/containernetworking/cni

daitangio 6y ago

Happy to see docker alternative, but the road is still long for it to replace docker

dathinab 6y ago
I don't think so. It's doing a better job for most of the use cases I had recently. You can run it rootless which makes it a far better use cases for a lot of dev use cases (where you don't need to emulate a lot of network aspects or heavy filesystem usage). Also you can run it as root without deamon which is much better for a bunch of use cases and eliminates any problems with fuse-overlayfs. This is the best way to e.g. containerize systemd services and similar (and security wise still better as e.g. it still works well with the audit subsystem). If worse comes to worse you can run it as a deamon in which case it's compatible with docker compose, and has more or less the same problems. In my experience besides some integration testing use-cases is either already a good docker replacement without running it as a deamon, or the use case shouldn't be handled by docker either way (sure there are always some exceptions). Lastly I had far less problems with podman and firewalls then with docker but I haven't looked into why that's the case. (But it seems to be related to me using nftables instead of iptables).
long_warmup 6y ago
I use podman in my projects, no problems. And since cli is compatible with docker all my work I have done using docker simply runs with podman.

shyim 6y ago

I still miss an easy way to setup multiple containers in a single network like with docker-compose. podman-compose is not really useable.

Spivak 6y ago
An Ansible playbook with the podman_container module is my go-to. It's not as easy because it's less magic but it's also less magic which I count as a win.
dathinab 6y ago
You can use docker-compose with podman. Through it has similar drawbacks as using docker-compose with docker. As far as I know there is currently no rootless way on Linux to setup custom internal networks like docker compose does (hence why podman-compose doesn't support such things).
- chippiewill 6y ago
  There definitely is a way as creating custom networks works perfectly fine with rootless docker. I'm surprised if podman doesn't support that since it uses rootlesskit (same as rootless docker) which does the heavy lifting for rootless networking.
pydry 6y ago
What's wrong with it?
- dathinab 6y ago
  With docker all container have their own ip address and (potential) internal DNS name. But this can't be done rootless. So with rootless podman all container map to the same ip address but different ports. This is for some use cases (e.g. spinning up a DB for integration testing) not a problem at all. For others it is. More over you can run multiple groups of docker containers in separate networks, you can't do so with rootless podman. Through you can manage networks with rootfull podman (which still has no deamon and as such works better with e.g. capabilities and the audit sub system then docker does). Through to get the full docker-compose experience you need to run it as a deamon (through systemd) in which case you can use docker-compose with podman but it has most of the problems docker has.

osipov 6y ago

Any plans for Docker to offer a daemonless version?

acec 6y ago

It is almost 100% compatible with docker. I only had to do minimal changes in my Dockerfiles to use it.

lmeyerov 6y ago
... but not docker-compose
lima 6y ago
But not Buildkit and docker-compose.
- westurner 6y ago
  Podman v3 is compatible with docker-compose (but not yet swarm mode, FWIU), has a socket and a daemon that services it. Buildah (`podman buildx`, `buildah bud --arch arm64`) just gained multiarch build support; so also building arm64 containers from the same Dockerfile is easy now. https://github.com/containers/buildah/issues/1590 https://github.com/containers/buildah/issues/1590 IDK what BuildKit features should be added to Buildah, too?
  - lima 6y ago
    > IDK what BuildKit features should be added to Buildah, too? Cache mounts are what we rely on for incremental rebuilds.

Xophmeister 6y ago

Isn't Singularity[1] also daemonless? [1] https://sylabs.io/ https://sylabs.io/

SEJeff 6y ago
Yes, but singularity is a gigantic suid root binary where as podman uses user namespaces for unprivileged stuff.

geerlingguy 6y ago

What's the status on ease of running on Mac? I know last time I seriously considered testing it out in my workflow, it was kind of crazy that I had to have a separate machine (VM or real) just to run container images... I see in the docs [1]: "Podman is a tool for running Linux containers. You can do this from a MacOS desktop as long as you have access to a linux box either running inside of a VM on the host, or available via the network. You need to install the remote client and then setup ssh connection information." And yes, I know that Docker (and any other Linux container-based tool) also runs a VM—but it sets it up almost 100% transparently for me. I install docker, I run `docker run` and I have a container running. [1] https://podman.io/getting-started/installation#macos https://podman.io/getting-started/installation#macos

throwaway894345 6y ago
The docker machine or whatever it’s called these days is pretty impressive but when we’ve tried to use it in developer workflows on mac it would eat all of the CPU allotted to it in order to marshal filesystem events over the host/guest VM boundary. We tried a number of workarounds but none solved the problem. I’m sure it’s possible, but we ultimately gave up and moved back to developing against native processes on MacOS. For other use cases I imagine it’s just fine.
- dzolvd 6y ago
  this is an issue I have battled as well. Docker is great for normalizing the execution environment; which should be a huge boon for developer tooling but on MacOS having your laptop sound like a hovercraft just to have some files watched for hot rebuilds is no bueno.
eadmund 6y ago
> I know last time I seriously considered testing it out in my workflow, it was kind of crazy that I had to have a separate machine (VM or real) just to run container images... Why does that seem 'kind of crazy' to you? A container is really just a namespaced Linux process, so … you either need a running Linux kernel or a good-enough emulation thereof. What seems kind of crazy to me is that so many folks who are deploying on Linux develop on macOS. That's not a dig against macOS, but one against the inevitable pains which arise when developing in one environment and deploying in another. Even though I much prefer a Linux environment, it would seem similarly crazy to develop on Linux and deploy on Windows or macOS. Heck, I think it is kind of crazy to develop on Ubuntu and deploy on Debian, and they are generally very very close!

corty 6y ago

Even less IPv6 support than docker. With docker you can at least get it to work somehow, even if it is totally different from IPv4, weirdly. Podman just has no IPv6 support to speak of.

3np 6y ago
For airgapped networks, this is great.
flowless 6y ago
Not entirely true. I was researching this few hours ago and according to https://github.com/containers/podman/issues/6114#issuecomment-632409804 https://github.com/containers/podman/issues/6114#issuecommen... it just works when you add another network. Docker registry having no IPv6 is another fun story tho.
- corty 6y ago
  Well, from that issue from 2020-12-24: "--ipv6 has not landed yet", so you cannot assign static external v6 adresses to a container. No activity on that bug since. There is tons of stuff in docker that works automatically, where in podman it is like "Please be aware you have to configure ip6tables (or whatever your OS firewall is) for forwarding.". Yeah, well, if i have to do everything by hand or program a ton of scripts around it, it is quite useless, isn't it? IPv6 support is always like this: half-baked at best, someone got it to work somehow, developers declare it done since it worked for someone. Then crickets... IPv6 support isn't done until you can replace IPv4 with it, with no changes to your config except addresses. Even docker isn't there yet. And podman's is still in a larval state.

lifeisstillgood 6y ago

I redid some docker builds yesterday with podman, and filled up my laptops 99GB entirely with layers. Docker was handling it somehow better. I used --squash, cleaned up the various apt-gets but boy it made me realise that for every abstraction we get leakage - sometimes serious leakage

SEJeff 6y ago

Under the covers, both podman AND docker use runc. Redhat is writing a new version named "crun" which is lower overhead and faster: https://github.com/containers/crun https://github.com/containers/crun

cashewchoo 6y ago
Adduser vs useradd is older than I am and I still can't keep them straight. Just put this with the rest of the fire, I guess, heh!
- lrossi 6y ago
  Indeed. After 30 years of Linux, we should have got better at naming things.
- SEJeff 6y ago
  I think it is sort of a play on words. Since runc is written in go and is slower, crun is written in c and is faster. Stupid nerd joke I know.
lrossi 6y ago
Upside: At least they used a different name. On other occasions, they just hijack the name. Like they did with dstat.
- dralley 6y ago
  That was a misunderstanding where they saw that there had been no commit activity for 3 years, and no responses to issues/PRs filed in the past 18 months, and assumed the upstream was dead. Making that assumption was reasonable - moving forwards without at least making an attempt to contact the maintainer "just in case" was not. It would have been courteous to at least try. But it's not exactly the deliberate hostile takeover that you make it out to be. https://bugzilla.redhat.com/show_bug.cgi?id=1614277#c9 https://bugzilla.redhat.com/show_bug.cgi?id=1614277#c9 >> To my knowledge, there is no need (legally) to obtain consent to use the name 'dstat' for a replacement command providing the same functionality. It might be a nice thing to do from a community perspective, however - if there was someone to discuss with upstream. >> However, dstat is dead upstream. There have been no updates for years, no responses at all to any bug reports in the months I've been following the github repo now, and certainly no attempt to begin undertaking a python3 port. >> Since there is nobody maintaining the original dstat code anymore, it seemed a futile exercise to me so I've not attempted to contact the original author. And as pcp-dstat is now well advanced beyond the original dstat - implementing features listed in dstat's roadmap for many years, and with multiple active contributors - I think moving on with the backward-compatible name symlink is the best we can do.

runeks 6y ago

Sincere question: why would I want to use Podman over Docker? What does being "daemonless" actually buy me in practice?

lmeyerov 6y ago
$ for your favorite infra oss startup or volunteer rhel ~forces you to do podman in rhel 8+. They make it hard enough to do a docker install that corp policies will reject docker (non-standard centos 7 repos etc), while allowing ibm's thing. Ex: Locks podman over docker for a lot of US gov where rhel 8 is preferred and with only standard repos, and anything else drowns in special approvals. subtle and orthogonal to podman's technical innovations.. but as a lot of infra oss funding ultimately comes from what enterprise policies allow, this a significant anti-competitive adoption driver for podman by rhel/ibm.
- op00to 6y ago
  Because people can’t install non-Red Hat software due to internal policy, that’s somehow anti-competitive?
  - lmeyerov 6y ago
    The main repos have plenty of non-rhel sw, and people can expend social capital working around IT policies, but that doesn't change it being against the grain. We are fine because we use Ubuntu for the generally better GPU support, but I don't have control over the reality of others. I repeatedly get on calls with teams where this comes up, and I feel sorry for the individuals who are stuck paying the time etc. cost for this. Docker for RHEL 8 is literally installing Docker for Centos 7, so this isn't a technology thing, but political: A big corporate abusing their trusted neutral oss infra position to push non-neutral decisions in order to prop up another part of its business. This is all bizarre to operators who barely know Docker to beginwith, and care more about say keeping the energy grid running. ("The only authorized thing will likely not work b/c incompatibilities, and I have to spend 2w getting the old working thing approved?") And again, none of this is technical. Docker is great but should keep improving, so competitive pressure and alternatives are clearly good... but anti-competitive stuff isn't.

williesleg 6y ago

Yay IBM!

fillest 6y ago

Well, "daemonless" is kind of marketing - there is still this daemon-per-container 'conmon' thing https://github.com/containers/conmon https://github.com/containers/conmon and I don't get why it is needed because 1) who actually needs to re-attach anyway? 2) container's streams are already properly handled by whatever supervisor (e.g. systemd). You can't disable conmon and I'm not sure if its usage is not hardcoded throughout the codebase. I would very much like to use Podman as a finally proper container launcher in production (non-FAANG scale - at which you maybe start to need k8s), but having an unnecessary daemon moving part in thousands lines of C makes me frown so far.

op00to 6y ago

Every complaint here boils down to: “it’s not docker, it doesn’t work exactly like docker, therefore it’s bad”.

jeffbee 6y ago
Which is a remarkable position considering how bad docker is. You’d think that not being docker would be a feature by itself.

curt15 6y ago

The ability to start containers as a normal user ("rootless mode") is quite interesting and imo the most significant feature missing from LXD, which requires you to either use sudo or be a part of the "lxc" group (equivalent to having sudo privileges).

mixmastamyk 6y ago
Wouldn't the lxc group have much more focused permissions than sudo?
- ta988 6y ago
  You can mount the root fs in a container and with uid mapping and because the daemon is running as root you can pretty much do whatever you want.
hda111 6y ago
You can afair configure LXD to disallow escalating privileges via group membership.

rq1 6y ago

What’s the difference with systemd-nspawn? (with or without -U)

eeZah7Ux 6y ago
...that nspawn is already available on 90% of Linux systems.
Klasiaster 6y ago
Podman can run under your user and use your UID (e.g., 1000) as UID 0 for the root user in the container. Any other user in the container gets mapped to a temporary unprivileged UID on your system (the range to pick from is specified in /etc/subuid).
mekster 6y ago
How are you meant to pull package from docker hub with nspawn?
- rq1 6y ago
  I don’t. I’m using Nix containers.

yamrzou 6y ago

I you'd like to know what it is like to use Podman, I've found those terminal recordings by Matthew Heon (Podman contributor) quite helpful: https://asciinema.org/~mheon https://asciinema.org/~mheon

symlinkk 6y ago

Who cares? It’s one command to start the daemon.

superbcarrot 6y ago

Can anyone explain what daemonless means in this case and what the advantages are? (I don't work in this space and my knowledge ends with knowing what a container is.)

_joel 6y ago
You don't need root access to run the containers
8fingerlouie 6y ago
IIRC, docker by default runs a daemon as root, which spawns all your containers. It is possible to run it rootless as well (https://docs.docker.com/engine/security/rootless/ https://docs.docker.com/engine/security/rootless/), though you’d need a separate daemon for each user you’d want to run containers as. Podman doesn’t have that. It spawns containers without the help of a controlling daemon, and can spawn containers both as root and rootless. Rootless is of course a fairly big deal considering if you run docker containers as root, and runc has a vulnerability, you could potentially escape the container and become root, where a rootless installation would just let you escape to whatever user is running the container.
- mekster 6y ago
  So, it's only about a tiny security concern but it got much bigger usability problems and what's worse RedHat is forcing people to use podman since RHEL 8 when it's not even ready.

yobert 6y ago

I've been trying to use podman in production and it is not working very well. I'm excited for this technology but it is not ready. - podman layer caching during builds do not work. When we switched back to docker-ce, builds went from 45 minutes to 3 minutes with no changes to our Dockerfile - fuse-overlayfs sits at 100% CPU all day, every day on our production servers - podman loses track of containers. Sometimes they are running but don't show in podman ps - sometimes podman just gets stuck and can't prune or stop a container (edit: formatting)

yobert 6y ago
Reading through the comments though it looks like I should try making a BTRFS mount for the container storage and that might help the fuse-overlayfs problems though.
- tw04 6y ago
  I know it tends to be a controversial technology around here, but I've had great luck with OpenZFS for container storage.
- znpy 6y ago
  It feels weird that a containerisation technology imposes requirements on the underlying storage technology to work well, tho. Now I understand that btrfs is not really a requirement. But I've been thinking about using podman to replace docker-in-docker on our fleet of gitlab runners where we build our images, and running btrfs is a deal breaker. We really don't want to add complexity to the mix. Fuse-overlay burning through the whole CPU is not really good looking either. Mh. I still have to properly dig deep into this, but I'm keeping my hopes high.
  - carlosf 6y ago
    Have you tried Kaniko? That's how I got hid of DinD in my Gitlab Runners.
    - horstmeyer 6y ago

Cu3PO42 6y ago

I have recently found out you can actually run Docker containers with LXC as well. I've yet to get deep into it, but it seems to work fine*. * Manual config tweaking may be required.

linkdd 6y ago
IIRC, Docker's first versions were only a high-level API to LXC. containerd/runc/... came after.

Brian_K_White 6y ago

100 years ago I wrote a simple sysv init script that started/stopped/queried/enabled/disabled lxc containers, graceful host shutdown (all containers shut down gracefully before the host shuts down), you could start/stop/query/enable/disable individual containers as well as the the meta-service of all containers, all from a few k of bash, and no daemon, not even in the form of a shell loop that stayed running. Right about then systemd came along (as in, my distro at the time adopted it) and essentially required a service to monitor (some process, tcp port, or file to monitor), but my efficient little script had no service by design, because none was needed and I hate making and running anything that isn't needed to get a given job done. (Plus about that time my company went whole-hog into vmware and a managed service provider and so my infant home-grown lxc containers on colo hardware system never grew past that initial tiny poc) I laugh so many years later reading "Hey! Check it out! You don't need no steeenkeeeng daemon to run some containers!"

maweki 6y ago

One very interesting piece of tech coming from it, is toolbox (https://github.com/containers/toolbox https://github.com/containers/toolbox). Basically throwaway (or keeparound) rootless containers with their own root directory but shared HOME. Install hundreds of dev-dependencies to build this one piece of software? Yeah, not gonna install those packages permanently. Spin up a toolbox, build it, install it in my home/.local. You have root in the container without having root in the host system. That takes care of a lot of issues as well.

johnchristopher 6y ago
I usually share a volume between containers (eg:: a volume for wp-cli cache, another for -g npm_modules). What benefits would toolbox add ?
- dnautics 6y ago
  a lot of software expects a usable home directory. There was another type of containerization (syos) which was designed for HPC use cases (think deploying 10k nodes doing one thing many times in parallel divvying up and pulling a shared dataset on a academic or industry (pharma), cluster with a high performance distributed filesystem underneath) that did this, however syos is not appropriate for most webscale use.
- maweki 6y ago
  You could, for example, host the whole dev environment for a project in a container and still develop on the code in home.
  - johnchristopher 6y ago
    Ah, I see ; put the whole dev toolchain in the container. I use bindfs to mount the volume. I have a $HOME/Dev folders with WPProjectA, WPProjectB folders. Each has a volume subfolder mounted like that (the script has more variables but that's the gist of it): /usr/bin/bindfs \ --force-user=johnchristopher \ --force-group=johnchristopher \ --create-for-user=www-data \ --create-for-group=www-data \ /var/lib/docker/volumes/WPProjectA-web/_data \ $HOME/Dev/WPProjectA/volume This setup allows using VSCode+xdebug and editing the code in the mounted volume while running the container and remote debugging.

minieggs 6y ago

I have a container I stick in Syncthing that I do 100% of my development within across all my machines. One day I decided to switch to podman on a whim and forgot I made the switch. That was about nine-ish months ago.

westurner 6y ago

Is the title of this page out of date? AFAIU, Podman v3 has a docker-compose compatible socket and there's a daemon; so "Daemonless Container Engine" is no longer accurate. "Using Podman and Docker Compose" https://podman.io/blogs/2021/01/11/podman-compose.html https://podman.io/blogs/2021/01/11/podman-compose.html

Conan_Kudo 6y ago
Podman does not run in a daemon-mode by default. There's a dbus-activated service unit that activates the daemon when you try to use Docker Compose with it.

TwoNineFive 6y ago

I recently passed the RHCSA exam, which as of last October added a containers/podman section to the test. I usually work with lxcs under libvirt and directly, so working with podman was new to me. The test requirements are very simplistic, but I went further and spent a decent amount of time finding annoyances and issues. I am not impressed with podman. It's buggy and slow. Documentation is very basic and the underlaying mechanics are not covered where needed. Want to reconfigure the command parameters your start your container with to add a exported variable? Gotta remake the entire setup because there's no edit command.

ta988 6y ago
I really enjoy working with LXC. And the fact that I can switch now between container mode or VM mode makes it even better. That and best ng able to do lxc in lxc easily to test an infrastructure locally.

gvalkov 6y ago

It's great to see how far Podman (and its sister projects) have come. I think it's a reliable tool and I'm a happy user both personally and professionally. We make heavy use of Podman in our infrastructure and it's mostly a pleasure. My current pet peeves are that: 1) Ansible's podman_container module is not as polished as docker_container. I regularly run into idempotency issues with it (so lots of needlessly restarted containers). 2) Gitlab's Docker executor doesn't support Podman and all our CI agents run on CentOS 8. I ended up writing a custom executor for it and it's working quite well though (we're probably not going back to the container executor even if it supported Podman, since the custom executor offers so much more flexibility). 3) GPU support is easier/more documented on Docker. For this reason, the GPU servers we have are all Ubuntu 20.04 + Docker since it's the more beaten path. 4) Podman-compose just needs more work. Luckily for us, it seems that Podman 3.x will support docker-compose natively [1]. As mentioned, our CI environment is very dependent on Podman. The first step of every Gitlab pipelines is to build the container image in which the rest of the jobs will run. I find that it's simpler to have a shell executor in a unprivileged, restricted environment (i.e. can only run `podman build`) than setting up dind just for building images. All jobs that follow are ran in rootless containers, for that nice added layer of security. Wishing all the best to the Podman, Buildah and Skopeo teams. [1]: https://www.redhat.com/sysadmin/podman-docker-compose https://www.redhat.com/sysadmin/podman-docker-compose

barosl 6y ago

I tried to use Podman a few months ago, but the time spent to execute the "podman" command was considerably higher than that of the "docker" command. I am worried if this is a fundamental problem of the daemonless design. If that is the case, it would not be possible to solve the speed problem of Podman.

hardwaresofton 6y ago

I was a big fan of Podman and switched to it wholeheartedly[0][1] but after a while I ran into a few issues that made me have to switch off of it and back to docker: - image unpacking issues (invalid tar header) - linking containers isn't supported [EDIT] - I wrote up the post real quick[2] [0]: https://vadosware.io/post/rootless-containers-in-2020-on-arch-linux/ https://vadosware.io/post/rootless-containers-in-2020-on-arc... [1]: https://vadosware.io/post/bits-and-bobs-with-podman/ https://vadosware.io/post/bits-and-bobs-with-podman/ [2]: https://vadosware.io/post/back-to-docker-after-issues-with-podman/ https://vadosware.io/post/back-to-docker-after-issues-with-p...