5 ms·

I think many of these issues are also solved by Podman and systemd depending on what kind of "production" you're building for. If you're building a linux-y appl

I think many of these issues are also solved by Podman and systemd depending on what kind of "production" you're building for. If you're building a linux-y appliance and you need to run a few containers I think Podman is a much better and more ergonomic way of doing so. I think perhaps that's less true for running a web service (where the linux environment is just a means to that end).

philipallstar 5mo ago
Is there a nice guide for podman that includes quadlets (or saying not to use them?) I find lots of guides stray into things that work on redhat, and on my Linuxes of choice, Raspbian and Ubuntu, things aren't straightforward.
- exceptione 5mo ago
  Can't comment on Raspbian, but Ubuntu LTS (has/had) a seriously outdated podman version. This is the kind of nuisance the Debian derivatives have been running into for more than 20 years: they are extremely conservative, and if that is all you need, then that is great, but if not, you'll have to either run the latest Ubuntu (not LTS), or you upgrade to something like fedora.
  - jiggunjer 5mo ago
    Is there no upstream package repo like docker has.
  - mr_mitm 5mo ago
    In many cases, Debian unstable is also a good choice.
  - skydhash 5mo ago
    > they are extremely conservative, and if that is all you need, then that is great You don’t need to live at the edge of new features. Do you upgrade your fridge and your oven every two months? It’s nice when you can have something running and not worry that the next update will break your software and/or your workflow.
    - exceptione 5mo ago
      Sure, but these are development dependencies we are talking about. Running old versions of these dependencies block your projects. But it isn't limited to self-developed software, quite often for of the shelve software you run at the same problem. To each their own, but this is the reason I advice newcomers to stay away from Debian based distro's. I don't intend a distro flamewar, it works perfect for `boring old and feature complete software´ like Dovecot. To add: containers would alleviate a good part of these concerns, but the stupid thing here is that precisely that is broken for up-to-date podman workflows.
      - skydhash 5mo ago
        Your test system should reflect your prod system. Why run Debian if you intend to deploy on the latest ubuntu? Unless you want to use VMs. For other stuff that does not alter the system that much, you can find more recent version in the backports.
        exceptione 5mo ago
        It has integration with systemd, but moreover, I think the promise of Debian-derivatives is one of "we are boring and old, but also boringly stable". Now, throwing in backports undermines that promise. I think one is better of with a distro that moves faster.
  - MrDrMcCoy 5mo ago
    So use a static build of podman. They're readily available.
    - exceptione 5mo ago
      Podman's quadlets have a deep integration with systemd. I guess that if you have that kind of risk appetite you would be better of with running Arch on auto-update.
      - MrDrMcCoy 5mo ago
        If by "deep integration" you mean "unit generator", then sure. There isn't much I've seen beyond that, nor are there many systemd features being used that version drift would cause an issue for. Static podman introduces no real risk that I can identify.
  - linsomniac 5mo ago
    Not sure if you consider 5.7.0 (6 months old) "seriously outdated", or are talking about Ubuntu 24.04 (the previous LTS). I recently looked and decided 5.8.2 (3 weeks old), didn't have anything compelling to make me want to try to shoehorn it in.
    - exceptione 5mo ago
      Ubuntu 24.04. The new LTS had dropped only two weeks ago. LTS users had a very outdated podman (4.9, two years old) and couldn't use quadlet types like build units (v5.2.0, aug 2024).
      - linsomniac 5mo ago
        We are switching our Docker systems over to using Podman, primarily to get rid of the machinations we have to do to keep "apt update" from taking down services if there's a new Docker version. We're rolling them up from 24.04 to 26.04 and just using the podman packages on 26.04.
        exceptione 5mo ago
        I see, at least the good thing with 26.04 is that you are set for a while.
- notme43 5mo ago
  I find the podman man pages quite readable and thorough if you've had experience configuring systemd services. Good examples as well. https://docs.podman.io/en/latest/markdown/podman-systemd.unit.5.html https://docs.podman.io/en/latest/markdown/podman-systemd.uni...
  - philipallstar 5mo ago
    Thanks - you're right. I just have never known whether to quadlet or not to quadlet, as the centre of gravity seems to be moving that way, but that might be a redhat-first feature.
    - MrDrMcCoy 5mo ago
      The quadlet feature is just part of podman, which consists of no more than a generator for systemd. Generators are just an executable "hook" that spits out unit files according to systemd-defined paths. Any systemd system with podman will be equivalent to any other for the respective versions of each. There is nothing distro-specific about it, especially since podman can be a single multicall binary like busybox to provide it to a system that has systemd and nothing else.
madspindel 5mo ago
Yes, I recommend this: https://www.redhat.com/en/blog/kubernetes-workloads-podman-systemd https://www.redhat.com/en/blog/kubernetes-workloads-podman-s...
pmig 5mo ago
What are the benefits of running Podman Compose instead of Docker Compose? I don't see how it helps with orphan containers, logs and mutable tags.
- whilenot-dev 5mo ago
  GP is talking about podman with generated systemd unit files (a.k.a. podman quadlet[0]), not the docker-compose-compatible podman-compose ...and I'd agree, systemd can manage services on a system just fine, and even better than any compose workload ever could. journald will help with logs, and the pull policy[1] helps with mutable tags. What help do you need with "orphan containers"? [0]: https://docs.podman.io/en/latest/markdown/podman-quadlet.1.html https://docs.podman.io/en/latest/markdown/podman-quadlet.1.h... [1]: https://docs.podman.io/en/latest/markdown/podman-image.unit.5.html#policy-always https://docs.podman.io/en/latest/markdown/podman-image.unit....
- gear54rus 5mo ago
  Then you learn podman can't even list containers for all users properly and it kind of starts smelling like the whole ip4 vs ip6 debacle: bunch of vocal proponents wanting you to subject yourself to endless torture for no discernible reason.
  - figmert 5mo ago
    What do you mean it can't list containers for all users?
  - tovej 5mo ago
    I mean, ipv6 is for not runnig out of IP addresses. There is a clear discernible reason. There are workarounds to make ipv4 work, but they complicate the system and make it more fragile.
- figmert 5mo ago
  I'm not OP, but the whole podman compose topic gets quite confusing, as initially Podman didn't seem to know what they were trying to do. I've given some more context around it in previous comments. You shouldn't be using podman compose. It's flimsy and doesn't work very well (at least it was last time I used it prior to Podman v3), and I'm pretty sure it doesn't have Red Hat's direct support. Instead, activate Podman's Docker API compatibility socket, and simply set your `DOCKER_HOST` env var to that socket, and from there you can use your general docker client commands such as `docker`, `docker compose` and anything else that uses the Docker API. There are very few things that don't work with this, and the few things that don't are advanced setups. For what it's worth, podman has also a thin wrapper (podman compose) which executes `docker-compose` or the old `podman-compose`. The docs should explain which it picks. Note: - `podman-compose` is an early attempt at remaking `docker-compose` v1 but for Podman. This used parsed the compose config and converts them to podman commands, and executes it. - Later Podman wrote a Docker compatible socket instead, which can work with most docker clis that accept a `DOCKER_HOST` argument, including `docker` and `docker-compose` (both v1 and v2) - `podman compose` is a thin wrapper that automatically selects `docker-compose` or `podman-compose` depending on which is installed. Generally all you need is podman, docker-compose (the v2 binary), and that's it. From there you can use `podman` and/or `podman compose`.
bityard 5mo ago
I desperately WANT to like podman quadlets and keep trying to find a use case for them. But I always got the impression that the developers who implemented quadlets never actually had to manage multiple containers in a real production environment. Having your whole application with its containers, volumes, and networks all defined together in one easy-to-read YAML file is a way better experience. Deployment is two steps: 1. `git clone foo` 2. `docker compose up -d`. You can see the state of the application containers with `docker compose ps`. You can run multiple compose applications on the same host and manage them separately by putting them in different directories. With quadlets, you delegate everything to systemd. You have to break the configuration up into a bunch of tiny unit files and then separately copy them to /etc or a dedicated user's dotfiles. An application with a handful of containers and multiple networks/volumes/etc can spiral into a dozen unit files. Good luck SSH'ing into an unfamiliar system and understanding at a glance what it's doing. It is far more annoying to predictably deploy and tightly couples your application configuration to the host system configuration. (Even moreso if you created dedicated users for each application, which I understand is the recommended solution.) If I'm just holding it wrong and there exists some better tooling to manage podman in prod that I don't know about, I'm happy to hear about it.
- throwaway894345 5mo ago
  Honestly with the hell I'm going through just trying to get Podman to run properly on macos, I can't imagine trusting the Podman people with a production deployment. I was not particularly impressed with the Docker tooling, but Podman has been even worse. This is a not-remotely exhaustive list of things I've run into in the last 24 hours: * Podman fails to build a 16GB container image (after 30 minutes of downloading dependencies) despite having 90GB free out of a 200GB podman virtual machine * Podman machine will, for reasons I don't understand, create a filesystem in a block device with wildly different sizes, and it seems like it's just random * Pushing podman images to a container image registry via the Podman Desktop UI gives no indication that it's doing anything or even recognized the "push image" click, a success or error notification _might_ appear several or tens of minutes later or possibly not at all * Starting a podman machine might work, but it fails ~75% of the time with not-particularly-exotic options (a bunch of ram and disk) and very cryptic error messages, frequently telling me to file bug tickets (I have) * Podman Desktop won't let me create a podman machine with more than 44GB of disk, but the podman machine CLI won't let me create a machine with fewer than 100GB (IIRC--it's some number larger than 44, in any case) Apart from the container image being absurdly large (Python developers love massive packages, I guess), I'm not doing anything exotic.
DeathArrow 5mo ago
> I think perhaps that's less true for running a web service (where the linux environment is just a means to that end). If some BSD would support OCI containers, I would run my apps on BSD.

philipallstar 5mo ago

Is there a nice guide for podman that includes quadlets (or saying not to use them?) I find lots of guides stray into things that work on redhat, and on my Linuxes of choice, Raspbian and Ubuntu, things aren't straightforward.

exceptione 5mo ago
Can't comment on Raspbian, but Ubuntu LTS (has/had) a seriously outdated podman version. This is the kind of nuisance the Debian derivatives have been running into for more than 20 years: they are extremely conservative, and if that is all you need, then that is great, but if not, you'll have to either run the latest Ubuntu (not LTS), or you upgrade to something like fedora.
- jiggunjer 5mo ago
  Is there no upstream package repo like docker has.
- mr_mitm 5mo ago
  In many cases, Debian unstable is also a good choice.
- skydhash 5mo ago
  > they are extremely conservative, and if that is all you need, then that is great You don’t need to live at the edge of new features. Do you upgrade your fridge and your oven every two months? It’s nice when you can have something running and not worry that the next update will break your software and/or your workflow.
  - exceptione 5mo ago
    Sure, but these are development dependencies we are talking about. Running old versions of these dependencies block your projects. But it isn't limited to self-developed software, quite often for of the shelve software you run at the same problem. To each their own, but this is the reason I advice newcomers to stay away from Debian based distro's. I don't intend a distro flamewar, it works perfect for `boring old and feature complete software´ like Dovecot. To add: containers would alleviate a good part of these concerns, but the stupid thing here is that precisely that is broken for up-to-date podman workflows.
    - skydhash 5mo ago
      Your test system should reflect your prod system. Why run Debian if you intend to deploy on the latest ubuntu? Unless you want to use VMs. For other stuff that does not alter the system that much, you can find more recent version in the backports.
      - exceptione 5mo ago
        It has integration with systemd, but moreover, I think the promise of Debian-derivatives is one of "we are boring and old, but also boringly stable". Now, throwing in backports undermines that promise. I think one is better of with a distro that moves faster.
- MrDrMcCoy 5mo ago
  So use a static build of podman. They're readily available.
  - exceptione 5mo ago
    Podman's quadlets have a deep integration with systemd. I guess that if you have that kind of risk appetite you would be better of with running Arch on auto-update.
    - MrDrMcCoy 5mo ago
      If by "deep integration" you mean "unit generator", then sure. There isn't much I've seen beyond that, nor are there many systemd features being used that version drift would cause an issue for. Static podman introduces no real risk that I can identify.
- linsomniac 5mo ago
  Not sure if you consider 5.7.0 (6 months old) "seriously outdated", or are talking about Ubuntu 24.04 (the previous LTS). I recently looked and decided 5.8.2 (3 weeks old), didn't have anything compelling to make me want to try to shoehorn it in.
  - exceptione 5mo ago
    Ubuntu 24.04. The new LTS had dropped only two weeks ago. LTS users had a very outdated podman (4.9, two years old) and couldn't use quadlet types like build units (v5.2.0, aug 2024).
    - linsomniac 5mo ago
      We are switching our Docker systems over to using Podman, primarily to get rid of the machinations we have to do to keep "apt update" from taking down services if there's a new Docker version. We're rolling them up from 24.04 to 26.04 and just using the podman packages on 26.04.
      - exceptione 5mo ago
        I see, at least the good thing with 26.04 is that you are set for a while.
notme43 5mo ago
I find the podman man pages quite readable and thorough if you've had experience configuring systemd services. Good examples as well. https://docs.podman.io/en/latest/markdown/podman-systemd.unit.5.html https://docs.podman.io/en/latest/markdown/podman-systemd.uni...
- philipallstar 5mo ago
  Thanks - you're right. I just have never known whether to quadlet or not to quadlet, as the centre of gravity seems to be moving that way, but that might be a redhat-first feature.
  - MrDrMcCoy 5mo ago
    The quadlet feature is just part of podman, which consists of no more than a generator for systemd. Generators are just an executable "hook" that spits out unit files according to systemd-defined paths. Any systemd system with podman will be equivalent to any other for the respective versions of each. There is nothing distro-specific about it, especially since podman can be a single multicall binary like busybox to provide it to a system that has systemd and nothing else.

madspindel 5mo ago

Yes, I recommend this: https://www.redhat.com/en/blog/kubernetes-workloads-podman-systemd https://www.redhat.com/en/blog/kubernetes-workloads-podman-s...

pmig 5mo ago

What are the benefits of running Podman Compose instead of Docker Compose? I don't see how it helps with orphan containers, logs and mutable tags.

whilenot-dev 5mo ago
GP is talking about podman with generated systemd unit files (a.k.a. podman quadlet[0]), not the docker-compose-compatible podman-compose ...and I'd agree, systemd can manage services on a system just fine, and even better than any compose workload ever could. journald will help with logs, and the pull policy[1] helps with mutable tags. What help do you need with "orphan containers"? [0]: https://docs.podman.io/en/latest/markdown/podman-quadlet.1.html https://docs.podman.io/en/latest/markdown/podman-quadlet.1.h... [1]: https://docs.podman.io/en/latest/markdown/podman-image.unit.5.html#policy-always https://docs.podman.io/en/latest/markdown/podman-image.unit....
gear54rus 5mo ago
Then you learn podman can't even list containers for all users properly and it kind of starts smelling like the whole ip4 vs ip6 debacle: bunch of vocal proponents wanting you to subject yourself to endless torture for no discernible reason.
- figmert 5mo ago
  What do you mean it can't list containers for all users?
- tovej 5mo ago
  I mean, ipv6 is for not runnig out of IP addresses. There is a clear discernible reason. There are workarounds to make ipv4 work, but they complicate the system and make it more fragile.
figmert 5mo ago
I'm not OP, but the whole podman compose topic gets quite confusing, as initially Podman didn't seem to know what they were trying to do. I've given some more context around it in previous comments. You shouldn't be using podman compose. It's flimsy and doesn't work very well (at least it was last time I used it prior to Podman v3), and I'm pretty sure it doesn't have Red Hat's direct support. Instead, activate Podman's Docker API compatibility socket, and simply set your `DOCKER_HOST` env var to that socket, and from there you can use your general docker client commands such as `docker`, `docker compose` and anything else that uses the Docker API. There are very few things that don't work with this, and the few things that don't are advanced setups. For what it's worth, podman has also a thin wrapper (podman compose) which executes `docker-compose` or the old `podman-compose`. The docs should explain which it picks. Note: - `podman-compose` is an early attempt at remaking `docker-compose` v1 but for Podman. This used parsed the compose config and converts them to podman commands, and executes it. - Later Podman wrote a Docker compatible socket instead, which can work with most docker clis that accept a `DOCKER_HOST` argument, including `docker` and `docker-compose` (both v1 and v2) - `podman compose` is a thin wrapper that automatically selects `docker-compose` or `podman-compose` depending on which is installed. Generally all you need is podman, docker-compose (the v2 binary), and that's it. From there you can use `podman` and/or `podman compose`.

bityard 5mo ago

I desperately WANT to like podman quadlets and keep trying to find a use case for them. But I always got the impression that the developers who implemented quadlets never actually had to manage multiple containers in a real production environment. Having your whole application with its containers, volumes, and networks all defined together in one easy-to-read YAML file is a way better experience. Deployment is two steps: 1. `git clone foo` 2. `docker compose up -d`. You can see the state of the application containers with `docker compose ps`. You can run multiple compose applications on the same host and manage them separately by putting them in different directories. With quadlets, you delegate everything to systemd. You have to break the configuration up into a bunch of tiny unit files and then separately copy them to /etc or a dedicated user's dotfiles. An application with a handful of containers and multiple networks/volumes/etc can spiral into a dozen unit files. Good luck SSH'ing into an unfamiliar system and understanding at a glance what it's doing. It is far more annoying to predictably deploy and tightly couples your application configuration to the host system configuration. (Even moreso if you created dedicated users for each application, which I understand is the recommended solution.) If I'm just holding it wrong and there exists some better tooling to manage podman in prod that I don't know about, I'm happy to hear about it.

throwaway894345 5mo ago
Honestly with the hell I'm going through just trying to get Podman to run properly on macos, I can't imagine trusting the Podman people with a production deployment. I was not particularly impressed with the Docker tooling, but Podman has been even worse. This is a not-remotely exhaustive list of things I've run into in the last 24 hours: * Podman fails to build a 16GB container image (after 30 minutes of downloading dependencies) despite having 90GB free out of a 200GB podman virtual machine * Podman machine will, for reasons I don't understand, create a filesystem in a block device with wildly different sizes, and it seems like it's just random * Pushing podman images to a container image registry via the Podman Desktop UI gives no indication that it's doing anything or even recognized the "push image" click, a success or error notification _might_ appear several or tens of minutes later or possibly not at all * Starting a podman machine might work, but it fails ~75% of the time with not-particularly-exotic options (a bunch of ram and disk) and very cryptic error messages, frequently telling me to file bug tickets (I have) * Podman Desktop won't let me create a podman machine with more than 44GB of disk, but the podman machine CLI won't let me create a machine with fewer than 100GB (IIRC--it's some number larger than 44, in any case) Apart from the container image being absurdly large (Python developers love massive packages, I guess), I'm not doing anything exotic.

DeathArrow 5mo ago

> I think perhaps that's less true for running a web service (where the linux environment is just a means to that end). If some BSD would support OCI containers, I would run my apps on BSD.