7 ms·

On the joy of podman and auto-updates (2022)

131 points by doublepg23 4y ago

jacooper 4y ago
Or just skip all the headaches and use Watchtower with docker or docker compose.
meitham 4y ago
It’s very nice to have the option for auto update, but in reality would anyone wants their containers to auto deploy newer versions? Aside from the security exposure (specially if you were on a Scandinavian’s holiday) there is the risk of backward incompatibility such as changes around config, db migration, cli arts or anything else the container might consume but sits outside the container!
- Arnavion 4y ago
  Using it with `:latest` is going to have the problem you described and is not the expected usage. It's more feasible to use it with a release line tag, like `postgres:15`, where you expect updated images to be backward-compatible. Alternatively, one place where `:latest` would make sense is if they're your images, and you're relying on podman-auto-update to update a fleet of servers to start running the latest version of a service as soon as you push the new image.
  - config_yml 4y ago
    How do people automate this kind of thing, like rolling out a specific image via CI? Generating the systemd files, copying via scp and systemd reload via scripted ssh?
    - txtsd 4y ago
      I would like to know this too. Is it stored in a dotfiles-like repo, then cloned, and `stow`ed?
    - zokier 4y ago
      I imagine Ansible could take care of the grunt work here, or something else like it
    - nine_k 4y ago
      Come on, scp :) Ansible / salt / puppet, etc would do this for you in a nicer way. Or maybe Nomad. Or k8s + ArgoCD would do that from the fact of a commit to a git repo.
      - config_yml 4y ago
        I guess in "Make it as simple as possible, but not simpler", scp is too simple for you? But you also mentioned five different beasts to master, so I'm not so sure ;)
        nine_k 4y ago
        If all you need is to update some files, with minimal error handling, scp is fine! (Well, rsync is probably a better option now, but scp would still work.) As you get progressively bigger, you can consider other options.
    - francis-io 4y ago
      I guess it would all depend on the scale. For my home servers which just run personal things (like a kanban board as a todo list) I just use watchtower[0]. This requires mounting the docker socket into this container, which is not ideal. In a production environment, id expect pinning of the docker sha and setting docker tags as immutable. Some software projects exist to scan for updates and draft PRs automatically for changes (I can't remember the name of the software but it begins with R). [0] https://containrrr.dev/watchtower/ https://containrrr.dev/watchtower/
      - cascandaliato 4y ago
        > I can't remember the name of the software but it begins with R Renovate maybe? https://www.mend.io/free-developer-tools/renovate/ https://www.mend.io/free-developer-tools/renovate/ I use it for my home server and I love it because it takes care of Dockerfiles too and version changes are saved in git, which means that a rollback is just a matter of switching back to a previous commit and rebuilding your containers (in addition to restoring a backup of your Docker volumes).
    - yrro 4y ago
      You could use Ansible with the podman_container task to keep your containers running.
  - noirscape 4y ago
    It varies a lot on what you're using. Postgres in particular is a bugbear because updating dockerized Postgres is hell at best and requires several backups in case you screw up. OTOH a bog standard python server backend can probably keep running for quite a while on alpine:latest without issue because it's not very reliant on much in terms of strange package availability (maybe old python versions that get deprecated every once in a while).
- deleted 4y ago
  [deleted]
- zokier 4y ago
  That is a problem if you do not have strong controls on the publishing side. Ultimately the point of cicd is that changes get deployed mostly automatically, it doesn't matter if its accomplished by servers using :latest tag lr some automation going and reconfiguring servers or something else
WirelessGigabit 4y ago
So one thing that I didn't read here is how it works with environment variables. Imagine you launch a container with FOOD=fries and DRINK=beer. The container doesn't set defaults for those variables, it does have one for DESSERT=ice-cream. The container runs with FOOD=fries, DRINK=beer & DESSERT=ice-cream. An update comes along, and the container now has a default for DRINK=wine and switches to DESSERT=creme-brulee. You update the container. What do you expect to happen with DRINK & DESSERT? DRINK remains beer, fine we chose that. But DESSERT also remains ice-cream, even though we didn't explicitly say that. The problem is that Docker (well, the surrounding tooling) cannot distinguish between your input and the container's default. They all get set in the `env` section of the container. So you update the container and end up with FOOD=fries, DRINK=beer & DESSERT=ice-cream. I get around this with using ansible with the Docker collection from the community [0]. This has a separate input and when it recreates the container it only takes into account the environment variables I set, and doesn't provide the other ones, so they become the container's default. Many containers have things like PYTHON_VERSION=3.11.1 PYTHON_PIP_VERSION=22.3.1 PYTHON_SETUPTOOLS_VERSION=65.5.1 PYTHON_GET_PIP_URL=https://github.com/pypa/get-pip/raw/66030fa03382b4914d4c4d0896961a0bdeeeb274/public/get-pip.py https://github.com/pypa/get-pip/raw/66030fa03382b4914d4c4d08... PYTHON_GET_PIP_SHA256=1e501cf004eac1b7eb1f97266d28f995ae835d30250bec7f8850562703067dc6 in there. PS: How do I get Podman 4 on Ubuntu? [0] https://docs.ansible.com/ansible/latest/collections/community/docker/index.html https://docs.ansible.com/ansible/latest/collections/communit...
- Arnavion 4y ago
  The container service unit will do `podman run -e FOOD=dries -e DRINK=beer whatever-image:latest` podman-auto-update.service will pull the new `whatever-image:latest` image, restart the container service (which `stop`s and `rm -f`s the container), and `image prune -f` to remove the previous `whatever-image:latest`. The restarted container service will continue to run `podman run -e FOOD=dries -e DRINK=beer whatever-image:latest`. So you end up with `FOOD=fries DRINK=beer DESSERT=creme-brulee`
  - WirelessGigabit 4y ago
    Thank you! I really appreciate this. It seems to do what I want it to do. I checked for Podman 4 on Ubuntu, but that's a whole other can of worms. And I need 4 for IPv6 support.
Jnr 4y ago
Nice post, but I really dislike the idea of creating unit files for each container. docker compose was a step forward, this is certainly going back and I will probably never use it no matter how many times someone says that this is the canonical way to do things, because it isn't, no one does that. It seems almost as if someone is deliberately crippling open source container tooling just to increase sales of cloud provider container management tools. It most likely isn't true but I still wonder about it because I haven't seen actually good contributions and innovation in this area for some years now.
- Already__Taken 4y ago
  user space container unit files kind of stuck to use and edit, I have to have notes to always add the socket too else I can't see what's running. just use k8s it's easier in the long run
- rkangel 4y ago
  This is certainly a canonical way of doing things. Podman allows you to treat a containerised process much more like any other process. In this world systemd manages the interdependencies (including on things like the network). You may think that other workflows are better, but there is a chunk of the engineering population that is happy with the tools they have for managing a set of bare processes on a machine.
  - nine_k 4y ago
    The key difference is that systemd is system-wide orchestration, and docker-compose controls a specific set of containers connected in a specific day, and likely otherwise isolated from the system. The latter allows to "ship" entire configurations that work together, as one unit.
    - mariusor 4y ago
      I think that existing tooling on nix environments can work much better than docker compose to aggregate a set of containers and resources. Setting resource limits using cgroups, isolating users/paths/networks using the capabilities of systemd and all together used from a common shell script seems more expressive and more versatile than using docker/docker compose and requiring an extra daemon to deal with everything.
lbayes 4y ago
I got somewhat excited about podman a couple months ago, and then learned that there are painful shenanigans between required versions and the (ancient?) one that is available in the Ubuntu 22.10 package manager [1]. It seemed great until I tried to launch an existing Postgres container that required a more recent Podman version. The Podman installation page [2] makes some effort to explain alternate installation steps, but doesn't make it clear why, or how far back you'll be. Not complaining here, just giving folks a heads up. Despite these issues, it does look like a great project. [1] https://github.com/containers/podman/issues/14065 https://github.com/containers/podman/issues/14065 [2] https://podman.io/getting-started/installation https://podman.io/getting-started/installation
- pydry 4y ago
  This is partly a problem with ubuntu. It sometimes has old software and makes it difficult to upgrade. I had the exact same problem with docker when it was newer - before the APIs stabilized.

131 points by doublepg23 4y ago

Or just skip all the headaches and use Watchtower with docker or docker compose.

It’s very nice to have the option for auto update, but in reality would anyone wants their containers to auto deploy newer versions? Aside from the security exposure (specially if you were on a Scandinavian’s holiday) there is the risk of backward incompatibility such as changes around config, db migration, cli arts or anything else the container might consume but sits outside the container!

Arnavion 4y ago
Using it with `:latest` is going to have the problem you described and is not the expected usage. It's more feasible to use it with a release line tag, like `postgres:15`, where you expect updated images to be backward-compatible. Alternatively, one place where `:latest` would make sense is if they're your images, and you're relying on podman-auto-update to update a fleet of servers to start running the latest version of a service as soon as you push the new image.
- config_yml 4y ago
  How do people automate this kind of thing, like rolling out a specific image via CI? Generating the systemd files, copying via scp and systemd reload via scripted ssh?
  - txtsd 4y ago
    I would like to know this too. Is it stored in a dotfiles-like repo, then cloned, and `stow`ed?
  - zokier 4y ago
    I imagine Ansible could take care of the grunt work here, or something else like it
  - nine_k 4y ago
    Come on, scp :) Ansible / salt / puppet, etc would do this for you in a nicer way. Or maybe Nomad. Or k8s + ArgoCD would do that from the fact of a commit to a git repo.
    - config_yml 4y ago
      I guess in "Make it as simple as possible, but not simpler", scp is too simple for you? But you also mentioned five different beasts to master, so I'm not so sure ;)
      - nine_k 4y ago
        If all you need is to update some files, with minimal error handling, scp is fine! (Well, rsync is probably a better option now, but scp would still work.) As you get progressively bigger, you can consider other options.
  - francis-io 4y ago
    I guess it would all depend on the scale. For my home servers which just run personal things (like a kanban board as a todo list) I just use watchtower[0]. This requires mounting the docker socket into this container, which is not ideal. In a production environment, id expect pinning of the docker sha and setting docker tags as immutable. Some software projects exist to scan for updates and draft PRs automatically for changes (I can't remember the name of the software but it begins with R). [0] https://containrrr.dev/watchtower/ https://containrrr.dev/watchtower/
    - cascandaliato 4y ago
      > I can't remember the name of the software but it begins with R Renovate maybe? https://www.mend.io/free-developer-tools/renovate/ https://www.mend.io/free-developer-tools/renovate/ I use it for my home server and I love it because it takes care of Dockerfiles too and version changes are saved in git, which means that a rollback is just a matter of switching back to a previous commit and rebuilding your containers (in addition to restoring a backup of your Docker volumes).
  - yrro 4y ago
    You could use Ansible with the podman_container task to keep your containers running.
- noirscape 4y ago
  It varies a lot on what you're using. Postgres in particular is a bugbear because updating dockerized Postgres is hell at best and requires several backups in case you screw up. OTOH a bog standard python server backend can probably keep running for quite a while on alpine:latest without issue because it's not very reliant on much in terms of strange package availability (maybe old python versions that get deprecated every once in a while).
deleted 4y ago
[deleted]
zokier 4y ago
That is a problem if you do not have strong controls on the publishing side. Ultimately the point of cicd is that changes get deployed mostly automatically, it doesn't matter if its accomplished by servers using :latest tag lr some automation going and reconfiguring servers or something else

WirelessGigabit 4y ago

So one thing that I didn't read here is how it works with environment variables. Imagine you launch a container with FOOD=fries and DRINK=beer. The container doesn't set defaults for those variables, it does have one for DESSERT=ice-cream. The container runs with FOOD=fries, DRINK=beer & DESSERT=ice-cream. An update comes along, and the container now has a default for DRINK=wine and switches to DESSERT=creme-brulee. You update the container. What do you expect to happen with DRINK & DESSERT? DRINK remains beer, fine we chose that. But DESSERT also remains ice-cream, even though we didn't explicitly say that. The problem is that Docker (well, the surrounding tooling) cannot distinguish between your input and the container's default. They all get set in the `env` section of the container. So you update the container and end up with FOOD=fries, DRINK=beer & DESSERT=ice-cream. I get around this with using ansible with the Docker collection from the community [0]. This has a separate input and when it recreates the container it only takes into account the environment variables I set, and doesn't provide the other ones, so they become the container's default. Many containers have things like PYTHON_VERSION=3.11.1 PYTHON_PIP_VERSION=22.3.1 PYTHON_SETUPTOOLS_VERSION=65.5.1 PYTHON_GET_PIP_URL=https://github.com/pypa/get-pip/raw/66030fa03382b4914d4c4d0896961a0bdeeeb274/public/get-pip.py https://github.com/pypa/get-pip/raw/66030fa03382b4914d4c4d08... PYTHON_GET_PIP_SHA256=1e501cf004eac1b7eb1f97266d28f995ae835d30250bec7f8850562703067dc6 in there. PS: How do I get Podman 4 on Ubuntu? [0] https://docs.ansible.com/ansible/latest/collections/community/docker/index.html https://docs.ansible.com/ansible/latest/collections/communit...

Arnavion 4y ago
The container service unit will do `podman run -e FOOD=dries -e DRINK=beer whatever-image:latest` podman-auto-update.service will pull the new `whatever-image:latest` image, restart the container service (which `stop`s and `rm -f`s the container), and `image prune -f` to remove the previous `whatever-image:latest`. The restarted container service will continue to run `podman run -e FOOD=dries -e DRINK=beer whatever-image:latest`. So you end up with `FOOD=fries DRINK=beer DESSERT=creme-brulee`
- WirelessGigabit 4y ago
  Thank you! I really appreciate this. It seems to do what I want it to do. I checked for Podman 4 on Ubuntu, but that's a whole other can of worms. And I need 4 for IPv6 support.

Jnr 4y ago

Nice post, but I really dislike the idea of creating unit files for each container. docker compose was a step forward, this is certainly going back and I will probably never use it no matter how many times someone says that this is the canonical way to do things, because it isn't, no one does that. It seems almost as if someone is deliberately crippling open source container tooling just to increase sales of cloud provider container management tools. It most likely isn't true but I still wonder about it because I haven't seen actually good contributions and innovation in this area for some years now.

Already__Taken 4y ago
user space container unit files kind of stuck to use and edit, I have to have notes to always add the socket too else I can't see what's running. just use k8s it's easier in the long run
rkangel 4y ago
This is certainly a canonical way of doing things. Podman allows you to treat a containerised process much more like any other process. In this world systemd manages the interdependencies (including on things like the network). You may think that other workflows are better, but there is a chunk of the engineering population that is happy with the tools they have for managing a set of bare processes on a machine.
- nine_k 4y ago
  The key difference is that systemd is system-wide orchestration, and docker-compose controls a specific set of containers connected in a specific day, and likely otherwise isolated from the system. The latter allows to "ship" entire configurations that work together, as one unit.
  - mariusor 4y ago
    I think that existing tooling on nix environments can work much better than docker compose to aggregate a set of containers and resources. Setting resource limits using cgroups, isolating users/paths/networks using the capabilities of systemd and all together used from a common shell script seems more expressive and more versatile than using docker/docker compose and requiring an extra daemon to deal with everything.

lbayes 4y ago

I got somewhat excited about podman a couple months ago, and then learned that there are painful shenanigans between required versions and the (ancient?) one that is available in the Ubuntu 22.10 package manager [1]. It seemed great until I tried to launch an existing Postgres container that required a more recent Podman version. The Podman installation page [2] makes some effort to explain alternate installation steps, but doesn't make it clear why, or how far back you'll be. Not complaining here, just giving folks a heads up. Despite these issues, it does look like a great project. [1] https://github.com/containers/podman/issues/14065 https://github.com/containers/podman/issues/14065 [2] https://podman.io/getting-started/installation https://podman.io/getting-started/installation

pydry 4y ago
This is partly a problem with ubuntu. It sometimes has old software and makes it difficult to upgrade. I had the exact same problem with docker when it was newer - before the APIs stabilized.