7 ms·

At my company, we run our CI/CD (Jenkins) using the Docker-in-Docker paradigm to facilitate easy maintainability of the CI itself and allow us to run containeri

At my company, we run our CI/CD (Jenkins) using the Docker-in-Docker paradigm to facilitate easy maintainability of the CI itself and allow us to run containerized builds. When we shifted to RHEL 8, we attempted to move this over to Podman and it went miserably (this was back in November 2019). The main reason being is that podman-in-podman doesn't work and had bugs (at least back in Nov 2019). Maybe it fixed now but this was our experience. We ended up doing quite a bit of analysis on podman only to conclude it's simply not there yet relative to docker (ecosystem and ergonomics). There are quite a few corner cases that docker quite simply supports out of the box beautifully that podman doesn't support or just has bugs. I like the what the project is trying to solve by being daemonless, but this is not as simple as a drop in replacement for docker that RedHat markets it as (alias docker=podman). We ended up sticking to docker professionally and personally, I am still using docker over podman. The ecosystem and ergonomics are just far too nice to give up over podman.

ctalledo 6y ago
If you are using Docker-in-Docker, you may want to checkout the new Sysbox runtime (find it on Github). It's a new type of runc that sits below Docker and creates rootless containers capable of running Docker, systemd, K8s, etc. All you have to do is "docker run --runtime=sysbox-runc" <some-image-with-docker> and you'll get a docker daemon that is fully isolated from the host. It's a great way of avoiding privileged containers or mounts to the host docker socket.
choeger 6y ago
Maybe you are just too far into docker. I noticed that a lot of default workflows (needlessly) depended on docker running with privileges. One big reason for that seem to be Mac users that only know docker from inside a VM. However, if you think about what you're really needing for CI you will easily see that docker-in-docker gains you nothing. You can as well use plain docker (or podman). The same holds for privileges. No CI operation should need privileges, if only for the reason that it should never alter the CI system itself. I encourage you to not take the standard workflows as a given and really think about what you need and I bet you either end up with a use case that can be covered by rootless podman or something that requires real VMs anyways.
- zomglings 6y ago
  Why do you say that docker-in-docker buys him nothing? It's not obvious at all and you go into no detail whatsoever to back up your opinion. In my experience, that is not true at all. Docker-in-docker allows me to deliver smaller images that can fit into a CI flow as language plugins instead of shipping a beastly 5G docker image with every possible language runtime I need to support for my CI tool.
  - core-questions 6y ago
    > beastly 5G docker image my beastly 12GB image that even includes Matlab wants a word with you
    - joana035 6y ago
      >> beastly 5G docker image > my beastly 12GB image that even includes Matlab wants a word with you Perhaps in the next 10 years we will be rediscovering packages. :P If you are in the business of charging complex prices per bits over the network, then docker seems to be quite a good investment and making it as popular as possible is a good strategy to print money. /s
      - core-questions 6y ago
        Well, to be fair, it is packages - I'm just using Docker (for this section of our stack) as a different sort of VM, essentially. It runs a service manager and a VNC X session, for chrissakes ;)
      - disgruntledphd2 6y ago
        > If you are in the business of charging complex prices per bits over the network, then docker seems to be quite a good investment and making it as popular as possible is a good strategy to print money. /s True, that. To be fair, at least it allows me to avoid lots of the brokenness of Python packaging.
        6y ago
joncp 6y ago
I've found your issues with podman to be a theme for Red Hat projects in general. Have you ever tried to configure Keycloak or OpenShift, for example?
- znpy 6y ago
  Just spent the last two days messing up with Keycloak on kubernetes... I feel you.
  - deleted 6y ago
    [deleted]
- ranadeep 6y ago
  This might be biased on my part but experiences like this is why I tend to openly avoid most Red Hat projects in general. There are definitely some great projects like RHEL, Ansible that are nice to work with. Even professionally, I tend to steer clear of immature Red Hat projects. We evaluated Keycloak but went with a vendor solution. OpenShift I believe was also evaluated at my firm and hit a dead-end.
  - dastx 6y ago
    Openshift is a great solution for a k8s setup on prem. From my experience it doesn't bring a whole lot when you're already on the cloud. All the additional services openshift provides, are already available as part of the major cloud offerings.
- donor20 6y ago
  Openshift years ago at least (last time I looked) - SHUDDER! I couldn't believe it was what big business / enterprise was standardizing on as best in class. It looks like it's now Kube / Docker connected, but back then I think it did it's own thing? The cost in time / $ / config to maintain and operate at the time was not pretty (all this no doubt has changed a lot).
  - oezi 6y ago
fredthomsen 6y ago
Heavy compose user here, and when I tried the podman equivalent of compose was missing some functionality that made me move back.
- ziftface 6y ago
  I had the same experience, and I'm not really sure why I can't use docker-compose with podman. If it's a drop in replacement, why can't I alias podman as docker and use docker-compose like normal? I did try that and it doesn't work, and podman compose isn't as useable. So I switched back to docker.
  - fuzxi 6y ago
    >If it's a drop in replacement, why can't I alias podman as docker and use docker-compose like normal? Podman is (mostly) a drop-in replacement for docker. However, docker-compose is a separate package from docker that requires explicit installation. So too is podman-compose separate from podman, though unfortunately it still needs more work.
    - fatherlinnux 6y ago
      I can explain this. Because docker-compose talks to the docker socket and uses the docker protocol. This was the last piece that we nailed with podman. As of Podman 2.0 with the new REST based API with a Docker compatibility layer, we are very close to just being able to use the docker-compose binary to talk to a socket (ran by systemd) which fires up podman. It doesn't quite work yet today because we are still implementing some of the REST verbs, but it's close. It's definitely a strategic direction for the roadmap. Stay tuned.
quietbritishjim 6y ago
> Docker-in-Docker paradigm Apologies for the patronising comment, but do you really mean that? Docker in Docker works but is intended for the developers of Docker to debug Docker itself. Usually for running Docker from within a container, you just hook up the Docker client to the TCP port of the Docker daemon running outside the container, which isn't strictly Docker-in-Docker. I ask this in case you're trying a wildly use case (if you really are running true Docker-in-Docker), or are making an unfair comparison (if you're just using the usual Docker client in container to daemon on host). In the latter case, I must admit I don't know what the idiomatic alternative would be for podman, given that I know nothing about it except that it's daemonless (and even that I only learned by reading your comment).
- felipelemos 6y ago
  There are some reasons to use docker in docker beyond debug docker itself. For example, in Jenkins case, select specific proxy settings for the daemon for a specific job without affecting other jobs in the same node.
- solotronics 6y ago
  I also needed to run docker commands from within a container (Jenkins container executing docker cmds on the host it's running on). I don't know how common this is but I had the same use case.
- flerchin 6y ago
  We docker-in-docker too, exactly as OP describes. In jenkins CI/CD.
- nhoughto 6y ago
  Docker in docker is a very common pattern in CI, in some cases is the only way to achieve certain tasks.
  - haberman 6y ago
    What kinds of tasks require two levels of containers?
peterwwillis 6y ago
Your life will be a lot less painful if you abandon docker-in-docker. The Jenkins solution is to simply execute containers using the host's Docker daemon, or via a container orchestration system which you call out to. The latter is the best option as it's much more scalable and doesn't require ugly proprietary hacks. Docker in Docker requires vertical scaling and more complex management for intermediate states/maintenance. Podman probably isn't ready to replace Docker yet, but rootless containers are the easy-mode for federation of clusters. If you can supply the other features you need (and I think most can) it's probably worth it in the long run.
stevekemp 6y ago
I'm not married to docker either, and love the idea of podman. In my case the blocker for migrating was related to docker-compose. podman-compose doesn't support as much as I needed for my current deployments - although offhand I cannot recall what was missing. I'm looking forward to trying things again in a few months, but those corner-cases can be real pains to deal with.
- fatherlinnux 6y ago
  We're working hard on docker-compose support. With the new Podman 2.0 REST API with a Docker compatibility layer, we are close. Not quite there, but definitely strategically on the roadmap.

ctalledo 6y ago

If you are using Docker-in-Docker, you may want to checkout the new Sysbox runtime (find it on Github). It's a new type of runc that sits below Docker and creates rootless containers capable of running Docker, systemd, K8s, etc. All you have to do is "docker run --runtime=sysbox-runc" <some-image-with-docker> and you'll get a docker daemon that is fully isolated from the host. It's a great way of avoiding privileged containers or mounts to the host docker socket.

choeger 6y ago

Maybe you are just too far into docker. I noticed that a lot of default workflows (needlessly) depended on docker running with privileges. One big reason for that seem to be Mac users that only know docker from inside a VM. However, if you think about what you're really needing for CI you will easily see that docker-in-docker gains you nothing. You can as well use plain docker (or podman). The same holds for privileges. No CI operation should need privileges, if only for the reason that it should never alter the CI system itself. I encourage you to not take the standard workflows as a given and really think about what you need and I bet you either end up with a use case that can be covered by rootless podman or something that requires real VMs anyways.

zomglings 6y ago
Why do you say that docker-in-docker buys him nothing? It's not obvious at all and you go into no detail whatsoever to back up your opinion. In my experience, that is not true at all. Docker-in-docker allows me to deliver smaller images that can fit into a CI flow as language plugins instead of shipping a beastly 5G docker image with every possible language runtime I need to support for my CI tool.
- core-questions 6y ago
  > beastly 5G docker image my beastly 12GB image that even includes Matlab wants a word with you
  - joana035 6y ago
    >> beastly 5G docker image > my beastly 12GB image that even includes Matlab wants a word with you Perhaps in the next 10 years we will be rediscovering packages. :P If you are in the business of charging complex prices per bits over the network, then docker seems to be quite a good investment and making it as popular as possible is a good strategy to print money. /s
    - core-questions 6y ago
      Well, to be fair, it is packages - I'm just using Docker (for this section of our stack) as a different sort of VM, essentially. It runs a service manager and a VNC X session, for chrissakes ;)
    - disgruntledphd2 6y ago
      > If you are in the business of charging complex prices per bits over the network, then docker seems to be quite a good investment and making it as popular as possible is a good strategy to print money. /s True, that. To be fair, at least it allows me to avoid lots of the brokenness of Python packaging.
      - 6y ago

joncp 6y ago

I've found your issues with podman to be a theme for Red Hat projects in general. Have you ever tried to configure Keycloak or OpenShift, for example?

znpy 6y ago
Just spent the last two days messing up with Keycloak on kubernetes... I feel you.
- deleted 6y ago
  [deleted]
ranadeep 6y ago
This might be biased on my part but experiences like this is why I tend to openly avoid most Red Hat projects in general. There are definitely some great projects like RHEL, Ansible that are nice to work with. Even professionally, I tend to steer clear of immature Red Hat projects. We evaluated Keycloak but went with a vendor solution. OpenShift I believe was also evaluated at my firm and hit a dead-end.
- dastx 6y ago
  Openshift is a great solution for a k8s setup on prem. From my experience it doesn't bring a whole lot when you're already on the cloud. All the additional services openshift provides, are already available as part of the major cloud offerings.
donor20 6y ago
Openshift years ago at least (last time I looked) - SHUDDER! I couldn't believe it was what big business / enterprise was standardizing on as best in class. It looks like it's now Kube / Docker connected, but back then I think it did it's own thing? The cost in time / $ / config to maintain and operate at the time was not pretty (all this no doubt has changed a lot).
- oezi 6y ago

fredthomsen 6y ago

Heavy compose user here, and when I tried the podman equivalent of compose was missing some functionality that made me move back.

ziftface 6y ago
I had the same experience, and I'm not really sure why I can't use docker-compose with podman. If it's a drop in replacement, why can't I alias podman as docker and use docker-compose like normal? I did try that and it doesn't work, and podman compose isn't as useable. So I switched back to docker.
- fuzxi 6y ago
  >If it's a drop in replacement, why can't I alias podman as docker and use docker-compose like normal? Podman is (mostly) a drop-in replacement for docker. However, docker-compose is a separate package from docker that requires explicit installation. So too is podman-compose separate from podman, though unfortunately it still needs more work.
  - fatherlinnux 6y ago
    I can explain this. Because docker-compose talks to the docker socket and uses the docker protocol. This was the last piece that we nailed with podman. As of Podman 2.0 with the new REST based API with a Docker compatibility layer, we are very close to just being able to use the docker-compose binary to talk to a socket (ran by systemd) which fires up podman. It doesn't quite work yet today because we are still implementing some of the REST verbs, but it's close. It's definitely a strategic direction for the roadmap. Stay tuned.

quietbritishjim 6y ago

> Docker-in-Docker paradigm Apologies for the patronising comment, but do you really mean that? Docker in Docker works but is intended for the developers of Docker to debug Docker itself. Usually for running Docker from within a container, you just hook up the Docker client to the TCP port of the Docker daemon running outside the container, which isn't strictly Docker-in-Docker. I ask this in case you're trying a wildly use case (if you really are running true Docker-in-Docker), or are making an unfair comparison (if you're just using the usual Docker client in container to daemon on host). In the latter case, I must admit I don't know what the idiomatic alternative would be for podman, given that I know nothing about it except that it's daemonless (and even that I only learned by reading your comment).

felipelemos 6y ago
There are some reasons to use docker in docker beyond debug docker itself. For example, in Jenkins case, select specific proxy settings for the daemon for a specific job without affecting other jobs in the same node.
solotronics 6y ago
I also needed to run docker commands from within a container (Jenkins container executing docker cmds on the host it's running on). I don't know how common this is but I had the same use case.
flerchin 6y ago
We docker-in-docker too, exactly as OP describes. In jenkins CI/CD.
nhoughto 6y ago
Docker in docker is a very common pattern in CI, in some cases is the only way to achieve certain tasks.
- haberman 6y ago
  What kinds of tasks require two levels of containers?

peterwwillis 6y ago

Your life will be a lot less painful if you abandon docker-in-docker. The Jenkins solution is to simply execute containers using the host's Docker daemon, or via a container orchestration system which you call out to. The latter is the best option as it's much more scalable and doesn't require ugly proprietary hacks. Docker in Docker requires vertical scaling and more complex management for intermediate states/maintenance. Podman probably isn't ready to replace Docker yet, but rootless containers are the easy-mode for federation of clusters. If you can supply the other features you need (and I think most can) it's probably worth it in the long run.

stevekemp 6y ago

I'm not married to docker either, and love the idea of podman. In my case the blocker for migrating was related to docker-compose. podman-compose doesn't support as much as I needed for my current deployments - although offhand I cannot recall what was missing. I'm looking forward to trying things again in a few months, but those corner-cases can be real pains to deal with.

fatherlinnux 6y ago
We're working hard on docker-compose support. With the new Podman 2.0 REST API with a Docker compatibility layer, we are close. Not quite there, but definitely strategically on the roadmap.