3 ms·

With the major caveat that rootless depends on /dev/fuse for the overlay filesystems (unless you hate yourself and want to use vfs), so you can't use it inside

With the major caveat that rootless depends on /dev/fuse for the overlay filesystems (unless you hate yourself and want to use vfs), so you can't use it inside a Kubernetes container unless it's privileged or has the SYS_ADMIN capability added. This is how we end up with beautiful hacks like Kaniko for building container images inside container workflows. Le sigh.

znpy 5y ago
Sorry for the vague reply, but I'm out with friends right now. But basically that's fixed because newest kernels (5.13 iirc) allow to work around that without requiring root privileges. So podman is going to stop requiring that too sooner or later.
- mikepurvis 5y ago
  Yeah, I am actually aware— I've been following it keenly via the buildah issue tracker, and I should have put that in my original post. But the current reality is still that building images on Kubernetes is a choice between several not-great options, especially if you're on managed k8s and can't use privileged mode as a stopgap. Anyway, even once this stuff all lands, there's still actually no way to do what I would consider to be the actual gold standard of k8s image building, which would be a method where you build the image starting from any base layers already on the kubelet. Because currently, whether it's kaniko, buildah, docker-in-docker, etc, you're basically always either downloading everything every time, or you're having to manually manage some scheme with a long-lived cache container that you volume-in each time and purge periodically, for example: https://github.com/GoogleContainerTools/kaniko#caching-base-images https://github.com/GoogleContainerTools/kaniko#caching-base-... In principle this should be possible with a Kaniko-like workflow, but you'd need a separate control pod / build pod setup, where the control pod would compute all the layer hashes and then repeatedly try to spawn from the bottom-up using `imagePullPolicy: never` until one of them succeeded, and then build the remainder of the container from there.
  - znpy 5y ago
    I feel your pain, as it's my pain too. Right now we have a pool of gitlab runners running as dedicated virtual machines but on the to-do list I have to test whether rootless podman + gitlab runner cache + $DOCKER_HOST pointing to podman's sock file can let developers use plain old docker-compose and the general docker tooling... Within a dumb pod in a kubernetes clusters, with all the bells and whistles (especially cluster autoscaling). A man can dream. Edit: we are a bit advantaged because we run on openstack, run our own registry (harbor) and our cloud provider doesn't charge us for bandwidth...
    - mikepurvis 5y ago
      I'm a huge fan of the kubernetes runner for GitLab CI, but I really wish there was a way to just to `image: path/in/repo/to/Dockerfile` and have it do the right thing as far as creating/using the supplied image.
      - deastman 5y ago
        Can you open an issue in the gitlab-runner project? It would be great if one of our engineers that focuses on the Kubernetes executor can weigh in. https://gitlab.com/gitlab-org/gitlab-runner/-/issues https://gitlab.com/gitlab-org/gitlab-runner/-/issues
        mikepurvis 5y ago
        Sure, done: https://gitlab.com/gitlab-org/gitlab-runner/-/issues/28195 https://gitlab.com/gitlab-org/gitlab-runner/-/issues/28195
  - rhatdan 5y ago
    You can volume mount in the host container storage into the container using additional stores. In this article, note sections on /var/lib/shared https://developers.redhat.com/blog/2019/08/14/best-practices-for-running-buildah-in-a-container# https://developers.redhat.com/blog/2019/08/14/best-practices...
rhatdan 5y ago
This is no longer true. As of kernel 3.11 native overlay can be used by rootless users.

znpy 5y ago

Sorry for the vague reply, but I'm out with friends right now. But basically that's fixed because newest kernels (5.13 iirc) allow to work around that without requiring root privileges. So podman is going to stop requiring that too sooner or later.

mikepurvis 5y ago
Yeah, I am actually aware— I've been following it keenly via the buildah issue tracker, and I should have put that in my original post. But the current reality is still that building images on Kubernetes is a choice between several not-great options, especially if you're on managed k8s and can't use privileged mode as a stopgap. Anyway, even once this stuff all lands, there's still actually no way to do what I would consider to be the actual gold standard of k8s image building, which would be a method where you build the image starting from any base layers already on the kubelet. Because currently, whether it's kaniko, buildah, docker-in-docker, etc, you're basically always either downloading everything every time, or you're having to manually manage some scheme with a long-lived cache container that you volume-in each time and purge periodically, for example: https://github.com/GoogleContainerTools/kaniko#caching-base-images https://github.com/GoogleContainerTools/kaniko#caching-base-... In principle this should be possible with a Kaniko-like workflow, but you'd need a separate control pod / build pod setup, where the control pod would compute all the layer hashes and then repeatedly try to spawn from the bottom-up using `imagePullPolicy: never` until one of them succeeded, and then build the remainder of the container from there.
- znpy 5y ago
  I feel your pain, as it's my pain too. Right now we have a pool of gitlab runners running as dedicated virtual machines but on the to-do list I have to test whether rootless podman + gitlab runner cache + $DOCKER_HOST pointing to podman's sock file can let developers use plain old docker-compose and the general docker tooling... Within a dumb pod in a kubernetes clusters, with all the bells and whistles (especially cluster autoscaling). A man can dream. Edit: we are a bit advantaged because we run on openstack, run our own registry (harbor) and our cloud provider doesn't charge us for bandwidth...
  - mikepurvis 5y ago
    I'm a huge fan of the kubernetes runner for GitLab CI, but I really wish there was a way to just to `image: path/in/repo/to/Dockerfile` and have it do the right thing as far as creating/using the supplied image.
    - deastman 5y ago
      Can you open an issue in the gitlab-runner project? It would be great if one of our engineers that focuses on the Kubernetes executor can weigh in. https://gitlab.com/gitlab-org/gitlab-runner/-/issues https://gitlab.com/gitlab-org/gitlab-runner/-/issues
      - mikepurvis 5y ago
        Sure, done: https://gitlab.com/gitlab-org/gitlab-runner/-/issues/28195 https://gitlab.com/gitlab-org/gitlab-runner/-/issues/28195
- rhatdan 5y ago
  You can volume mount in the host container storage into the container using additional stores. In this article, note sections on /var/lib/shared https://developers.redhat.com/blog/2019/08/14/best-practices-for-running-buildah-in-a-container# https://developers.redhat.com/blog/2019/08/14/best-practices...

rhatdan 5y ago

This is no longer true. As of kernel 3.11 native overlay can be used by rootless users.