5 ms·

What I like most about it is that it can run as any user. There's no need for a daemon that starts as privileged user. Another benefit is that all images and co

What I like most about it is that it can run as any user. There's no need for a daemon that starts as privileged user. Another benefit is that all images and container reside on a hidden directory in $HOME.

tacobelllover99 5y ago
Tell me you are a Dev without telling me you are a Dev
judge2020 5y ago
Docker also has a rootless mode as of 19.03: https://docs.docker.com/engine/security/rootless/ https://docs.docker.com/engine/security/rootless/
- GordonS 5y ago
  Wow, how did I not know about this! Unless I'm missing something, the limitations mentioned in the docs don't seem too limiting. Is there anything that needs to be changed or watched out for in real life?
  - foresto 5y ago
    It requires a kernel that allows unprivileged user namespaces. Docker images that run as uid 0, which many of them do, could potentially exploit their way out of the container, since kernel code running as (actual or namespaced) uid 0 hasn't been extensively tested to be safe and bug-free. You might have to experiment with different "drivers" to get the overlay filesystem and cgroups components to work on any given host. (These are not hardware drivers, but something more like docker subsystem plugins.) By default, it expects a dbus user session, which a headless server might not have. You can either enable one or configure a different `cgroupdriver`.
    - staticassertion 5y ago
      TBH the Linux kernel isn't capable of providing an isolation boundary anyways, so while it's a meaningful regression if your assumption is "attacker in the container" I highly recommend using gVisor or Firecracker, or otherwise reducing the need to trust the Linux kernel for anything like resisting hostile local attackers.
- znpy 5y ago
  Last time I checked that was experimental, and still not production ready -- has that changed?
  - foresto 5y ago
    The fourth sentence on the page linked above: "Rootless mode graduated from experimental in Docker Engine v20.10."
    - znpy 5y ago
      Thanks!!
mikepurvis 5y ago
With the major caveat that rootless depends on /dev/fuse for the overlay filesystems (unless you hate yourself and want to use vfs), so you can't use it inside a Kubernetes container unless it's privileged or has the SYS_ADMIN capability added. This is how we end up with beautiful hacks like Kaniko for building container images inside container workflows. Le sigh.
- znpy 5y ago
  Sorry for the vague reply, but I'm out with friends right now. But basically that's fixed because newest kernels (5.13 iirc) allow to work around that without requiring root privileges. So podman is going to stop requiring that too sooner or later.
  - mikepurvis 5y ago
    Yeah, I am actually aware— I've been following it keenly via the buildah issue tracker, and I should have put that in my original post. But the current reality is still that building images on Kubernetes is a choice between several not-great options, especially if you're on managed k8s and can't use privileged mode as a stopgap. Anyway, even once this stuff all lands, there's still actually no way to do what I would consider to be the actual gold standard of k8s image building, which would be a method where you build the image starting from any base layers already on the kubelet. Because currently, whether it's kaniko, buildah, docker-in-docker, etc, you're basically always either downloading everything every time, or you're having to manually manage some scheme with a long-lived cache container that you volume-in each time and purge periodically, for example: https://github.com/GoogleContainerTools/kaniko#caching-base-images https://github.com/GoogleContainerTools/kaniko#caching-base-... In principle this should be possible with a Kaniko-like workflow, but you'd need a separate control pod / build pod setup, where the control pod would compute all the layer hashes and then repeatedly try to spawn from the bottom-up using `imagePullPolicy: never` until one of them succeeded, and then build the remainder of the container from there.
    - znpy 5y ago
      I feel your pain, as it's my pain too. Right now we have a pool of gitlab runners running as dedicated virtual machines but on the to-do list I have to test whether rootless podman + gitlab runner cache + $DOCKER_HOST pointing to podman's sock file can let developers use plain old docker-compose and the general docker tooling... Within a dumb pod in a kubernetes clusters, with all the bells and whistles (especially cluster autoscaling). A man can dream. Edit: we are a bit advantaged because we run on openstack, run our own registry (harbor) and our cloud provider doesn't charge us for bandwidth...
EnigmaCurry 5y ago
I am a bit concerned with the security of rootless podman: https://news.ycombinator.com/item?id=28393949 https://news.ycombinator.com/item?id=28393949
truffdog 5y ago
It's especially great for laptops, dockerd can be a little expensive power wise.

tacobelllover99 5y ago

Tell me you are a Dev without telling me you are a Dev

judge2020 5y ago

Docker also has a rootless mode as of 19.03: https://docs.docker.com/engine/security/rootless/ https://docs.docker.com/engine/security/rootless/

GordonS 5y ago
Wow, how did I not know about this! Unless I'm missing something, the limitations mentioned in the docs don't seem too limiting. Is there anything that needs to be changed or watched out for in real life?
- foresto 5y ago
  It requires a kernel that allows unprivileged user namespaces. Docker images that run as uid 0, which many of them do, could potentially exploit their way out of the container, since kernel code running as (actual or namespaced) uid 0 hasn't been extensively tested to be safe and bug-free. You might have to experiment with different "drivers" to get the overlay filesystem and cgroups components to work on any given host. (These are not hardware drivers, but something more like docker subsystem plugins.) By default, it expects a dbus user session, which a headless server might not have. You can either enable one or configure a different `cgroupdriver`.
  - staticassertion 5y ago
    TBH the Linux kernel isn't capable of providing an isolation boundary anyways, so while it's a meaningful regression if your assumption is "attacker in the container" I highly recommend using gVisor or Firecracker, or otherwise reducing the need to trust the Linux kernel for anything like resisting hostile local attackers.
znpy 5y ago
Last time I checked that was experimental, and still not production ready -- has that changed?
- foresto 5y ago
  The fourth sentence on the page linked above: "Rootless mode graduated from experimental in Docker Engine v20.10."
  - znpy 5y ago
    Thanks!!

mikepurvis 5y ago

With the major caveat that rootless depends on /dev/fuse for the overlay filesystems (unless you hate yourself and want to use vfs), so you can't use it inside a Kubernetes container unless it's privileged or has the SYS_ADMIN capability added. This is how we end up with beautiful hacks like Kaniko for building container images inside container workflows. Le sigh.

znpy 5y ago
Sorry for the vague reply, but I'm out with friends right now. But basically that's fixed because newest kernels (5.13 iirc) allow to work around that without requiring root privileges. So podman is going to stop requiring that too sooner or later.
- mikepurvis 5y ago
  Yeah, I am actually aware— I've been following it keenly via the buildah issue tracker, and I should have put that in my original post. But the current reality is still that building images on Kubernetes is a choice between several not-great options, especially if you're on managed k8s and can't use privileged mode as a stopgap. Anyway, even once this stuff all lands, there's still actually no way to do what I would consider to be the actual gold standard of k8s image building, which would be a method where you build the image starting from any base layers already on the kubelet. Because currently, whether it's kaniko, buildah, docker-in-docker, etc, you're basically always either downloading everything every time, or you're having to manually manage some scheme with a long-lived cache container that you volume-in each time and purge periodically, for example: https://github.com/GoogleContainerTools/kaniko#caching-base-images https://github.com/GoogleContainerTools/kaniko#caching-base-... In principle this should be possible with a Kaniko-like workflow, but you'd need a separate control pod / build pod setup, where the control pod would compute all the layer hashes and then repeatedly try to spawn from the bottom-up using `imagePullPolicy: never` until one of them succeeded, and then build the remainder of the container from there.
  - znpy 5y ago
    I feel your pain, as it's my pain too. Right now we have a pool of gitlab runners running as dedicated virtual machines but on the to-do list I have to test whether rootless podman + gitlab runner cache + $DOCKER_HOST pointing to podman's sock file can let developers use plain old docker-compose and the general docker tooling... Within a dumb pod in a kubernetes clusters, with all the bells and whistles (especially cluster autoscaling). A man can dream. Edit: we are a bit advantaged because we run on openstack, run our own registry (harbor) and our cloud provider doesn't charge us for bandwidth...

EnigmaCurry 5y ago

I am a bit concerned with the security of rootless podman: https://news.ycombinator.com/item?id=28393949 https://news.ycombinator.com/item?id=28393949

truffdog 5y ago

It's especially great for laptops, dockerd can be a little expensive power wise.